Skip to content

R
Recommendation system based on Tamil-English code-mixed text analysis
Commit b91c49fe authored by Vijayakumar Sajeevan's avatar Vijayakumar Sajeevan
Browse files
Options

Commit by Jaseenthiran_IT19233536

parent e2663fec
Hide whitespace changes
Inline Side-by-side
Showing with 552 additions and 0 deletions
IT19233536_Jaseenthiran.S.ipynb 0 → 100644
View file @ b91c49fe
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "2a415cbb",
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"import re"
]
},
{
"cell_type": "markdown",
"id": "4ee3bfba",
"metadata": {},
"source": [
"## Get Data Set"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "a0e2bfd1",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>tanglish_cmt</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Vera Leval Thalapathi action Vijay Sethupathi ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Nalla thiraipadam....Parkkalam.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Neruppu veraleval</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>kathai vera level screen play.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Oru murai inta muviyai parkkalam.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>supper matrum nalla patam.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>innum orumurai paarkalam.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>enakku pidithathu pa.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Master Mersal</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Nalla Padam matrum Thiraikathai</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>Vijay sethupathy nadippu vera level.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>Kathai suththama sariyillai.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>Wow Vijay sir moththama senjudar</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>marana maas</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>Padam kevalam</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>mokka comedy scences</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>kevalamana mokka seruppu padam</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>vijay vijaysethupathy malavika and lokesh comp...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>Arumaiyana padam guys</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" tanglish_cmt\n",
"0 Vera Leval Thalapathi action Vijay Sethupathi ...\n",
"1 Nalla thiraipadam....Parkkalam.\n",
"2 Neruppu veraleval\n",
"3 kathai vera level screen play.\n",
"4 Oru murai inta muviyai parkkalam.\n",
"5 supper matrum nalla patam.\n",
"6 innum orumurai paarkalam.\n",
"7 enakku pidithathu pa.\n",
"8 Master Mersal\n",
"9 Nalla Padam matrum Thiraikathai\n",
"10 Vijay sethupathy nadippu vera level.\n",
"11 Kathai suththama sariyillai.\n",
"12 Wow Vijay sir moththama senjudar\n",
"13 marana maas\n",
"14 Padam kevalam\n",
"15 mokka comedy scences\n",
"16 kevalamana mokka seruppu padam\n",
"17 vijay vijaysethupathy malavika and lokesh comp...\n",
"18 Arumaiyana padam guys"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.set_option('display.max_rows', None)\n",
"pd.set_option('display.max_columns', None)\n",
"data_set = pd.read_excel(r'tanglish _data_set.xlsx')\n",
"data_set"
]
},
{
"cell_type": "markdown",
"id": "c2dfa699",
"metadata": {},
"source": [
"## Tokenization "
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "6e4891a2",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>tanglish_cmt</th>\n",
" <th>tokenize_cmt</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Vera Leval Thalapathi action Vijay Sethupathi ...</td>\n",
" <td>[Vera, Leval, Thalapathi, action, Vijay, Sethu...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Nalla thiraipadam....Parkkalam.</td>\n",
" <td>[Nalla, thiraipadam, Parkkalam, ]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Neruppu veraleval</td>\n",
" <td>[Neruppu, veraleval]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>kathai vera level screen play.</td>\n",
" <td>[kathai, vera, level, screen, play, ]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Oru murai inta muviyai parkkalam.</td>\n",
" <td>[Oru, murai, inta, muviyai, parkkalam, ]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>supper matrum nalla patam.</td>\n",
" <td>[supper, matrum, nalla, patam, ]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>innum orumurai paarkalam.</td>\n",
" <td>[innum, orumurai, paarkalam, ]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>enakku pidithathu pa.</td>\n",
" <td>[enakku, pidithathu, pa, ]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Master Mersal</td>\n",
" <td>[Master, Mersal]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Nalla Padam matrum Thiraikathai</td>\n",
" <td>[Nalla, Padam, matrum, Thiraikathai]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>Vijay sethupathy nadippu vera level.</td>\n",
" <td>[Vijay, sethupathy, nadippu, vera, level, ]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>Kathai suththama sariyillai.</td>\n",
" <td>[Kathai, suththama, sariyillai, ]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>Wow Vijay sir moththama senjudar</td>\n",
" <td>[Wow, Vijay, sir, moththama, senjudar]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>marana maas</td>\n",
" <td>[marana, maas]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>Padam kevalam</td>\n",
" <td>[Padam, kevalam]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>mokka comedy scences</td>\n",
" <td>[mokka, comedy, scences]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>16</th>\n",
" <td>kevalamana mokka seruppu padam</td>\n",
" <td>[kevalamana, mokka, seruppu, padam]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>17</th>\n",
" <td>vijay vijaysethupathy malavika and lokesh comp...</td>\n",
" <td>[vijay, vijaysethupathy, malavika, and, lokesh...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>18</th>\n",
" <td>Arumaiyana padam guys</td>\n",
" <td>[Arumaiyana, padam, guys]</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" tanglish_cmt \\\n",
"0 Vera Leval Thalapathi action Vijay Sethupathi ... \n",
"1 Nalla thiraipadam....Parkkalam. \n",
"2 Neruppu veraleval \n",
"3 kathai vera level screen play. \n",
"4 Oru murai inta muviyai parkkalam. \n",
"5 supper matrum nalla patam. \n",
"6 innum orumurai paarkalam. \n",
"7 enakku pidithathu pa. \n",
"8 Master Mersal \n",
"9 Nalla Padam matrum Thiraikathai \n",
"10 Vijay sethupathy nadippu vera level. \n",
"11 Kathai suththama sariyillai. \n",
"12 Wow Vijay sir moththama senjudar \n",
"13 marana maas \n",
"14 Padam kevalam \n",
"15 mokka comedy scences \n",
"16 kevalamana mokka seruppu padam \n",
"17 vijay vijaysethupathy malavika and lokesh comp... \n",
"18 Arumaiyana padam guys \n",
"\n",
" tokenize_cmt \n",
"0 [Vera, Leval, Thalapathi, action, Vijay, Sethu... \n",
"1 [Nalla, thiraipadam, Parkkalam, ] \n",
"2 [Neruppu, veraleval] \n",
"3 [kathai, vera, level, screen, play, ] \n",
"4 [Oru, murai, inta, muviyai, parkkalam, ] \n",
"5 [supper, matrum, nalla, patam, ] \n",
"6 [innum, orumurai, paarkalam, ] \n",
"7 [enakku, pidithathu, pa, ] \n",
"8 [Master, Mersal] \n",
"9 [Nalla, Padam, matrum, Thiraikathai] \n",
"10 [Vijay, sethupathy, nadippu, vera, level, ] \n",
"11 [Kathai, suththama, sariyillai, ] \n",
"12 [Wow, Vijay, sir, moththama, senjudar] \n",
"13 [marana, maas] \n",
"14 [Padam, kevalam] \n",
"15 [mokka, comedy, scences] \n",
"16 [kevalamana, mokka, seruppu, padam] \n",
"17 [vijay, vijaysethupathy, malavika, and, lokesh... \n",
"18 [Arumaiyana, padam, guys] "
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def tokenize(cmt):\n",
" tokens = re.split('\\W+', cmt)\n",
" return tokens\n",
"\n",
"data_set['tokenize_cmt'] = data_set['tanglish_cmt'].apply(lambda x: tokenize(x))\n",
"data_set"
]
},
{
"cell_type": "markdown",
"id": "f6307627",
"metadata": {},
"source": [
"## Create List for positive & negative Comments"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "a27d7978",
"metadata": {},
"outputs": [],
"source": [
"positive_list = ['seme', 'sattapadi', 'arumaiyana', 'arumai', 'supper', 'sirappu', 'marana', 'vera', 'wow', 'maas', 'mass', 'maass', 'veramathi', 'nalla', 'senjudar', 'nallam', 'paarkalaam', 'senjudar', 'senjudanga', 'neruppu', 'mersal', 'next level', 'level', 'vera kaddam', 'verelevel']\n",
"\n",
"negative_list = ['mokka', 'vaippilai', 'seruppu', 'kanravi', 'kevalam', 'asinkam', 'kevalamana', 'asinkamana', 'bore', 'boomer', 'beep', 'nallave illa', 'nalla illa', 'poor', 'mayiru', 'bad', 'failure', 'fail', 'sleep']"
]
},
{
"cell_type": "markdown",
"id": "08876a1a",
"metadata": {},
"source": [
"## calculation for rating"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "c19a8602",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Positive comments count : 12\n",
"Negative comments count : 3\n"
]
}
],
"source": [
"positive_comments_count = 0\n",
"negative_comments_count = 0\n",
"\n",
"for cmt in data_set['tokenize_cmt']: \n",
" try: \n",
" for word in cmt:\n",
" if word.lower() in positive_list:\n",
" while word.lower() not in positive_list:\n",
" continue\n",
" positive_comments_count += 1 \n",
" # print(word)\n",
" break\n",
" \n",
" for word in cmt:\n",
" if word.lower() in negative_list:\n",
" while word.lower() not in negative_list:\n",
" continue\n",
" negative_comments_count += 1\n",
" # print(word)\n",
" break\n",
" except:\n",
" pass\n",
" \n",
"print('Positive comments count : ', positive_comments_count )\n",
"print('Negative comments count : ', negative_comments_count )"
]
},
{
"cell_type": "markdown",
"id": "f339acf9",
"metadata": {},
"source": [
"## get rating"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "525516d5",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Positive Comment = 80.0 %\n",
"Negative Comment = 20.0 %\n"
]
}
],
"source": [
"positive_rating = positive_comments_count/(positive_comments_count + negative_comments_count)\n",
"print( 'Positive Comment = ', round(positive_rating*100, 2), '%')\n",
"\n",
"negative_rating = negative_comments_count/(positive_comments_count + negative_comments_count)\n",
"print( 'Negative Comment = ', round(negative_rating*100, 2), '%')"
]
},
{
"cell_type": "markdown",
"id": "ff337b7a",
"metadata": {},
"source": [
"## final decision"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "23d49581",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"This movie has more \"positive\" review than negative review...\n"
]
}
],
"source": [
"if positive_rating > negative_rating:\n",
" print('This movie has more \"positive\" review than negative review...')\n",
"else:\n",
" print('This movie has more \"negative\" review than positive review...')"
]
},
{
"cell_type": "markdown",
"id": "94a10141",
"metadata": {},
"source": [
"## If you want to write new data set...."
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "65621b71",
"metadata": {},
"outputs": [],
"source": [
"df = pd.DataFrame(data_set)\n",
"\n",
"writer = pd.ExcelWriter('sentiment_analysis_tokenization.xlsx', engine='xlsxwriter')\n",
"df.to_excel(writer, sheet_name=\"tokenization_list\")\n",
"\n",
"writer.save() "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5c5833cb",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.8"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment