# 2020-101 ## Introduction This project will be created under the guidelines provided by the Research Project module offered in 4th Year by the Faculty of Computing. The title of the project is "Student and Lecturer Performance Enhancement System". This Research Project comes under the domain of Artifical Intelligence and Machine Learning along with the support of Computer Vision. ## Main Objective Main Objectives that will be targetted in this Research Project will be listed as follows. * Tracking student attendance using facial detection and facial recognition and notify and getting opinion from the students who are absent and students who are leaving the lecture in the during a specified time period (Attendance Register). * Monitoring student behavior within the classroom during lecture periods to identify potential problem areas in retaining student attention and adapting the lecturing styles to suit the student needs (Monitoring Student Behavior). * Summarize the converted text and present as a document and identify the important points of the lecture to display them highlighted (Lecture Summarizing). * Monitoring lecturers’ behavior by analyzing the teaching style in a lecture hall during the lecture hours (Monitor Lecturer Performance). ## Main Research Questions This Research Project will be conducted to find answers for the following 4 research questions. * “What is the optimum way of tracking student attendance in a much efficient way and how does the lecturer analyze reasons for student absenteeism?” (Q1). * “Does a correlation exist between lecturing style and student behavior in the classroom and how can Computer Vision and Artificial Intelligence be incorporated in determining this relationship?” (Q2). * “How to summarize the lecture content to enable students to pay more attention to the lecture and reduce time spend for taking notes?” (Q3). * “How to evaluate lecture performance by tracking their behavior during a lecture and analyzing the quality of the lecture content which is delivered by the lecturer?” (Q4). ## Individual Research Questions The main research questions came up for the project were based on each individual component that will be implemented through the course of the project. Hence, the individual research questions could be listed down as follows. ### Attendance Register *What is the optimum way of tracking student attendance in a much efficient way and how does the lecturer analyze reasons for student absenteeism?* ### Monitoring Student Behavior *Does a correlation exist between lecturing style and student behavior in the classroom and how can Computer Vision and Artificial Intelligence be incorporated in determining this relationship?.* ### Lecture Summarizing *How to summarize the lecture content to enable students to pay more attention to the lecture and reduce time spend for taking notes?* ### Monitoring Lecturer Performance *How to evaluate lecture performance by tracking their behavior during a lecture and analyzing the quality of the lecture content which is delivered by the lecturer?”* ## Individual Objectives The main objectives of the research project could be further split into individual objectives which elaborates purpose of the project in much detail. ### Attendance Register * Tracking student attendance using facial detection and facial recognition. * Notify and getting opinion from the students who are absent and students who are leaving the lecture in the during a specified time period. * Setting up student data and respective course data in an adaptive manner. * Provide an overview for the lecturer regarding student attendance through a dashboard and predicting future attendance. ### Monitoring Student Behavior * Monitoring student behavior within the classroom during lecture periods to identify potential problem areas in retaining student attention and adapting the lecturing styles to suit the student needs. * Identifying the concentration level, emotion expressions and general behavior of students within 2-hour lecture duration using facial recognition, emotion recognition and motion detection employing computer vision technologies. * Determining the dependency between student behavior and the lecturer behavior by analyzing overall behavior status of students and lecturer for a given time frame during the lecture. ### Lecture Summarizing * Understand lecturer’s voice and filter the lecturer’s voice by removing background noises and silent pauses * Convert the noise-filtered audio into text * Summarize the converted text and present as a document. * Identify the important points of the lecture to display them highlighted. * Display important notices mentioned in the lecture. ### Monitoring Lecturer Performance * Monitoring lecturers’ behavior by analyzing the teaching style in a lecture hall during the lecture hours. * Identifying the lectures emotional state when the lecture is conducting. * Generating an end-semester feedback of the lecturer giving a summary of their performance. ## System Architecture ![System Acrhitecture of SLPES](SLPES_System.jpg) As mentioned in the given figure, this system will mainly consist of a mobile and web application. When the students and the lecturer enter the classroom, their behavior will be monitored by two separate video-cameras and a microphone. The collected data will transmit to the server through raspberry-pi and stores the gathered data in the MongoDB cloud database service. While the video camera tracks both facial expression and body movements, the microphone will record the lecture for further analysis. The student attendance will register by capturing facial identity using deep learning and computer vision mechanisms. Students will be monitored by capturing their facial and body movements by Computer Vision and Deep Learning algorithms. The lecturer audio and video details will be gathered and stored for analyzing as such lecture summarizing and monitoring lecturer performance using machine learning, deep learning, and computer vision technologies. While the students have access to the mobile application, the lecturers and the higher management have the authority to manage the web-application. Frontend components will communicate with the backend using Django REST API. ## Module Description ### Branch structure The following list contains the respective branch names for each individual * **db_and_monitoring** - IT17097284 * **monitoring_student_behavior_IT17138000** - IT17138000 * **IT17100908** - IT17100908 Note that the **QA_RELEASE** branch is the merged branch of all the working components. The **master** contains only the common folder strcuture of the project. ### Folder structure This folder structure adheres to the general project structure of a django web project. The following list describes the folders and files in use throughout the entire project. * **AttendanceApp** - comprises the work of *Attendance Register* component. * **FirstApp** - contains the implementations of *Monitoring Student Behavior* component. * **LectureSummarizingApp** - constitutes the workings of the *Lecture Summarization* component. * **MonitorLecturerApp** - comprises the implementations of *Monitoring Lecturer Performance* component. * **integrated_slpes** - this contains the general settings and url configurations of the entire project. * **manage.py** - this file is required to run the project (done by the comman *python manage.py runserver*) * **requirements.txt** - contains the list of dependencies required to run the project. ### Application strcuture As mentioned above, the folders such as *AttendanceApp* and *FirstApp* are the applications reserved for each individual module. The following list will elaborate the common folders and files in an application folder. * **logic** - this folder contains the basic logic implementation for the application (implementations of deep learning models, processing etc...). * **migrations** - comprises of the database migration files (*migration* in django means a way of propogating the changes made to your model, to the database schema). This is a folder generated by the django framework. * **templates** - this folder contains the html files (frontend) of the relevant application. However, the templates should be added inside another folder, with the same name as of your application. * **admin.py** - registering the models to the django admin site. * **api.py** - contains the RESTful API for the application. * **models.py** - all the database models are created within this file. * **serializers.py** - django provides a library to serialize the database models to facilitate the communication through REST API. The serialized class for each model is created here. * **tests.py** - test case implementations * **urls.py** - contains the specific url mappings for an application. * **views.py** - implements the frontend logic for each template file. ## Installation ### Prerequisites The following software tools needed to be installed before running this project 1. Anaconda 3, (click [here](https://docs.anaconda.com/anaconda/install/) to download. Select Python version as 3.7 or above). 2. PyCharm Professional Edition with Anaconda Plugin 2019.3.1 (click [here](https://www.jetbrains.com/pycharm/download/other.html) to download). ### Procedure To run this project, you will need to follow the given steps. 1. Clone the repository to a desired location in your machine. 2. In this location, open the command line and create a virtual environment by typing *conda create -n <ENV_NAME>*. (if this command does not work, type *CALL conda.bat create -n <ENV_NAME>*) 3. To download all the dependencies, run the command *pip install -r requirements.txt*. 4. To download the classifiers, visit [here](https://drive.google.com/drive/folders/1xIZfX47owHUXYqN13hchQ4waVW5UE5rp?usp=sharing) 5. Once the classifiers are downloaded, include them in the application folders as follows. * Create a folder named *classifiers* inside the *FirstApp* folder. * From the downloaded classifiers, include the content in *monitoring_student_behavior* folder inside this *classifiers* folder. * Create a folder named *models* inside the *MonitorLecturerApp* folder. * From the downloaded classifiers, include the content in *monitoring_lecturer_behavior* folder inside this *models* folder. 6. To download the static content of the project, visit [here](https://drive.google.com/drive/folders/1CtoRFxXRThE9vhsP2b0xJXhUTyPh8kN-?usp=sharing) 7. Once the static content is downloaded, include them in the project folder as follows. * Create a folder named *static* inside the project root folder. * Put the static content inside this folder. * Then open a command line in the project root folder and type *python manage.py collectstatic*. This will create another folder named *assets* inside the root folder. 8. Now open this project in the PyCharm IDE. 9. Before running the project, the url of the database needs to be configured. The steps are as follows. * Go to *integrated_slpes/settings.py* file and locate the *DATABASES* variable. * Inside the variable named *default*, copy the value under the variable *HOST*. * Then in the left sidebar, select the *project* tab and from the dropdown the upper right corner, select the option *project*. * Now select the *External libraries* section and your set of Python libraries. * Go to *site-packages* folder and find the library named *pymongo*. * From this library, open a file named *mongo_client.py* in the IDE. * In this file, there is a variable named *HOST* inside the *MongoClient* class. Now the replace the value in this variable with the copied url value. * Then go back to the terminal and type the command *python manage.py runserver*. The server will be opened in the address *127.0.0.1:8000*. ### Dependencies The required dependencies could be found in [requirements.txt](requirements.txt) file. The main dependencies required for this project are listed below. * *Python* (version 3.7+) * *Django* (version 2.2.11) * *Djongo* (version 1.3.1) * *djangorestframework* - (version 3.11.0) * *tensorflow* - (version 2.1.0) * [ hello]