# 2020-101


## Group Details

### Members

*  **Supervisor**: Mr. Samantha Rajapaksha
*  **Co-supervisor**: Ms. Dinuka Wijendra
*  **Leader** - I.K Seneviratne
*  **Member 1** - B.A.S.D Perera
*  **Member 2** - R.S.D Fernando 
*  **Member 3** - L.K.B Siriwardana




## Introduction

This project will be created under the guidelines provided by the Research Project module offered in 4th Year by the Faculty of Computing. The title of the project is "Student and Lecturer Performance Enhancement System". This Research Project comes under the domain of Artifical Intelligence and Machine Learning along with the support of Computer Vision.

## Main Objective

Main Objectives that will be targetted in this Research Project will be listed as follows.


* Tracking student attendance using facial detection and facial recognition and notify and getting opinion from the students who are absent and students who are leaving the lecture in the during a specified time period (Attendance Register).
* Monitoring student behavior within the classroom during lecture periods to identify potential problem areas in retaining student attention and adapting the lecturing styles to suit the student needs (Monitoring Student Behavior). 
* Summarize the converted text and present as a document and identify the important points of the lecture to display them highlighted (Lecture Summarizing).
* Monitoring lecturers’ behavior by analyzing the teaching style in a lecture hall during the lecture hours (Monitor Lecturer Performance).  


## Main Research Questions

This Research Project will be conducted to find answers for the following 4 research questions.


* “What is the optimum way of tracking student attendance in a much efficient way and how does the lecturer analyze reasons for student absenteeism?” (Q1). 
* “Does a correlation exist between lecturing style and student behavior in the classroom and how can Computer Vision and Artificial Intelligence be incorporated in determining this relationship?” (Q2). 
* “How to summarize the lecture content to enable students to pay more attention to the lecture and reduce time spend for taking notes?” (Q3). 
* “How to evaluate lecture performance by tracking their behavior during a lecture and analyzing the quality of the lecture content which is delivered by the lecturer?” (Q4).  


## Individual Research Questions

The main research questions came up for the project were based on each individual component that will be implemented through the course of the project. Hence, the individual research questions could be listed down as follows.

### Attendance Register

*What is the optimum way of tracking student attendance in a much efficient way and how does the lecturer analyze reasons for student absenteeism?*

### Monitoring Student Behavior

*Does a correlation exist between lecturing style and student behavior in the classroom and how can Computer Vision and Artificial Intelligence be incorporated in determining this relationship?.* 

### Lecture Summarizing

*How to summarize the lecture content to enable students to pay more attention to the lecture and reduce time spend for taking notes?*

### Monitoring Lecturer Performance

*How to evaluate lecture performance by tracking their behavior during a lecture and analyzing the quality of the lecture content which is delivered by the lecturer?”*


## Individual Objectives

The main objectives of the research project could be further split into individual objectives which elaborates purpose of the project in much detail.

### Attendance Register

* Tracking student attendance using facial detection and facial recognition.
* Notify and getting opinion from the students who are absent and students who are leaving the lecture in the during a specified time period.
* Setting up student data and respective course data in an adaptive manner.
* Provide an overview for the lecturer regarding student attendance through a dashboard and predicting future attendance.

### Monitoring Student Behavior

* Monitoring student behavior within the classroom during lecture periods to identify potential problem areas in retaining student attention and adapting the lecturing styles to suit the student needs.
* Identifying the concentration level, emotion expressions and general behavior of students within 2-hour lecture duration using facial recognition, emotion recognition and motion detection employing computer vision technologies.
* Determining the dependency between student behavior and the lecturer behavior by analyzing overall behavior status of students and lecturer for a given time frame during the lecture.

### Lecture Summarizing

* Understand lecturer’s voice and filter the lecturer’s voice by removing background noises and silent pauses
* Convert the noise-filtered audio into text
* Summarize the converted text and present as a document.
* Identify the important points of the lecture to display them highlighted.
* Display important notices mentioned in the lecture.

### Monitoring Lecturer Performance

* Monitoring lecturers’ behavior by analyzing the teaching style in a lecture hall during the lecture hours.
* Identifying the lectures emotional state when the lecture is conducting.
* Generating an end-semester feedback of the lecturer giving a summary of their performance. 


## Summary of Individual Components

### Attendance Register

There are three processes, namely, attendance marking, getting feedback, and displaying information. To identify students, video is recorded from a front-mounted camera and it is used to capture a frame. OpenCV was used to record as well as capture a frame. The image enhancement algorithm clears out the captured image to run the face detection algorithm. After detecting the faces in the frame, the system runs the face detection algorithm to identify students.  OpenCV and Scikit-learn were used for the processing of images and the classification of face images. Detected students' attendance details were stored in the database as well as late coming activities and skipping activities. The mobile app helps achieve the task of gathering face recognition data. Face data collection is a mandatory step in the registration process once a student downloads and sets up the application. The user interface, similar to setting the face to unlock in smartphones, makes the system exceptional in terms of usability. After a lecture, using the mobile application, reasons for student behaviors track down. The lecturer's dashboard in the mobile application presents a detailed student attendance, students' feedback, and reasons.

### Monitoring Student Behavior

Once the students are recognized, the overall student behavior will be evaluated based on Activity, Emotion recognition, and gaze estimation components. A pre-trained deep learning model was used for identifying the student emotions under categories such as *Happy*, *Sad*, *Anger*, *Surprise*, and *Neutral*. The Activity Recognition was also based on a deep learning model. This model was trained under 3 classes: *phone checking*, *listening*, and *note-taking*. Prior to running this model, the students needed to be separately identified. Hence, an object detection model named MobileNet-SSD was used. To perform the student gaze estimation, a combination of facial landmarks, deep learning and the application of *perspective-n-point* algorithm was used. The head gaze was calculated based on directions such as *looking up and right, looking up and left, looking down and right, looking down* and left and *looking front*. Some threshold values were set to vertical and horizontal directions by calculating the angle relative to the respective planes.

Once the above stages are completed, the data visualization process takes place. For this the lecture video will be divided into 5 equal time segments and all the student behavioral aspects will be measured for the given time segments. These calculations will be saved into the database for future reference. In order to find the correlation between the student behavior and lecturer performance, the python data manipulation and analysis library named *pandas* is used.

### Lecture Summarizing

Once the lecture is Completed the original audio file will be saved in the Database. As the initial step of "lecture summarizing" the audio file will be processed and the unnecessary parts such as background noises, long pauses will be removed from the audio file using the spectral grating algorithm. Then the filtered audio will be converted into a text format using the "speech-to-text approach". This text file will be used to summarize the lecture and identify important things such as exam dates, assignment details, etc., mentioned by the lecturer. As the 1st step of this process using three keywords: Exam, assignment, and important, the notices, exam/assignments dates mentioned will be identified and saved in a different document. 
Then the rest of the document will be summarized using extraction-based summarization. After the summarization, the summarized lecture will be converted into a pdf and provided to students. The important notices and important points of the lecture will be displayed separately. The recorded lecture will also be provided as they can use the recording if they want to know more details after reading the summary. 


### Monitoring Lecturer Performance

The concept of this component is not to judge the lecturer by their actions but to distribute accurate data among the higher management in order to help them make better decisions in lecturer evaluation. Once the lecture is initiated, the back camera will start monitoring the lecturer and start tracking the body movements by detecting the postures. This procedure will categorize the lecturers' behavior under 3 postures which are sitting, standing, and walking. By analyzing those postures by calculating from the number of frames that the video has captured, showing the percentage of each posture with analyzing each frame when the lecture has been ongoing. The main purpose of the above process is to identify the behavior of the lecturer in the classroom. In the emergence of the lecture, the lecturers’ audio data will be recorded and converted to text for further analysis. It will be analyzed under two main methods named quantitative and qualitative analysis 


## System Architecture

![System Acrhitecture of SLPES](SLPES_System.jpg)

As mentioned in the given figure, this system will mainly consist of a mobile and web application. When the students and the lecturer enter the classroom, their behavior will be monitored by two separate video-cameras and a microphone. The collected data will transmit to the server through raspberry-pi and stores the gathered data in the MongoDB cloud database service. While the video camera tracks both facial expression and body movements, the microphone will record the lecture for further analysis. The student attendance will register by capturing facial identity using deep learning and computer vision mechanisms. Students will be monitored by capturing their facial and body movements by Computer Vision and Deep Learning algorithms. The lecturer audio and video details will be gathered and stored for analyzing as such lecture summarizing and monitoring lecturer performance using machine learning, deep learning, and computer vision technologies. While the students have access to the mobile application, the lecturers and the higher management have the authority to manage the web-application. Frontend components will communicate with the backend using Django REST API.


## Module Description

### Branch structure

The following list contains the respective branch names for each individual

*  **db_and_monitoring** - IT17097284
*  **monitoring_student_behavior_IT17138000** - IT17138000
*  **IT17100908** - IT17100908
*  **IT17098960_RSD_FERNANDO** - IT17098960 

Note that the **QA_RELEASE** branch is the merged branch of all the working components. The **master** branch contains only the common folder strcuture of the project.

### Folder structure

This folder structure adheres to the general project structure of a django web project. The following list describes the folders and files in use throughout the entire project.

* **AttendanceApp** - comprises the work of *Attendance Register* component.
* **FirstApp** - contains the implementations of *Monitoring Student Behavior* component.
* **LectureSummarizingApp** - constitutes the workings of the *Lecture Summarization* component.
* **MonitorLecturerApp** - comprises the implementations of *Monitoring Lecturer Performance* component.
* **integrated_slpes** - this contains the general settings and url configurations of the entire project.  
* **manage.py** - this file is required to run the project (done by the command `python manage.py runserver`)
* **requirements.txt** - contains the list of dependencies required to run the project.

### Application strcuture

As mentioned above, the folders such as *AttendanceApp* and *FirstApp* are the applications reserved for each individual module. The following list will elaborate the common folders and files in an application folder.

* **logic** - this folder contains the basic logic implementation for the application (implementations of deep learning models, processing etc...).
* **migrations** - comprises of the database migration files (*migration* in django means a way of propogating the changes made to your model, to the database schema). This is a folder generated by the django framework.
* **templates** - this folder contains the html files (frontend) of the relevant application. However, the templates should be added inside another folder, with the same name as of your application.
* **admin.py** - registering the models to the django admin site.
* **api.py** - contains the RESTful API for the application.
* **models.py** - all the database models are created within this file.
* **serializers.py** - django provides a library to serialize the database models to facilitate the communication through REST API. The serialized class for each model is created here.
* **tests.py** - test case implementations
* **urls.py** - contains the specific url mappings for an application.
* **views.py** - implements the frontend logic for each template file.

## Installation

Please Note that, the following instructions are only applicable for Windows 10 operating systems.

### Prerequisites

The following software tools needed to be installed before running this project

1.  Anaconda 3, (click **[here](https://docs.anaconda.com/anaconda/install/)** to download. Select Python version as 3.7 or above).
2.  PyCharm Professional Edition with Anaconda Plugin 2019.3.1 (click** [here](https://www.jetbrains.com/pycharm/download/other.html)** to download).

### Procedure

To run this project, you will need to follow the given steps.

1.  Clone the repository to a desired location in your machine.
2.  In this location, open the command line and create a virtual environment by typing `conda create -n <ENV_NAME>`. (if this command does not work, type `CALL conda.bat create -n <ENV_NAME>`)
3.  To download all the dependencies, run the command `pip install -r requirements.txt`.  
4.  To download the classifiers, visit **[here](https://drive.google.com/drive/folders/1xIZfX47owHUXYqN13hchQ4waVW5UE5rp?usp=sharing)**
5.  Once the classifiers are downloaded, include them in the application folders as follows.
       * Create a folder named *classifiers* inside the *FirstApp* folder.
       * From the downloaded classifiers, include the content in *monitoring_student_behavior* folder inside this *classifiers* folder.
       * Create a folder named *models* inside the *MonitorLecturerApp* folder.
       * From the downloaded classifiers, include the content in *monitoring_lecturer_behavior* folder inside this *models* folder.

6.  To download the static content of the project, visit **[here](https://drive.google.com/drive/folders/1CtoRFxXRThE9vhsP2b0xJXhUTyPh8kN-?usp=sharing)**
7.  Once the static content is downloaded, include them in the project folder as follows.
       * Create a folder named *static* inside the project root folder.
       * Put the static content inside this folder.
       * Then open a command line in the project root folder and type `python manage.py collectstatic`. This will create another folder named *assets* inside the root folder.
    
8. Now open this project in the PyCharm IDE.    
9. Before running the project, the url of the database needs to be configured. The steps are as follows.
      * Go to *integrated_slpes/settings.py* file and locate the *DATABASES* variable.
      * Inside the variable named *default*, copy the value under the variable *HOST*.
      * Then in the left sidebar, select the *project* tab and from the dropdown the upper right corner, select the option *project*.
      * Now select the *External libraries* section and your set of Python libraries.
      * Go to *site-packages* folder and find the library named *pymongo*.
      * From this library, open a file named *mongo_client.py* in the IDE.
      * In this file, there is a variable named *HOST* inside the *MongoClient* class. Now the replace the value in this variable with the copied url value.
      * Then go back to the terminal and type the command `python manage.py runserver`. The server will be opened in the address *127.0.0.1:8000*.


### Dependencies

The required dependencies could be found in [requirements.txt](requirements.txt) file. The main dependencies required for this project are listed below.

*  `Python` (version 3.7+)
*  `Django` (version 2.2.11)
*  `Djongo` (version 1.3.1)
*  `djangorestframework` - (version 3.11.0)
*  `tensorflow` - (version 2.1.0)

