<h1align="center">Interactive, visual learning-based tool for hearing impaired children to improve language skills.</h3>
## Main Objecive
To implement an interactive Mobile and Web-based learning tool for hearing impaired children and to improve the overall language proficiency while bridging the skills gap between the hearing peers. This should be accomplished by using visual learning and speech therapy in an engaging way.
## Main Research question
Hearing impairment is a sensory disability that hinders linguistic development. Language proficiency among the hearing-impaired community is reported to be low compared to hearing peers [2]. Research has proven that if a child has not acquired a first language in early childhood when the brain plasticity changes, there is a risk that the child might not be able to be completely fluent in any language in their life [3]. Language acquisition at an early age is automatic if it is engaged regularly and meaningfully [1].
Children with disabilities have different mental operations compared to hearing peers, which leads to sensory and/or motor deficits, making reading and writing more challenging. Researchers have found that hearing impaired people typically find reading and writing challenging compared to hearing peers [4].
Reading and writing involve a complex thinking process. Writing involves the choice of words, organization, purpose, audience, clarity, sequence, and transcription. Competency in writing is greatly influenced by the vocabulary, knowledge of synthetical structures, planning composition, reviewing, and revising. Researchers have found a link between disabilities and linguistic skills. Missing “phonological processing” is responsible for poor reading and writing skills among hearing-impaired children. Poor phonological awareness holds responsible for the difficulty in identifying and remembering orthographic forms of words and difficulty in guessing the order of the sounds and poses [4].
In order to develop the linguistic skills of hearing-impaired children, primarily visual learning methods are used. Most existing systems provide translation between spoken language to sign language which does not necessarily aim to develop linguistic competency. To teach young hearing-impaired children, software-based visual learning tools are used in schools. But they often have a limited vocabulary and are does not provide complex learning materials that are crucial to learning the choice of words, word organization, synthetical structures, etc. Those systems do not support explorative learning. Explorative learning at an early age is crucial as if the child does not develop communication and language, it is difficult to achieve later [5]. Being able to keep the visual attention of the child is crucial to success in visual learning which most of the existing systems have not addressed.
## Individual Research components
### IT19188928 – M.M.S. Udbhasa
Lip reading and speech therapy
- Lip reading is the main mode of communication among hearing impaired and normal community. The input from the camera captures the facial landmarks and lip movements. The audio will also be captured concurrently to assess the similarity of pronunciation to they will be assessed using convolutional neural networks (CNN).
- A generated text will be shown, and corresponding lip movements also visualized for the pronunciation.
- According to the user’s pronunciation a similarity score will be calculated based on lip detection and the audio processing.
- Using the above-mentioned technologies, several speech therapy practices can be implemented with the help of therapists.
Contribution to develop the virtual character
Contribution to the crowdsourced data insertion system
### IT19001718 – L.L.P.S.M. Lelkada
Learning material generation and NLP module
- Based on the previously learned words and newly learned words, generate meaningful sentences. Using those generated sentences, they are presented in a way to master the ability to understand the synthetical structures, ability to guess, and improve the vocabulary.
- Orchestration of visual engagement between the texts, images, and sign language presenter according to the level of understanding of the user.
- Generating content that interests the user to keep engaged with the application and maintain visual attention.
Course content development
Contribution to the crowdsourced data insertion system
### IT19954974 – P.D.M.K. Priyanka
Student progress tracking and performance based adaptive lesson planning
- A student can learn basics through a course series optimized to the level of user.
- There are quizzes designed for determining the level a user belongs to. Reinforcement learning is used in this and with the use of this, a user is leveled up upon successful completion of a quiz. Furthermore, a user is leveled down and given a lesson series upon unsuccessful completion of a quiz.
- Rewards are given based on the level completion and a digital certificate will be offered to the user on completion of all required levels and lessons which are needed to be recognized as an expert in using sign language.
Course content development
Contribution to the crowdsourced data insertion system
### IT19142838 – B.W.E.K. Senarathna
Surrounding object detection and identification for visual learning
- YOLO algorithm will be used to identify multiple objects at once and user may have the opportunity to select the relevant object need to learn.
- For highlighting a specific object in the surrounding using computer image segmentation will be used, and a pixel-wise mask will be used to identify an object distinctly.
- Incorporating the identified objects with the sign language signals and showing them to user with visual means and storing them in the database.
Contribution to develop the virtual character
Contribution to the crowdsourced data insertion system
