In a project that exemplifies technical prowess as well as a commitment to social impact, the students of Shiv Nadar School, Gurgaon, have succeeded in creating a smart glove that can translate sign language into spoken words. Communica, the winning project at the school's annual Colloquium, aims to revolutionize communication for the hearing impaired by addressing the critical need for accessible communication methods in India.
Developed by the team of Aaditya Seth, Samaira Taneja, Armaan Srivatsan, and Sehraj Kapoor under the mentorship of teachers Nimesh Kumar and Nitesh Kumar, the glove integrates advanced sensors, machine learning, and web applications to offer a practical solution for the hearing-impaired community.
In this interview with TheCSRUniverse, the student team and their mentors discuss the conceptualization, development, challenges and future prospects for their prototype.
Scroll down to read the full interview:
Student Team:
Q. What inspired you to create Communica? What roles did each member play in the project?
A. The Capstone Project is a 10-month initiative where all grade 10 students work in teams of five to identify and solve community problems using technology. The inspiration for our project, Communica, came when one of our team members visited a petrol pump and encountered a hearing-impaired employee struggling to communicate. This experience deeply moved us, and we decided to address this issue.
Our team consists of four members: Aaditya Seth, the Project Executive and Programmer; Samaira Taneja, the Web Developer who also completed a certified course in Indian Sign Language to interact deeply with the hearing and speech-impaired community; Armaan Srivatsan, the Content Writer and Editor; and Sehraj Kapoor, the Product Developer and Financial Advisor.
Q. Please walk us through the steps you took from ideation to implementation of the project? What were some of the biggest challenges you faced and how did you overcome them?
A. Our team began ideating after discussing an experience shared by one of our members. To better understand our target audience and the challenges faced by the hearing and speech-impaired community, we decided to engage directly with them. We visited a petrol pump and spoke with the hearing-impaired employees and the manager to gain insights into their daily tasks and communication difficulties.
The manager explained the problems the employees faced, highlighting that the only means of communication with them was through written text due to his lack of sign language skills. These interactions inspired us to create an intelligent glove designed to help these employees communicate with customers in a more inclusive manner.
We began developing the glove's design and visited the petrol pump regularly to get feedback from the employees. One significant challenge we encountered was the variation in hand movements, which caused different readings for the same gesture on different hands. To address this, we rigorously tested the glove on many people to obtain average values for each gesture, ensuring accuracy and reliability.
Through continuous iteration and feedback, we aimed to refine the glove to meet the needs of the hearing-impaired employees effectively. This process underscored the importance of direct engagement with the community to create a practical and impactful solution.
Q. Can you explain how the smart glove works? What technologies and tools did you use?
A. The Communica gloves primarily consist of a Flex Sensor, MPU6050 (gyro sensor), speaker, haptic feedback motor, web app, and ESP32 (MCU). Here’s how they work:
Deaf users wear the gloves and make custom gestures. Each gesture produces outputs from the flex sensor (measuring the bend of each finger) and the gyro sensor (capturing hand orientation). Multiple readings from these sensors are averaged to predict different gestures and train a Machine Learning Model.
The ESP32 is configured as an asynchronous web server and serves as the main control system, connecting the flex and gyro sensors. TensorFlow Lite is used to train the ML model for the edge computing device, which is then deployed on the ESP32. After deployment, the ESP32 tests the gloves by asking the deaf person to make gestures similar to those recorded, ensuring accurate predictions.
A user-friendly web app connects to the ESP32, allowing the user to visualise in real-time how much each finger is bent. Once a gesture is recognised, the ESP32 speaks the corresponding sentence or word and provides haptic feedback to the user, indicating that a word has been spoken.
For the final prototype, three different gloves were designed and tested with various MCUs (Arduino Nano, Arduino Lilypad, and Arduino Uno) with different deaf individuals. The ESP32 was found to be the best fit for the requirements, so the final prototype was designed using it.
The Communica glove is rechargeable, lasting up to 6 hours on a single charge, and includes a battery charge indicator.
Q. How did you ensure the accuracy and reliability of the sign language translation?
A. The accuracy and reliability of the Communica glove is ensured through extensive testing with the target audience and continuous improvements based on their feedback. Unlike traditional sign language translators, Communica does not convert standard sign language gestures. Instead, it creates a customised repository of gestures tailored to the user's work profile and specific needs.
Q. Have you received any feedback from the hearing-impaired community? If so, what has been their response?
A. We tested our project with several hearing-impaired employees at a Bharat Petroleum station. The gloves significantly assisted them in communicating with customers during their shifts. They provided valuable feedback on various aspects, such as comfort and battery life, which we used to improve the glove's design. According to their feedback, if our project is developed into a full-fledged product with mass production, it could greatly enhance the social and work lives of hearing-challenged individuals.
Q. Can you discuss the real-world application and potential impact of Communica beyond the school environment? What are the possibilities of introducing the device for broader use in the future?
A. Communica has significant real-world applications and the potential to create a substantial impact beyond the school environment. By facilitating effective communication for deaf individuals, the glove can enhance their ability to interact with others in various settings, such as workplaces, public services, and social environments.
The possibility of introducing Communica on a larger scale is promising. Mass production and distribution of the glove could make it accessible to a wider audience, significantly improving the social and work lives of hearing-impaired individuals. This could lead to more inclusive workplaces and public spaces where communication barriers are minimised.
Furthermore, the adaptability of Communica allows it to be customised for different work profiles and personal needs, making it a versatile tool for various industries. By continuing to refine the technology and gather feedback from users, we can ensure that Communica remains a reliable and effective solution for enhancing communication for the deaf community.
Overall, the broader implementation of Communica holds the potential to foster greater inclusivity and accessibility, making a positive difference in the daily lives of individuals who are hard of hearing.
Student Mentors:
Q. How did you guide and support the students throughout the development of Communica? What teaching or mentoring strategies did you find most effective in helping the students achieve their goals?
A. Capstone is a student-led project where students identify a problem, and we, as mentors, guide them through each step of the process. Our role as facilitators is to provide direction and support by brainstorming challenges and solutions together at each phase of the project, from ideation to prototyping, testing, and implementation. We also help them develop strategies to approach their problem, plan their project timeline, allocate resources effectively, and iterate on their designs based on feedback.
Having brainstorming sessions, challenging students with open-ended questions, facilitating meetings with their target audience, and providing clues when they are in doubt are all crucial strategies that help them achieve their goals. Brainstorming sessions encourage creative thinking and collaboration, allowing students to explore a wide range of ideas and solutions. By posing open-ended questions, we challenge them to think critically and consider multiple perspectives, deepening their understanding of the problem.
Q. Can you share your initial impressions of Communica and how the project evolved over time? What were some common challenges faced by the students, and how did you help them address them?
A. My initial impressions of Communica were very positive. The project was inspired by a real-world problem, demonstrating the students' empathy and commitment to making a difference in the lives of the hearing-challenged community. The concept of creating a glove that translates custom gestures into speech was innovative and showed great potential.
As the project evolved, the students faced several common challenges. One significant challenge was ensuring the accuracy and consistency of gesture recognition, as hand movements can vary widely between individuals. To address this, we encouraged the students to test the glove on a diverse group of people and gather extensive data to create a reliable machine learning model. This iterative process of testing and refining helped improve the glove's accuracy.
Another challenge was the technical complexity of integrating various components like the Flex Sensor, MPU6050 gyro sensor, and ESP32 MCU. We facilitated brainstorming sessions to break down the problem into manageable parts and provided guidance on how to effectively combine these components. By posing open-ended questions, we encouraged the students to think critically and explore different solutions.
Comfort and usability were also significant concerns. The students received feedback from the hearing-impaired employees about the glove's comfort and battery life. We helped them prioritise these aspects by suggesting ergonomic designs and efficient power management strategies.
Throughout the project, we held regular meetings to discuss progress, address doubts, and provide clues to overcome obstacles. We also facilitated interactions with the target audience to ensure the solutions were user-centric and practical.
Overall, by fostering a supportive environment and encouraging iterative development, we helped the students transform their initial concept into a functional and impactful solution. Their perseverance and willingness to learn from feedback were key factors in the project's success.
Q. How does the integrated IT program at Shiv Nadar School prepare students for projects like Communica? How do such capstone projects contribute to the overall educational experience of the students?
A. The integrated IT program at Shiv Nadar School prepares students for complex projects like Communica by equipping them with essential technical skills, fostering critical thinking, and encouraging real-world application. Through a curriculum that includes programming, electronics, and data analysis, Robotics, 3D Modelling & animation and Sound Engineering, students gain hands-on experience with the tools and technologies needed for innovative projects. Emphasis on problem-solving, collaboration, and creativity helps students approach challenges methodically and work effectively in teams. Capstone projects allow students to apply their learning to real-world issues, enhancing their skills in project management, research, and iterative testing. These projects also foster empathy and social awareness by addressing community problems, teaching students resilience and adaptability. Overall, such projects integrate various disciplines, providing a holistic educational experience that prepares students for future academic and professional success while contributing to their personal growth.
Q. What advice would you give to other educators looking to implement similar programs in their schools?
A. For educators looking to implement similar programs in their schools, it is crucial to shift the focus from marks-oriented education to solving current Sustainable Development Goals (SDG) problems, starting from primary classes. Emphasising real-world applications and interdisciplinary learning can make education more relevant and impactful. By integrating subjects like IT, science, and humanities, students can develop holistic problem-solving skills. Project-based learning encourages hands-on experience, critical thinking, and creativity as students tackle open-ended challenges related to SDGs.
Providing access to necessary resources and strong mentorship helps guide students through their projects, fostering a sense of purpose and direction. Collaboration and teamwork should be emphasised, teaching students effective communication and leveraging each other's strengths to address global issues. Engaging with the community enhances the relevance and impact of student projects, making learning more meaningful.
