Authors

Mayar Osama, Rawan Sherif, Alaa Yehia, Salma Mohamed

Publishing Date

April 21, 2021

Abstract

Many people assume that visually impaired people can not live alone or work independently. Vision impairment forces people to depend on others and require constant assistance. The objective of this project is to offer more independence to visually impaired people through developing new features or improve existing ones and integrate it into a compact and accessible mobile application. In this context, the project will focus on four main use cases/features: photo-to-speech feature that allows the user to “read” text from images of printed documents, currency detection feature that identifies the amount the user carries, food detection feature that identifies the type of food served on a plate and color and texture recognition feature that identifies colors and textures and offers a recommendation of colors that could match. To offer optimal accessibility, the user will be able to use voice commands to tell the system the action they want to do and the system will analyze the command and execute the action. The application will use the picture taken by the user, using a smartphone, process the image and with the voice command given by the user the system will be able to execute the command. The output given at the end through all the features will be read to the user through Text-to-Speech, so the user can hear the final outcome of the executed command.

1.1 Background

WORLDWIDE, THERE ARE AROUND 285 MILLION PEOPLE WHO ARE VISUALLY IMPAIRED [1], OUT OF WHICH 3 MILLION OF THEM ARE IN EGYPT [2]. VISUALLY IMPAIRED PEOPLE ARE CAPABLE OF LEADING A NORMAL LIFE, BUT THEY STILL FACE TROUBLES DUE TO LIMITED AIDING TOOLS. THEY USUALLY STRUGGLE WITH SIMPLE DAILY DEALINGS, LIKE KNOWING THE TYPE OF FOOD THEY’RE SERVED OR READING A MENU OR BE AWARE OF THE AMOUNT OF MONEY THEY HAVE OR EVEN MATCH THEIR CLOTHES. THE AIM OF THIS SYSTEM IS TO FACILITATE SOME OF THE SIMPLE ISSUES THEY MIGHT FACE, WITHOUT REQUIRING THEM TO CARRY AROUND ANY TOOLS OR GADGETS, AND HAVE IT ALL HAPPEN THROUGH THEIR SMARTPHONE. OUR SYSTEM WORKS ON THE CAPTURED IMAGE THE USER UPLOADS. THE FEATURES IN THIS APPLICATION ARE GIVEN SPECIFIED VOICE COMMANDS, SO WHEN THE COMMAND IS GIVEN THE CORRESPONDING ACTION WILL BE EXECUTED ON THE IMAGE. THE IMAGE WILL GO THROUGH SOME IMAGE PROCESSING/MACHINE LEARNING TECHNIQUES AND THE OUTPUT WILL EITHER SHOW THE EXTRACTED TEXT FROM THE IMAGE, OR THE AMOUNT OF MONEY THE USER HAD, OR THE IDENTIFIED COLORS AND TEXTURES. AND ALL THE OUTPUTS WILL THEN BE READ TO THE USER THROUGH TEXT-TO-SPEECH.

1.2 Motivation

ASSISTIVE TECHNOLOGY MARKET FOR THE VISUALLY IMPAIRED IS SET TO HIT 6,105.7 MILLION DOLLARS BY 2025. WITH THE INCREASE IN AWARENESS FOR ASSISTIVE TECHNOLOGIES, THE MARKET HAS UNDERGONE A SUBSTANTIAL GROWTH [11]. YET, BECAUSE OF THE HIGH COST OF TECHNOLOGIES, VISUALLY IMPAIRED PEOPLE STILL DO NOT HAVE ENOUGH ACCESS TO SUCH TECHNOLOGIES. OUR VISIT TO AL NOUR AMAL SCHOOL FOR VISUALLY IMPAIRED STUDENTS HAVE SHOWED US THAT EVEN WITH THE SURGE IN TECHNOLOGY, THEY STILL FACE HARDSHIPS IN THEIR DAY TO DAY LIFE ACTIVITIES. AS EXPLAINED BY ONE OF THE TEACHERS AT THE SCHOOL, SOME OF THE STRUGGLES VISUALLY IMPAIRED PEOPLE FACE, THAT ARE NOT THAT RELEVANT IN THE TECH MARKET INCLUDE NOT KNOWING EXACTLY WHAT FOOD IS PRESENTED ON THEIR PLATE OR EVEN HOW TO READ THE MENU AT A RESTAURANT. THESE STRUGGLES ARE VERY RELEVANT AND OCCUR CONSTANTLY IN THE LIFE OF A VISUALLY IMPAIRED PERSON, SO THERE IS A STRONG NEED FOR THIS TECHNOLOGY. AS WELL, BETTER THE ASSISTIVE TECHNIQUES THAT ALREADY EXIST, LIKE COLOR AND TEXTURE MATCHING OR RECOGNITION OF EGYPTIAN MONEY BILLS, AND DEVELOP THEM ALL IN A WAY WHERE THEY BECOME ACCESSIBLE TO BLIND PEOPLE. AS WELL, NOT REQUIRE THEM TO BUY MORE EXPENSIVE ASSISTIVE GADGETS AS THE HELP COULD ALL BE GIVEN THROUGH THEIR SMARTPHONES.


1.3 Problem Statement

VISUAL IMPAIRMENT DOES NOT ALLOW BLIND PEOPLE TO BE COMPLETELY INDEPENDENT, WHERE THEY HAVE TO RELY ON OTHER PEOPLE TO EVEN PERFORM SIMPLE DAILY ACTIVITIES. ONE OF THE MAIN CHALLENGES THEY FACE WHEN DEALING WITH MONEY IS THAT EGYPTIAN MONEY BILLS DO NOT HAVE BRAILLE ON THEM. HAVING BRAILLE ON BILLS HELP VISUALLY IMPAIRED PEOPLE TO IDENTIFY THE VALUE OF THE BILL THEY HOLD. SO THEY’RE MORE LIKELY TO BE EXPOSED TO FRAUD THAN OTHER PEOPLE. AS WELL, THE USERS USUALLY DEPEND ON OTHER PEOPLE TO CHOOSE AND COORDINATE THEIR OUTFITS, BECAUSE THEY DON’T KNOW WHAT IS THE COLOR AND TEXTURE OF THE ARTICLE OF CLOTHING THEY’RE PLANNING TO WEAR. ALSO, THE USERS STRUGGLE WITH READING MENUS AND WRITTEN DOCUMENTS. AS WELL WHENEVER THEY’RE SERVED FOOD THEY NEED SOMEONE TO TELL THEM WHAT WAS SERVED