Mobile Apps and Mobile As a powerful tool

Hello all, welcome back to our course on digital accessibility. ility and today we will talk about uh mobile and mobile apps and how they can be leveraged in uh making or innovating new accessibility devices. accessibility innovation and how it is uh a very powerful accessibility tool by itself just by its nature of multimodality and just by its nature of being largely available at a consumer level. So first is definitely we've been iterating throughout this course that why accessibility matters and accessibility is about inclusion and not just compliance. Uh it's not just about um you know following the law or um you know not uh uh following some compliance and guidelines of WAG. It is beyond that because we are enabling access to more than a billion people living with disabilities according to who right and disabilities can be uh of various kinds permanent temporary situational and mobile devices uh by its very nature of being now readily available at different price points considering it has uh different kinds of sensors. Now it is uh also has internet enabled and all of those facilities and plus it's a small handheld lightweight device. By its very nature it um plays a very key role not just in everyday interaction uh but also in everyday access. So uh why we should as innovators as engineers as designers think about mobile accessibility to begin with. One is uh as I mentioned that they are personal devices. They are portable devices. They are available with you uh at all times. Most of them most of the individuals have separate devices each one to their own where customization personalization is also possible. So that gives us a very strong advantage uh in any design in any um uh new technology, new media that we would like to build. Um because we are constantly talking about customization and personalization and um how um technologies which are adaptable uh to uh varying needs of varying individuals can be of more benefit. Uh then the second most important uh aspect being that it is a very low cost uh device compared to dedic dedicated assistive devices. So when we say dedicated assistive devices in some of the uh previous sessions we have spoken about something like uh refreshable braille display something like note takers they are some devices which are specific devices and the scale uh doesn't justify bringing down the cost a lot. uh plus they may be high technology investment high R&D devices also that also carries in the price point. So uh but mobile devices are largely very um you know multifaceted plus they are available at different price points right right at the nominal starting point of uh 5k 3k uh uh rupees in India and then of course you can also buy um a used or a refurbished model which also has a higher processing uh capability at a much lower cost as well. And today we are all aware that um mobile phones uh is used across most verticals like uh education, healthcare, finance, communication etc. And um all of these industries are leveraging individual um uh you know information, individual access and uh that can be a very important entry point if we are thinking of innovating for the um special use cases as well. So smartphones uh can um you know replace or augment uh assistive hardware primarily because it uh can support multiple accessibility needs. It has softwares which can be continuously improved. Customization is possible and personalized access is possible uh using mobile technologies. There are also uh various builtin accessibility features uh in mobile phones uh particularly at the OS level itself. So I'm sure like we have spoken about talkback. We have spoken about um assistive touch in the iOS. We have spoken about voice over and there are other accessibility features like um uh dark background, high contrast ratio and all of those things or multiple multimodel feedback mechanisms. Right? So uh if a call comes you can have flashing lights as well as vibration virotactile feedback as well as a sound feedback. So depending on your needs you can activate or deactivate the different uh feedback systems. Plus, it is also uh contains several um you know sensors which can uh garner or gather information about the environment um including your camera, your microphone, your GPS, uh your Bluetooth uh and all the other uh you know balancing sensors and distance sensors and all of LAR sensors which we can leverage in order to um know more about the surrounding which the human our user is interacting with and use that information into processing and delivering an appropriate feedback. So models such as obstacle detection and all of that is possible to implement on a mobile phone. So a person for example who is walking and navigating um a space in um where you know he or she is a person with visual impairment or blindness he or she wants to navigate through that space uh but there may be certain obstacles. So if there are several such apps available which are leveraging uh your camera input, they are processing it in real time and giving the user the feedback that okay so and so is there in front of you. Right? You you may not always want to touch and then identify what is there in front of you. Right? So it may be an animal, it may be a dirty uh dump truck, it may be um a person who doesn't want to be touched, it may be a stranger, right? So uh it's not always uh you know touching and then identifying. So then in those situations an obstacle detection or a obstac object identification algorithm which can be implemented using the camera input of your mobile phone can be a very useful tool as an assisted technology tool again simply implemented on your mobile phone. So we are leveraging the hardware and just augmenting the existing sensors with a software layer so that they can uh enable uh access to the uh environment or any other information. So for example in an upcoming uh product demo we will also showcase one such smartphone based uh assisted technology system where we are again leveraging the use of uh camera input and uh we are trying to read um the you know NCRT CBSC textbooks where the user is actually just moving their hand on the page It's a printed textbook. There is no tactile feedback. The tactile feedback is just for the page flipping and the corners etc. So that they get some feel of reading a book. But then the based on the gesture recognition through computer vision models um the the application is able to then narrate the whatever is written on the paragraph where you are pointing. And um similarly we are able to annotate uh different variety of um geographical information on one uh uh geography map. So for example uh there may be a crops map. I'm sure you all might have studied in school. Um there is a crops of India maps. There is a industries. There is a political map. There is a physical map. All of them are different maps. So but I if what if I tell you that there can be an application where there is one tactile outline of the India map and a person or a student who is blind can just explore that map and select which information they want to hear and based on the location of their finger that application is able to tell them okay this is where cotton is grown this is where highest rainfall happens this is where Maharashtra is etc etc. So now let us talk about this is these are few examples where we uh can think about leveraging the hardware uh and augmenting it with some uh another layer of developed software in order to develop an innovative assistive technology solution and you will see a demo in a upcoming session. Um now let us talk about some of the accessibility features which are inbuilt on the mobile phone and which are very helpful uh for users in their day-to-day needs. So one is definitely your screen readers for persons who are blind and low vision. So uh which is again your talkback etc. So you can just explore those features on your phone right now. You can go in the settings and uh of your phone and then there is an accessibility setting. You can turn on the talk back and it will start uh telling you each word and the navigation keywords uh in an audio format. Then there is magnification and display adjustment. So the fonts can get bigger and you can uh change that setting on your phone which is your master setting. So any app that you open uh will also follow that master setting of your phone. Then there can be voice or speech input. Now we all have you know iOS users have your Siri and uh uh Android users have your okay Google um you know interfaces where you can actually you don't need to even navigate to a certain app. So, for example, I want to schedule a meeting for Friday 400 p.m. I can just say, "Hey, Siri, please schedule a meeting uh with so and so on Friday at 5:00 p.m. uh so and so location." And Siri will be able to do that in your calendar and set a reminder for the same, right? You don't even need to navigate uh spend the cognitive energy, spend the time to do it. uh and that itself is an as accessibility feature. Uh then there can be switching uh between alternative input access. So voice uh of course voice and speech is one talk thinking about linguistic um inclusion. So your keyboard also has Hindi or other uh Indian languages. There are of course global languages enabled. So Spanish, French etc. and you can choose in what way you want to interact with uh the system the mobile phone. So I think that itself uh has these are some of the inbuilt features maybe you can discuss on the forum about what other features you might have recognized or identified on your mobile phone which uh we might have missed mentioning here. So now let us talk about each one of them in a little bit more detail. So uh what are screen readers on a mobile? So they convert onscreen content into speech or braille. Uh they support gesture based navigation. They support gesture based navigation. So if you want to go to the next item, next interactive item. So there are four buttons on your interface. I am currently in heading one, button one, whatever. I want to go to heading one, button two. So I can just keep swiping right. So the keyboard interaction for a screen reader is tab. But on a phone it is swipe. So it is gesture based. And um but to in order to make sure that your application uh is well aligned with this um mobile screen reader you have to uh work with evaluate with these screen readers uh at the code at the development level itself. So we will talk about this much in much detail in the upcoming week where we'll talk about evaluation methodology and evaluation pipelines. Uh so the the point is that we need to have well structured UI elements. So of course examples are that uh Android phones have talkback uh iOS phones have uh voice over. Both can be just simply enabled by going to settings and uh just turning them on through accessibility dialogue box. Magnification and visual adjustment. So what happens in this case is that uh you are able to zoom onto the screen. Uh there can be gestures which are magnification gestures where you are you know kind of pulling your two fingers apart and it is able to zoom. You can adjust font sizes or other display scale. Uh you can change the color contrast. You can do color inversion also. So for people with low vision or light perception they can use color inverted modes. Uh so these kind of features can support users not just with low vision but also with um uh people uh users who are elderly. So many a times uh they would require larger touch targets for example. So magnification can help us in making the touch targets bigger as well. Uh the font sizes if they're too small, the screen is too cluttered in terms of reading. Um and sometimes uh uh elderly users may be facing hypertropia uh which is in layman language called nearsightedness. Uh so uh there um you are unable to read when something is at a reading level right so if it's something very close if it is too small you're not able to read it properly. So then those font adjustments can be utilized and fonts can be made bigger easily. Then we spoke about the voice based interaction. So uh that enables handsfree interaction. So it is a boon for uh people with motor impairments. We will uh talk about this aspect uh in the next session as well. And it is very useful for people who may have motor impairment or motor fatigue issues or people with cerebal pulsy where the fine motor skills or gross motor skills are largely affected uh by their uh impairment. And uh it can include your voice typing. So dictation you don't need to type a message. You can uh you know just voice um press the mic mic button and um ask it to you know write the message or whatever you want to say. Hi, let us meet at so and so at so and so uh and or voice commands as I mentioned that hey Siri say book uh book a meeting slot for me for Friday uh something like that or uh it reduces reliance on precision touch input right which is also again people who are elderly may face problems with precise touch targets precise iz touch uh interaction requirements and then uh people with cereal pulsy or um other motor impairments may have similar uh uh issues which can be easily overcome by using uh voice- based interaction which a mobile phone even the ones which are very uh available at a very low cost can afford all of these um u features. Now then touch interaction challenges. So small touch targets create barriers, right? So complex gestures are hard to remember sometimes. Uh so and motor tremors may affect uh accuracy. So design your application design should think about you should think about at the time of design itself about tolerating error about all of these variability of interaction and uh if you um again go back to your basics and look at uh the W CAG guidelines there is a lot about uh you know touch targets etc. uh but uh I think in this um uh the uh because largely WKAG is for web based interfaces the new uh version WCAG 3.0 O talks a little bit more about um mobile interfaces and you know a portrait based smaller screen interface largely. Uh so I think some of the aspects can be picked up from there but one should not just rely on compliance one should also keep uh evaluating with users at a continuous uh at a continuous basis. if particularly you are developing uh new solutions. So mobile accessibility uh is very useful particularly in education because accessible ebooks and learning apps are available. Audio supported learning content is available. Support for captions and transcripts are available. uh it allows for independent learning. So then there are multiple examples I'm sure you also might have used of accessible learning apps. So uh for example screen reader friendly navigation they're looking at that uh audio feedback for interaction to adjustable text and layout they support diverse learning needs. So all of these are some of the examples of learning app uh learning apps which have all of these features uh ingrained in their interaction. Another important vertical where um a mobile phone can be very useful in terms of accessibility is your navigation and mobility. So uh since this is a portable mobile device it carry you can carry it uh with you at all times uh and largely I'm sure all of us have seen uh people when we are driving we are putting it for map access or when we are riding a bike we are putting it for map access. The same can be true for uh people with diverse needs and um it is a very uh reliable uh source for GPS-based navigation assistance already. It's a uh GPS-based navigation is in its very robust stage. All of us have used Google maps or Apple maps I'm sure to navigate their way through a new city or uh if they are going to a new place in the same city. uh so I think that is there then public transport and wayfinding apps are also there particularly say for bigger uh cities like Delhi uh there is u like a metro rail network which is quite complex so they have their own apps so okay if you are here which station you can board which is the nearest station and if you want to go to say a red fort which station do you deboard how many times do you change the train and all of that. Uh then it also kind of tells you audio directions, gives you vibration cues. So multimodal feedback again and again we are saying that a mobile device is capable of doing that. So the app design should leverage the all of these feedback mechanisms and it is very critical for independent mobility because if somebody uh has a sensory deprivation one kind of sensory deprivation somebody's hard of hearing they can rely on the visual or the viroactile feedback. If somebody's uh has low vision or uh visual impairment, they can rely on the audio and virotactile feedback. Uh same is the case with um if you are distracted. For example, you're driving, you're distracted. Uh your your attention to both the visual or the audio stimulus is not there. Then your third uh uh feedback system which is your vibration can pull your attention back to uh the navigation system maybe. So it is very uh critical. Other than that uh communication and expression in terms of speech to text for users with hearing loss uh text to speech for reading support then there are AAC apps for expressive communication. Uh this is largely for people with autism etc. who are who are not able to kind of remember words for everything. Uh they can you know kind of communicate through such applications or people with cable policy or people who are who have speech impediment they can uh you know use such apps. It also supports social inclusion definitely because these are not specialized products. Everybody today has a smartphone and if we are able to leverage that hardware, it's not something weird looking. Uh it is not something only I am using. If there is a specialized solution, we may think that yes, it is a good innovation. It is going to help uh the person who is in need. But if you take we have discussed about this that even a specialized solution is actually not inculcating equality. It is still it is enabling access probably but it is still inequality. Right? So then because it's not the same as what everyone is using. I am getting a special treatment. Right? So then that is also something uh people with um uh impairments are not uh you know they're very careful about that. So they're also looking for social inclusion that I am part of uh the society. I want to fit in. I am same as everybody. I don't need any special treatment. Right? So that uh individual positive mindset is also enabled through something which is an inclusive product rather than a specialized product. Then there can be some other multiple everyday accessibility scenarios where uh a application a mobile application can be of much use. One is of course your mobile banking and payments um your health care access and tele medicine your shopping ego governance social media participation it's so I want to buy something I want to pay you know just by scanning um you know a QR code I don't need to you know go through the long tedious uh uh route of entering somebody's phone number or account detail to transfer money. It's a quick way of uh you know giving somebody money and or if I want to buy something from an e-commerce website. I mean all of that uh is enabled through um uh mobile phones. Now let us talk about so of course uh when we are designing for mobile applica for mobile when we are designing applications for mobile there may be some uh inclusive design principles which have to be kept in mind. So we have to design for diversity and not the average user. We have discussed it in detail in some in one of the earlier sessions. You can maybe go back to that also. uh you we can provide multiple ways to interact. Uh so multimodal interaction is the key. You have to provide options so that it is inclusive to a larger number of audience. Avoid reliance on a single modality. This is something very important. And allow user control and customization. So personalization of the settings, the personalization of uh display your voice, which modality I prefer, which one I want to interact with, what uh screen contrast I would uh prefer, what font size I would prefer. All of those should be part of your design. And of course, uh referring to your W CAG guidelines can be a good starting point. Then our design should also be responsive and adaptive. So now with AI and AI based solutions, it has become u um you know need of the hour that our designs are adaptive. Uh it contains dynamic content. It can uh kind of iterate itself based on the personalized need of the user. So and other than that there can be some basic robustness related aspects as well when we talk about responsiveness. So for example supporting different screen sizes or different orientations. Uh it is able to respect the OS level accessibility settings. So when you are designing your application always remember that it should your system accessibility setting should uh take priority it should not override your uh um you know uh visual design or something. So if the user has enabled certain font size on their mobile phones but your app doesn't respect the OS level accessibility settings then your app will have smaller font and they will not understand why um it is happening and there is a good chance that they may not use it. So if uh in order to enable the fact that um you are able to retain those customers as well uh make sure that your app is able to respect the system accessibility settings. Um so then uh other than that uh semantic meaning of the content uh is also important uh particularly from a screen reader sequence perspective. uh and we have spoken about this uh um in one of uh another session where we have spoken about parallel architecture and all of that uh essential for inclusive mobile experiences. So yeah, so it is all of these aspects are quite essential to create inclusive by experience. So some of the commonly occurring accessibility barriers may be your low color contrast that's the most common then your uh missing labels u particularly in form fields or your uh all text is missing in buttons etc or your gesture only recognition gesture only interaction is there but if I want to do uh through the u GUI I am not able able to do it. So poor focus management. So if you have too many visual elements on the application including ads etc. it may be very difficult to focus on what to do or where is the call to action. So why it is important to understand the barriers. So because um small issues like these can fully block access and while they are they seem to be very obvious barriers. Uh unfortunately most of the applications are failing uh in compliance just because of these very bare minimum barriers and many times at a coding level the developers are not able to understand the correct use of labels. uh context uh dependent use of labels or um um you know they are not able to understand how to leverage area labels for example in a web interface. So it is very important to think about all of that and barriers compound for disabled users. So all of these uh may be uh you know one kind of a barrier like for example for a low vision user uh the color contrast may be an issue. Um but I mean for somebody who can see they may it may be a hurdle but they might be still able to interact with it but for somebody who is low vision uh then it is almost impossible for them to interact with your system. So such kind of accessibility failures they reduce independence and good design should focus on removing all of these friction. So of course um uh all of these uh interactive elements amount to a major chunk of the accessibility barriers and accessibility compliance related aspects. But uh another very important aspect when we which we have to keep in mind particularly when we are developing for mobile users is uh in specifically in uh you know rural and remote areas there may be low resources or low limited internet connectivity or um older or low-end uh low processing power devices are used or many a times people don't have their own uh uh individual mobile phones and maybe one mobile phone is being shared among four family members. So it is important that uh we should able we should not rely too much on um cloud-based interaction so that every interaction is uh fetching some data from the cloud. That means that there there's a lot of dependency on internet right so one should think about leveraging offline uh solutions and uh leveraging lightweight solutions so that uh even in limited connectivity or low processing power uh mobile phones your applications are able to run smoothly. Then of course uh we have spoken at length about WAG principles and how they um you know can be divided under the POU are perceivable, operable, understandable and robust. the same principles. Although these principles were initially built for the web web interfaces, but they can definitely be extended to at least from a design principle perspective very well applies to your mobile uh application design as well. We have spoken in depth about um WCAG principles each one each category in uh one of the earlier sessions. So you can go go back to that session and go kind of revise it because this is one important aspect uh which is coming up again and again. So WAG is something by the end of the course you should be thorough uh with. So other than that uh what we can think about is uh we u we can think about content descriptions uh for UI elements. We should think about logical focus order uh particularly for uh the screen reader sequences. uh then the minimum touch sizes should be either uh um you know scalable as per the requirement or to be on the safer side. It can if it allows or your visual design allows and if you can make it you know slightly bigger as compared to the required minimum sizes why not uh voice over integration or talkback integration which is your screen reader mobile of the mobile phone. You have to keep testing with those uh tools in order to understand uh you know about the logical focus about um all text etc. What is whether the all text is making sense or not? Are multiple labels being read at once? Uh so all of those uh things uh matter and dynamic type for scalable text uh accessible controls and settings uh and con consistent focus behavior. So one should uh so what is the meaning of consistent focus behavior that for example there is a workflow where there is um you know few questions on one page and then you have to go to next page uh or and also there is a back button. So if on every page the position of the next button keeps changing it will be very difficult for users to interact with that interface. So that consistency is required or the color of the button keeps changing or uh the labeling is not there uh in some of the pages then it becomes very difficult if it's a long form content particularly. So now um beyond all of this there is definitely ethical aspects related to mobile accessibility as well. one is of course your compliance related laws. uh and of course other than that uh now accessibility has been identified by the government of India as a human right and respecting you user dignity and autonomy should be the primary uh goal of compliance and um particularly targeted exclusion through design it can be punishable as well by law and design choices have to think about social impact in the long run. So some of the future directions uh which are possible in mobile accessibility is of course your AI enabled personalization uh context aware uh assistance is possible multimodal interaction we are anyways doing uh in in your mobile accessibility all of these aspects can allow for greater independence for uh users of varying needs. So at this stage if you are a designer or a developer uh or a student who is just learning about accessibility um just learn about accessibility think about accessibility from early stages of design or development itself. Plugging it in at a later stage can be very difficult. And we have spoken about this point multiple times that when you are developing uh any particular product, it can be an application or a website, any digital product, you have to keep testing it with a variety of users. um and test with assistive technologies like screen readers. Uh test on various platforms or play various kinds of devices to understand low resource settings as well. um learn from real users and always remember that accessibility uh is an ongoing process and uh even if your even if your application is kind of passing the accessibility guidelines or criteria success criteria there may be some aspects of interaction which are getting affected which can be improved. improved which is coming from user feedback. Please don't hesitate to in include that as well. And as the technology is developing and our phones are getting smarter by the day as well as cheaper by the day, uh I'm sure it'll become even more robust and even more uh easier to leverage this technology in our day-to-day requirements more so with AI powered personalization and context aware solutions. So uh that is it. So some of the key takeaways from this session can be that uh mobile devices can democratize accessibility because it is uh largely now available with everyone across the board. uh be it a remote low remotely located person or a person uh from high income to a low-inccome strata everyone is carrying a device and that is a very very strong um aspect which we can as developers leverage. We can think about largecale deployment. We can think about large scale inclusion. We can uh think about including the diverse linguistic, the diverse um uh age related dem demography, the diverse cultural demography as well as the users with special needs into our um use cases and the list of use cases in order to make um applications more accessible and more user friendly. and uh um built-in features on the mobile phones anyway support diverse needs. So even if you are working for organizations which are developing hardware aspects or which are developing the OS uh or OS related aspects for your mobile phone that can also be uh you can also think about that aspect and it is always good uh to have multi-model interaction enabled in any uh mobile device and any good design removes barriers and accessibility is going to benefit a large chunk of users. So uh I hope this this session can kind of open your mind towards leveraging mobile mobile devices for accessibility uh and innovation of uh assistive technologies uh in your you know thought process or if you're already working for organizations which are developing large scale deployed um apps. So you know like uh like the ones from Google's or uh the blinket or zomato and all of these which are large scale deployed applica applications. So maybe you can also uh kind of share this knowledge with your team as well and inform them about accessibility and how compliance is now regulated by law. And it is important to not just uh comply but also include a wider chunk of audiences to drive profit profitability as well. Right? So um good to talk to you uh in today's session. I will see you in the next session. Thank you.