Emerging Input Technologies

Good morning everyone. Uh welcome back to our course on digital accessibility and today we will discuss a little bit about emerging input technologies. So let us start with our uh current scenario and what are input technologies in general. Let's try to understand that first. So uh in a computer computer based system or an interactive system usually the commonly known uh um input devices can be your keyboard where you are entering text uh your mouse which as which has a button or a scroll interface which helps you in uh pointing to or clicking in some onto some buttons. There can be graphic tablet-based interfaces where there is a pen which is commonly called Wacom. The company Wacom makes graphic tablets where there is a pen based mouse pen shaped mouse and you can draw onto the Wacom tablet and it directly translates into a graphical uh digital uh drawing. Then there can be mic which captures your audio. Uh biometric systems which can uh actually scan your fingerprints or other iris related uh things. There can be scanners which is you know part of your printers where it is almost like clicking a photograph of your um paper or your document. There can be webcams which are uh basically camera based devices which capture live videos or uh helps you in video recording. Then there can be other devices like barcode scanners which help you which is like an um um RFbased scanners and then they uh help you in scanning um barcodes and QRS in real time very quickly. So these are some of the traditionally known um input devices. Of course, the keyboard and the mouse are the most commonly and the mic we can say uh are the most commonly used interfaces uh by most of us in our day-to-day interaction with uh computer systems. Then there is of course your mobile screen interface which we are missing in this traditional uh uh landscape is something which is a touchscreen based interface. It can be it can be a tablet can be a phone and uh that has capacitive touch where you can actually touch on the screen itself. So basically it's a it's a version of a mouse where the screen itself is um uh touch sensitive. So basically like a graphic tablet where there is an external pen and an external surface or a mouse where there is an external external device and then that device is rubbed or moved on an external surface where the movement is only translated on the screen. The same thing kind of translates in a capacitive touchbased interface where your finger movements and your finger based pointer and touch is the cursor is the pointer and it it you can also do um continuous movement scan. So basically you can draw on your phone also. Uh so capacitive touchscreens are one of additional input devices which we interact with on a day-to-day basis. So traditionally uh the input technologies like uh keyboards, mice, mice is a plural for mouse just in case you're not aware pose accessibility issues for people with motor visual or cognitive impairments. We'll have a deeper discussion about what kind of motor, what kind of visual or what kind of cognitive impairments uh would a mouse or a keyboard incur for um uh you know usage. Motor impairments can make it difficult to use these devices while visual impairments can be hindered by a lack of keyboard navigation or support or screen reader compatibility. We'll talk about these aspects in a bit more detail. Cognitive impairments can be challenged by complex layouts and poor design like that can create barriers for all users such as using unclear links. And of course we've all experienced um you you know touchscreen uh resolution and touchscreen um uh because there's a layer of capacitive touch sensitive um you know front on top of the LCD that kind of reduces the brightness of um the touchscreen devices and that is the reason we actually struggle to see the um screens in bright daylight, right? And I think that is something we've all experienced and that is a situational impairment. So what are the issues with current input technologies? We'll talk about them in primarily three heads. The first is motor and physical accessibility issues. So motor impairments means dexterity loss or uh fine motor skill issues, hand paralysis or conditions like arthritis where you are struggling to really hold uh you know bend your fingers in an in a desired uh format and keep pressing kind of buttons uh for longer durations or fine motor skills. In fact, even when you are moving um the mouse or you're moving your finger on the touch screen, there there may be multiple buttons placed next to each other and uh the uh pressing of the button assumes or requires fine motor capabilities. That means you're able to uh you know navigate your hand uh between two very close uh stimulas. Right? So such people with limited dexterity, hand paralysis, people who are aging uh or may have conditions like arthritis may struggle with the fine motor control needed. Alternative inputs are needed. Uh so we uh in our uh discussion so far we have talked about this uh several times when we look at multimodel interfaces we always think about multimodel kind of input technologies as well so that you have alternative ways to enter your uh input details. So if I want to uh enter my name in a given field, of course the the first and the assumed way would be to uh navigate to the field first probably using your mouse. Once the cursor is placed there, you are able to type your name. That is the base level interaction which is assumed and then there can be other alternative ways. So in case uh I am not able to interact with uh a mouse or a keyboard in an intended manner uh there should be multiple ways to interact with the system to enter your input. Uh so such as um you know people who have who are amput head head pointers uh or eyetracking so right how you look at the screen and where you are looking at based on that the cursor is placed the pointer uh doesn't require uh hand movements for the mouse um or switch access which may not be supported by all software. So basically we ideally would need uh alternative inputs but that is not something which is supported by a majority of our uh interfaces. The third issue which is there is can be of limited navigation. So interfaces that require precise mouse specific movements like drag and drop or complex gestures can be impossible for users with mobility issu you issues to complete. Right? So again precise movements fine motor skills required where you are supposed to uh you are supposed to click on something uh and drag it. So basically what is expected is that you press a button you keep it in the pressed condition in order to drag it and then release it. Uh so those kind of um uh motor expectations all kinds of users may not be able to fulfill. The second category of issues with uh current input technology uh occurs in the domain of cognitive accessibility issues. So uh under the cognitive accessibility issues uh of course there can be information processing related aspects. So like individuals with cognitive disabilities like uh ADHD, dyslexia or other intellectual disabilities may have difficulty in processing complex information and navigating through confusing layouts. So and following unclear instructions. So if um uh you know some instructions are given but they are not precise they may not be able to navigate through your content with ease. Then of course this overwhelming interfaces complex designs frustrating and are frustrating and they create barriers making it difficult to focus on the task at hand. uh this particularly is uh you know a major concern for the elderly as well. Then jarens and complexity. So content that is overly technical or uses complex language can be in accessible to those with cognitive impairments. So um so we we have uh you know looked at all of these aspects earlier as well but looking at it from a input interaction perspective we are uh you know trying to see what are the different issues in the current input technology uh state-of-the-art. The third category of uh issues uh come under the assisted technology and software integration aspects. So uh for example screen reader is an assisted technology which is available and so-called makes softwares and websites and apps accessible to persons with visual impairment. But if your software is not compatible, it's not the architecture is not built in a way that it a screen reader is able to access, the labels are not present, the headings are not present. That means it is largely incompatible with a screen reader and even though uh access assist assisted technologies like screen reader may be present in the market, they may not be useful uh for your product unless and until you make it compatible with a screen reader. And then there may be some complex kind of um information such as tables or poorly marked tables which um which can uh you know get misinterpreted and misrepresented in a screen reader if they're not labeled properly. So keyboard navigation problems. So users who cannot use a mouse uh may be unable to navigate an application or website if interactive elements do not respond correctly to keyboard inputs. Then there can be dynamic content issues. So um dynamic content may include videos or you know live audios. If they're not properly announced using area live regions or they're not uh there they're missing uh descriptive alt text then they're largely not accessible. There can be some solutions. Some of these solutions might have already kind of come uh come up in some earlier session. But this again looking at it from an input interaction perspective and for special use cases uh motor impairments, visual impairments etc. It's good to have a discussion that way. So uh for motor impairments so like for example with physical limitations and conditions like paralysis, arthritis, limited dexterity make it difficult to use a mouse and a keyboard then the solutions can be ensure that interfaces are fully navigable with a keyboard and that complex mouse specific interactions are not required. Right? So things like drag and drop or um you know very fine pressing of fine buttons or pixel level um um you know kind of segregation of information should not be there. Support for alternative input devices like headp pointers or eyetracking systems can also be beneficial. So if your software or your website is open to um compatibility with other kind of input devices like headp pointers and eyetracking systems. Uh it is it's a great uh way to make it more inclusive for visual impairments. Um keyboard navigation uh may be the only way to navigate because keyboard kind of offers a very nice tactile kind of a layout and uh the button islands are separated very nicely and you know basically you can count buttons or if you are an expert in keyboard usage probably uh you will be you know very well aware of what is there in which region and uh and because it's a standardized tactile layout, right? Most most of the keyboards are quarterty keyboards and they are the layout is standardized with so and so keys on the left, so and so keys on the right, so and so keys on the top. Uh so then that is a very useful uh input uh wherever people with visual impairment go, they can use that keyboard. But in if interactive elements do don't respond to keyboard inputs or if skip to main content links are missing. Right? So for example there is the dashboard uh there is a dashboard on top. There is an introductory tile on top on your website. There are many tabs on the website. uh and the moment you turn you open this um the website and the screen reader is also on uh the the screen reader starts reading from the top right but you don't want that clutter that audio clutter and there is no skip to the main content link so every time you open a page that dashboard will get read every time you open it. So uh having a skip to a main content links are very important because if you're using it through a keyboard it by definition becomes a sequential interaction. Mouse interaction you can directly go and um you know click to some arrow or some button or direct the main content you scroll down uh instantly as you land on the page using a mouse. But uh using a keyboard all you can do is press tap tap tab. So it may take like 15 tabs or 20 tabs to reach to the main content which can be frustrating sometimes. Screen readers websites and applications must be compatible with screen readers. This requires semantic HTML code level markup for elements like tables etc to ensure screen readers can interpret the content correctly. Uh so as a solution of make sure that you provide skip to main content links ensure all interactive elements are responsive to a keyboard interaction uh and um so I mean despite the fact that keyboard are we are still talking about the traditional keyboard uh input u means and still most of the interfaces are not compatible with those uh traditional in input devices. So uh at a code level semantic HTML needs to be implemented uh to structure content logically for screen readers. So continuing on the same lines for cognitive impairments complexity is a problem. Individuals with cognitive impairment like dyslexia uh ADHD may struggle with complex overwhelming layouts. So solution is that please design uh clear simple layouts with straightforward instructions. Uh link text. Always add links to the text. Ambiguous uh link text like click here or more information can be confusing. keep it uh meaningful. So for example, if the click here is meant to uh submit a button, it should be rather submit or click here actually leading you to a table or a statistical diagram. Um so it should be diagram here, right? Instead of click here. So that there's some meaning to the button and then because the screen reader would also uh be able to u kind of announce it that this button is for the diagram access. So it should be contextual and it should be meaningful. So use descriptive link text etc. uh that clearly indicates the destination of the link. Then other general accessibility issues like dynamic content. So for announcements for dynamic content are often missing. So make sure that you use area labels to announce changes to the users. Real world testing is a big barrier and lack of testing with actual users is not really happening in a lot of cases. So engage users with disabilities in the testing process to identify and fix accessibility issues in its core. So this uh sets a bit of a background about uh what are the existing technologies uh in terms of input only technologies we are talking about right now we are not talking about the output technologies we'll talk about the output technologies in the next session uh any form of interaction is basically that you you are able to input some information there's some pro processing and to get a feedback or you are able to perceive a task and you are able to perform an action, right? Uh so both the input and output devices are uh there and um in terms of the POR principles also I think we have um we have discussed this earlier. O which stands for operability. Operability which can increase uh the accessibility of any system digital systems per se is directly linked to your input devices. So uh operability uh is directly linked to your input devices only when when you are able to input your choices you can respond to a call to action you will be able to operate the system. Most of our current uh interaction looks like this only. Uh and uh we have a screen with a lot of cluttered icons, maybe some background images and then we are supposed to kind of navigate away to the app that is needed and you know press it with our fingers. But this basically assumes a lot of uh abilities and uh including cognitive, including um motor, including physical, including visual uh in the absence of which this kind of basic interaction also becomes a little bit of uh difficulty. So uh most of our present interaction looks like this where uh we have um you know some icons and we are supposed to kind of navigate away to the app that we are um we're requiring to use and we then suppose supposedly press the button. So based on our previous discussion Nabi uh we have some idea that um uh if you have cognitive impairments, if you have visual impairments, um if you have um motor impairments, physical impairments, then it becomes very difficult to do this basic interaction as well. Right? And we also have spoken about several times that those impairments can appear in the form of a situational impairment as well and may not be just limited to a permanent impairment. Some organizations like Apple have uh worked on iOS assistive touch where there is an enlarged version of it's like a lens which kind of goes around with your finger. It's not entirely for a permanent visual uh permanently visually impaired person, but it can definitely um help people who are elderly in help in navigating or reading what the icon says underneath it or pressing the right uh button especially when they are placed so close to each other in a table format. people with cognitive issues, people with low vision or people with low dexterity also. Right? So such kind of technologies are there which enhance or add a layer on top of your input experience and helps you in uh clicking the uh right kind of interface that kind of builds a little bit of a background into what are the different ways now we are looking at in the future. What are the different uh ways to make input more accessible using emerging input technologies? Some of uh the aspects we had a discussion about in the AI and accessibility sessions. uh but here we'll now talk particularly in terms of input technologies and how uh the emerging uh aspects can enable better multimodal access access accessible ways to interact with digital interfaces. So emerging input technologies are transforming accessibility by offering diverse personalized and handsfree interaction. So this is very important personalized and handsfree and diverse right and now we'll see in the upcoming slides how in what ways it is personalized uh it is hands-free or uh it is diverse. So these innovations are largely leveraged uh le leveraging AI technologies uh sensors and robotics to go beyond the traditional keyboard and mice and providing greater independence of uh use and ease of use. So let us see the different categories under which they fall. So AI and machine learningdriven inputs. So artificial intelligence is primary a driver in enhancing accessibility by creating more intuitive and adaptive input methods. So first is something which may be very popular uh which is something called advanced speech recognition um and NLP natural language processing. So uh some examples may include uh say the dictate feature which we spoke about uh in one of the previous sessions on AI and accessibility. Uh so AI powered speech recognition has become highly accurate and speechbased input uh and uh models models driving speech to text speech to text have become very robust. So now uh doing a voice based search or dictating content to a word uh to make a word document uh doing um voice based um um you know uh control is has become fairly easy. So you can control devices, dictate text and kind of bypass uh you can even issue complex commands and the systems are able to understand even if you tell them in a long form long paragraph they are able to semantically kind of summarize it and uh tell you what um the action it and kind of squeeze the action item out of it and do it for you, right? uh the other aspects I think we we did discuss about uh which needs to be looked at is probably uh speechbased issues like stutter and uh all of those things lists etc. But as the models are becoming more robust the those problems will I'm sure get solved in the near future. The next set of kind of input devices uh which are driven by AI and machine learning are image recognition and computer vision based models. So these technologies provide alternative input for visually impaired users. So applications like Microsoft's seeing AI or Google's lookout use device cameras to recognize and describe objects, text, people, people in the surroundings or converting uh visual information into audio feedback. So basically you can wear your smart glasses and run these models and uh you can just look at you can just point your camera or just head onto a text and it the the image recognition models will be able to read the text for you and kind of convert it in real time into audio output. It can also be used in uh obstacle detection, realtime obstacle detection to avoid um you know kind of stumbling or um banging into any object. Then predictive text and AI keyboards. So AIdriven predictive text anticipates words and phrases significantly reducing the physical efforts required for typing for users with motor limitations. So uh predictive text is similar to autocorrect or when you write in the mail in the Gmail it based on the uh the email it is able to predict a pause uh a response. So something like thanks for the update or something like that. You don't even need to type any longer and you can just uh select one of those options and it's a faster and a more accessible way of uh uh you know interacting reduces the physical effort to a big extent. Then there are definitely AI powered virtual volunteers or assistants or systems like uh the one developed by OpenAI and Be My Eyes use AI to provide real time virtual visual assistant where a virtual agent can see what a visually impaired uh person sees through their camera and provide guidance. Right? So basically uh there is a virtual assistant who can attend meetings for you can tell that okay there's a room with around 15 people and uh so and so person is um you know sitting on the right so and so person is sitting on the left which you know kind of adds to your social inclusion as well. Yeah. So something like a co-pilot is also an example of a virtual assistant and dictate feature on word for example is is an example of AI powered speech to text input. speech to text input. You can just dictate and uh make a document. There can be other alternative hardware input devices which are now being developed. the robustness etc are being tested and innovations in those hardware are now looking at um different input methods for users with special needs physical special needs. So for example eye tracking systems. So these systems enable users with severe physical disabilities or motor impairments to control computers and mobile devices only using their eye movement. So so it also looks more natural. So basically for a third person uh it would appear that so for example here is my phone and I would also look at the screen and interact with it. Right? Maybe I also required to use my thumb or my fingers. But for example, uh if those eye tracking systems are in place, I can just look at the device and kind of navigate through my gaze and select uh a certain option based on a longer gaze. For example, if a gaze stares for on one of the options for a longer duration, it is able to open that. Then BCI or brain computer interactions though still largely in early development or research stages. BCI where you can directly control interfaces using brain waves. So all you need to do is think and that kind of bypasses all kinds of modalities right all kinds of impairments as well. All you need to do is think that this need this is this needs to be done and this button needs to be pressed and it will happen. So that is going to be potentially a revolutionary handsfree input method uh for those with severe physical limitations. Then adaptive switch uh devices and joysticks. So customizable input devices such as sip and puff switches operated by breath. Specialized joysticks like those used in wheelchairs offer single switch or minimal movement control for navigation and task completion. So basically what we are saying is that uh we the older input devices like a joystick used to assume that you have again that dexterity and that had movement but say um you you have uh you have a person who who doesn't have both of their hands. So breath operated joysticks. So if you sip it is the car for example there's a joystick controlled car which is a breath control car if you uh sip then the car moves backwards if you puff it moves forward or specialized joysticks especially for those who uh use wheelchairs. Then again a a very interesting emerging assistive devices area is variables. So uh variables of course I'm sure most people might have heard about smart glasses. Uh then there are hearing aids which have like really evolved uh today's day and age and other variables that are providing novel input and feedback methods. For instance, the OrCam II variable device that reads the text aloud and is able to recognize faces also and you know kind of tell you uh in an audio format that wherever so and so person is standing in front of you. Neimano uh robot gloves assist users with limited hand mobility in gripping objects. So Romano robotic gloves or haptic gloves are able to assist users with limited hand mobility in gripping objects. Then there can be interaction new interaction paradigms which are so the ones which we discussed earlier there are more applications of AI or variables or all of these other hardware technologies. Uh but then there can be new methodologies itself which are coming up in order to uh you know drive these emerging input technologies and output technologies. So gesture and motion control. So the rise of sensors in mobile and varial devices in in enables interaction through natural body movements. Uh and this is being researched extensively for controlling devices without the traditional inputs with applications in smart wheelchair and navigation systems. So uh basically for example your phone now has a gyroscope and uh there are other mo motion controls which are there and even the same can be incorporated sensors. The cost of sensors the cost of incorporating sensors has become very low. Uh so it is it can be something which can be explored in a lot of other applications like navigation. So gesture and motor gesture based control where uh you are just indicating something instead of pressing buttons uh you say you know kind of swipe uh in front for the wheelchair to move forward or you just swipe back and the systems around uh the wheelchair is able to detect those movements and uh kind of help you in navigation itself. So gesture and motion are one of the two key areas in which there's a lot of research going on. Then other handsfree interaction. A central theme across many emerging technologies is this hands-free interaction which can involve you know voice based input, gesture based input or the other alternative inputs like tongue movements, breath movements uh which can address a wide range of uh mobility or situational impairments also. So in the future it may actually look like that we are you know interacting with a a robot and we just telling the robot to uh do things for us. Uh he or the robot is able to decipher the semantic meaning of what we are saying and do actionable um you know you is able to act accordingly. they have a robotic arm as well integrated in the system. So if you think about all of these uh input technologies if you combine them it becomes a robot only. So basically there is a camera there is a microphone there is uh you know a robotic arm there is um uh which can kind of fill in for the loss of physical movements instead of you. So basically we are moving away from this kind of an interface toward this kind of an interface in the near future. So then uh this brings us to a very interesting video on emerging tech for accessibility. Uh it's a longer video kind of a TED talk mode. Uh you can watch it on YouTube. Uh the link is right here on the slide. Um, I can play it a little bit. So basically uh Joe Dewan is talking about the future of technology and how it runs through accessibility. A lot of the developments are happening in order to address accessibility issues and then eventually all of those things will uh move into broader consumption and no longer be specialized technologies. And that is the basically the premise overall premise of uh the talk and the interaction. You can go ahead and watch it. It's almost 50 minutes. Um and uh I hope you gain uh something from it. So in order to summarize today's session uh we want to say that emerging technologies uh when developed with inclusive design principles from the onset they are vital for breaking down barriers and ensuring equal opportunities for individuals with disability in an increasingly digital world. Right? So more of our interactions are becoming digital and how all of these digital interfaces can enable wider access is something of key importance and criticality in the current hour. So some of the search topics that you can utilize for your uh for gaining more insights on this topic is uh emerging input technologies, AI based input technologies, variable input technologies, adaptive input technologies etc. And uh we will talk about emerging output technologies in the next session. uh keep learning and let's see you in the next session. Thank you for joining.