Evaluation Tools – Pros and Cons + Intro

Hello all, welcome back to our course on digital accessibility and As discussed in the previous sessions, we spoke about the WCAG guidelines and what are the BA u you know the criteria for fulfillment the compliance and all of those things and we also spoke about at the end of that lecture that um it is it would be good if we have some automated tools for evaluation uh to begin with so that we understand or as designers or developers we know what are the failures is where is our uh product lagging? So today in this session this is what we are going to talk about what are the basic evaluation tools the popular ones and uh what are the pros and cons of using them and uh after this session we will also have um a tutorial of sorts where um you know we will kind of upload the site or upload the uh application and then it gives us u um an evaluated report of uh the accessibility. So let us uh begin with again our PUR principles of accessibility and I think we discussed it in detail again in the previous lecture and the WAG uh guidelines uh where we spoke about that how they themselves are divided in four chapters or four sections again based on the same PUR principles perceivable operable understandable and uh robust and Each section has a list of criteria which also carries a list of techniques in order how to make them um compliant with that particular criteria. What are what is the need for that criteria to begin with? What are the various examples in which this kind of a criteria is required? All of those things are there to understand it from a criteria perspective or when uh to educate yourself basically. But for example, you are already using a product, you are al already developing a product. So there is also a faster way of getting around. So automated evaluation tools to check the accessibility compliance or errors or failures are very much available. there are some common or popular ones which we will discuss in the sec in in this uh session itself and then there are now with the um uh AI based um you know applications becoming quite popular I'm sure there are many tools available we will not be able to cover all of them uh of course uh as part of the course but uh some of the common ones we will cover and uh you can get an idea about how they function. And of course, if there are several paid and free tools available which you can use at your own convenience. If you just like try to do a Google search uh or like any search engine for that matter uh saying that okay uh give me options for web accessibility evaluation tools or digital application accessibility evaluation tools. It will give you or direct you to uh various sites or various tools. So uh let us try to understand what they are and some of the common ones and what are the pros and cons for using those evaluation tools first. So automatic tools can play an important role in the field of digital accessibility uh offering efficiency and scalability in testing. However, they're not a complete solution and they come with their own set of advantages and disadvantages. uh and uh automated tools are a powerful component of the accessibility strategy and they must be used as a part of comprehensive approach but it should not be used as the only approach right you uh as accessibility experts one should educate themselves on the WA criteria list uh to begin with but this can be just a quick tool to assess your uh level of compliance science of any digital given product in in a quick uh manner but it doesn't claim to be uh comprehensive but it is definitely a part of it. So let us keep that in mind before jumping onto u you know the evaluation tools themselves. Let us also try to understand some of the pros and cons before we jump into using it. So what are the pros? So of course the efficiency and speed they are quite efficient they're very fast they can give you like a full-fledged report of every page in say 10 minutes 5 minutes etc. Now with AI based u um you know applications it may be lesser which saves a lot of time uh you know human effort um where you have to cross check with each criteria etc. it can be a lot of work. So, and this speed is crucial for very large websites where there's a lot of content or a lot of pages or uh you know websites or applications which get frequently updated. So, it is not possible to keep doing it uh manually again and again. So, for that matter it is definitely uh a very useful tool that it offers speed and it is uh very efficient. definitely scalability. So they can be easily integrated into continuous integration or continuous delivery pipelines. So I'm sure you are aware of these CICD pipelines if you are a IT uh engineer or an IT based developer. Uh so the they these kind of products are digital products actually have as we said like continuous updates or there is a continuous feeding or accessing of some server based data uh server based data or other aspects. So then that means it is continuously integrated or continuously getting updated. Uh so in that case um and of course a large set of audiences are using it. So in that case uh using automated accessibility valuation tools can be um a very useful tool because it can be enabled into large scale uh continuous delivery mechanisms as well. It can help in early detection. So uh before diving into a full-fledged deep analysis or an audit of accessibility uh of a digital product, people can do just like a quick scan and get an idea about what is the status of the accessibility of this particular product and teams can just quickly fix some common uh smaller issues which are um you know um you know level A or smaller ISS issues like color contrast etc. uh very quickly even you know before presenting it to stakeholders or senior management etc. And it also actually reduces the remediation costs after launch. So right before launch also if those quick uh remedies can be done after launch once it is used at a very large scale it becomes a little bit more costlier to do it or integrate it in the continuous pipeline for that matter but it although that is possible but uh early detection can be a more of a preventive method which can um cost much less to the company or the stakeholders themselves. Other pros uh consists of consistency and objectivity. Although uh these tools um these web accessibility guidelines if people are very familiar with and they are there are experts uh they can provide uh a lot of information help in a lot of uh accessibility compliance etc. But human variability uh can occur, human error can occur uh during manual reviews. So such kind of tools can offer consistency and objectivity in developing reports etc. And then it also helps us in identifying obvious errors. So for example, I think we discussed this earlier that sometimes we may just look at uh look at the interface and feel yeah I mean this looks nice or this this looks the color contrast looks fine only because uh it looks seems like a high contrast. So I don't think it'll be a problem. So but uh it although it may be obvious that just by looking at it you are able to see it's a color contrast but it may be failing the criteria by little uh small percentage. So for example if the criteria says 4.5 is to 1 uh maybe the contrast is 4 is to 1 but is still legally it is not fulfilling the criteria right so for those kind of checks which are which look obvious to the eye maybe running uh an automated uh evaluation can help the designers or uh the developers just get an idea okay maybe it looks fine but just a minute change in the color contract can make it legally compliant. Similarly, um um some very clearcut uh WAG violations such as missing all text which is like the very first uh criteria uh can be very obvious right so I mean just by looking at it you may not be able to see that uh there is all text missing or descriptive text is missing etc. So um running that uh evaluation can help you detect them on time and fixing them. It can be a kickstarter for an indepth manual search or manual evaluation as well. So automated reports can uh provide a good starting point for manual evaluators and manual testers helping them to focus their human expertise on more complex issues. So I think this these this is a very important pro of using an accessibility tool and this is how we should also look at it that it is it is a supporting tool and not by itself comprehensive. So a complete evaluation should always include um human accessibility experts as well as accessibility tools and then only it can become a complete uh comprehensive evaluation. Let us now look at some cons of using automated accessibility tools and why we should not entirely rely on only automated accessibility tools. So it has limited scope and it can give you false negatives. I think this is very important to understand that automated tools can only detect a fraction of all WAG issues. They are not uh able to catch very complex or AAA level comp uh complexity issues and they cannot accurately evaluate elements that may involve that may require human judgment. uh or like particularly around the understandability part of of it where where cognitive uh semantic values or semantic understanding is needed and semanticism is very much a human or psychology concept. So what is the meaning of semanticism and why am I saying this that because under understandability there are labels that okay so and so information should have some meaning in context of the rest of the page or rest of the website right or in the workflow itself. If it doesn't have any meaning then it fails to fulfill the understandability criteria or criteria listed under the understandability principle. So it is important and many a times those understandability issues are very humane issues which may not be uh captured under this evaluation automated evaluation softwarebased tool. So that requires human judgment. it cannot evaluate context. So tools understand the meaning and intent uh they cannot understand uh the meaning or the intent of the content. So for example, it can verify uh whether a image whether an image has an alt text or not but it cannot determine whether the text is meaningful, accurate or helpful. For example, uh if all the uh images on a website are just say all text, right? There is no real meaning or no real words. Actually, even if you just type all text in the box or of all text for the evaluation for the evaluator tool, it is passing the criteria, right? It has the um it has the evaluation, it has the all text. So it can only check the presence or absence of an alt text whether that uh text is actually a meaningful description of the image itself that needs a human judgment. Other than that there can be false positives also. So sometimes so we we spoke about we spoke about false negatives but there can be false positives also. So sometimes uh tools flag issues which are not actual WAG violations. So this can lead to a wasteful um investigation and fixing of non-existent problems actually. So sometimes it can give you uh errors which are not really there. So it has to be double checked uh by uh a human as well. Although with increasing uh AI integration I think uh they will become more efficient and all of these the probability of getting a false negative or a false positive will also gradually decrease with more uh complex algorithms which is able to find semantic understand the image itself or find some semantic meaning or you know assess the description for instance uh to some extent I'm sure In the future these tools will also become a little bit more intelligent but so far they are not as comprehensive and should not be used uh in isolation over reliance and I think that is where the last point ended when we say it should not be used in isolation that is what we mean that it should be the teams should not uh over rely on such tools right there's there's a risk Risk uh of teams becoming over reliant and assuming that uh that a 100% automated pass means that their site is fully accessible while eventually it may lead to uh you know lawsuits or fines etc for the company. So the company should make sure to invest human expert uh invest in human experts who can also follow up with u um you know quality uh assessment of the accessibility or employ services of an accessibility auditor uh for that matter to assess their product followed by the automated tool assessment. So that uh other than that these tools are not really uh usability testing tools. They are just cross-checking some of the parameters which have quantitative uh thresholds. So for example, it can check uh whether uh so and so buttons or so and so text is uh the contrast against the background is 4.5 is to1 or not because it has a quantitative uh threshold. Similarly, the size of the font or availability of um uh all text or not. The answer is a binary, right? Yes or no. Whether it's not there or there. But it cannot really evaluate the overall user experience is for persons with disability or whether the application or the website itself is um providing that ease of learning that um uh engagement that multimodal uh engagement with the content uh understandability or have an idea about what is familiar to the target audience uh understand the context in which it will be used all of those it's not able to do. So it requires expertise to interpret results. So it just generates a numeric report. Uh we will see we will see a few uh examples of what kind of reports it generates and uh the results also need to be interpreted by development or design experts who understand what is uh who have some idea about accessibility accessibility criteria only then they'll be able to understand because it says the number of the criteria. So it'll say you violation of 1.1.1. So if you don't know what is the meaning of 1.1.1 or you don't know what is a W CAG list of guidelines and you can go back to that list and see okay which one is 1.1.1 uh without that information you will not be able to even interpret uh the report generated by these tools. So I think it is very important to educate yourself uh first then proceed to use it because otherwise it is very difficult to even understand what the report says. So I think in the previous um uh session we spoke a little bit about uh we we spoke in depth really about the W CAG guidelines and it has been developed by worldwide web consortium uh and it has been part of their web accessibility initiative. So again here's the same link which leads you to the main W CAG repository. Uh and this is a table which uh can kind of tell you a little bit about we did discuss conformance level A and AAA being the highest. So uh A incorporates uh most basic web accessibility features. These should be followed for any website. DoubleA deals with major accessibility barriers and developers should follow standard coding procedure to satisfy compliance. So these aspects may need uh code level changes. If it is not compliant at W uh at double A level, it may need code level changes for make it compliant. This one may be more CSS, HTML based changes or you can just include a script which is there suggested in the techniques and it will become um the a compliant. Um AAA's conformance is like a good to have. If it is good if it is there it will enhance the usability for people with special uh abilities. It is non-mandatory. if the web uh website could be compliance with uh uh AAA but then if it is there it will be more accessible and based on this only we discussed right GIGW only mandates mandates double A compliance which is this level of compliance if you have again if you have AAA it's great good to have it will of course en ensure uh good level of engagement, better user experience, better user accessibility. But double A is legally mandated. It has to be there. So particularly so uh we will look at a few examples where uh government websites were evaluated uh whether they are compliant to these guidelines. While automated tools are in this uh these examples we are only using automated tools for validation and report generation. No tool is completely accurate. So I think we have discussed it in the pros and cons in depth but it helps us give uh in giving an overview of accessibility compliance before diving into an indepth analysis by a team of expert human evaluators. So two of the most commonly used uh evaluation tools is one is Google lighthouse and one is a tool called wave. So in this example uh 10 government websites in which their homepage and five random pages have been evaluated and automated evaluator is used on all pages and their accessibility scores were noted. So um if you see the uh Google lighthouse it generates a report in the format where it gives you a percentage or a score right. So if it is 100% uh it is fully accessible it is 100 it if it is u slightly more than 90 between 90 to 100 it is considered accessible uh maybe small issues are there which can be fixed. If it is between 90 to 80 it is partially accessible and uh if it is less than 80 it's not accessible. that means it's not compliant with a lot of uh lower level compliance criteria as well. So these say for example passport sava and make in India these two websites were also evaluated. I'm sure you might have visited both of these websites. So uh do you have any idea or can you guess what the scores would have been for these two? I mean they they are one of the cleaner good-looking websites right so passport sava is sadly 66 and make in India is very sadly 59 which is much lower than the minimum compliance threshold also so now let us um yeah I mean since the score is below 70 that means it is non-compliant at a very deeper level and then coding issues need to be resolved to make it accessible. So um as I said the lighthouse kind of gives you uh a score table for as and uh for each website five pages were evaluated um and kind of uploaded. So for uh each evaluation you put paste a link in the interface and it generates a score and gives you a list. So we'll we'll do a tutorial after the session as well but let's just try to understand what is uh what does a score mean before actually generating a report for our work. So for example in income tax e India e filing government.in in they reason had reasonable scores uh only one page with less than 80. So most of them um some of them are really good. So for example upsc.in has reasonable uh accessibility. Similarly um income tax India government or some of them some of the pages are really low. So for example this EPF India has a page which is score of 39. This is really like unacceptable. So I mean uh you can just evaluate and see which pages need more work and so as we discussed so this is like kind of a report overview which is generated. Now you have to see that okay which pages need more attention which pages need more number of hours. It can help you in operational division of your work or your team's work and um uh can can help you um u develop priorities like okay for example page three has the lowest score let's look at this first index page seems to be accessible let's leave it for the time being but this page is like very low let's work on this make it more accessible similarly page number five in The EPF India gov.in has a score of 50 which is very pathetic. So let's go and work on that one. So it gives you an idea of where to start, what to do first. So if you see some of as I mentioned some of them are reasonably accessible and some of them are somewhere in the middle. Remaining are uh some pages are very bad. So for example the passport saver this page or index landing page is really bad. EPF we saw has a couple of pages very bad. Make in India the land page itself is very bad. Most of the other pages are very low. So uh it gives you a certain sense of uh understanding of what to fix which pages need to be worked on uh more diligently. So this is a average lighthouse uh accessibility score for all of these uh pages. So this is like an average of the five pages. This is just a data valuation. So UPSA seems okay. Seems to be doing okay. India post seems to be doing okay. Um some of them more I mean income tax, kamigov. Uh so these can be reasonable um um you know report generation and all of those if you want to present it to external uh stakeholders or you want to make a report about u particular set of products of your company and present it to your senior management. I think these kind of uh tools can be really helpful. Second evaluation is a little bit more in-depth. So the lighthouse gives you a set of scores. It also tells you how to fix um the issues but not down to uh each issue. Uh it so it can give you it it has a whole area library and repository but uh the wave tool is a little bit more comprehensive because it you know tells you a little bit more about what kind of errors is it. So the moment you add um the link to the interface, it gives you like how many errors, how many contrast errors, how many alerts, how many features are compliant, how many area issues, how many structural elements, right? And it gives you links to like missing form table, document language is missing, there's an empty link. So it gives you all of these things or details about that page. So it it provides a graphical representation of the location of issues which can be a very useful next step. Indicates all of the components of accessibility violation but this is not really a comprehensive tool. We're not saying that all all of the semantic aspects are being covered. It is just able to identify quantitative aspects like whether a label is missing or not. So if for example it says zero errors, labels are there but they are semantically not meaningful still your website may fail accessibility valuation by an auditor because they are not semantically compliant and they fail the understandable uh principle related criteria. So similarly for all those websites a wave report was also generated and uh each page and the kind kind of like say contrast issues how many were there missing label how many were there in each page or something empty link how many were there so make in India apparently has 218 empty links for some reason um although this is old data uh this is not a new data but yeah I mean probably at that time they had a lot of empty links things. So just trying to understand what kind of issues are more common um in all of those. So contrast which seems to be very obvious right but seems to be a most common issue empty link again which seems to be very uh um common or very obvious that okay if there is something clickable it needs to take you somewhere right? If it is if it is clickable and not taking you anywhere that means it is an empty link. So then or it uh so you might have noticed a lot of um links that you click gives you a 404 error. Why? Because it's empty. So um uh so I think that uh gives these reports the they give you a decent idea of what to do and where to start. So WAG violation uh so for example this is just an example uh your product may be violating a separate set of uh violations. So this is just u all of these issues all of these issues what criteria they fall under that is what the uh here it is listed. So for example for contrast it is 1.4.3 4.3 minimum contrast level level double A similarly uh missing label non-ext content 1.1.1 which we discussed a lot in the previous class similarly there are two three more uh missing label criteria related to it similarly for all the other all text empty button empty link link purpose is missing. So for example error errors related to missing label all text empty button is like a 1.1.1 criteria nonext content again for all these numbers uh after you see the report you have to have some knowledge of where to go what to find how to fix it so you again go back to the WAG library and refer to it and uh see the techniques how to fix it and then fix it similar Similarly, double A conformance uh 1.4.3 contrast 2.4.6 headings and labels. Uh level A. So headings and labels uh I think when you'll see this criteria for example it is not just visually they should have bigger font or thicker font or color differences etc. In the architecture, in the code they should be labeled as heading one followed by subheading, head subheading one, subheading two because then the screen reader reads it accordingly like a heading and heading lab uh labeling has to be there right. So again ba as we discussed all the legal mandates are around A and double A levels and both of these level related criteria have to be fixed in order to make your uh web legally compliant with the mandate. So similar so let's like this is just one uh more example of like common issue as we discussed that visibly it may look like okay it is decent contrast but if it is not compliant it will uh the automated tool will tell you it is not compliant proper color contrast is very essential. So as per standard color contrast ratio should be 4.5 for smaller text and 3.1 for larger text. So make sure that it is there. And there are also other color color contrast uh analyzers. So which you can search for. You can just type color contrast analyzers for web accessibility or digital accessibility in your search engine and it'll show you because these kind of analyzers as I said again they are evolving and a lot of AI based tools are coming up. In fact, there are several plugins which are also now available for you know um systems like uh WordPress or even um Adobe Premere which is used widely for making the websites itself the front- end design part. So there are a lot of such uh things and even illustrator I think has uh an inbuilt color contrast analyzer. So all of these plugins are already available. So this is just a depiction uh of an um of a good contrast. So for example as I said this one is 4 is to 1 right. So it looks like yeah I mean it is quite a good contrast but it is just missing 4.5 by a little bit. So legally it will again fall in the category of non-compliant. So make sure that it is falling numerically following the criteria to the tea otherwise uh the company or the product can eventually fail legal u issues. Similarly sometimes it's also uh so sometimes it's also just not about the numeric contrast. It is also about what color you are using. So for example, numerically all of them are following the 4.5 is to one. But um you know this these two this is something which is there on the road as well. Black on yellow the sinages uh white on blue is also very highly visible but u u blue on green has low visibility despite the fact that it may have the right color contrast and similarly green on red also the edges become vibrating which again becomes a little bit of a difficulty to be read. So again uh here for example buttons so few things aesthetically may look better to you but the require it's not passing the required ratio so it should be u you know changed to some other contrast so that it passes the required ratio. However we can see that visually many a time so it's again a tradeoff sometimes. So contrast issue it is passing but it is harder to read. So there are again as we were saying there were uh there were a lot there are lot of usability aspects which also need to be looked at by experts and try to understand. So for example, can we change the font here? Right? So if the this color contrast is better so that the required ratio is uh legally as we discussed right for bigger font it is 3 is to 1 for smaller font it is 4.5. So if we can change the size big to bigger so that it is legally able to follow a 3 is to1 because this contrast is already quite uh close to three. Um so with little tweak it can be made into three. So all of these nitty-gritty and nuances need to be understood by designers and developers while looking at the reports itself. So to summarize we in this session discussed uh the use of automated tools for accessibility evaluation. We spoke about the pros and cons of accessibility evaluation tools. We also looked at some examples and we also looked at the two commonly used tools. Google Lighthouse the link is right here and uh wave tool the link is here you can check out these both of these resources at your end as well and in the following session we will have a tutorial of sorts where we'll try to show you how to upload a docu upload a website or an application to both of these um uh tools and use them and generate a report. So thank you for joining us in this session and thank you for sticking around with digital accessibility. We will see you again in the next session.