Scoring Explanation

Automated accessibility tests can only catch about half of accessibility issues, while the other half can only be found manually.

Companies can hire independent auditors to check for accessibility issues that aren’t covered by automated tests. The report of their findings is called a VPAT.

The method I use to create the accessibility score depends on whether or not a VPAT has been published for the plugin.

If a VPAT has been published, I give a score of 10 if it passes the WCAG AAA tests, a 9 if it passes AA, and a 6 if it doesn’t pass. (Ideally there will be scores between 9 and 6, but I haven’t reviewed enough VPATs to determine a fair scoring system for those cases).

If a VPAT has not been published, I try to test as much of the plugin output as I can, using the Accessibility Checker plugin from Equalize Digital. Starting with a default score of 5, I reduce the score by 0.5 for every issue and 0.25 for every warning in the test results. If I see an issue in the report that appears to be a reporting error (for example, the plugin couldn’t calculate the background color, so it reported a contrast failure), I remove that from the list of issues so it doesn’t count against the score.

The humility score is an attempt to quantify whether a plugin is a respectful guest in the WordPress dashboard or an entitled owner. Does it do its job quietly, keep things tidy, and stay out of the way, or does it rearrange the furniture and repeatedly call out for attention? In short, does it prioritize the plugin authors or the plugin users?

The name was inspired by one of my favorite plugins of all time, Mark Jaquith’s Menu Humility plugin (which looks for themes and plugins that add their menu items at the top of the dashboard menu and moves them to the bottom).

Starting with a humility score of 10, the following things will cause deductions to the score:

UI

• Appears mostly native to WordPress dashboard. No learning curve (no affect on score)
• Appears partially native to WordPress dashboard. Small to medium deviations from the native dashboard UI, due to either branding or interface changes. Minimal learning curve.  (-1)
• Appears completely foreign to WordPress dashboard. Custom UI, requires learning a new user interface. (-2)

Advertisements

• Persistent: universal ad that can’t be dismissed (-8)
• Universal: displayed on all dashboard screens, but can be dismissed (-3)
• Misplaced: displayed outside plugin settings, or to wrong user roles. Ads should never be displayed to non-admins because they shouldn’t be making responsible for choosing/installing software. (-2)
• Prominent: designed to attract attention, not using the native dashboard UI. (-2)
• Discreet: Uses native styling or other non-attention grabbing styling. (-1)

Notices

• Persistent: universal notices that can’t be dismissed (-8)
• Universal: displayed on all dashboard screens (-2)
• Misplaced: displayed to wrong user roles (-1)
• Non-native:  uses non-native notification styles (-1)

Other deductions

• Misplaced menu item: If the plugin creates a CPT, that can be placed in the post type section of the menu along with posts and pages. Otherwise, menu items should either be in the tools submenu, the settings submenu, or at the bottom of the dashboard menu. (-1)
• Colored menu item: unless there is an error that requires user attention, plugins should not use color in an attempt to bring attention to their menu item. (-1)
• Gratuitous admin bar link: unless the functionality of the plugin is built into the admin bar (like the Query Monitor plugin), the plugin should either a) not place a link in the admin bar or b) include a setting to remove the link from the admin bar. (-1)
• Colored admin bar link: unless there is an error that requires user attention or the functionality of the plugin is built into the admin bar (like the Query Monitor plugin), the plugin should not use color to draw attention to itself in the admin bar. (-1)
• Misplaced dashboard widget: If the plugin includes a dashboard widget, it should not be placed at the top of the first column by default. It should be placed at the bottom of a column. (-1)
• Overrides native notice styles: The WordPress dashboard includes styling for notices. Plugins should use these (and definitely not override the styling of notices it didn’t create). (-1).
• Data sharing/telemetry on by default: data sharing/telemetry should be opt-in, not opt-out. (-1)
• Data sharing not defined: requests for data tracking should include an explanation of all data collected (or a link to the explanation). (-1)
• Other: This could be anything else that prioritizes the plugin authors over the plugin users. The deduction will depend on the severity of the offense. Also, this doesn’t have to be part of the plugin itself. For example, if a plugin subscription is impossible to cancel without contacting the authors, or if the cancellation process uses dark patterns to try to discourage cancellations. If anything like this is used in the scoring process, a description will be included.

I can’t catch every offense, but when I see one, I try to screenshot it and add it to the site.