Three IT strategies for implementing AI hiring tools

As employers turn to AI tools to sift through a flood of résumés and find qualified candidates for IT jobs, many are designing “knockout criteria” to help determine who gets an interview.

But establishing that criteria isn’t just a job for HR, according to Patrice Williams-Lindo, CEO and founder of workforce and AI governance advisory Career Nomad: “I think that HR owns the why. IT owns the how.”

For IT pros grappling with that how, Williams-Lindo highlighted three important strategies that can prevent AI and human hirers from accidentally filtering out top candidates.

Audit your rejections—not just your hires. Run what’s known as the four-fifths test, which analyzes if the selection rate for a demographic (including race, sex, or ethnicity) reaches at least 80% for the highest-rate pool. For example, if 25 out of 50 male applicants are hired (a 50% selection rate), and 20 out of 50 female applicants are hired (a 40% rate), the selection rate is 80% (i.e., 40% divided by 50%), which passes the test.

“Is there any group clearing the screen at less than 80% of your top-clearing group? That’s not a gray area. It’s a legal standard for adverse impact. The tool is already telling us where potentially there’s a problem,” Williams-Lindo said.

Make explainability part of procurement. A lot of companies buy the AI screening tool first, and then ask about how it makes decisions. Wrong order, according to Williams-Lindo.

“Find out from their vendor in plain language why a specific candidate would be knocked out,” Williams-Lindo said, adding that those questions are meant to be explored at the negotiating table, not to be “a mystery to solve after it’s live,” Williams-Lindo advised.

Log any changes to knockout questions. Depending on the organization’s culture and needs, there are plenty of questions that could eliminate an employee. Some are simple: Do you have a four-year degree? Are you willing to work fully onsite?

Others are potentially more nuanced: Have you ever been terminated from a position?

“A flat ‘yes’ knocks candidates out whether it’s contextually a layoff or for cause,” Williams-Lindo said. But if the organization decides that a previous layoff isn’t enough to trigger a knockout, those questions must be updated, removed, or exchanged, and everyone involved in the interview process updated.

Williams-Lindo spoke more with us about strategies for monitoring and changing the knockout questions.

This interview has been edited for length and clarity.

What would your advice be to an IT or HR pro as they’re designing these knockout questions?

Number one: Tie every question directly to a legal requirement or genuine deal-breaker, not a preference. They should ask themselves: If someone answers “no” to this, could they still do the job well? If it’s “yes,” it’s not a knockout question; it’s a scoring criteria. So, reserve those knockouts for things like licensure, work authorization, and physical requirements that are truly non-negotiables and not “nice to have.”

Another thing to keep in mind is making sure that they are clear on what is required versus what is preferred, and never knock out unpreferred. If they say they’re looking for “five-plus years of experience with Salesforce” as a hard cutoff, it eliminates someone with four years and exceptional results, or even someone with equivalent experience in a comparable CRM. Preferred qualifications should adjust the score, not end a candidacy.

How else can you test a knockout question?

Don’t use absolute cutoffs. Use ranges where possible. So, instead of “you should have X salary expectation or below,” use a range that lets the recruiter or whoever’s talking with the potential candidate have a conversation. Instead of “must have degree,” for example, I would say “have this degree or X number of years experience,” so kind of an either/or.

How frequently should you test your knockout questions?

At least annually, they should do a self audit of every existing knockout question. Times change. Requirements that made sense maybe a year ago may not make sense now, right? Think about the ones that said “must come into office five days a week.” During the pandemic, that changed…You just want to make sure that you put some sort of dating or expiration review on every question, in the same way that you would review a policy for your organization.

What does a healthy hiring strategy look like that does involve the use of [AI] tools?

Use AI to expand the pool, not narrow it first. Let it surface candidates that a keyword search could have missed: non-traditional résumés, adjacent industries, career pivoters, before you ever let it screen anyone out, so you kind of get an idea of where it can detect real value. I’d say start with the edge case first, because that sourcing is where that AI tool can add real value, and elimination is potentially where it can do damage…Never let AI make the final call, only the first pass. Have checks in place because you want to make sure that a human right is actually seeing who got filtered out.