How to test IT job candidates in the era of AI

At the start of the year, while others pledged to go to the gym more often or eat healthier, Botpress CTO Michael Masson had his own resolution: remove take-home assessments from his company’s hiring process.

The reasoning was simple: He had witnessed candidates leveraging AI to conquer the entire test without actually using their technical skills.

“We saw it again and again, where people were just using AI without really thinking,” Masson said. “We didn’t see any work. We just saw them complete the assessment really easily.”

The end of take-home assessments. Botpress is one of several organizations that is reassessing the value of take-home assessments in today’s increasingly AI-driven world.

These assessments were once a crucial part of the hiring process for IT leaders to assess how devs and IT professionals work in their own environment. “We want to see them work with their own tools, with their own software, and what they’re familiar with,” Masson said. “It gives us a really good understanding on their skills and how they can apply them on a specific problem.”

But as Strider CTO Breno Oliveira told IT Brew, AI has introduced “noise” into the hiring process, with applicants now using chatbots to complete coding assignments in mere minutes.

“We reached a level with generative AI that you can literally just copy and paste a whole take-home description, and it can give you something that is somehow workable,” Oliveira said.

Live coding assessments. This shift has led some companies to part ways with take-home assessments altogether. AI proficiency platform Larridin is one of them, according to co-founder and CTO Ameya Kanitkar, who told IT Brew he made the switch to live assessments in January.

“The way our setup works is that we bring the candidate on site and we do three or four different rounds. One of the rounds is essentially 90 minutes, where we provide them $100 or $200 in AI credits, and give them a problem,” Kanitkar said.

During those 90 minutes, candidates are allowed to ask questions. Afterwards, they spend time explaining the various choices they made, allowing Larridin to assess their technical acumen.

“Basically, what we are really assessing is their ability to work with the agents and produce a quality work in a reasonable amount of time [and] semi-supervised setting,” Kanitkar said.

Botpress also leverages live coding assessments as part of its hiring process. Masson said these sessions allow him to observe crucial things about the candidate, including their communication and problem-solving abilities: “It gives us a really good insight into the technical skills that you have.”

Pairing sessions and paid trials. Companies are turning to other testing alternatives to assess serious job-seekers. Oliveira said he has seen some clients host code pairing sessions, which is when two engineers work on the same project in real time.

Paid trials, where companies pay developers for a small project before hiring them, is another option for companies, Oliveira said: “It wasn’t that common, but we’re seeing that clients are more open to pay for a trial now.”

Giving the take-home assessment a makeover. IT pros also have ideas for revamped versions of the take-home assessment that take AI into account. Masson, for example, told IT Brew companies can assess a candidate’s skillset by giving them a large codebase and asking them to add a new feature. While AI can likely build out a feature by itself, applicants will still need to understand and read the codebase themselves. Companies can throw in elements into the codebase that make using AI a little trickier, Masson added.

“If you steer AI in the wrong direction, the person working in the code base has to fix all of the small problems [and] has to understand the code base,” Masson said. “[It] gives us a good understanding of how someone works in a messy environment.”

He added companies can have candidates talk through code reviews, which he said are taking up bigger chunks of developers’ days due to AI-generated coding. In May, a Harness survey found that 81% of software engineers spend more time on code review after adopting AI coding tools.

“It’s a skill that we need to develop as the industry is moving towards adding and pushing more and more code,” Masson said. “Seeing someone review [and] talking through code that was done by AI or was done by humans is something that would be really interesting.”