The Quest for a More Structured Virtual Technical Interview

Imagine a world where your virtual technical interviews are efficient, reliable, and unbiased. A world where you know candidates are being evaluated on the things that matter, and everyone is getting a consistently fair shake. This magical land is what hiring scientists call the land of “more interview structure.”

Decades of research on hiring interviews has found that adding structure, in the form of more standardized questions, instructions, training, or ratings, generally adds greater reliability to interviews. And more reliable interviews are generally fairer and more accurate.

Keeping the caveat in mind that interview quality doesn’t go up and to the right with more structure forever, we know that more structured hiring interviews lead to better predictions and outcomes than less structured interviews. What we don’t know, however, is whether this applies to a virtual technical interview used to assess the skills, problem-solving abilities, and communication skills of your software engineering job candidates. Despite the tech interview’s ubiquity, there’s little research on how structured these interviews are and to what extent structure components affect candidate evaluations.

Thankfully, we have reams of anonymized and aggregated virtual technical interviewing data. So, we decided to conduct our own research to answer these questions. Looking at ~16,000 virtual tech interview sessions conducted on our CodeLive platform, we compared three groups: (1) Report mode sessions, interviews based on pre-completed coding tasks; (2) Template mode sessions, interviews based on uncompleted coding tasks and (3) Whiteboard mode sessions, interviews based on open-ended programming in our virtual IDE. Report and Template mode sessions represent ‘more structured’ interviews, as these sessions use pre-populated content assumed to be asked of all candidates, while Whiteboard mode sessions are ‘less structured’ in that these most likely include open-ended questions and answers. 

Surprisingly, out of the nearly 16,000 sessions we reviewed, 70% were conducted using Whiteboard mode. Of these sessions, about 34% didn’t have any task description included, meaning about 1 in 4 virtual technical interviews in our sample could be labeled as ‘completely unstructured.’ We also discovered that “technical skills” and “problem-solving skills” rated on our built-in interview scorecards impact whether a candidate receives an overall hiring recommendation greater than the candidate’s communication skill rating. What do these findings tell us?

We now have data to show that even with virtual interviewing tools, most virtual technical interviews are falling into the “less structured” bucket. And despite what hiring managers might assume, technical skills are still driving the lion’s share of their hiring recommendations. If we should be living in the land of “more structure”, the reality seems to be that most recruiters and hiring managers are still on the quest to reach it.

With these findings in hand, what should we do now if we want to continue on the quest for more tech interview structure? Here are 3 things:

1. Start with adding a little structure to your virtual technical interview

Interview structure can come in many forms: interview training, standardized questions, or clearly defined ratings. But it’s often unreasonable to assume you can introduce all of these elements overnight (there’s even a case to be made that ‘too much’ structure is a thing). Instead, focus on areas where you can introduce even a modicum of structure.

For example, encourage hiring managers to use Template mode instead of Whiteboard mode in CodeLive and make intentional, standardized task choices candidates will be asked to complete. A little can go a long way to show hiring managers that their decisions become easier and more accurate when small structure improvements are made.

2. Don’t assume interviewers are using available interview structure components

We know our research sample may be slightly biased in that these are interview sessions conducted on Codility’s virtual interviewing platform. As such, we may be dramatically underestimating the true percentage of unstructured tech interviews in the wild.

However, what’s striking about our findings is that even when given access to more structured interviewing features, a majority of tech interviews are still fairly open-ended. Encourage your hiring managers to learn and understand the tools at their disposal – this could save everyone time from having to add more interview structure from scratch!

3. Remember tech skill ratings are driving hiring recommendations

Our research shows that tech and problem-solving skill ratings are the main driving factors behind hiring recommendations – at least at this stage. Is measuring mostly technical skill what you intend for your engineering interviews? It’s probably safe to assume the answer is yes, but if you want soft skill evaluations to play a bigger role in recommendations, an intentional effort should be made to define and evaluate these skills during the interview.

Also, keep in mind that reviewing tech skills during the interview may be redundant with a more automated evaluation like what’s provided in CodeCheck, a scalable and objective skill proficiency measurement. Moving technical skill evaluation to this phase could leave more room for evaluating other soft skills that matter for performance on the job!

Interested in learning more about how to add structure to your virtual technical interviews? Be sure to stop by our Product Tour page to see how CodeLive can help you on your quest to the land of more interview structure. 

The full research study referenced in this blog article will be presented at the 2022 Society for Industrial and Organizational Psychology (SIOP) Annual Conference on April 27-30 in Seattle, WA.