Myth-Busting: The Truth About Validity in Coding Assessments

For the lucky ones, the term “validity” recalls some distant memories from a high school statistics class. But for most of us, validity, especially as it applies to decision-making tools used for hiring, is an abstract and new idea that often comes with a few misunderstandings. Before we dispel some myths surrounding validity, let’s refresh our memories on what it is and why it’s essential to make good tech hiring decisions.

In a hiring context, validity is the extent to which the scores on an assessment are job-related. For instance, a coding assessment that intends to measure the ability of a software developer or a personality test that measures the interpersonal skills of a salesperson.

In the coding assessment example, your code testing scores are “validated” if they represent actual coding skills in a way that’s reflective of what’s required on the job. With accurate measurements of job-related skills, you can make better inferences about whether or not candidates will be able to do their jobs at a required or outstanding level.

We’ve talked a lot about code testing, but validity holds several practical benefits if you use any decision-making tool in your hiring process. Not only does validity evidence protect you from a legal standpoint, but it also provides you with information that a candidate’s capabilities are a good “fit” with the job requirements.

Despite the clear advantages of establishing the validity of hiring procedures, many people who first learn about validity often come to the wrong conclusions about how it works and why it’s essential. To help make sure “valid” assessments are on your list of must-haves in your hiring process, here are a few common myths about validity that we need to clear up: 

Myth #1. “Validity only matters if you use a coding assessment.”

Employers learning about validity for the first time often assume it only applies to assessments with right and wrong answers, such as cognitive ability, job knowledge, or work sample tests. 

However, the Uniform Guidelines on Employee Selection Procedures (UGESP), drafted in 1978, set the Equal Employment Opportunity Commission’s (EEOC) standards and methods for determining whether a hiring practice is discriminatory. The UGESP keeps U.S. employers accountable when hiring individuals from minority groups protected by Title VII of the Civil Rights Act of 1964 and the court cases that followed. 

What’s important here is that the UGESP applies “…to all selection procedures used to make employment decisions, including interviews, review of experience or education from application forms, work samples, physical requirements, and evaluations of performance.”  

Which of these procedures does your organization use? You likely use several decision-aids that fall under the UGESP’s purview. This means that even if you used a dance contest to make a hiring decision for an investment banker (which I highly discourage), it must be validated to comply with the UGESP. 

You might also like: Bringing Industrial & Organizational Psychology to Tech Hiring

Myth #2. “Validation is a nice-to-have and can wait.”

When the topic of validation is brought up, many people think it is a compliance issue for big companies to worry about. Well, the EEOC would beg to differ. The EEOC is a federal agency that enforces anti-discrimination laws in the U.S. for all companies with more than 15 employees. 

Without validation evidence on the books, organizations lack protection if their hiring processes are ever legally challenged. A 2013 study found that 90% of employers who had personnel decisions taken to trial lost their case, with an average payout of $1.5 million. The moral of the story? Most employers should make collecting validity evidence a priority to avoid costly legal challenges. 

As an aside, the U.S. currently has the most fully-formed legal framework for non-discriminatory hiring, but this framework is slowly being adopted in other countries and regions such as the EU, so local statutes and laws should always be consulted.

Myth #3. “The validity of our coding assessments is something for our vendor to worry about.”

Though it may be in the vendor’s best interest to worry about collecting validity evidence for the hiring tools they bring to market, the employer is ultimately responsible for the decisions made from the selection procedures they use. 

Validation provides evidence about whether the inferences from the scores gathered from a coding assessment are justified. Therefore, as an employer, the decisions you make based on a candidate’s test score should be supported by validity evidence.

Not only that, employers who invest in validation efforts also gain two competitive advantages. First, having validity evidence is helpful for organizations striving for a diverse workforce that is hired fairly. Second, using a code testing system that shows strong validity evidence can give employers the confidence to hire the best employees.

Extra reading: Artificial Intelligence & Human Learning In Recruitment Software

The organization should see increased productivity and engagement among their employees and increased fairness perceptions among their job candidates. Therefore, employers who leave vendors to deal with validation concerns are not only opening themselves up to legal challenges; they’re also missing out on the whole business case for evidence-based hiring. 

Ultimately, validation is a key ingredient to successful hiring, which is why it is so crucial that these “validity myths” get busted. The sooner employers hop on the validation train, the better their hiring decisions will be, and, as a result, the more potential for success their organizations will have. 

Want to learn more? Check out these resources on Codility’s approach to code testing validation and how we can help you validate your selection procedures in your technical hiring process.