How interview scorecard peeking can create bias in hiring

4 mins, 17 secs read time

Very few organizations choose whether to hire somebody after just one interview. In a structured hiring process, potential candidates often speak to several members across the organization before a hire or denial.

While this process can be time-consuming, companies benefit from giving many employees the chance to evaluate candidates and share their perspectives before making a final hiring decision.

In Greenhouse, every interviewer is given one or more scorecards to summarize their evaluation of a candidate. As the process continues, interview scorecard feedback is combined across interviews to provide a complete picture of the candidate.

While this collaboration is beneficial, it is important to control the flow of information so interviewers are not influenced by one another. This is why we have a specific set of permissions our customers can choose from to determine who can see scorecards and when:

When a customer opens their Greenhouse account for the first time, this permission is set to “Only after submitting their own.” In our support documentation, we explain why we chose to make this a default for all customers, with this hypothetical scenario:

Imagine that Deidre has finished her interview but hasn't completed her scorecard yet. Deidre has access to other interviewers' scorecards. She looks at Carl's submitted interview scorecard and sees that he gave the candidate poor feedback. Deidre had a positive impression of the candidate after her interview, but now she's having second thoughts. When she fills out her scorecard, she changes her overall recommendation from "Yes" to "No Decision." In this example, Deidre was biased by Carl's scorecard, which caused her to give the candidate poorer feedback on her own scorecard despite her positive interview experience.

The Deidre scenario is a prime example of interview scorecard peeking. Using data from scorecards submitted for over 10 million panel interviews (interviews where the candidate is interviewed by two or more people at once), we are able to understand to what extent scorecard peeking can create bias among interviewers.

To estimate bias, we want to see how often two interviewers give the exact same rating, and understand whether the frequency changes when one interviewer “peeked” at another’s interview scorecard. [Note: This is a narrow definition of bias, because we don’t know what rating an interviewer would have given if they didn’t peek. We also can’t measure instances where peeking brought one interviewer’s rating closer to another’s without making them identical, but we think that is another possible biased outcome.]

In our analysis, we found that interviewers who peeked were 3.6% more likely to give the exact same rating compared to interviewers who submitted their scorecards independently of one another. While that may sound like a small change, the magnitude of the difference becomes significant once you consider the scale of our product. Across 1 million interviews, a 3.6% increase would mean that interview scorecard ratings were biased for 36,000 interviews (by the narrow definition used in our analysis, this is likely an underestimate). Greenhouse customers conducted approximately 2.6 million panel interviews in 2022, and 18.8 million additional interviews, bringing the potential volume of biased interviews to over 750,000. This scale and potential impact is why it is important for us to suggest sensible default settings for our customers; 85% of customers use one of our settings that prevents scorecard peeking, thereby minimizing the risk of this bias.

In our analysis of panel interviews, we can see that the likelihood of passing increases as pairs of ratings become more positive. However, we also see that a negative rating is not always decisive when the other rating is positive. Rather, a disagreement in ratings likely prompts a conversation among interviewers and the hiring team.

To continue our example with Deirdre, by making her answer more like Carl’s, she is amplifying his feedback, and the candidate’s chance of passing drops from 14% to less than 6%. Had she submitted her original assessment, the candidate may still not have passed, but the discrepancy would have provided the candidate another opportunity behind the scenes by inviting the hiring team to take a closer look.

The benefit of a hiring process with multiple interviews is that it allows an organization to collect feedback from different interviewers, generating a more comprehensive view of candidates by combining perspectives. However, this is only useful when each interviewer’s feedback is provided independently from others. Respectful disagreement is a healthy part of decision making, and surfacing disagreement is necessary to resolve it. Deidre should have trusted her impressions, and that the collaborative process would resolve her and Carl’s differing views – but her company could have helped her do this by making it so she couldn’t peek to begin with. By doing this, they would be helping themselves hire better candidates.

To leverage the wisdom of the panel, companies need to make sure every individual’s honest opinion is represented – which means no interview scorecard peeking should be allowed.

Interested in finding more areas to mitigate bias in hiring? Find out how anonymous take home tests can help.

Find out now