For six years, we ran the largest blind eng hiring experiment of all time. Here’s what happened.

This is part 1 of our series about eng hiring in the age of AI. Now that hiring is back, so are we! Want to hire with us? Just fill in this form.

Many people have heard of interviewing.io, but most people think of us as a mock interview platform. What you probably don’t know is that for six years, we ran the largest, live blind eng hiring experiment of all time.

You’ve probably heard about the blind orchestra auditions described by Malcolm Gladwell in Outliers. When orchestras had musicians play behind a screen during auditions, effectively hiding their gender, the likelihood of women advancing to later rounds increased by about 50%, and over time, the proportion of women in top symphonies nearly tripled.¹

After I read Outliers, the idea got stuck in my head, and I started interviewing.io so we could do the same thing.

The experiment

For six years, from 2016 to 2022, we ran the software engineering version of the orchestra experiment, except instead of hiding gender, we hid resumes.

Our users did anonymous mock interviews on our platform, and we aggregated their interview outcomes to surface top performers.

Then, top performers could book anonymous interviews at their companies of choice. The twist: companies didn’t see any resumes. Their first interaction with each candidate was a fully anonymous technical interview with an engineer from that company. The interview was exactly the same as the company’s usual technical phone screen, except for the anonymity. At the end of the interview, both sides decided whether to move forward; only then would the candidate unmask.

With this approach, from 2016 to 2022, we hosted about 10,000 real (but anonymous) interviews.

Why did we do this?

Simply put, hiring is broken, and we wanted to fix it. “Hiring is broken” has become a bit of a cliche though. How is it broken? What does broken actually mean?

I’ve spent years trying to simply articulate everything that’s wrong with the status quo’s approach to hiring. Here everything that’s wrong with hiring in one diagram.

Essentially, everyone is chasing the same set of candidates who look good. Unfortunately, many of those candidates aren’t looking right now, and many aren’t good. (We have a lot of data showing that pedigree doesn’t really predict performance – check out our post about how to spot great non-traditional talent).

But, there isn’t a straightforward way to filter for good candidates.

So, instead of being able to look directly for the right people, you have to rely on proxies. Some obvious proxies might be where someone has worked before and usually turns into looking for people from FAANG and FAANG-adjacent companies (and maybe a few very high-profile top-tier startups as well).

But, those people may not be good, and they’re probably not looking right now. So, hiring takes forever.

Our customers (without us) spend something like 200 staff hours and $40,000 per hire, split between a lot of time on sourcing (because not that many people respond) and a lot of time interviewing people who fail (because people who look good aren’t necessarily good).

And, of course, without us great non-traditional candidates languish in the online application black hole without getting a chance to show what they can do.

That’s what this experiment was trying to fix.

The results

With our blind approach, over six years, we placed thousands of engineers at FAANG and FAANG-adjacent companies and top-tier startups.

Of our users who got offers from top-tier companies, 46% (almost half!) didn’t have either a top school or a top company on their resume. In a normal (not blind) hiring process, these candidates wouldn’t even have gotten an interview.

In fact, many of these non-traditional candidates had been rejected by the very same company where they later got an offer through us. Many of our customers were surprised to see that, after the candidate unmasked, that the candidate had already existed in their ATS and had previously been rejected at the resume stage. One of our candidates had been rejected three times by the same company, at the resume stage. With us, he “snuck in” because he got the chance to interview anonymously and ended up getting hired. He’s still there today.

Our blind model wasn’t charity. It was good business. Because we used interview performance data, not resumes:

• Our average hire had 10 years of experience (lest you think good senior engineers don’t go for stuff like this).
• Our candidates (both traditional and non-traditional) converted 3X better than other sources (including internal referrals). Not surprising; we used performance data to predict performance!
• On average, cost per hire went from $40k to about $10k for our customers. This was driven by less time spent on everything, especially sourcing and interviewing – when you know who’s good, you don’t need to interview nearly as many people.
• Our non-traditional candidates were 2X more likely to accept offers and stayed at their job 15% longer than their traditional counterparts after placement.

In the six years we did hiring, we confirmed something that I had hoped was true at the start: it’s possible to make money and do good business while doing the right thing. Hiring great people, rather than people who look great, not only makes hiring fairer, but also cheaper, faster, and better.

We sadly had to put our hiring product on ice when the hiring freezes and the resulting downturn started in 2022, but, now that hiring is back, we’re relaunching it. You can sign up for early access here.

Why use us?

You don’t have to use us! But we are the only recruiting marketplace out there that uses interview performance data to find great engineers. And having performance data is a big deal and helps make hiring way more efficient.

It’s the only way we see to fix broken hiring. Here’s why.

With data from mock interviews, we know who’s good because we know how they do in interviews. And we know who’s looking because they’re practicing on interviewing.io. About 10,000 engineers sign up for interviewing.io every month. We know who’s good and looking, and we can help you hire faster, cheaper, and more efficiently than ever before. In other words, you’ll be hiring from the green part of the Venn diagram instead of the red.

Do you want to be hiring from the green part rather than the red part? Try us out. You can use us as a source of excellent candidates and/or you can use our predictive model² to help you surface the best people from your inbound applications. Just fill in this form for early access, and we’ll be in touch.

Footnotes: