One of the things that sucks most about technical interviews is that they’re a black box—candidates (usually) get told whether they made it to the next round, but they’re rarely told why they got the outcome that they did. Lack of feedback, or feedback that doesn’t come right away, isn’t just frustrating to candidates. It’s bad for business. We did a whole study on this. It turns out that candidates chronically underrate and overrate their technical interview performance, like so:
Where this finding starts to get actionable is that there’s a statistically significant relationship between whether people think they did well in an interview and whether they’d want to work with you. In other words, in every interview cycle, some portion of interviewees are losing interest in joining your company just because they don’t think they did well, even when they actually did. It makes sense… when you suspect you might not have done well, you’re prone to embark on a painful bout of self-flagellation, and to make it stop, you’ll rationalize away the job by telling yourself that you totally didn’t want to work there anyway.
Practically speaking, giving instant feedback to successful candidates can do wonders for increasing your close rate.
In addition to making candidates you want today more likely to join your team, feedback is crucial for the candidates you might want down the road. Technical interview outcomes are highly non-deterministic. According to our data, only about 25% of candidates perform consistently from interview to interview. Why does this matter? If interview outcomes are erratic, it means that the same candidate you reject this time might be someone you want to hire in 6 months. It’s in your interest to forge a good relationship with them now and be cognizant of and humble about the flaws in your hiring process.
I thought this tweet captured my sentiments particularly well.
Great teams treat candidate rejections with just as much care as offers.— Danny Trinh (@dtrinh) February 4, 2020
It’s insane to see folks screw this up, especially with younger talent.
Why? You have no idea how they’ll grow in 18 months.
For all you know, you just benched MJ in high school 🙏🏼
So, despite the benefits, why do most companies persist in giving slow feedback or none at all? I surveyed founders, hiring managers, recruiters and labor lawyers (and also put out some questions to the Twitterverse) to understand why anyone who’s ever gone through interviewer training has been told in no uncertain terms to not give feedback.
As it turns out, feedback is discouraged primarily because companies are scared of getting sued… and because interviewers fear defensive candidate backlash. In some cases, giving feedback is avoided just because companies view it as a no-upside hassle.
The sad truth is that hiring practices have not caught up with market realities. Many of the hiring practices we take for granted today originated in a world where there was a surplus of candidates and a shortage of jobs. This extends to everything from painfully long take-home assignments to poorly written job descriptions. And post-interview feedback is no exception. As Gayle Laakmann McDowell, author of Cracking the Coding Interview, explains on Quora:
“Companies are not trying to create the most perfect process for you. They are trying to hire—ideally efficiently, cheaper, and effectively. This is about their goals, not yours. Maybe when it’s easy they’ll help you too, but really this whole process is about them… Companies do not believe it helps them to give candidates feedback. Frankly, all they see is downside.”
Look, I’m guilty of this, too. Here’s a rejection email I wrote when I was head of technical recruiting at TrialPay. This email makes me want to go back in time and punch myself in the face and then wish myself the best in my future endeavors to not get punched in the face.
These kinds of form letter rejections (which I guess is better than just leaving the person hanging) make a lot of sense when you have a revolving door of disposable candidates. They are completely irrational in this brave new world where candidates have more leverage than companies. But, because HR is fundamentally a cost center tasked with risk mitigation (rather than a profit center tasked with, you know, making stuff better), and because engineers on the ground only have so many cognitive cycles to tackle hard stuff outside their job descriptions, we continue to march forward on autopilot, perpetuating outdated and harmful practices like this one.
In this hiring climate, companies should move toward practices that give candidates a better interview experience. Is fear of litigation and discomfort legit enough to keep companies from giving feedback? Does optimizing for fear and a few bad actors in lieu of candidate experience make sense in the midst of a severe engineering shortage? Let’s break it down.
While researching this piece, I spoke to a few labor lawyers and ran some Lexis Nexis searches to see just how often a company’s constructive feedback (i.e. not “durrrr we didn’t hire you because you’re a woman”) to a rejected eng candidate has resulted in litigation.
Hey, guess what? IT’S ZERO! THIS HAS NEVER HAPPENED. EVER.1
As some of my lawyer contacts pointed out, a lot of cases get settled out of court, and that data is much harder to get. But in this market, creating poor candidate experience to hedge against something that is highly unlikely seems… irrational at best and destructive at worst.
At some point, I stopped writing trite rejection emails like the one above, but I was still beholden to my employer’s rules about written feedback.2 As an experiment, I tried giving candidates verbal feedback over the phone.
For context, I had a unique, hybrid role at TrialPay. Though my title was Head of Technical Recruiting, which meant I was accountable for normal recruiter stuff like branding and sourcing and interview process logistics, my role had one unique component. Because I had previously been a software engineer, to take the heat off the long-suffering eng team, I was often the first line of defense for technical interviews and conducted something like 500 of them that year.
After doing a lot of interviews day in and day out, I became less shy about ending them early when it was clear that a candidate wasn’t qualified (e.g. they couldn’t get through the brute force solution to the problem, let alone optimize). Did ending interviews early cause candidates to fly off the handle or feel particularly awkward, as many people suspect?
In my experience, cutting things off and saying nothing about why is a lot more awkward and leads to more defensiveness than letting candidates know what the deal is. Some candidates will get defensive (at which point you can politely end the call), but if you offer constructive feedback—let them know what went wrong, make some recommendations about books to read, point them to problem repositories like Leetcode3, etc.—most will be grateful. My personal experience with giving feedback has been overwhelmingly positive. I used to love mailing books to candidates, and I formed lasting relationships with many. Some became early interviewing.io users a few years later.
Anyway, the way to avoid negative reactions and defensiveness from candidates is to practice giving feedback in a way that’s constructive. We’ll cover this next.
When I started interviewing.io, it was the culmination of what I had started experimenting with at TrialPay. It was clear to me that feedback is a Good Thing and that candidates liked it… which in this market means it’s also good for companies. But, we still had to grapple with prospective customers’ (pretty irrational) fears about the market being flooded with defensive candidates with a lawyer on speed dial.
For context, interviewing.io is a hiring marketplace. Before talking to companies, engineers can practice technical interviewing anonymously, and if things go well, unlock our jobs portal, where they can bypass the usual top-of-funnel cruft (applying online, talking to recruiters or “talent managers,” finding friends who can refer them) and book real technical interviews with companies like Microsoft, Twitter, Coinbase, Twitch, and many others… often as early as the next day.
The cool thing is that both practice and real interviews with companies take place within the interviewing.io ecosystem, and wrt feedback, you’ll see why this matters in a moment.
Before we started working with employers, we spent some time building out our practice platform and getting the mechanics right. For practice interviews, our post-interview feedback forms looked like this:
After each practice interview, interviewers fill out the form above. Candidates fill out a similar form rating their interviewer. When both parties fill out their forms, they can see each other’s responses.
If you’re curious, you can watch people practicing and read real feedback that they got in our public showcase. Here’s a snapshot:
When we started letting employers hire on our platform, we just recycled this post-interview feedback format, told them they should leave feedback to help us calibrate and because it’s good for candidate experience, and fervently hoped that they wouldn’t have an issue with it.
To our surprise and delight, employers were eminently willing to leave feedback. On our platform, candidates were able to see whether they passed or not and exactly how they did, just a few minutes after the interview was over, stopping the rising tide of post-interview anxiety and self-flagellation in its tracks, and, as we’ve said, increasing the likelihood that a great candidate will accept an offer.
And if a candidate failed an interview, they got to see exactly why they failed and what they needed to work on, probably for the first time ever in interview history.
On interviewing.io, interviews are anonymous: an employer knows literally nothing about the candidate before and during the interview (employers can even enable our real-time voice masking feature). Candidates’ identities are only revealed after a successful interview, i.e. after the employer has already submitted feedback.
We insist on anonymity because about 40% of our top-performing candidates are non-traditional, and we don’t want lack of pedigree or an unusual background to open them up to bias. Because interviews are anonymous, it’s often impossible to discriminate on the basis of age or gender or background. Therefore, feedback has to be constructive, by design, because the only info the interviewer has to go on is how well the candidate is performing in the interview. In addition to helping candidates get a fair shot, this anonymity provides something of a safety net for employers—it’s harder to build a discrimination case out of the feedback when the employer doesn’t know the identity of the candidate.
In many other contexts, anonymity can be destructive because of reduced accountability. But in the interview process, we’ve discovered over and over that anonymity sets free the better angels of our nature and creates a kinder, more inclusive interview experience for candidates and employers alike.
So, how can you fold these learnings into your process? Of course, the easiest way to get your feet wet is to start using interviewing.io. We’ll get you candidates you wouldn’t be sourcing without us, and we’ll empower you to give them the best interview experience you can.
But even if you don’t use us, based on how unlikely it is that you’re going to get sued or deal with angry candidates, we strongly recommend having your interviewers provide constructive feedback (like in the examples above) after every interview for all candidates, whether they pass or not, over email.
Here are a few strategies for delivering constructive feedback:
If you know of such a case, please tell me, and I’ll update this post/associated content accordingly ASAP. ↩
This is a rule pretty much every company has, ever. It’s not just TrialPay, which was a great place to work and whose defensive HR policies, like every other company’s, were in no way indicative of their workplace culture. ↩
Whether algorithmic problems of the type you’d find on Leetcode are the best way to interview is a question worth asking… and I’ve since come to feel pretty strongly that many of them are not. But that’s out of scope for this piece. ↩
Given the root of a binary tree, imagine yourself standing on the right side of it, return the values of the nodes you can see ordered from top to bottom.
Given a binary tree, extract all the leaves in repeated succession into a list of lists by starting at the bottom and working your way upwards.
Given a 2-D integer array mapping the letters ABCD and their costs. Calculate the smallest cost to make a string of length n.
Interview prep and job hunting are chaos and pain. We can help. Really.