The other half, or why interviewers aren't always great and how to make them better

Hey, Aline (founder of interviewing.io) here. Time for another awesome guest post.

Btw, if you have strong opinions about interviewing or hiring that you’ve been itching to write about, we’d love to hear from you. Please email me at aline@interviewing.io.

Jos Visser

Jos Visser is a consummate IT professional with over four decades of experience interviewing and being interviewed. He has over 500 interviews under his belt at companies like Google, Facebook, Amazon, and OpenAI. On top of that he was a member of hiring committees and created and delivered many different training programs aimed at making people better interviewers. Jos can be reached at josvisser66@gmail.com or through LinkedIn.

It is high time we start talking about interviewing. I know it seems like we are talking about interviewing all the time, but we are usually talking about only one half of the equation: How to be a good candidate. What about the other half? What about the interviewer?

In the last decade, there has been an explosion of attention for candidates and how to improve their interview performance. There are blogs, sites, companies, and books that help you improve your interviewing skills. You can sharpen your coding skills, learn how to approach a system design interview, read entire volumes of standard answers to standard questions, do mock interviews, and learn how to use the STAR method to answer behavioral questions. These days, candidates are often very well prepared for a standard interview and therefore, bar a brain fart, do well at that.

This stands in stark contrast to the preparation of the interviewer. If you are lucky, your interviewer might have gotten a two hour class on how to ask only bona fide work related questions and has sat through two shadow interviews. Maybe they have even done a few interviews! Consequently, there is a lot of bad interviewing being done. That needs to change.

Let’s start by acknowledging that interviewing is very hard. The average interview is a 45 minute video conference in which you need to get the audio and video hardware working, engage in introductory chit chat, give the candidate some time to ask you any questions , and ascertain whether they got what it takes to be successful in the role. Given all the overhead involved, you might only get 35 minutes for the meat of the interview. This is not a lot of time, especially if the candidate is, as I indicated above, often very well prepared.

Don’t get me wrong, it is good that candidates are prepared. It shows an interest in doing well and displays the kind of methodical approach to reach a goal that I want to see in the candidate. But it does make interviewing a lot harder.

In the olden days, you could surprise a candidate by asking how to remove a file called -f or how to sort one million eight-digit numbers with only 1MB of RAM. It would unfailingly be the first time the candidate ever saw that question and so you were immediately seeing how the candidate dealt with solving a problem that they had never solved before and probably never even thought about. This often prompted interesting discussions and you could see the candidate’s brain at work in solving a novel problem.

We live in different times now. A lot of candidates have seen all the common questions and are exquisitely prepared for them. As a result, whenever they give the right answer, you have learned nothing about the candidate, other than that they were probably well prepared for this particular question. Take the min-stack question. I used to like this question because it is not entirely trivial and has some half-way interesting trade offs between time and space complexity. Then some genius came up with a non-trivial solution that solves the question in O(1) time and O(1) space. It is a super cool trick and I would not expect anyone to come up with it from scratch in 35 minutes. If you ask the question today and people immediately knock out the optimal solution, you know that they memorized the answer and you have learnt nothing. Same for how to design a URL shortener or a key/value store.

This experience might give rise to the idea that all the current questions are known and that we just need a collection of new questions. That idea is wrong; it is not about the questions. Exaggerating a bit, I am of the opinion that all interview questions that can be asked within the scope of a 45-minute interview are known already. Coming up with a “new” question typically just means replacing the gift wrapping around the core algorithm of an old question with new gift wrapping, thereby making it look new. Even if, through a stroke of genius, you manage to come up with a brand new unique never-before-seen question, there is no way to keep it secret. As soon as you give it to a candidate, the question is out and the entire candidate prepping industry will cover it on their sites and blogs, and in their mock interviews and books. It’s like keeping the new Taylor Swift album secret, it ain’t gonna happen.

So interviewing is hard to begin with and even harder with well-prepared candidates. How to deal with that?

The non-answer to this question in my opinion is to replace (some) interviews with take-home assignments. I wrote about this before and by and large I consider it a mistake. Not only does it put the burden of getting a better signal solely on the candidates, it also is a method to ensure that great candidates, with lots of places to go, do not bother to enter your interviewing process. It’s just another example of penny wise and pound foolish. I am not saying it cannot be done right, but what I have seen so far does not inspire me with confidence.

Instead, my answer is to match like with like: If the candidates are becoming better at being candidates, the interviewers need to become better at being interviewers.

One of the ways in which a lot of interviewers need to become better is through getting rid of the idea that good interviewing starts and ends with good questions. Nothing could be further from the truth. Surely, a good question is always better than a bad question, but a good interviewer can still get lots of signal from handling a bad question well, whereas a bad interviewer cannot get any signal from a great question. It really is not about the question, and that is one of the things I think good interviewer training needs to cover.

But if good interviewing is not about the question, then what is it about? The answer is: Digging.

Let’s take the world famous “how to remove a file called -f” interview question. There is a canonical answer, which is to use rm -- -f. If the candidate knows this, that’s okay, but not more than that, because you haven’t learned anything about the candidate’s knowledge and thinking yet (other than they know this “trick”). When I get this answer, I would be wondering whether the candidate understands why this works. Do they understand if rm -f? Or rm '-f'? Or rm ./-f? And what about X=-f rm $X? And if so, or if not, why, or why not? And why exactly does a plain “rm -f” not work? Would rm -* work? And does it matter what Unix system you are on? Or which shell you are using? And does python -c "import os; os.unlink('-f')" work? And if so why? And if rm -- -f works, does that automatically mean that touch -- -f works to create that file?

You can use this very simple question as a starting point for an extensive discussion about shell argument expansion, command line parsing by Unix binaries, the getopt(3) library function, the unlink(2) system call, and the many interesting ways in which the GNU, BSD, and System V versions of common utilities differ in this respect. While you are at it, you might as well throw in follow up questions about $IFS, spaces in file names, hidden files, inode numbers, device files, hard links and symbolic links If -f is a symbolic link, does rm -- -f remove the link or the file linked to? And what if -f was a hard link?

Interviewing is about digging, about never taking the candidate’s answer at face value, and about taking the answer and using it as a jumping board to ask follow-up questions. I really don’t care if the candidate knows the answer to my questions up-front. What I care about is whether they can reason towards the answer using the knowledge they have and the additional knowledge I might give them in the course of the interview. If the candidate comes in knowing next to nothing about the Unix shell, but they can deduce what is going on from the examples I give them and propose experiments to test the hypotheses they come up with, I am as happy as a clam at high tide.

Interviewers can learn how to dig. They can learn how to write better feedback. They can learn how to evaluate a candidate correctly against the hiring bar. It might not be easy, but it is not not rocket science either. That said, learning how to interview well is real work; real work that you can only expect from people who are interested in being great interviewers and who are duly recognized.

It is probably a dirty secret that in many companies, a lot of engineers are seeing interviewing as a chore to be dispensed with quickly and with as little effort as possible: They often get little to no training, there is no quality control, no feedback on feedback, and there are no incentives to be great or to get better. No wonder that a lot of interviews are suboptimal.

Fortunately, in most companies, you don’t need all engineers to be interviewers, so you can restrict yourself to the ones who want to interview. You can then create a culture of interviewing; a culture where great interviewing is the norm, and where that great interviewing is rewarded.

Rewards are essential here. To quote Charlie Munger: “Show me the incentives, and I’ll show you the outcome.” It doesn’t have to be money. Or at least, not a lot of money, as rewards come in many forms: Public recognition, swag, a “Great Interviewer” mug, a badge on the internal phone book pages, or a shout out at the all-hands. All of these conspire to give people the idea that they are doing important and meaningful work. Plus it gives the managers of the interviewers the idea that they are enabling an essential company function. And of course: Great interviewing needs to help you get promoted. If it doesn’t, then you might as well go write another design document rather than spending your time on attending your local interviewing club.

There is unfortunately no cheap and easy way for your interviewers to become better. It is not hard to improve interviewing quality and effectiveness, but it does require time, money, and effort. Plus it helps if you know what you are doing (as a company hiring engineers).