Hire by Jousting Tournament
In the legend of King Arthur, Merlin presented the British knights with a sword thrust through an anvil and into the stone beneath. He then claimed that the true king of England, chosen by God, would be able to pull the sword straight out. When, after a time, no man surfaced who was able to perform such a feat of strength, the knights decided that the next best choice was to hold a jousting tournament, and anoint the winner as their king.
On the surface, this strikes us moderns as a silly way to choose a government. But is it, really? Choosing a king is always an uncertain endeavor. But the knights lived in an age of war. With a jousting tournament, they could at least be certain of choosing a mighty warrior as their leader. And there's more layers to it. Having a jousting tournament let them make a clear, unambiguous decision in a short time window with all the knights gathered. Winning a national jousting tournament is a hard thing, and leading a people to victory is also a hard thing. They were choosing someone with a proven history of accomplishing hard things.
I wonder if a similar logic applies to job interviews for software engineers, and in particular, whiteboard interviews. This is an interview format where, in a series of ~45 minute time slots, the candidate tries to solve algorithmic problems by writing code on a whiteboard. We love to hate them. I admit I have the beginnings of a post in my blogging drafts folder, cataloguing why they are a bad idea. But that's a whole subgenre of blog posts already, and it seemed unsporting to beat that horse any deader.
But there are a couple things I've come across recently which are leading me to reexamine some of my assumptions. I'm struck by how universal this format is for job interviews at the top tech companies. It leads me to ask myself, "What if I was wrong about them? What if they actually are the best way anybody knows to hire great engineers, such that its worth investing time in playing the game?"
So, call this the opposite of a "why whiteboard interviews stink" post. It's the "Why whiteboard interviews might actually be quite effective" post.
First, a dose of perspective
In an ideal world, you'd have some sort of interview process which would allow you to rank candidates from best to worst, with perfect clairvoyance. Then, you'd hire the top candidate. The trick must be creating an interview process which is as close as possible to that perfectly clairvoyant one.
Not so fast! Unless you have a reputation as an extremely compelling place to work, the true "Top Candidates" aren't in your pile of applicants at all. Even if they were, they'd politely scoff at your offer when they realized how little you were able to pay. Even if you were somehow able to bamboozle one into joining, he'd be horrified by what he found. He'd be bored out of his skull, make some attempt at a hopeless crusade to "Turn this place around", and then he'd leave for greener pastures within a few months.
If you really, truly want to hire the best engineers, the interview process is secondary. The most important aspect is being the sort of place where the best candidates want to work.1
Companies are constantly trying to find some kind of shortcut to being more compelling (think foosball tables). But in reality, there's no substitute for an ambitious growth trajectory, a culture of engineering excellence, autonomy for engineers, management which cares about people's well-being, and interesting problems to work on.
I take it back. There is one way to bump yourself to a higher league, which is within everybody's reach.
Every manager has some leeway to make her workplace more attractive. But at the same time, her ability might be limited if she works for a bit-player insurance company, the team’s whole raison d'etre is maintaining an arcane SAP integration, and the VP she reports to is a lunatic. The VP might thump his chest about their “amazing culture”, but she's really better off not hiring "The Best". Far better to hire that developer who's good enough, and has a bottomless well of patience for corporate idiocy.
But! Assuming you do have a truly exciting company, and great engineers are applying
What would it look like if we tried building a hiring process from scratch? Imagine we live in a world which has never heard of "Job Interviews". Rather, when you want to hire, you collect a stack of resumes. You pick out the one who looks most qualified, and make him an offer, sight unseen. If you can't decide between a few top candidates, pull a name out of a hat.
But lets take it a step farther. Wouldn't it be nice to actually meet the guy before you hire him? Talk shop a little bit. You're not really looking to sort out the great from the merely good; this is just a sanity check. Is he so disagreeable that you can't chat for an hour without becoming frustrated with him? That he can't last even that long without making racist jokes or going off on a tirade about how much he hated his last boss? When you describe the job you have for him in more detail, does it seem like its a job he actually wants? Are there other issues which are obvious meeting in person, but don’t quite come out in a resumé?
We'll call this our baseline. It's low effort, and my suspicion is, you could do a lot worse.
Years ago, I read a series of short articles on LinkedIn2, where various managers shared their techniques for conducting job interviews. It was fascinating, because every manager was convinced that he or she had some sort of secret interviewing sauce. And there was no rhyme or reason to it. The aggressive sales manager liked to hire people who would pick up the gauntlet when he started criticizing their interview performance, while the quiet HR manager liked to hire people who gave thoughtful, introspective answers when asked about their greatest weakness. Individually, it all seemed like sane advice, and in aggregate, it made me realize that none of them were any good at interviewing. At best, they were hiring people who reminded them of themselves.
I am doubtful that this sort of thing, where each hiring manager has a short list of favorite interview questions, adds any value at all over the baseline. It might well be worse, ruling out great people who are just a little different. I've certainly seen it result in weird office monocultures, where everyone on the team talks, acts, and looks like the boss.
Goals for interviewing
But what are some things we can add on top of our baseline interview process, which we can be confident will actually improve it?
Avoid the single point of failure
In Lazlo Bock's book on the hiring practices of Google, he claims they discovered that it was always better to average several people's hiring decisions. The average decision was better than any individual who made up the average
How is this possible? I can only think of one explanation. I've had a pet theory for a while about decision-making.
Whenever you have a single, "decider", if he's very good, he will make wise, thoughtful decisions about 80% of the time. However, no matter how good he is, about 20% of his decisions will be genuinely bad. Maybe its that we all have blind spots. Maybe once in a while, we get focused on one aspect of a problem, and start to ignore all other aspects. Whatever the reason, having hiring decisions as a consensus of multiple people makes it less likely that one person's lapse of judgement to have serious consequences.
Weed out the non-programming programmer
It's an open secret in our industry. A small but significant percentage of the people you find employed as programmers.... can't program! Not at all. Not even a little bit.
Anybody reading this who hasn't worked with programmers will think I'm exaggerating3. "Oh, surely you just mean they're underperformers". The rest of you have a rueful smile.
For a hiring manager, this is horrifying. Hiring the guy who's not just an underperformer, but a zero or even a net-negative contributor, is a six-figure mistake if you go through all the heartache of firing him as fast as possible, and it can sink your group in the dead sea effect if you don't. They are also surprisingly difficult to avoid. They often have senior titles and years of experience with known companies on their resumés. Often, they know the lingo well enough to fake it for job interviews4.
There's really only one way - have them program, at least a little, as part of the job interview. It's not foolproof. Some perfectly capable people will get a case of nerves and freeze up when put on the spot, and some non-programmers can code a few lines for a job interview, they just fall apart on a real project.
It's something I hate to admit, but it's probably smartest to be a little conservative with hiring. If you are such a compelling place to work that you have a strong pipeline of candidates, it's alright not hiring the absolute best. If perfect clairvoyance would reveal that the person you hired was only the 3rd best in the pile, she's still of a very high caliber. Passing on a couple great candidates might be a worthwhile tradeoff if it enables you to confidently rule out the non-programmer.
But at the same time, try not to repel the top candidates in the pipeline
You can afford a few false negatives. But to the extent you weed out candidates, it's important to do it in such a way that it doesn't bring down the average. The best candidates are the ones with the most alternatives in front of them. They are the ones you will lose first if the interview process is insulting or demeaning, or seems to be conducted without respect for their time. In other words, that applicant tracking system your HR department has, where candidates need to fill out 7 pages of online forms to even apply? That needs to go.
By the same token, its important to be able to make a rapid decision. The very best candidates are likely to have competing offers, and if you require them to come in for multiple days of interviews, or dither for weeks before making an offer, you're going to lose them.
Have the candidates do a hard thing
FizzBuzz questions are all well and good, but it's a stronger signal if you ask candidates to do something genuinely hard. There's not enough time to perfectly simulate the months and years of work they'll actually perform on the job, but if it correlates reasonably well with capability, it's a win. Ideally, it should be difficult enough that non-programmers can't possibly fake it, even good programmers will be pushed to their limits, and great programmers will have a chance to shine. I bet there would be value in it being any hard thing - even running a marathon, or heck, winning a jousting tournament, but it's better yet if its a hard programming thing.
It strikes me that with all these goals, Whiteboard interviews accomplish them surprisingly well.
By being concise, they allow several people to interview the candidate separately. You avoid the single point of failure problem
They're respectful of the candidate's time. The allow for a decision after a single day of interviews
They're difficult enough, they ought to be impossible for a non-programmer to BS his way through
Most hard projects take time to do. Whiteboard interviews offer a unique way to do something difficult, in a way which fits within a short timeslot.
They have the unique advantage that they're not about specialized knowledge. The problem with digging deep into specialized knowledge in a job interview, is that even for the best candidates, the amount they know is tiny compared the amount of knowledge that's out there. If you ask a question about exotic algorithms, you'll rule out the guy who knows every detail of the JVM, but hasn't studied algorithms since college. If the job is focused on Ruby there's some value in hiring someone with deep knowledge of Ruby. But the larger a company becomes, the more of their infrastructure is unique to them, so that advantage starts to fade. If you hire someone who's smart and able to accomplish hard things, he can pick up any new technology he needs to.
There are alternative approaches of course, but they come with their own disadvantages. It might be better yet if you were able to bring every candidate in on a contract-to-hire basis, or at at take them on for a two-week trial period. But a long-term contract-to-hire is almost indistinguishable from a plain old hire, and a nobody who isn't desperate is going to pause her job search or quit her old job in exchange for the uncertainty of a short-term trial.
Take-home coding challenges are an alternative which seems to work reasonably well. However, they risk vetting for people who are willing to spend the most time with them, not the most capable people. They are also too easy for the non-programmer to cheat.
A Schelling Point
Dr. Depravio has you locked in his Volcano fortress, and he wants to play a game! Pick a number, between 1 and 1000. Ah, but the doctor wants to make the game more interesting. He has your brave assistant Passepartout locked in a room down the hall, and he's being asked the exact same question. Dr. Depravio is going to pull the lever to activate his Tesla cannon and vaporize Manhattan, unless you independently guess the exact same number.
Is Manhattan facing 99.9% certain destruction? Well..... not quite. With the stakes so high, you're not going to pick a random number. You'll try to predict what Passepartout will guess, knowing that he's a clever lad and he'll be trying just as hard to predict your guess. You're best off picking something distinctive from the range - probably one of the boundaries, either 1 or 1000, or 500 right in the middle. Once it’s narrowed down to just those options, you can be reasonably confident that 1000 is the most distinctive value
In Game Theory, these values are known as Schelling Points, and they come up all the time in real life. It's like, the best place to open a new restaurant is probably on Main Street. Everyone who wants to eat out goes there because that's where all the restaurants are, and its the best place to build a restaurant because that's where all the hungry customers know to look. Or Hollywood - it's the best place make a movie because that's where all the actors live, and all the actors live there because that's where all the movies are made. It's better for everyone if all the movie-makers have a place to gather, instead of being spread out between studios across the country.
It seems that Whiteboard interviews have become a Schelling point at Silicon Valley tech companies.
If we lived in a world where Google asked algorithmic problems, but Apple gave candidates a take-home coding challenge, Meta quizzed you on esoteric compiler knowledge, Microsoft asked about manhole covers, and Amazon wanted to know about your greatest weakness 🙄 nobody would have any idea how to prepare. Everybody would lose - Google would miss out on fantastic engineers whose only flaw was that they invested their time practicing for Microsoft-style interviews, and vise-versa.
Even if whiteboard interviews are a flawed means of hiring (they certainly are), the fact that all the big companies use them means that they are probably worthwhile to study for. The fact that engineers who see themselves as contenders for the most prestigious jobs study for whiteboard interviews, means that IF you want to hire the top engineers, you're probably best off conducting whiteboard interviews too.
I can count a number of times when I've watched a brilliant colleague resign, and I've looked around and realized "This was my failure, and the failure of the whole company. She was an awesome coworker, and we simply weren't compelling enough to keep her". The follow up question, of course, was "Wait a minute. Why am I still working here?"
I’ve been trying for years to find the series again, but no cigar
Well, maybe just a smidge. Typically, it's not quite that they've never seen a line of code. It's that for practical purposes, their ability is so low that there exists no task small and simple enough that they can accomplish it.
You might ask - Who are these people? HOW can you do this for years without being found out? Of the few I've encountered, I haven't even been able to peg a particular type. One guy was extremely charming, to the extent that he'd always been able to take the "Project Manager / Expert Collaborator" role in team projects. One had nearly a decade of experience, but with a government contractor which was so bureaucratic, it seemed he'd been able to get by as a full-time meeting attender. One was self taught, and could work simple example problems, but seemed to sincerely not realize there was more than that to creating software. Weirdest of all, one would freeze up into an analysis paralysis loop, endlessly circling around debating unit test coverage and documentation and refactoring and semantic error handling, to such an extreme that he was unable to complete even simple tasks. Like, he clearly knew his stuff in theory, but couldn't put it into practice. I wonder if engineers are like Tolstoy's families - All successful engineers are alike; each unsuccessful engineer is unsuccessful in his own way.