Hiring is Broken And Yours Is Too

I can’t keep track of the number of articles I’ve read about hiring in the past few years. Inevitably, they all follow the exact same format. First, they claim that hiring is broken. Next, they describe the hiring practices used by XYZ company. The present an extremely thorough analysis of every possible way in which such a practice can be gamed, or produce bad results. Finally, they wrap up by saying that hiring should instead be done on the basis of ABC criteria, only for someone else to write an essay the following week attacking ABC as a broken hiring method.

At this point, I’ve seen almost every single combination of ABC and XYZ. As a fun little exercise, I thought I’d take some time to compile every one of them.

I’m sure lots of people have very passionate opinions about the right and wrong way to hire. Quite frankly, I’ve lost all interest in hearing people’s opinions and anecdotal experiences. Until and unless someone does a rigorous scientific study evaluating different interviewing techniques, preferably using a double-blind randomized trial, there’s no point in beating this dead horse further. Everyone’s hiring practices are broken, and yours aren’t any better.

Look at which University they attended

Pros:

• University admissions departments spend an enormous amount of effort and manpower in identifying and attracting the most talented individuals. Why not leverage this investment and expertise for your own company
• Top universities generally speaking have top professors, other top students, and challenging course content, all of which will rub off on the candidate
• Getting admitted into a top university usually takes a lot of hard work and intelligence. This can be a good proxy for job success

Cons:

• There are idiots present in every university. Yes, even at Stanford
• University admissions unfairly favors the wealthy and well-connected. By rewarding this, you’re discriminating against the poor, the not-well-connected, and minorities
• The skills required for getting into Stanford at 17 (extracurriculars, SAT prep etc) do not correlate to job success as a software developer
• How good a student you were at 17, is not very relevant to who you are at 25

Look at which companies they’ve worked at in the past

Pros:

• See all the pros listed above, and replace “Stanford” with “Google”
• Companies are usually better proxies for the technical skills you need in software developers

Cons:

• Those companies have hiring practices that are broken (hence why you’re writing this post). Therefore, you shouldn’t be relying on their hiring process
• This doesn’t work for new-college-grads
• You’re creating a self-perpetuating loop where someone who got rejected by Google the first time, will never get a 2nd chance ever again

Look at their GPA

Pros:

• It’s a very comprehensive metric, which takes into account performance across a large number of courses and exams, over 4 years, covering a mix of homework, projects and exams
• It’s a good proxy for how hardworking and dedicated an individual is
• It’s a reasonable proxy for how smart an individual is

Cons:

• Einstein had a bad GPA, so clearly, the metric is broken
• Many 10x hackers are so focused on their passions, that they neglect unrelated coursework, which leads to a bad GPA
• For experienced candidates, GPAs are extremely out of date
• Someone’s grade in Economics shouldn’t matter in a software interview

Ask them for references

Pros:

• Gives a view of their actual on-the-job performance

Cons:

• People will only ever list references who will say good things about them
• Someone may provide a good reference just to avoid getting sued
• Rewards people with great social/networking skills
• Punishes people who’ve worked with bad/abusive managers in the past

Do back-channel references

Pros:

• Gives an unbiased picture of actual on-the-job performance

Cons:

• You will be considered evil for doing this
• Hard to consistently find the right people who can provide accurate references
• Rewards people with great social/networking skills
• Punishes people who’ve worked with bad/abusive managers in the past

Take home projects

Pros:

• Emulates on-the-job work
• Eliminates any false stress from the interview setting
• Get to see and evaluate the actual work-product that they produce

Cons:

• This is basically unpaid work. You will be considered evil for doing this
• Very time consuming. Discriminates against people with lives and families
• Extremely easy to game. People can get their friends to help them. People can spend 30 hours doing something which should be a 10 hour exercise, just to look really good

Audition for the role as a contractor

Pros:

• Actual on-the-job work
• See what they are like to work with
• See their actual work product

Cons:

• Depending on their current employment agreement and immigration status, this could be illegal
• You’re asking them to take a week of holiday, just to work for you. People will hate you for doing this
• Discriminates against people who have lots of companies showing interest. They can only afford to do this audition at 1 or 2 places
• Discriminates against people with lives and families

Ask them to describe their past projects

Pros:

• If they’ve worked on challenging, relevant projects in the past, they can likely be successful at your company as well
• Building a successful product can be a reflection of talent

Cons:

• Anyone can glamorize and exaggerate the work they have done, and make it sound a lot more impressive than it actually was
• Working on a “cool project” often comes down to being in the right place at the right time
• Product success or failure is often reflective of PM talent, not just Dev talent
• Evaluation is very subjective. Punishes candidates whose interviewers have had a bad day. Punishes candidates who aren’t very eloquent or good at schmoozing. Punishes candidates who don’t fit the ideal-developer-stereotype

Ask for source code from significant projects

Pros:

• Actual work product, that’s a good reflection of their capabilities

Cons:

• Most work product is confidential. Google will never allow you to show your work-source-code
• Not everyone has the time to work on open-source projects. Punishes people with families
• Not everyone is interested in spending their free time on open-source projects. Punishes people with diverse interests

Quiz them on their knowledge of various technologies, tools and frameworks

Pros:

• If they know about, and how to use technology X, they will be able to use X to successfully fulfill their job role
• Shows that they care enough to research this stuff

Cons:

• Tools and technologies can be learnt. Hence, they shouldn’t be used as a filter
• Fundamental development skills aren’t well correlated with specific tools/technologies, and the former is much more important
• There are too many things for any one person to know. Immense luck involved in your interviewer quizzing you on the things you know, and avoiding the things you don’t
• You don’t want to be that company that hires for “brown”

Brain teasers

Pros:

• Tests their overall intelligence and problem solving skills

Cons:

• Either you know it or you don’t. Relies on you being lucky enough to get a flash of inspiration, or you having heard it before
• Not directly related to programming
• Even Google says it is useless

Live Coding Exercises

Pros:

• Live test and demonstration of coding skills, that’s not susceptible to the above cons
• The fact that FizzBuzz is so effective, demonstrates the need for more in-depth live coding tests
• Completely fair and impartial. If you solve the problem, you’re scored highly. If you don’t, you’re scored poorly. Such meritocracy

Cons:

• Many questions suffer from the same pros/cons as brain teasers: they rely on flashes of inspiration, and you being lucky enough to have them
• Very stressful. Many programmers can’t handle the stress of coding under the clock, with someone watching them
• Not reflective of practical coding. Questions are artificial and contrived
• People will hate you for doing this
• “90% of our engineers use the software you wrote, but you can’t invert a binary tree on a whiteboard so fuck off”

Pair programming on a “real” problem

Pros:

• You’re working on a real problem, that resembles the day-to-day responsibilities
• There is no “known-right” answer. Collaborative exploration

Cons:

• It takes developers weeks to get ramped up on a specific code-base, domain expertise, and technology stack. Expecting someone to drop in one afternoon and start making an impact, is not very realistic
• Highly variable. Depending on the day they come in, some candidates can be given very easy issues to work on, involving languages/domains/technologies they’re already familiar with. And others can have the complete opposite experience.
• In order to assess someone’s potential, you have to give them problems that push them to the limit, even if it isn’t “realistic”. There’s a reason Intel CPUs are evaluated using synthetic benchmarks, and not MS Word
• If you gave every candidate the same “real issue”, of challenging but not insurmountable difficulty, in a sandboxed environment, requiring minimal prior domain-knowledge, with flexibility on language/technology-choice: Congrats, you’ve reinvented live coding exercises.

---

The German translation of this essay has been published in golem.de.
And a Russian translation can be found here on habr.com.
Discussion thread on Hacker News.
A variant of this essay was earlier published here.