How To Interview Software Engineers
July 02, 2019
Interviews are a very vulnerable display of ourselves and I am extremely humbled by the fact that some 400 odd engineers sat with me to interview, and taught me how to do it right.
I have previously put together a team of brilliant engineers from the likes of Google, Facebook & Amazon for a seed-funded startup as the product manager and set up an entirely new tech team from scratch for FusionCharts as a product consultant. Now I am doing it at scale, as the co-founder at Adaface where we automate first round technical screening with our chatbot, Ada.
Since it has been a core part of my job for several years now, I have been working on designing an interview process that I would want to go through myself, one that helps us find the engineers best suited for the role, and most importantly an interview where candidates leave the building with a smile.
This post is a summary of what I have learned so far, and I will lay out actionable insights that hiring managers can incorporate into their hiring process. I will divide my learnings into 3 sections:
So how do we decide what questions to ask? Do not ask trick questions just for the sake of a filter. Candidates get pissed. And it doesn't give you any meaningful signal about the ability of the candidate. No one is happy.
Just because there is a question that only 5% of engineers can solve, doesn't mean that those candidates are the top 5% most suited for the role. Also, this way of measuring a developer's skill has an inherent bias against more experienced developers.
After a bit of iteration, I chanced upon what I like to call assume-the-candidate-is-exceptional way to score. Here's how it works:
Most candidates know how good they are with a particular skill, but they usually self-report incorrectly for 2 reasons: bias and misaligned scales.
There's an easy way to fix them both:
Bias → ask them to prove it with data.
Misaligned scales → tell them about your scale and ask them to grade to your scale.
Most engineers are smart to recalibrate if you explain your scale.
I typically explain my skill expertise scale to candidates:
And I ask candidates how they would grade themselves in this scale for all the skills they know. Almost all of the candidates score themselves within an error range of -0.5 of what I end up scoring them at. Use this to your advantage to know what skills to focus more on and what to spend less time on.
This might sound mean to some extent and maybe it is. But if you ask a candidate 10 questions and they solve all 10 easily, they will perceive it to be an easy interview and will associate that with how challenging the work might be. That might lead to them not being very excited about the role. The harder it is to get somewhere, the more we appreciate it. Also, it gives you a lot of meaningful data on how they react in a difficult/ stressful situation.
Obviously, that does not mean that we have to ask the hard questions and make them feel like they lost. The intention is quite the opposite. The candidate and the interviewer are evaluating each other. Let's push each other to have at least one challenge that's new so that the discussion has taught us something new, irrespective of the result.
So I end up asking at least one question (I have a handy list of some questions that are really good/ tough) to good candidates and leave them with that.
Don't write the scorecards post-interview. You will forget a lot of what happened during the interview, the scoring won't be as granular. At the same time, make sure you check for bias at the end. Did you score someone higher just because they are from a tier 1 college? Did you score them higher or lower because they are of the opposite/ same gender? Or because they were very similar to you?
We don't like to take ownership of things we didn't signup for. It is hard to take a stance of rejecting or hiring someone and a lot of the times we end up saying 'maybe' so that someone else needs to make that decision. Force yourself to not do that. One way to do that is by choosing a good scale. If you choose a scale of 1-5, most of the interviewers choose 2 or 3 or 2.5 or 2.75 (anything that makes them comfortable to be in the zone of 'I don't know let someone else make the decision').
Here's the scale I think interviewers should use for the final yes/ no decision: 1 - 4
The pros of this scale are that there are no maybe's and there are polarizing yes's or no's. When every interviewer follows this scale, it makes easy to discuss a candidate in the context of hiring them.
There are hiring managers who never compromise on the "bar"/ "quality" of candidates. But at the end of the day, we need to execute. We need to move closer to the company's mission. So how do we decide whether to hire or not? Build a decision framework that gives results (actual hires) and course-correct as you go. To this end, I believe a humane version of hire fast, fire fast strategy is better than not hiring anyone.
If we always hire everyone with a rigid framework, we'll end up with similar thinking, non-diverse and eventually a boring group. Sometimes, go with the gut and make few 'maybe' or even 'no' hires if the project needs that but be very mindful of how many such hires you're making. Who will their manager be? What are their responsibilities? Setup 1-1s to check how they're performing. Set up processes/ training to help them turn their negatives around or double down on their positives.
Here's a hiring strategy that has worked quite well for me in the past:
We need to set meetings on the calendar to visit our hiring framework and its results. Analyze if our assumptions have been correct. Are the "perfect" hires performing as expected? What happened to the "maybe" hires- how are they performing?
Your friends probably know if your current partner is good for you (or not) and might foresee your breakup way before it happens. It is the same with a team and a new hire. Your team might give you good data on what they think about the hire culturally. It also gives the candidate a chance to experience the company's culture before making a final decision.
What's a pre-requisite or a must-have skill? If a candidate doesn't have this skill, we are not going to hire them. Ideally, we won't even interview them if we can help it.
This is a bit of work. But do it. Ideally, you need to have this locked down with all interviewers. Do not let each interviewer decide for themselves what's important for the role.
Share it in the interview invitation email that you send to candidates. Once I started doing this, the conversation with candidates was much more effective. Remember that this might be constantly changing for the same role, so talent acquisition teams and hiring managers need to keep iterating this.
For example: hiring the first iOS engineer? Probably want them to must-have knowledge of Swift. Second iOS engineer? Maybe Swift isn't a must-have this time? They can learn on the job.
Tricky spot: Hiring for a role with no must-have skills like a full-stack engineer. In cases like these dig deeper into what kind of work they should have previously done. How many projects/ delivered-impact/ time did the person work on any particular skill? If the person kept switching between languages every quarter without any high-impact projects, they might be a beginner in everything.
Once we have narrowed down on the must-have skills, we also need to define the expertise range. Should they be experts in the skill, and be able to deploy new libraries by themselves? Should they already be doing code reviews in that skill? Should they have experience with one production application in that? What is the bare minimum we need?
It takes time for candidates to settle in. So we shouldn't expect them to start talking about themselves. Have a pitch about yourselves, the company and the role handy (write it down). Preferably the short, medium and long versions. Start with a short introduction of yourself and set the stage for them.
This is probably one of the most important things in an interview, and what most of us get wrong. Especially because every how-to-smash-a-tech-interview guide out there tells candidates that they need to think out loud. But not everyone works that way. Some interviewers expect candidates to walk them through their thought process as they solve a question, and that's detrimental for the category of engineers who don't work that way.
I understand why that is, and it makes the interviewer's job easy if the candidate talks out loud about what they are thinking, it's much easier to see how they are thinking about optimizing step by step and score accordingly. But, we need to learn how to work in a way that's most conducive for the candidate to perform at their best.
I believe the best way is to give the candidates all options upfront. "Here's a few ways we can do this, what would you prefer?" They usually ease up immediately.
Ideally everyone, more so if the candidate could be a great fit. Talk about what impact YOU created in the company so far and what your next projects are. Nothing beats honesty and personal experience. Tell them why are you excited to be in the company and what inspires you. Prove with data. If you are not excited/ inspired every day about your job, you probably shouldn't interview. That is one reason, I would never ask someone who has spent only 3-4 months so far in the company to interview.
I'll end with this, if you do nothing else right, please make sure you're not insulting a candidate irrespective of circumstances. An investor might have pulled out at the last moment, you might be going through a breakup or whatever else, nothing is a good enough justification to make a candidate feel bad in an interview setting.
The part of your life where you're interviewing for roles can be extremely stressful. And a bad interview experience can kill someone's confidence, you never know what someone is already going through. Irrespective of how qualified (or not) the candidate is for the role, please make sure each candidate is leaving the interview with a smile. That's the least we can do for someone who took so much time out just to interview with us.
If you would like a step-by-step procedure to conduct great interviews, you should read Adaface's Guide: How to conduct an interview, which explains the best methods to conduct an interview effectively.
There are enough companies in the world that have ridiculous interview processes that cause undue stress to candidates. Please be an example of how it can be better.