Interviewing: Ask them to show, not tell!

08 Jun 2020

Picture this. You are taking a Calculus test in High School. Instead of a written assessment, it’s a different type of evaluation. It’s an oral test. Huh? And, instead of asking to solve Calculus problems, the test is about how you handled Calculus problems you solved in the past. [brain explodes]

Do you know how to calculate the derivative of a curve? You look to both sides, somewhat surprised, and answer, “yes, I do.” The test-taker makes a note. Tell me about a time you had a hard Derivative problem. You are dumbfounded, but you answer about a problem you had two weeks ago. How did you go about solving it and what were the results of your actions? You answer it. There are three more sets of questions in the same line. You see the smile from the test taker as if you did well in the assessment. You are proud of yourself and a little confused.

Wouldn’t our academic life be much easier if we were asked to tell about our skill and knowledge instead of having to show it? Instead of having to answer the questions to the problem, we talked about the solution. Instead of having to show the results of a project, we tell you how we did the project.

Sadly, this is the state of many job interviews nowadays.

STAR/SOAR

STAR stands for Situation, Task, Action, and Result. SOAR stands for Situation, Obstacle, Action, and Results. They are the same thing. They are a way to formulate a set of interview questions to understand how a person handles a situation. It’s great to evaluate their thinking process, their ability to adapt and learn, their inner motivators, and a lot more. It can even be used to get a superficial view of a candidate’s domain knowledge.

Tech companies widely adopt this interview methodology. They are often referred to as “behavioral interviews.”

STAR and SOAR are a terrible way to evaluate functional skills. And there lies the problem.

“HIRE” vs. “NO HIRE”

I believe it’s critical to evaluate a candidate’s adaptability, motivations, desire to learn, behavior under adverse situations, etc. But it’s just a piece of the puzzle for a complete assessment of a candidate’s fit for the role and the company.

The team should be trying to answer two questions when assessing a candidate:

Will they be able to do the job they are being hired to do today and tomorrow?
Will they make other people at the company better at doing their job?

The second question is a topic for a different post.

For a team to assess the first question above, there are four things to evaluate:

Current skillset and ability to acquire more skills.
Knowledge needed and the ability to learn it.
Motivation to get the work done.
Ability to adapt as things change.

If a candidate fails (1), (2), or (3) is an automatic “no hire.” If they fail on (4), it’s up to the hiring manager to decide how stable the role will be over the expected lifetime of the employment relationship.

SOAR and STAR are suitable to evaluate some of those, but it’s not acceptable to assess skills (#1), and it falls short for a proper evaluation of knowledge (#2).

Show, not tell

The only two ways I believe are possible for a candidate to show their skills and knowledge is for them to show it in front of an interviewer or to take an assessment test.

An assessment test can come in the form of an online test or a take-home project. It’s also possible to look at someone’s previous body of work. The latter gets tricky because you won’t know how much was their doing versus collaborative teamwork. But if possible, it should be taken into consideration (even better if you can corroborate with a previous coworker).

The best way to evaluate someone’s skills and knowledge is by seeing them do their craft and show it on the spot. It’s harder for some roles, easier for others. I believe this is doable for most individual contributor (IC) roles in a tech company: software engineers, product managers, designers, copywriters, customer support personnel, content marketers, PR managers, salespersons, etc.

Yes, I’m aware of the flaws in asking a software engineer to write code on the whiteboard. I’m fully aware of how designers hate having to come up with something with so little context and research. I know how salespeople feel trapped in simulated sales scenarios where they don’t have a good grasp of the value proposition, target market, and met/unmet needs of their prospective customers.

Using on-the-fly “show me” is an equalizer. Every candidate starts at the same point. It might feel there is a disadvantage to certain personality types or “ableness” and you can compensate for that with some homework project. That said, I find that homework projects have a negative impact on the pipeline more often than not, so use it cautiously. As an interviewer, you learn how to adjust for a candidate’s introversion or even speaking abilities.

I also find a candidate’s ability to solve a problem in a time-constrained and stressful situation (interviews tend to be stressful for most people) extremely valuable. It’s a great way to measure how fast they are, how good they are at asking the right set of questions, and how they deal with frustration. Yes, I know there are flaws in that too. You don’t want to make it psychologically unsafe for them to do the best they can.

The ideal interview loop

I concluded the perfect interview loop for individual contributors is four interviews only. There is a diminished return in assessing candidates after the 4th interviewer. Just three interviewers aren’t enough to cover the assessment scope. The four must be good interviewers. If you have someone with only a few years of experience in interviewing, you might want to add a 5th interviewer.

In addition to the main interview loop, I also expect for 3–4 additional “talks” with a candidate:

A recruiter screening: 50%-ish of candidates should be eliminated at this screening. If too few are eliminated, it’s putting an unnecessary burden on the hiring manager. If too many are eliminated, it’s either an indication of a pipeline/job description problem or a problem with the screening process itself (too hard or poorly trained recruiter).
An “informational interview” with the hiring manager before deciding to bring a candidate for a full interview loop. This is the time for the hiring manager to lay it bare the team, the work, the company values, the culture, etc. since you want candidates self-selecting out of the process.
Optionally, there is a “lunch interview” involved. I can’t stress this enough. Lunch interviews must not be assessment interviews. If your company asks for the lunch interviewer to provide feedback, something is wrong. The exception is if the candidate says something that would disqualify them (e.g., he uses a racial slur or talks trash about a profession that he doesn’t like to work with).
A post-interview “sell chat” by someone above the hiring manager if the candidate is a HIRE. This “interview” is not for assessment (although red flags might be raised). It lets the candidate know they will fit well with the company and let them ask whatever questions they might have lingering in their mind. It’s also an opportunity to anchor the role expectation, prime the candidate for an offer, and get them excited about the company.

The “On-site” Interview Loop

Back to the four interview loops and what I consider a great loop looks like, independent of function.

Interviewer #1

Of the four interviews, I believe the first interviewer needs to focus on the candidate’s previous experience. Candidates get anxious if they can’t talk about their awesomeness in the past. You want to get that out of the way right at the start. Let them speak for 15–20 minutes of *recent* previous experiences that are relevant to this context. Sometimes, this is the hardest interview because candidates can ramble in the wrong direction. Then the interviewer can spend 20–30 minutes in SOAR/STAR mode focusing on the past.

Interviewer #2 and #3

The two middle interviewers, interviews 2 and 3, are critical to evaluate skills and knowledge. They should be 100% focused on presenting the candidate with a problem or task, and seeing how they go about it and how far they get. These must be (progressively) hard, so the candidate feels stretched. It must be as close as possible to the work they will be doing, and it must have the necessary leeway if they don’t have the domain knowledge or the knowledge of tools, processes, or methods used by the company.

Interviewer #4

Finally, the last interviewer has two relevant jobs to do. First, use the STAR/SOAR method to evaluate a hypothetical future scenario where the primary goal is to assess a candidate’s adaptability, motivation, critical thinking, other soft skills, and, as importantly, alignment with the company values (all interviewers should be looking at this). Second, it’s the opportunity to get the candidate wowed by the company and ask some final questions. Ideally, the last interviewer is the hiring manager. In some companies, they are people who have shown exceptional candidate assessment abilities (“Bar Raisers” at Amazon, “As Appropriate” at Microsoft).

Wrapping

There is a lot more about the recruiting and interviewing process that I didn’t describe here; otherwise, I’d have to publish a book. The point is that interviews need to be structured and the interview process deliberate. And, they need to focus on the key questions you are trying to answer:

Will they be able to do the job they are being hired to do today and tomorrow?
Will they make other people at the company better at doing their job?

If you have the confidence that you answered those questions with a “Strong Yes,” go ahead, send that offer!