Conducting a High Quality “Problem Solving” Interview

Problem solving interviews have been a mainstay in many industries for decades. Consulting firms and investment banks are well-known for using “business case” interviews to evaluate candidates’ analytic horsepower and structured thinking. Technology companies use live coding exercises extensively in evaluating software developers — in fact, many companies use little else when evaluating engineering talent.

Problem solving interviews have an important role to play in evaluating candidates for certain types of roles, but they do have limitations. They are notoriously prone to generating false negatives (rejections of candidates who would actually be quite capable of performing well in the job). This happens because the interview format is inherently contrived—few companies actually require employees to solve problems in a matter of minutes via a high pressure, 1:1 interaction and without any supporting computational tools or reference materials.  Therefore, these interviews are best used when the cost of false negatives is lower, e.g., there is an ample supply of candidates, or with more junior-level candidates don’t have a great deal of career history to evaluate.

Problem solving interviews can be quite useful when used appropriately, and I strongly recommend them as a supplemental lens to evaluate candidates who will be hired into an intellectually or analytically demanding role. There are several important guidelines for making them work:

  • KEEP IT RELEVANT. Make sure the problem you are presenting to the candidate is highly relevant to the role they are interviewing for. Avoid random brain teasers or riddles—stick with problems that closely resemble those they can expect to face on the job. In fact, the best problem solving questions are often directly derived from actual work in the company. Not only does this ensure you are testing relevant skills, but it also gives the candidate some direct exposure to the kinds of problems your company solves every day.
  • MAINTAIN OPTIONS. Ideally, choose a problem in which there are multiple potential paths to solve the problem. It is common in software development “coding” interviews to present a problem that can be coded up in several different ways, some being (for example) more flexible, scalable or efficient than others. This allows you to make finer distinctions between good vs. great candidates, and increases the probability that weaker candidates will come out of the room feeling they did a good job.
  • USE PHASING. Whenever possible, structure the problem in distinct phases or steps, allowing you to see different elements of the candidate’s thinking process. Many such interviews (both business and technical) are set up three steps—(1) diagnose the problem, (2) design the best solution and (3) implement/execute on it.
  • KEEP IT INTERACTIVE. Stick with problems that can be broken into components and discussed interactively. Encourage your candidates to explain their reasoning and talk through the tradeoffs they are making as they go along. Much of the data you will gather about candidates will come from their process, not just their solution. An overly “black box” approach robs you of this valuable information.
  • CREATE SAFETY NETS. Make sure you have plenty of “nudges” handy for candidates who may struggle with one or more parts of the problem. Doing so ensures you avoid awkwardness with weaker candidates, and ensures all candidates come away feeling as good as possible about their performance. I have seen situations where a fairly simple nudge turned a potential crash-and-burn situation into a solid overall performance.
  • MAINTAIN RAPPORT. In biographical interviews, great rapport comes from showing curiosity about the candidate him/herself. In problem solving interviews, that same curiosity can be channeled into your collaborative exploration of the problem with the candidate. Even though you may have presented this problem dozens of times before, it is important to maintain believable passion and interest throughout the interview. Similarly, be very careful not to express disapproval, judgment or frustration if the candidate gets off track—candidates can detect this and it can taint the entire hiring process.
  • STANDARDIZE. Maintain objectivity and consistency by using a standard set of problem solving interviews across your team. Get clear on what constitutes top 10% vs. merely top 50% performance on a given problem, and track candidates’ performance over time to ensure you are well-calibrated. Periodically check Glassdoor to ensure your questions have not inadvertently leaked—this is a frequent problem that can create obvious bias.

Problem solving interviews are not the be-all, end-all approach to vetting candidates, even in roles with a heavy analytic or problem solving component. They are merely one of many acceptable interview formats, and should be viewed as a supplement to a deep understanding of your candidate’s unique career history. When misused, problem solving interviews can deliver “noisy” data and can create a bad candidate experience. But when used appropriately, they can provide insights into an individual’s intellectual capabilities and thought processes that no other interview format can match.