What do we know about hiring? We’re data scientists.

I’ve been a line engineer since 1999, and in those 19 years, I’ve had my fair share of interviews, both sides of the table. I’ve interviewed as a candidate and recruited as a hiring manager at small companies, startups, large companies, and government agencies. It’s a familiar flow; recruiter sets up a phone screen, you go through some basic checks, move onto the obligatory technical screen, and if you sorted a string or fibonacci’d the hell out of terminal buffer like a pro, you get invited to a gate drill, the highlight of the hiring process, an 8 x 1hr test of will where the interviewer assesses your abilities across a set of problems you’ll most likely never apply in role or are used to assert technical and intellectual dominance over you.

After this, you’ll proceed in one of three ways:

  • You’ll get an offer – Hopefully within 24-36 hours, or within 6-8 weeks, it depends on how far the hiring manager is into the latest season of Game of Thrones and whether Mercury is in retrograde.
  • You’ll get ghosted by the recruiter –  Everyone wants to avoid the uncomfortable conversation of trying to objectively explain why someone was passed over due to subjective measures.
  • We can’t share the reasons – My absolute, hands-down fave. This is the best shield for people who have zero clue about a topic but are tasked with explaining it to someone else. I say this all the time when people ask me why I didn’t wash the dishes or how does space travel work; “can’t share the reasons“.

This was the process for me for the first ~ 5 years of my career, but there is a downside to it. It takes a lot of time. Time to read resumes, time to book phone screens, time to do interviews, all tolled, it takes about ~ 4-6 hours per candidate. Also, I had to take my best engineers off the line to do the interviews, which is terrible for team productivity and also it’s not really what they signed up for, which resulted in lots of “I really don’t want to do anymore interviews”. Then there was the subjectivity. At the end of the process, no matter how good our interview tool was, we’d get in a room and debate the merits of each candidate. There was no matrix, no candidate by row and attribute by column, no empirical process to selecting a candidate, just good old horse trading.

Now, at this point of the story, most people start to feel the attack coming on, and either get into the defensive posture, as if ready to jump out of the closest window yelling, “My process isn’t like thaaaaaaat”, or they assume the attack position, ready to discredit the assertion that all that time, people, money, could not yield the best possible outcome. As they like to say, “weight is a sign of reliability”. But it’s not, it’s just a proxy in the absence of something better.

Enter data science, more specifically, deep learning and embedding. I’m a huge fan of embeddings, and in my most recent role at Microsoft, the power of embeddings in every part of the business from cloud computing to content relevance was epic.

So what do we know about hiring? We’re data scientists. 

Well, I can tell you that the most impactful way to objectively assess a technical candidate is to observe them during a coding exercise. Observation is a spectrum, you can observe someone:

  1. Solving a problem on a whiteboard
  2. Doing a simple problem on a tool like HackerRank
  3. Sit them down at an existing code base on a live running system and give them a set of tasks to complete

I think we can all agree number 3 is optimal, it produces the richest stream of data to analyze, but the downside is it’s expensive, complex, and doesn’t scale well. You’d have to have lots of experienced engineers looking over the shoulder’s of a candidate, taking incredibly detailed and uniform notes, and then have a process to compare each candidate across cohorts and across all candidates who have ever taken the test.

So what do we know about hiring? We’re data scientists. 

You know what I love about machine learning? Other than by pledging my allegiance early I’m less likely to be slaughtered at the hands of my robo vacuum cleaner? It doesn’t suffer from fatigue. It doesn’t get bored. It doesn’t get distracted. And it can see things humans cant. This is great when taking lots of noisy data on a regular basis and producing an low bias signal. I mean, imagine if we could take the technical hiring process today, and invert it. Engineers first start with an adaptive, live-fire coding exercise where every possible data point is captured. At the completion of the assessment, data is fed into a sophisticated deep learning network that is capable of computing millions of data points and relationships, and can produce a number which objectively tells you how this candidate will perform as an engineer against a set of baselines. This data is then used not just during the recruit phase, but after the engineer starts, data is continually captured and modeled to proactively predict performance issues. Sounds pretty awesome.

So what do we know about hiring? We’re data scientists. 

As an applied data scientist, I’m hyper-conscious of hyper-parameters (see what I did there?). What’s a hyperparameter? It’s something you can’t learn. For example, if I have some data, like, the time of day and the number of donuts Dave eats. I can model this and one parameter might be hours from lunch time, and this might describe how likely Dave is to eat a donut. However, if I know that Dave only eats chocolate donuts after 12pm, and glazed before, then that’s a hyperparameter. It’s not learned from the model data, it’s a fixed parameter that is part of the model but not controlled by the model.

Technical recruiting has a few hyperparameters, but the one I dislike the most is pattern matching. Because I’ve never met a recruiter who also earned a living as a full time engineer, I’ve never met a recruiter who was competently able to screen/filter/vet a candidate on their technical ability. So instead, they filter on hyperparameters. Hyperparameters I’ve experienced personally are:

  • Did the candidate go to MIT? (or equivalent name brand “tier 1” school)
  • Has the candidate ever worked at Google, Facebook, Amazon, Microsoft, etc?
  • Does the candidate profess to know every buzzword tech under the sun at a core contributor level?

Switching gears, you know what my fave hyperparameter in data science is? Regularization. It’s this cool thing that helps you from over fitting your model. What’s over fitting? It’s where you train your model so well for one specific thing, that it’s amazing at that thing, but sucks at anything else. Regularization relaxes this and makes the model better at generalization.

You can see where I’m going right? If you’re model is hypertuned with heavily subjective bias like the school someone went to, or if they ever worked at company X or Y, then any hope for diversity or objective evaluation is gone. The very start of your process is broken, and like any data science process, garbage in, garbage out. The current model needs to be remade, with new hyperparameters that eliminate subjective bias and help our hiring models become more generalized and powerful.

So what do we know about hiring? We’re data scientists. 

As it turns out, as data scientists, we actually know a fair bit about hiring, only, we don’t think about it the same way as most of the industry. As data scientists, we embrace data, objectivity, fairness, and an empirical process. Don’t get me wrong, the human side is super important too, but we also believe that that can be made more scientific and efficient too.

Is it  impossible? No, it’s just complex and hard.

Is it the right thing to do? Yes, because every engineer deserves to be judged on their technical features and not their demographic features.

Will the revolution be televised? You can bet on it!