When it comes to making long term decisions, I like to collect a variety of data across a meaningful period of time. Why? Between two entities in a relationship, you are almost always going to see variables change over time. So when it comes to recruiting, retaining and developing engineering talent, observing someone over the course of an 8 hour interview loop gives you a contracted period of time with limited variables from which to draw conclusions and make predictions about the long term arc of you and them. Add to this the unique nature of AI engineers, and you’re going to need a new playbook to guide you in building a world class data science team.
Before I dig into the playbook, let’s talk about the archetypes at play. There are three distinct types of data science engineer:
This fresh grad has more course credits in AI/ML than Prof. Ng could teach and has just put the finishing touches on their thesis solving Falconer’s conjecture written in pure R. They are light on software carpentry skills and the only tests they have ever written were as a TA in college.
This up and comer has been on the line for years, mastered no less than 3 languages including one named after facial hair and uses painfully esoteric IDE’s that border on ASCII art. They’re “self-taught”, meaning they’ve read every README.md on Github that mentioned AI/ML and can wax lyrical with utmost conviction on why Deep Learning beats the stuffing out of SVMs without personally having ever used either.
Wha? Machine learning? Artificial Intelligence? Who? Speak into the good ear!!
Who’s the best type to build a team with? Well, you kind of need elements of all three. I’ve built AI/ML teams for the past 5 years and while the methods for recruitment and selection have changed, the personalities are almost always the same. You need someone who has a reasonable grasp for what’s happening under the covers, otherwise they’re going to shy away from the complexity and end up practicing coincidental coding rather than making rational informed design decisions. They also need to have a hackers spirit. Engineering AI features is heavily iterative and takes the patience of a saint, so if they’re not capable of working in ambiguous environments they will burn out. And lastly, they need have a healthy level of scepticism regarding the overall field. Machine Learning was born out of a schism between AI purists and those who had to work for a living, and as AI has taken the brand forefront once again, there is a daily deluge of information on AI and ML. If you don’t have good bullshit filters, you’re going to spend a lot of time moving your eyes left to right instead of your fingers up and down.
Make it interesting, make it relevant, make it honest. Omit words like “rockstar”, “ninja”, “guru”, and replace these with actual frameworks and platforms you are using or intend to use. Discuss the expectations of the role within the framework of:
- Feature engineering – Where does the data come from, how much work is required to get it into a quality standard for training models?
- Model selection and training – How are you expected to develop hunches? What are you expected to deliver and in what time frame? When is done, done?
- Model maintenance and troubleshooting – What does care and feeding look like? Is this mission critical or best efforts?
The JD is crucial to inviting the right candidate into the funnel while ensuring the looky-loos and tire kickers don’t clog your pipes.
The Coding Exercise
If you’re like me, when you interview someone for a role on your team, you’re thinking long term, or at least you should be. So back to the 8 hour interview. I worked with my last AI/ML hire for around 6 months, but my longest relationship has been over 5 years. So let’s say for me x is somewhere around 960 < x < 9600. So at a min of 960 hours of engineering time spent, I have 8 hours of data to set up my vector. At 9600 it gets worse.
Also the one thing I’ve always hated about the 8 hour interview, is as engineers, we spend so much time in code, alone, together, in the SO hive mind. So pulling someone in for a series of cold start coding sessions is down right bizarre, and frankly, feels like the kind of interview process a non-engineer would come up with (stares directly at IBM middle management). So I like to put together a comprehensive coding exercise which touches the major elements of the job description including any engineering and infrastructure pieces required to be successful in the job. I like to give that to a candidate on a Friday evening and expect it back before work starts Monday morning. The process is very straight forward:
- Clone an assignment repo – This contains a seed project in a monolithic form with all the major project folders in place with a detailed README.md regarding what is expected and any data required to perform the task. For example, it might be some product data in a json file where the task is to create an API to expose an endpoint where incoming product descriptions are matched with similar product descriptions from the json file and delivered through the API.
- Encourage frequent and small commits – As the candidate is making progress, encourage them to make commits with decent commit messages. This is great to provide insight into their reasoning and thought process, and also gives the interviewing team concrete places to discuss code with the candidate should they make it to the next stage.
- Use the SCCS to collaborate and discuss – Github and Bitbucket both have great PR and comment capabilities. Leave comments, ask questions of the candidate, leave notes to your team mates. All of this provides a great place to have a code conversation with your candidate within a meaningful context.
Come Monday morning, you’re going to have a great insight into your candidate, and they’re going to have a realistic sense for the job they’re applying for.
The Follow Up
Don’t tell me you thought there wasn’t an 8 hour interview? Of course there is, it just isn’t a random expression of technical strength designed to make your candidates feel like dousing themselves with petrol. Instead, it’s a joyous confluence of engineering expression facilitated through code. Code that the candidate wrote, that your team is familiar with, and actually relates to the job they’re applying for. During this time you can dig into their reasoning, design decisions, but also provide positive feedback, let them know if they did something exceptional or delightfully unexpected.
A Data Scientist is not just for Christmas
So you’ve vetted 90+ candidates over the last 3 months and are ready to offer 1 a job. They accept and you’re in AI/ML engineering nirvana. Now what?
Don’t drop the ball. It’s really that simple. The best AI/ML teams recognize that it’s not an exact science or an engineering discipline, it’s a bit of both with magic in the middle. So always make sure your checkpointing your processes, reviewing your practices, and tracking against your high standard. By keeping the tempo similar to the interview process, learning new methods, exploring innovative technologies, solving unique problems, recognizing great work, you’ll not only build a great team, you’ll keep it.
And if you have any questions on how all this works in the wild, drop me a line.