Code Pilot Manifesto

Hiring Science

A boring bit about me, my name is Scott Fletcher, and I am one of the founders of Code Pilot. I am fundamentally a systems and a problems person. Systems that give the desired outcome with a minimum of fuss. I enjoy systems that are streamlined and make sense. I really enjoy solving problems, and when I look to solve problems, I usually think from a systems-oriented approach. How should this work? How could this be done better? What is wrong with this situation etc…

This happens to me every day when I am waiting for a coffee and I think the shop could improve their service, profit margins, and customer satisfaction. My mind follows the same path when hearing client’s stories, or waiting at a bar for a drink. Most of my business ideas stem from a frustrating system or process.

Earlier this year, a dear friend of mine, Dave, was talking to me about his idea on how to improve the hiring process, specifically with hiring software engineers. I have felt this pain, I have wanted it to be better for a long time, but never really put my attention to the problem, it has just been a frustration while working on core operations of various companies. Hiring is just an external annoyance – albeit a very annoying annoyance.

I was even once so frustrated that I wrote to Reid Hoffman about how he could make Linkedin better if it focused on projects people worked on, instead of being role based. It is really hard to know what someone does in a role at a company, but knowing what projects they are working on, or what they actually do was my hypothesis to make it easier to determine if someone would be suitable.

Spending the last 3 months with Dave thinking, talking, researching and analyzing and building solutions for this problem has been enlightening. Our understanding is much deeper than I ever imagined it would be about this area – something that has never previously interested me. Luckily it is an area full or problems and broken systems.

There is the added topic of diversity, and the makeup of companies is currently a very hot topic internationally. Are there enough women in senior management? Why is the company monocultural? Why do we prefer certain schools? How do we deal with different people? With different ideas? What is best for our company, or any company and how do we get there?

These problems are deep and multifaceted, but I believe people are looking at the problem with the wrong lens. They are thinking inside the constraints of the existing system, the existing paradigm. The way companies function, the way company’s culture develops, the changing nature and expectations of society.


My Hypothesis: All these problems stem from the following –

  1. The inability of a company to objectively measure and analyze people and their talent to achieve a specific tasks/ set of tasks.
  2. The inability to define a set of tasks/competencies for a role that is complete and sensible.
  3. The inability of people (applicants) to articulate their talent, and measure their talent and abilities in different areas.


I will cover broad areas and why they are important to consider:

  1. Hiring Workflow
  2. Parsing Problem
  3. Subjective Bias
  4. Sunk Cost Fallacy
  5. Am I good at hiring? Or, just not terrible?
  6. What a company needs
  7. What a person needs
  8. Limited Potential


Hiring Workflow – Points to a symptom

This thread on Hacker News is very interesting:

Why is it so hard to hire good engineers? (nothing to do with us, just interesting)

I think Git-Pull starts his post with a very relevant piece of information, then trails off….:

‘Maybe it’s hard for good engineers to find good employers and bosses. 9/10 of the technical screens I take are bunk. It boils down to what’s fresh on my mind that moment; random trivia.’



What is the problem here?

The problem in a nutshell:

  • Applicants are terrible at articulating what they are good at and have no way to let people objectively know how good they are.
  • Companies are terrible at saying what they want, what skills they actually need from applicants, and have no way of measuring whether someone is good or not at what they can’t describe.
  • The process creates further pain because applicants are trying to work out how best to communicate with the other side and pass their filters, which have developed due to the inability to assess people – see Subjective Bias below.

Hence the application process is a big bulb of viscous hell that everyone is stuck in, whether looking for a job and looking for new employees. This is big business, some say it is now a $400Billion industry. I have not seen any new entrants to this industry try to remove the bulb, they are just trying to make it more efficient.


Currently, Applicants need to do a CV / Resume and write some kind of introduction message. This in itself is a daunting task, and typically can take up a month of procrastination, I know some people who prefer not to move jobs because they don’t want to do this part. What value does it add to the process? It helps people tick boxes. Typically people list 28K of programming languages they know intimately and have worked with every system around. I find it funniest when they say they have been doing some things longer than those things have been around – very common these days with new things being invented every year.

People are doing this because they want to get a chance to tell someone they are good and capable. How does this help ESL (English Second Language) people? How does it help people who are not good self-promoters? How does it help companies who need to differentiate between people? What should you say you can do? How do you distill 4 years into 5 bullet points? Is that too much? Or not enough?

Ultimately the CV and cover letter will cease over time. It will become more like a trade card summary – I have 10 years experience, top 3 languages are: 1,2,3 and top environments are: 1,2,3 and the objective talent score combined with experience on each of these things.


How should you describe a job? It can sometimes be very hard, and sometimes very simple – but can we write something simple? ‘We need a full stack developer to work with our product team to add features and remove bugs.’ – This would very accurately describe a lot of software jobs.

It is important to start to use relevant information for a job, Does it matter what the company does? Does it matter what the product does? Does it matter about the applicant’s theoretical expertise or practical ability? All these questions have two sides, and sometimes they are relevant and sometimes not – depending on what you need and who you want to apply. I know a lot of applicants that say I don’t really care what the company does, I want to work with fun people.

Today if you think about it, engineers are hired the same way as an executive assistant – they jump through some hoops, and then slowly get a decision. This is insane, and the process of doing all this has barely changed over the last 50 years.

From an engineering perspective, recruiters provide no value apart from shuffling papers from one pile to another, as they have no ability to assess technical talent. Recruiters have essentially become tour guides for the big bulb of viscous hell – also known as the application process.

Companies should ask themselves what they need from an engineer, not everyone wants to work on an anti-gravity machine – a lot just want money so they can enjoy their life. Is there room to grow? There is often a lot of emphasis on theoretical knowledge – I know of companies who push this a lot, and have debates in interviews of things that will never be used in the job.

Ultimately most companies need someone solid who can get things done in a timely manner. This requires knowledge of their trade, ability to learn, ability to be adaptable and able to solve problems in a sensible way. They might have some prerequisites on language and environment, but unless these are advanced situations, they too can be learned by talented people.

This will come down to a company articulating what they need and matching that with the talent from applicants. Then both need to work out if there is a personality clash if all good – then start work!

Parsing Problem

The parsing problem is one that causes more resources, software and money to be thrown at the problem. Companies looking to capitalize on hiring industry have started creating and solving their own problems. Here is the way I think about it:

  1. You can apply for an unlimited number of jobs in a very short space of time – in fact, job boards encourage this
  2. Companies can not sift through all the applications – it becomes overwhelming
  3. They deploy software to do this – we now have ATS (Applicant Tracking Systems) – and for some bizarre reason charge you on the headcount of your entire company, not applicants or open positions
  4. They make this a little more manageable
  5. When this doesn’t work, we should talk to a recruiter – they will get us the best people
  6. Repeat Steps 1-4

When we get all this noise in the system, we have created a very large parsing problem that can’t easily be solved. There are too many resumes to really read – we skim almost all unless something stands out. Then we start to invent things that will scan the resumes and work out who is more suitable based on their content – this assumes that the content is optimal – or otherwise it may highlight people who wrote resumes that the system likes.

We are at a place where there is so much noise, that we skim read them all (giving them 10-30 seconds), and then shortlist and spend a bit more time. We can not effectively parse the information contained in these applications so we rely on Selective Bias.


Subjective Bias

According to my good friend Google:

‘Subjective validation, sometimes called personal validation effect, is a cognitive bias by which a person will consider a statement or another piece of information to be correct if it has any personal meaning or significance to them.’

I see a subjective bias as crucial in this area, and something very interesting.

People are full of bias, every single person on the planet makes many (I would say 100’s) of decisions every day based on bias (or learned behavior, which can be another mental shortcut). This is not always a bad thing. It is the way that people shortcut the multiple sets of complex information that they are required to process constantly. Some biases could be:

  • A restaurant with a queue has better food
  • More reviews and highers stars on TripAdvisor mean something is better
  • Harvard is the best school
  • The smartest people go to Harvard
  • That coffee shop does not make good coffee (based on one experience 4 years ago)

These all seem silly, but they are all little mental rules that we develop to make things easier. The problem is when these are applied to hiring and hiring decisions, the outcomes can be less than ideal. Hiring bias I see all the time – women are not good engineers, that school is no good, never heard of the previous company, this programming language is better than that etc.

This can go on forever, and it polarises the whole hiring process. At a company where I used to work, it was very multicultural which was great. But there was a habit of people to fill their teams with people from their country. It got to the point where I was mandating discrimination based on their bias – ‘we have too many of people from country X, no more’.

The only way to remove the bias is to take them all out of the equation, and just measure talent. If you measure talent first (before opening an application), it is amazing how different the lens is. If I put three people in front of you and said number 2 is most talented, your bias would still be active. If I put three people behind doors and said door 2 was most talented, which door would you open?


Sunk Cost Fallacy

According to my good friend Google:

‘The Sunk Cost Fallacy. The Misconception: You make rational decisions based on the future value of objects, investments, and experiences. The Truth: Your decisions are tainted by the emotional investments you accumulate, and the more you invest in something the harder it becomes to abandon it.’

In my experience when hiring people it was such a tremendously painful experience. This comes down to many different reasons, a few being – I was always so busy, I don’t have the time to invest in all the work (even with an internal sourcing team or recruitment company). There was no way to work out if people were good, we used some coding tests – often different, often as a filter, not really trusting that as technical vetting.

It would always seem to take a huge amount of time until the person would sign their contract, 2-3 months easily. Often if people told me they were leaving, I would give people a pay rise knowing they would leave anyway just so it would give me 3-4 months longer until I could replace them.

We would get several people to interview them – maybe 3-5 people, the largest I was ever involved in was 11 interviews – insane on the company and applicant. Google said it costs them $4,000 for every person they evaluate for a job.

Why do we do this? To me now it seems as though we do it from fear of full ownership and taking the risk, not wanting to go through the process again, it takes so long that we want to make sure everyone is happy, we are scared to pull the trigger – so lets get a committee.

When we can shortcut this process with objectivity, this will be replaced in the future by 1-2 x 30-minute culture chats – that is it. Hired. It can be easy. It is far more productive to spend that time and effort working with someone on problems. If they are the wrong person swap them out faster. The fastest hire we have achieved went from assessment to accepted offer in 27 hours. It doesn’t have to be painful. This guy has his shit together:


Am I good at hiring? Or, just not terrible?

I was recently talking to a friend and colleague, and asked him what did he think his success rate was in hiring people. He said 70-80% success rate. Upon thinking about this, I think it means that 20-30% were terrible / mistakes. 20-30% scored an F. How many received an A, B, C, D?

Maybe my scale is a little off, but I feel the principle is correct. How many people are in a job just filling space, or worse, being destructive? Without articulating the measure for success and defining a role very well, it is hard to draw this out accurately, but if you asked people what they wanted in an employee when they were justifying their need for headcount, compared to what they ended up with 1 year after they hired someone – it would be a very interesting comparison.

In the future, there will be no work to do when hiring. All you need to do is get better at articulating what you want and then be able to get an opinion on whether you can work with someone after a 30min chat. This will streamline the whole process, and also remove a lot of variables from the equation.


What a company needs

When thinking about what a company needs, it is important to really understand the base work that a person will be doing. When looking at this base work, and understanding what is the desired outcome things become clearer. When thinking about say a full stack developer, rarely are people inventing something new. Most work is done automating processes, making things easier for customers or scale problems.

Does this require someone in the top 1%? I would surmise that it requires an engineer that understands their domain, does not make a lot of mistakes and is happy grinding out work. Most full stack engineers simply do not come across impossible problems that have not been solved.

The other key element is what kind of organization is this applicant coming into? Are they coming into a company that is full of amazing engineers, or company with engineers, or a company with below average engineers? It is important to have an understanding of what you are looking for and how this compares to what you already have. It is hard to mix really talented people and put them in the same role as someone less talented without an ability to express themselves or change things in the company.


What a person needs

When we look at what a person needs, most people say that they want a fun team, and to be learning. They don’t like politics, and rules that don’t make sense – which invariably come in any organization once they reach a certain size.
Matching the needs of an applicant is very important, very rarely have I seen people want money as their primary driver (more in sales positions.) If we can articulate what the company needs, and their current set up, it will become easier for applicants to understand what they are walking into. Matching the requirements of the applicant will really help drive long-term engagement.


Limited Potential

A key area that can be quite disheartening is that everyone has a limited potential. When promoting people, the biggest mistake that I have made in the past is promoting people who are amazing at their job, with my idea being their influence will then make more people amazing. The problem here is that a lot of the people fail at the next level up, and then you have to remove them as they would not take a demotion.

It is very hard to identify where the potential will stop, however, this is important to think about, if organizations are some kind of triangle with someone at the top and more workers at the bottom, not everyone can go to the top. So when thinking about who to hire, or what kind of people you want in your company, think about the fact that you might hire someone to sit in the same role for 5 years and love it. Not every hire needs to have potential to be a manager or leader of a company.


Code Pilot

Dave, Myself, Caleb, and Cara started Code Pilot as we were sick of the broken system, the constant bias and the inability of the current system to run efficiently. We are a data science company, everything we do is based on data and science. Hiring Science is our application of data science in the way Software Engineers are hired. Drop us a line if you want to do things in an easier way.

Scott, Cara, Dave and Caleb (they are actually conjoined twins).

For Australian friends, I am using US Spelling as we are based in Austin, there are no doubt many examples which are incorrect.