AI in Hiring: Bias, Black Boxes, and the ATS Triopoly
Founder, ResumeAI
Kayvan is the founder and lead engineer of ResumeAI. He writes the product and the code, and recently open-sourced the State of ATS 2026 dataset that maps the Applicant Tracking System used by 743 large employers.
View on LinkedInLast updated
A friend sent 1,600 applications
A friend sent 1,600 job applications last quarter. She got 78 interviews. The vast majority of those 1,600 were rejected by software she never saw — and in three-quarters of cases, the software came from a single vendor.
I open-sourced the data behind that claim. The dataset covers 743 of the largest US employers (Fortune 500, Global 2000, and Series-C+ private companies above $1B) and the Applicant Tracking System (ATS) each one uses on its public careers portal. Workday alone screens 75.4% of those employers. The top 3 vendors together cover 93.5%. The "AI in hiring" conversation, when you look at the actual screens that touch real candidates, is mostly a conversation about three companies.
This is what those three companies — and the AI layered on top of them — actually do.
Why this matters now
The job application process is no longer a human reading a resume. With hundreds to thousands of applications coming in for a single role, no team can manually triage every candidate. AI fills that gap: parsing resumes, ranking candidates, scheduling interviews, even conducting some of them.
The pitch is reasonable. AI is faster, cheaper, and supposedly more consistent than a tired human at 4 PM on a Friday. The reality is more complicated.
Amazon's 2018 lesson (and why nobody learned from it)
75% of resumes are rejected by ATS
Build an optimized resume in minutes or check your current score — both free.
In 2018, Reuters revealed that Amazon had been building an internal AI recruiting tool meant to identify top engineering talent by ranking resumes one to five. The tool was scrapped in 2017 because it had a problem: it systematically downgraded resumes that contained the word "women's" — as in "women's chess club captain."
The cause was the training data. The model was trained on a decade of resumes Amazon had received between roughly 2008 and 2017 — a period when the tech industry's applicant pool was overwhelmingly male. The model learned that the patterns associated with hiring decisions correlated with male-coded resumes, then encoded that correlation as a hiring preference. It didn't know what "women's" meant; it just knew the word didn't appear in the resumes of people Amazon had previously hired.
Amazon dropped the project. But the underlying problem — training data that reflects who was hired in the past — applies to every supervised learning system in hiring. The lesson wasn't isolated to Amazon. It just got publicized.
How bias survives even when you try to remove it
The obvious fix is to remove sensitive fields before training: drop gender, race, age, name. If the model never sees those features, it can't discriminate on them.
It doesn't work. Models discover proxies.
Neighborhood predicts race. Attended school predicts socioeconomic status. Resume keywords ("sorority," "fraternity," "women's college") predict gender. Name predicts national origin and often religion. Years of employment gaps predict caregiving status — which predicts gender. A model trained on outcome data will pick up whichever combination of features best predicts the historical outcome, regardless of whether those features were intended as proxies.
This isn't a bug you can patch by removing more fields. The more fields you remove, the more aggressively the model leans on the remaining ones. Every meaningful feature is, in some statistical sense, correlated with every protected characteristic.
There's a related problem on the candidate side. ATS systems work by matching resume keywords against the job description. That mechanism is gameable. Candidates who understand the system can inject keywords aggressively and rank highly even when they're not actually qualified. Candidates who don't game it rank lower even when they're a better fit. Both groups are evaluated by a system that doesn't measure either skill — it measures keyword density.
Speed vs fairness — there's no free option
The most common defense of AI hiring is efficiency. A human takes weeks to review the resumes from a single popular job posting. An AI takes hours. That's real, and it matters.
The trade-off is that human reviewers come with human subjectivity — but they also come with human accountability. When a human decides not to interview a candidate, that decision is, in principle, reviewable. The human can be asked why. When an AI makes the same decision, the answer is usually some variant of "the model assigned a low score" — which isn't an answer, it's a description of the output.
Speed is a real benefit. Fairness is a real cost. There is no version of this trade-off where you get both for free.
Three things that would actually help
Mandatory third-party audits. Hiring AI should be subject to the same regulatory scrutiny we apply to credit-scoring algorithms — periodic, independent review with the audit results published. If a vendor can't demonstrate that its system passes the EEOC's four-fifths rule, it shouldn't be used to screen candidates. Human-in-the-loop for rejections. AI is reasonable for sorting and prioritizing candidates. It is not reasonable as the sole signal for a rejection. The simplest structural fix is to require a human review for any rejection AI flags — humans handle the no-decisions, AI handles the ranking. Cost goes up, but only on the marginal cases. Transparency about which system is being used. Candidates today have no way to know whether they're being screened by Workday, Greenhouse, Taleo, Lever, or a proprietary system — each of which parses resumes differently. They can't optimize what they can't see. The single highest-leverage change available to candidates today is knowing which ATS each employer uses, because optimization rules differ meaningfully across them.Hiring is just the tip
AI in hiring is concrete and visible. The same dynamics apply to systems that approve loans, set insurance premiums, recommend medical treatment, route 911 calls, predict recidivism, allocate housing assistance. In each case, training data encodes historical decisions, the model picks up the patterns, proxy variables survive cosmetic removal of sensitive features, and accountability dilutes when the decision moves from a person to a probability score.
Hiring is the easy case. It's regulated, individual, and publicly contested. If we can't make AI hiring fair under those conditions, the deeper deployments will be much harder.
What this means for your job search
Until the regulatory situation catches up to the technology — which will be years, not months — the practical advice for candidates is unromantic but real: figure out which ATS each of your target employers uses, optimize your resume specifically for that system, and don't assume the same resume works across vendors.
The full dataset of 743 employers and which ATS each one runs is at withresumeai.com/reports/state-of-ats-2026. You can scan any resume against any job description at withresumeai.com/ats-checker and see a system-specific score.
It's an unsatisfying answer to a structural problem. But it's the answer that's available right now.