HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

Original Article ↗ Hacker News Discussion ↗

Scope and Purpose of the Tool

  • The open‑sourced system is not a full ATS in production use; it was built to rank tens of thousands of intern resumes so humans know what to read first.
  • Demo config uses a tiny local model (gemma3 4B, temp 0.1); actual internal use reportedly uses larger Gemini models and very low cutoffs, with only the bottom tail auto‑discarded.

LLM Non‑Determinism and Scoring Variance

  • Multiple people reproduce large score swings on the same resume (e.g., 48–99/100) across runs and models.
  • Long subthread debates whether temperature 0 or 0.1 should be deterministic, touching on softmax, floating‑point non‑determinism, batching, PRNG seeding, GPUs vs CPUs vs TPUs.
  • Consensus: you can engineer near‑deterministic inference, but most cloud setups don’t, and determinism would not fix fundamentally weak evaluation criteria.

Rubric Design and Embedded Biases

  • Scoring heavily favors open source (35 points) and personal projects (30) over work experience (25).
  • Many argue this systematically penalizes: people with families or non‑coding hobbies, those at startups whose “extra work” isn’t public, older or self‑taught engineers, and those not active on GitHub.
  • Some see it as selecting for “intrinsically motivated” or obsessive coders; others see it as selecting for overwork and privilege.

Effectiveness, Legality, and Fairness of AI Screening

  • Hiring managers describe overwhelming applicant volume and see any auto‑filter that advances ~30–35% as an improvement over human‑only triage.
  • Critics counter that a filter no better than random just discards qualified candidates earlier, and randomness could be achieved more cheaply and transparently.
  • There is concern that opaque LLM scoring may introduce unlawful discrimination (EU GDPR Article 22, US equal‑employment law), especially since inputs like names, schools, and locations correlate with protected classes, even if prompts say “don’t use them.”

Prompt / System Design Critique

  • The main prompt bundles many subtasks (OSS, project complexity, skills) into one step with vague 0–35 ranges and many adjectives (“significant,” “substantial”), pushing hard judgment onto the model.
  • Suggestions: split into smaller tasks, use additive or categorical rubrics, link scores to explicit evidence, strip sensitive fields before prompting, and rely on structured outputs.
  • Some note hallucinations (e.g., awarding Google Summer of Code points not present in the CV).

Broader Reflections on Hiring

  • Many see resume‑based and ATS‑driven hiring as already arbitrary; LLMs amplify opacity and reward buzzword stuffing and AI‑generated CVs.
  • Alternatives discussed: work‑sample tests, capped or time‑bounded applications, random lotteries among minimally qualified candidates, or screening based on written questionnaires rather than resumes.