Dayjob: Tax – evaluating AI agents on 50 professional tax assignments

Original Article ↗ Hacker News Discussion ↗

AI models are increasingly being used to prepare or review complex tax returns, with some users reporting that tools like Claude or GPT caught expensive mistakes made by human accountants. Commenters weigh this potential against serious concerns: taxpayers remain legally liable even when professionals or AI err, sharing financial data with LLM providers raises privacy and regulatory issues, and current models still struggle with edge cases, math, and cross‑border tax rules. Several contributors argue that the real bottleneck is the complexity of systems like the U.S. tax code and call for more deterministic, open tax software rather than fully autonomous AI agents.

Human Accountants vs AI Agents

  • Multiple commenters report significant errors by tax accountants (misfilings costing $20k–$50k, incorrect loss carryforwards, wrong amortization periods, mishandled K‑1s, deductions, etc.).
  • Several say running past returns through Claude/GPT revealed mistakes, leading to amended returns and refunds.
  • Others, including experienced CPAs, push back: many “failures” are underreported income or misunderstanding audits, not necessarily preparer incompetence or fraud.
  • Emerging pattern: AI is often used as a second reviewer to catch human mistakes, not as a sole preparer.

Liability, Regulation, and Fairness

  • Consensus: in many jurisdictions (esp. US), the taxpayer is ultimately liable, even if a professional prepares the return.
  • You may sue or recover penalties/interest from a negligent preparer, but that’s costly and uncertain.
  • Some see this as unjust given tax code complexity; call for simplification or automatic filing.
  • A bill in the US would regulate paid preparers; many think it doesn’t go far enough.

Using LLMs for Taxes: Promise and Limits

  • Users report LLMs:
    • Spotting exchange-rate errors, misapplied real-estate losses, missed offsets, and structural issues in returns.
    • Handling complex cross‑border situations as research/checking tools, but not reliably computing exact liabilities.
    • Preparing spreadsheets, filling forms (even dozens of pages), and walking users through portals and procedures.
  • Limitations:
    • Poor raw math and numeric reasoning; need for deterministic calculators or tool calls.
    • Weak on edge‑case treaty issues and some withholding/deduction details.
    • Not ready to fully automate complex returns without human review.

Privacy and Data Concerns

  • Strong discomfort about sending full financials to cloud LLMs; fears of leaks, training use, government access, or resale.
  • Others argue the time savings justify the risk, especially for small businesses drowning in admin.
  • Suggested mitigations: local inference, zero‑retention providers, building local tools that only use models for code generation or structural checking.

Benchmarks, Formalization, and Tooling

  • Some question the benchmark’s value without strong scaffolding; argue real systems would use tax‑specific harnesses and tools.
  • Others highlight projects formalizing IRS forms and decision trees (using Lean, YAML, etc.) and note IRS’s own open‑sourced Direct File artifacts.
  • General agreement that much of tax prep should be a deterministic rules engine, with AI assisting interpretation and data wrangling.