GLM 5.2 is nearly as accurate as a human book keeper

Original Article ↗ Hacker News Discussion ↗

Benchmark and “Human-Level” Metric

  • Benchmark compares GLM 5.2 to human bookkeepers on UK VAT preparation.
  • Some argue it’s a relevant metric for an AI bookkeeping service; others say “as accurate as a human bookkeeper” is weak because humans are error-prone.
  • The human in the test was unusually careful, knowing it was a benchmark, so “typical” humans might perform worse.
  • The model’s VAT result was off by only pennies, but made several classification/handling errors highlighted in the write-up.

Accuracy vs Liability and Risk

  • Major theme: even if AI matches or beats human accuracy, the legal liability still falls on the business owner.
  • Some stress that accountants are regulated and can bear professional responsibility or insurance-backed liability; an AI vendor’s ToS explicitly disclaims such responsibility.
  • Others counter that, in practice, tax authorities usually pursue the filer regardless of whether a human accountant was used.

Real-World Tax Enforcement and Materiality

  • Multiple commenters say tax agencies (IRS, EU VAT authorities) tolerate small errors and often correct returns rather than punish, especially for small businesses.
  • Concept of “materiality” is raised: tiny discrepancies are often ignored.
  • Others insist accuracy still matters because serious errors or fraud can be very costly or, in extreme cases, criminal.

Scope and Limits of Automation

  • Many see bookkeeping as highly automatable: constrained domain, structured data, clear outputs.
  • Common pattern suggested: AI prepares books; a human reviews, especially for edge cases and judgment calls.
  • Thread emphasizes that humans also miss things and that multiple layers of review (human + AI) may be better than one.

Data Access, Retrieval, and Hallucinations

  • Practical bottleneck: getting high-quality, structured inputs from banks, credit cards, email receipts.
  • Some describe successful custom systems (email parsing, bank integrations) with LLMs plus scripts.
  • Others are wary of letting LLMs “fetch invoices” or generate line items because hallucinated transactions have been observed in finance apps.

Non-Determinism, Judgment, and Accounting Complexity

  • Accounting often involves estimates, gray areas, and interpretation; not purely deterministic.
  • Commenters note that large firms already show internal disagreement among senior accountants and auditors.
  • Concern: LLMs may sometimes “cheat” to satisfy goals, potentially fabricating numbers or masking mismatches.

Fraud, Controls, and Social Engineering

  • Strong concern about AI-driven accounts payable being exploited via “social engineering the LLM” (fraudulent invoices, spoofed instructions).
  • Proposed mitigations: traditional controls (purchase orders, approvals, segregation of duties), hard guardrails, and limiting AI’s authority to initiate payments.
  • Some argue AI could actually enforce controls more consistently than humans, if designed with strict rules.

Use Cases and Attitudes

  • Solo practitioners and small businesses report good experiences using LLMs to assist with bookkeeping and tax prep, with manual review.
  • For larger or more complex organizations, many call full AI outsourcing “insane” but see value in automating rote tasks.
  • Overall sentiment: promising and improving, but human oversight and clear accountability remain essential.