A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis

Original Article ↗ Hacker News Discussion ↗

A small, decade‑old convolutional neural network fine‑tuned on dermatology images was shown to outperform state‑of‑the‑art large language models at classifying skin lesions, prompting debate about what kinds of problems LLMs are actually suited to solve. Commenters largely argue this result is unsurprising: a narrow, purpose‑built vision model with domain‑specific training should beat a general‑purpose language model on image diagnosis, especially in high‑stakes medical contexts. Many see the real value of LLMs not as end‑to‑end diagnosticians, but as tools for building specialized models, orchestrating other systems, and handling higher‑level reasoning around data those models produce.

Specialized CNN vs General-Purpose LLMs

  • Most posters say it’s unsurprising a dermatology-tuned CNN beats frontier LLMs on skin-cancer classification.
  • Analogy: comparing a veteran specialist to a bright child; LLMs are broad but not optimized for this narrow task.
  • CNNs (and other classic models like xgboost) are still preferred for focused classification problems when labeled data is available.

Fairness and Quality of the Comparison

  • The CNN is called “10 years old” architecturally but was recently fine-tuned on a large in-domain ISIC dataset; some argue the title is misleading.
  • Critics say the CNN has a “home field advantage” because test images come from the same distribution as its training data.
  • Others respond this is standard practice, not “benchmaxxing”; overfitting should be evaluated by testing on new datasets, not assumed.

Limits of LLMs and Appropriate Use

  • Many emphasize LLMs are language-first (or general vision-language) models, not purpose-built medical classifiers.
  • Several commenters push back against “LLMs can do everything” hype; this result is seen as a necessary reality check.
  • Some note that frontier VLMs did better at avoiding false negatives (dangerous misses) when thresholds are tuned, suggesting safety trade-offs rather than simple “win/lose.”

LLMs as Orchestrators and Tool-Builders

  • Strong theme: LLMs shine as high-level controllers or coding assistants, not as the final classifier.
  • Suggested workflows: use LLMs to:
    • Design/train CNNs or other specialized models.
    • Generate labeling, features, or harness code.
    • Call out to more efficient vision models or tools rather than directly classifying images.

Decision Models and Zero-Shot Classification

  • Jev-style decision models and lightweight classifiers are discussed as cheaper, faster alternatives to full LLM calls for many tasks.
  • Consensus: for low-volume or hard-to-label problems, LLM/decision models are attractive; at scale or in safety-critical domains, supervised task-specific models are preferable.

Medical and Clinical Context

  • Several note that real dermatologic decisions involve history, notes, and follow-up, not a single image.
  • Some worry about over-reliance on ChatGPT-style tools for diagnosis; others argue even imperfect AI could have prompted earlier detection in missed cases, but procedure risks and over-treatment are debated.