GPT-4 Turbo with Vision is a step backwards for coding

Original Article ↗ Hacker News Discussion ↗

Perceived regression and “lazy coding”

  • Many report GPT‑4 Turbo with Vision is worse for coding than earlier GPT‑4 models, especially on larger edits.
  • Common behavior: omitting method bodies, inserting TODOs or comments instead of full implementations, even when explicitly asked not to.
  • Some see similar regressions in base GPT‑4 and GPT‑3.5 recently; others say the older “base” GPT‑4 in ChatGPT Classic still feels better.

Speculated causes

  • Hypotheses include: cost control (shorter answers to save GPU time), performance/latency, or copyright concerns limiting long continuous code blocks.
  • Some users see “laziness” as useful: less boilerplate, more high‑level guidance, fewer giant dumps of code they won’t paste directly.

Model comparisons and alternatives

  • Several users shift coding work to Claude 3 (especially Opus), citing:
    • More willingness to output long, complete code.
    • Less verbosity in explanations.
  • Downsides mentioned: stricter content filters, occasional hallucinations (non‑existent packages/APIs), weaker follow‑up edits, account bans, regional access limits, and a weaker consumer UI.
  • Other options mentioned: GPT‑4‑32k for large code output, smaller/cheaper models or open‑source models for some tasks.

Benchmarks and reliability

  • Thread notes conflicting evals:
    • Aider’s benchmarks show GPT‑4 Turbo with Vision underperforming and more “lazy.”
    • LiveCodeBench leaderboard has it at the top for certain coding tasks.
  • Some argue LLM benchmarking is fragile: narrow task sets, prompt sensitivity, potential test overfitting.

Prompting and workflow adaptations

  • Techniques suggested:
    • Using unified diffs and multi‑pass refactoring to fight laziness.
    • Strong system prompts requesting concise or fully detailed answers.
    • Backtracking, asking for multi‑section critical analyses to break out of shallow replies.
    • Taking multiple samples and choosing the least “lazy.”

API, UX, and technical issues

  • Desire for explicit model version selection in ChatGPT, mirroring API flexibility.
  • Reports that gpt‑4‑turbo‑2024‑04‑09 often “thinks” its cutoff is 2021 unless reminded of its model ID in the system prompt; described as a grounding/anchoring bug.
  • Output token limits (~4k) are a major constraint for large code generation; users chain calls or prefer 32k‑context models.
  • Some rely on third‑party UIs (VS Code extensions, custom chat clients, big‑AGI, etc.) rather than the default web chat.

Safety, moderation, and support

  • Anthropic/Claude criticized for aggressive filtering (e.g., knife image translation leading to account suspension) and opaque, slow appeals.
  • Concern that over‑zealous bans and minimal customer support are risky as these tools become infrastructure.

Broader reflections

  • Debate over whether “laziness” is regression or a feature depends on use case (boilerplate vs full code).
  • Skepticism that current LLMs represent genuine “understanding”; described more as powerful, shallow statistics engines that are transformative yet far from AGI.