GPT-4 Turbo with Vision is a step backwards for coding

Original Article ↗ Hacker News Discussion ↗

OpenAI’s new GPT-4 Turbo with Vision model is reported to perform worse at coding than earlier GPT-4 variants, with many users noting a tendency toward “lazy” outputs that omit full implementations even when explicitly requested. Commenters speculate this may be driven by cost-saving or safety constraints and point to broader issues like inconsistent behavior, hallucinated knowledge cutoffs, and limited control over model versions in consumer products. Some developers are switching to alternatives such as Claude or older GPT-4 models via the API, emphasizing that each model now has a distinct “personality” and must be evaluated for specific coding workflows.

Perceived regression and “lazy coding”

  • Many report GPT‑4 Turbo with Vision is worse for coding than earlier GPT‑4 models, especially on larger edits.
  • Common behavior: omitting method bodies, inserting TODOs or comments instead of full implementations, even when explicitly asked not to.
  • Some see similar regressions in base GPT‑4 and GPT‑3.5 recently; others say the older “base” GPT‑4 in ChatGPT Classic still feels better.

Speculated causes

  • Hypotheses include: cost control (shorter answers to save GPU time), performance/latency, or copyright concerns limiting long continuous code blocks.
  • Some users see “laziness” as useful: less boilerplate, more high‑level guidance, fewer giant dumps of code they won’t paste directly.

Model comparisons and alternatives

  • Several users shift coding work to Claude 3 (especially Opus), citing:
    • More willingness to output long, complete code.
    • Less verbosity in explanations.
  • Downsides mentioned: stricter content filters, occasional hallucinations (non‑existent packages/APIs), weaker follow‑up edits, account bans, regional access limits, and a weaker consumer UI.
  • Other options mentioned: GPT‑4‑32k for large code output, smaller/cheaper models or open‑source models for some tasks.

Benchmarks and reliability

  • Thread notes conflicting evals:
    • Aider’s benchmarks show GPT‑4 Turbo with Vision underperforming and more “lazy.”
    • LiveCodeBench leaderboard has it at the top for certain coding tasks.
  • Some argue LLM benchmarking is fragile: narrow task sets, prompt sensitivity, potential test overfitting.

Prompting and workflow adaptations

  • Techniques suggested:
    • Using unified diffs and multi‑pass refactoring to fight laziness.
    • Strong system prompts requesting concise or fully detailed answers.
    • Backtracking, asking for multi‑section critical analyses to break out of shallow replies.
    • Taking multiple samples and choosing the least “lazy.”

API, UX, and technical issues

  • Desire for explicit model version selection in ChatGPT, mirroring API flexibility.
  • Reports that gpt‑4‑turbo‑2024‑04‑09 often “thinks” its cutoff is 2021 unless reminded of its model ID in the system prompt; described as a grounding/anchoring bug.
  • Output token limits (~4k) are a major constraint for large code generation; users chain calls or prefer 32k‑context models.
  • Some rely on third‑party UIs (VS Code extensions, custom chat clients, big‑AGI, etc.) rather than the default web chat.

Safety, moderation, and support

  • Anthropic/Claude criticized for aggressive filtering (e.g., knife image translation leading to account suspension) and opaque, slow appeals.
  • Concern that over‑zealous bans and minimal customer support are risky as these tools become infrastructure.

Broader reflections

  • Debate over whether “laziness” is regression or a feature depends on use case (boilerplate vs full code).
  • Skepticism that current LLMs represent genuine “understanding”; described more as powerful, shallow statistics engines that are transformative yet far from AGI.