Advancements in machine learning for machine learning

Original Article ↗ Hacker News Discussion ↗

Machine learning is increasingly being used to optimize itself, from compilers that auto-tune tensor graphs across hardware backends to systems like FunSearch that use models to search for better algorithms. Commenters debate how transformative these advances will be: some see them as the next phase of industrialization that will reshape programming jobs, drug and materials discovery, and software tooling, while others argue current models like GPT‑4 and Gemini remain expensive, brittle, and constrained by human-defined goals and real-world bottlenecks such as clinical trials. There is broad agreement that AI-driven optimization will significantly boost throughput and accessibility, but skepticism that it will quickly replace expert humans or that transformer-based LLMs can scale indefinitely without new architectures.

Future of Programming and Jobs

  • Many expect programming practice and most software jobs to change drastically within 2–5 years, with “AI jobs” becoming the norm and salaries declining.
  • Others are skeptical, citing legacy systems, slow-moving industries, and viewing ChatGPT as a better search engine and Copilot as advanced autocomplete, not paradigm shifts.
  • Debate over whether AI will replace programmers vs. merely augment them. Some note that non-programmers using AI to code is already analogous to non-artists using image models.
  • A recurring concern: people may still enjoy programming, but far fewer will be paid for it, similar to other crafts displaced by technology.

AI Capabilities, Singularity, and Moving Goalposts

  • Some see work like FunSearch and ML-for-ML as signs that a “singularity” or AGI-like phase may be closer than expected.
  • Others argue that AI still fails at high-level decision-making, framing problems from vague requirements, and coping with messy human organizations.
  • Several comments note that milestones once treated as defining “AI” (e.g., strong conversational ability / Turing Test) are now discounted, and complain about “moving goalposts,” while others argue that redefining goals is natural as capabilities improve.

Real-World Impact: Science, Medicine, and Materials

  • Discussion on whether AI must fully simulate humans to design safe drugs; consensus in-thread is that it will more likely propose candidates, with humans and trials filtering them.
  • Some argue AI already significantly accelerates parts of pharma and materials R&D (e.g., protein folding, crystal structures), even if clinical trials remain the bottleneck.
  • Others remain unimpressed until AI visibly “cures” hard diseases or reduces suffering in clear ways.

Model Competition: GPT‑4, Gemini, and Data

  • GPT‑4 is perceived as consistently ahead of other large models (Llama, Claude, Gemini, etc.).
  • Explanations proposed: first-mover advantage, better data and data cleaning, heavy use of human annotation/RLHF, and possibly hyperparameter “lottery.”
  • Gemini Pro is seen as closer to GPT‑3.5; Gemini Ultra (claimed GPT‑4‑class) was not yet available in this discussion.
  • Some think transformer LLMs may be near architectural peak, with future gains mostly incremental and focused on efficiency.

ML Compilers and Performance Prediction

  • “ML compilers” (e.g., torch.compile, XLA, JAX, IREE, OpenXLA) aim to optimize computation graphs across hardware (CUDA, AMD, TPUs, CPUs) via intermediate representations.
  • Near-term promise: automatic speedups (e.g., ~20–50% in PyTorch 2.0 benchmarks) and portability without custom vendor support.
  • A detailed summary notes Google’s work on predicting run-time performance of large tensor graphs on TPUs using graph neural networks and large datasets of XLA graphs; currently focused on prediction, not automatic graph rewriting.
  • Skeptics argue ML compilers are overhyped: expert humans still beat them on core kernels, compilers can generate “efficient garbage,” and they don’t yet feed back higher-level algorithmic insights.
  • Supporters counter that this is the same trade-off as traditional compilers—massive throughput and good-enough performance—and expect ML-driven optimizers (analogous to neural-evaluated chess engines) to grow in importance.

Access, Education, and Society

  • Some foresee AI dramatically lowering the barrier to programming, with more non-specialists building software and schools eventually teaching “how to use models” rather than traditional coding alone.
  • Others frame AI as another transformative “gold rush” like web, mobile, and cloud—overhyped for many firms, but structurally important and likely permanent.