Fly.io has GPUs now

Original Article ↗ Hacker News Discussion ↗

Fly.io has introduced GPU-backed virtual machines with scale-to-zero billing, aiming to make on-demand AI workloads easier to run alongside existing Fly-hosted apps. Commenters probe whether the pricing and cold-start costs (model loading, large images, volumes) are competitive with alternatives like DigitalOcean, Runpod, and Vast.ai, and debate how useful “edge” GPU inference really is for typical LLM and image workloads. A recurring concern is Fly.io’s reliability and support maturity for production use, though some users report smooth experiences and note complementary features like emerging S3-compatible storage and flexible VM lifecycle control.

Pricing and Competition

  • Many see Fly’s GPU prices as high compared to some competitors (DigitalOcean, AWS, various “race-to-zero” GPU startups), especially without long-term commitments.
  • Others note that on-demand GPU supply is tight everywhere; “cheaper” headline prices often require multi‑year commitments or are unavailable in practice.
  • Some compare costs favorably to platforms like Modal and Replicate, welcoming more competition in hosted inference.

Performance, Cold Starts, and Model Loading

  • Spin-up time is driven less by VM boot and more by GPU images and model weights.
  • Large base images (1–3+ GB) and downloading model files can add 30–120 seconds; loading multi‑GB models into VRAM is a major factor.
  • Weights stored on local NVMe volumes can be reused, reducing repeated downloads; remote/network-style storage for models is considered a bad idea by several commenters.

Scaling to Zero and “Keep Warm” Behavior

  • Billing starts when a machine boots and ends when it stops, with no enforced minimum.
  • Machines scale down by exiting with code 0 under the right restart policy; “keep warm” is implemented in user code via delayed exit or custom logic.
  • Runtime options like kill signals and timeouts give some control over shutdown behavior.

Target Use Cases and Market

  • Intended users include existing Fly apps needing GPUs, people building hosting/AI platforms, and workloads that benefit from occasional GPU bursts and scale-to-zero economics.
  • Some question the size of the “needs GPU but also needs scale-to-zero” market and whether edge GPU inference meaningfully differs from standard datacenter inference.

Infrastructure and Virtualization

  • GPU VMs use Cloud Hypervisor (not Firecracker) with PCI passthrough, not vGPU.
  • Operationally, Cloud Hypervisor and Firecracker are described as similar from Fly’s perspective.

Reliability and Support Concerns

  • Thread contains sharp disagreement: some report smooth multi‑month or year‑long use; others describe Fly as “not production ready” with outages, flaky deploys, non‑spinning machines, and weak or forum-only support.
  • Fly’s own messaging emphasizes that their Postgres offering is not fully managed; some users were surprised and consider that a drawback.

Storage / S3 Replacement

  • Lack of a first‑class S3-compatible service was a blocker for some.
  • Multiple comments point to an in‑beta, Fly-integrated S3 replacement (Tigris / regional object store).
  • Licensing of suggested AGPL-based S3 alternatives is contentious due to corporate policies against AGPL.