Show HN: Attaching to a virtual GPU over TCP
A new cloud service lets developers attach to a remote NVIDIA GPU over TCP so that their local or cloud VM appears to have a directly connected card, with only GPU API calls and data forwarded over the network. Commenters probe how this works technically (user-space API interception via LD_PRELOAD), question performance and bandwidth limits for training versus inference, and note that latency makes it unsuitable for gaming or highly interactive graphics. Many see value in pay‑as‑you‑go and GPU sharing for development and batch workloads, but express strong interest in a self‑hosted version and clearer pricing compared to existing tools like Juice, rCUDA, or traditional GPU instances.
How it works
- Client runs code on a CPU-only Linux machine; a userspace library intercepts CUDA / GPU API calls.
- Intercepted calls and GPU commands are forwarded over TCP to a remote GPU, making the local system believe a GPU is directly attached.
- No kernel drivers or eBPF are involved; it’s LD_PRELOAD-style API remoting, not PCIe tunneling.
- Only GPU-related state lives remotely; files, packages, and CPU execution stay on the client instance.
Performance, latency, and bottlenecks
- Network adds latency, especially for GPU VRAM ↔ RAM transfers and training workloads.
- Provider claims inference performance (e.g., BERT) is close to local, with training slower but being optimized.
- Concerns that datasets larger than VRAM would cause heavy I/O each epoch; creators acknowledge and say they’ve added optimizations.
- Some say 10–30 MB/s per GPU is typical for their workloads, so 10 Gbps links may suffice.
- Gaming and highly interactive graphics are considered impractical due to latency.
Use cases discussed
- On-demand GPUs for development without paying for GPU time when only doing CPU work.
- Transparent acceleration for GUI and desktop apps (Matlab, CAD, Blender render farms).
- ML inference, some training, video transcoding, and potentially hash cracking.
- Potential for finer-grained GPU sharing or fractional GPUs.
Comparisons and related tools
- Compared to AWS GPU instances, Nitro, TPUs, Ray, Juice Labs, rcuda, qCUDA, virtio-gpu, and other GPU-over-network systems.
- Some see it as a more generic or more transparent alternative to cluster frameworks.
Pricing, access, and hosting
- Currently beta and free (T4 GPUs); future pay-as-you-go model planned.
- A100/H100 not yet generally available; pricing “to be determined.”
- No self-hosting yet, but strongly requested and considered for the future.
Limitations and open questions
- Linux-only for now; no Vulkan/OpenGL or DirectX; partial library support (tested with PyTorch/HF; issues with some others).
- Behavior and cleanup on GPU reset, VRAM residue, and network instability are raised but not fully resolved.
- Some skepticism about real-world value for serious training workloads and concerns about subscription-style “cloud everything” models.