JIT WireGuard

Original Article ↗ Hacker News Discussion ↗

WireGuard-based VPN tools and AWS access

  • Netmaker is discussed as a self-hostable WireGuard-based mesh similar to Tailscale; Netbird is also recommended but criticized for extensive data collection and risky daemon behavior (e.g., creating root SSH tunnels).
  • Some ask why not use AWS-native options (VPC peering, PrivateLink, AWS VPN, SSM). Replies: those are aimed at inter-VPC connectivity, not cheap, simple SSH-style access across many accounts; AWS VPN egress can be costly for heavy data, while SSM + SSH works but needs agents/permissions and more tooling.

JIT WireGuard concept and implementation details

  • The blog’s “just-in-time” WireGuard peers solve the problem that kernel WireGuard has no native “on-demand peer install” API.
  • They intercept only initiation packets using eBPF (udp[8] = 1), then decide in userspace whether to authorize and install a peer; stale peers are later removed.
  • Some suggest using NFQUEUE to queue packets to userspace, then verdict them (drop/allow) after config, enabling fail-open and scaling to multiple workers.
  • Discussion on replaying the initial handshake packet after peer install; considered possible and a neat optimization, though not implemented in the article.
  • The approach aims to support many potential peers without preloading them all into the kernel; peers live primarily in a database and are installed on demand. Similar techniques are used by other WireGuard-based systems to stay within resource limits.
  • JIT is clarified as “just-in-time configuration,” not compilation; some note the term is widely used outside compilers (e.g., manufacturing).

Debate on WireGuard’s design and limitations

  • One line of critique: WireGuard is tightly bound to a network interface; some wish it were a generic “filter” usable with arbitrary file handles, unlocking more flexible compositions.
  • Others defend its design goal: minimal configuration surface for easier auditing and practical security, versus IPSec’s complexity.
  • Complaints focus less on the protocol and more on tooling: .ini-style configs, wg vs wg-quick mismatch, and awkward client UX (especially on Windows).
  • Several note that WireGuard is not a full “VPN solution”: no built-in route/DNS pushing or centralized policy distribution. Workarounds include running BGP or writing custom clients, but that’s too complex for many, especially on mobile.
  • The “cryptokey routing” model is seen as fundamental; higher-level products (Tailscale, Netmaker, etc.) exist precisely to add management, policy, and route distribution on top.

Browser, userspace, and alternative networking stacks

  • Running WireGuard in browsers is constrained by lack of raw sockets; only HTTP/WebSocket-based paths work. Some projects use relays (e.g., WebSocket over specialized relay servers) and WebAssembly to approximate VPN behavior.
  • OpenZiti/BrowZer is presented as an example: a service-worker + WebAssembly model that lets browsers join a private overlay network via WebSockets, after a bootstrapped script injection.
  • Fly’s flyctl uses a userspace TCP/IP stack plus userspace WireGuard; the host OS is not the WireGuard peer, only the CLI process is. A separate “real” peer mode is also available for OS-level access.
  • Some find userspace TCP/IP stacks conceptually odd, but others note they were common historically and are increasingly practical.

Control plane messaging (NATS vs HTTP)

  • Commenters are surprised that NATS-based coordination had enough message loss to hurt reliability.
  • It’s noted that “core” NATS does not provide durability/redelivery; JetStream adds such features but may not have been configured or available when designed.
  • Fly staff say they likely “held NATS wrong” and realized a message broker added complexity without extra expressiveness for their simple, single-message use cases; a direct HTTP request model is easier to reason about and monitor.

Apple/macOS VPN restrictions

  • There’s a claim that the official WireGuard macOS GUI must ship via the App Store because Apple withholds necessary VPN entitlements from web/self-distributed apps; this is framed as privacy-hostile, since App Store use ties the software to user identity and hardware.
  • Others dispute this, pointing out non-App Store VPN clients that work fine, implying the original statement is overstated.
  • Clarification: newer Network Extension/System Extension APIs require specific entitlements, which Apple now grants to Developer Program members, so building WireGuard’s exact app from source as a non–program member won’t fully work without disabling security features. Older or different VPN APIs remain usable for non–App Store apps.

Fly.io platform experience

  • Several users praise Fly.io’s developer ergonomics: quick global deployments, very fast spin-up for workloads, Anycast networking, and features like FLY_REPLAY and LiteFS that make global clustering and low-latency access easier than on traditional VPS providers.
  • Others report serious reliability issues: multi-day outages for critical services, flaky “db”/Postgres offering that’s explicitly “not managed,” and API/control-plane outages blocking deployments. This leads some to migrate to GCP (Cloud Run or GKE) for fewer surprises and better documentation.
  • A few continue using Fly.io despite rough edges, hoping the platform matures into a stable, Anycast-centric alternative.

Transport choices and connectivity

  • For flyctl, WireGuard is often tunneled over WebSockets by default to survive restrictive networks where UDP (even on 443) is blocked or unreliable; pure UDP WireGuard is supported as an opt-in.
  • This default is chosen to minimize hard-to-debug UDP failures; doing “happy eyeballs” probes to choose between UDP and WebSocket is considered unnecessary complexity given Fly’s focus (cloud platform, not general VPN service).
  • Discussion confirms the JIT handshake trick works through NAT as long as the 4-tuple (source/dest IP+port) is consistent, since typical UDP NATs don’t inspect packet contents.

Miscellaneous

  • Some confusion about Fly’s explanation of kernel WireGuard not supporting “on-demand” peers is resolved: peers can be added at runtime, but the JIT scheme requires authenticating and deciding before adding, to avoid storing large numbers of never-used peers.
  • Various userland WireGuard/Noise projects (e.g., Go libraries) are mentioned, with ideas for application-level tunnels, magic-wormhole-like file transfer, and service-mesh-style overlays.
  • Deploying Dockerized apps on Fly is described as straightforward: run flyctl launch in the directory with the Dockerfile.