JIT WireGuard

WireGuard-based VPN tools and AWS access

• Netmaker is discussed as a self-hostable WireGuard-based mesh similar to Tailscale; Netbird is also recommended but criticized for extensive data collection and risky daemon behavior (e.g., creating root SSH tunnels).
• Some ask why not use AWS-native options (VPC peering, PrivateLink, AWS VPN, SSM). Replies: those are aimed at inter-VPC connectivity, not cheap, simple SSH-style access across many accounts; AWS VPN egress can be costly for heavy data, while SSM + SSH works but needs agents/permissions and more tooling.

---

JIT WireGuard concept and implementation details

• The blog’s “just-in-time” WireGuard peers solve the problem that kernel WireGuard has no native “on-demand peer install” API.
• They intercept only initiation packets using eBPF (udp[8] = 1), then decide in userspace whether to authorize and install a peer; stale peers are later removed.
• Some suggest using NFQUEUE to queue packets to userspace, then verdict them (drop/allow) after config, enabling fail-open and scaling to multiple workers.
• Discussion on replaying the initial handshake packet after peer install; considered possible and a neat optimization, though not implemented in the article.
• The approach aims to support many potential peers without preloading them all into the kernel; peers live primarily in a database and are installed on demand. Similar techniques are used by other WireGuard-based systems to stay within resource limits.
• JIT is clarified as “just-in-time configuration,” not compilation; some note the term is widely used outside compilers (e.g., manufacturing).

---

Debate on WireGuard’s design and limitations

• One line of critique: WireGuard is tightly bound to a network interface; some wish it were a generic “filter” usable with arbitrary file handles, unlocking more flexible compositions.
• Others defend its design goal: minimal configuration surface for easier auditing and practical security, versus IPSec’s complexity.
• Complaints focus less on the protocol and more on tooling: .ini-style configs, wg vs wg-quick mismatch, and awkward client UX (especially on Windows).
• Several note that WireGuard is not a full “VPN solution”: no built-in route/DNS pushing or centralized policy distribution. Workarounds include running BGP or writing custom clients, but that’s too complex for many, especially on mobile.
• The “cryptokey routing” model is seen as fundamental; higher-level products (Tailscale, Netmaker, etc.) exist precisely to add management, policy, and route distribution on top.

---

Browser, userspace, and alternative networking stacks

• Running WireGuard in browsers is constrained by lack of raw sockets; only HTTP/WebSocket-based paths work. Some projects use relays (e.g., WebSocket over specialized relay servers) and WebAssembly to approximate VPN behavior.
• OpenZiti/BrowZer is presented as an example: a service-worker + WebAssembly model that lets browsers join a private overlay network via WebSockets, after a bootstrapped script injection.
• Fly’s flyctl uses a userspace TCP/IP stack plus userspace WireGuard; the host OS is not the WireGuard peer, only the CLI process is. A separate “real” peer mode is also available for OS-level access.
• Some find userspace TCP/IP stacks conceptually odd, but others note they were common historically and are increasingly practical.

---

Control plane messaging (NATS vs HTTP)

• Commenters are surprised that NATS-based coordination had enough message loss to hurt reliability.
• It’s noted that “core” NATS does not provide durability/redelivery; JetStream adds such features but may not have been configured or available when designed.
• Fly staff say they likely “held NATS wrong” and realized a message broker added complexity without extra expressiveness for their simple, single-message use cases; a direct HTTP request model is easier to reason about and monitor.

---

Apple/macOS VPN restrictions

• There’s a claim that the official WireGuard macOS GUI must ship via the App Store because Apple withholds necessary VPN entitlements from web/self-distributed apps; this is framed as privacy-hostile, since App Store use ties the software to user identity and hardware.
• Others dispute this, pointing out non-App Store VPN clients that work fine, implying the original statement is overstated.
• Clarification: newer Network Extension/System Extension APIs require specific entitlements, which Apple now grants to Developer Program members, so building WireGuard’s exact app from source as a non–program member won’t fully work without disabling security features. Older or different VPN APIs remain usable for non–App Store apps.

---

Fly.io platform experience

• Several users praise Fly.io’s developer ergonomics: quick global deployments, very fast spin-up for workloads, Anycast networking, and features like FLY_REPLAY and LiteFS that make global clustering and low-latency access easier than on traditional VPS providers.
• Others report serious reliability issues: multi-day outages for critical services, flaky “db”/Postgres offering that’s explicitly “not managed,” and API/control-plane outages blocking deployments. This leads some to migrate to GCP (Cloud Run or GKE) for fewer surprises and better documentation.
• A few continue using Fly.io despite rough edges, hoping the platform matures into a stable, Anycast-centric alternative.

---

Transport choices and connectivity

• For flyctl, WireGuard is often tunneled over WebSockets by default to survive restrictive networks where UDP (even on 443) is blocked or unreliable; pure UDP WireGuard is supported as an opt-in.
• This default is chosen to minimize hard-to-debug UDP failures; doing “happy eyeballs” probes to choose between UDP and WebSocket is considered unnecessary complexity given Fly’s focus (cloud platform, not general VPN service).
• Discussion confirms the JIT handshake trick works through NAT as long as the 4-tuple (source/dest IP+port) is consistent, since typical UDP NATs don’t inspect packet contents.

---

Miscellaneous

• Some confusion about Fly’s explanation of kernel WireGuard not supporting “on-demand” peers is resolved: peers can be added at runtime, but the JIT scheme requires authenticating and deciding before adding, to avoid storing large numbers of never-used peers.
• Various userland WireGuard/Noise projects (e.g., Go libraries) are mentioned, with ideas for application-level tunnels, magic-wormhole-like file transfer, and service-mesh-style overlays.
• Deploying Dockerized apps on Fly is described as straightforward: run flyctl launch in the directory with the Dockerfile.