DeepSeek V4 Flash on a Single AMD MI300X

Hardware availability and form factors

• Single MI300X units are hard/expensive to obtain; commonly sold in 8‑GPU boxes around ~€250k.
• Some claim you can find individual OAM modules on secondary markets, but without backplanes/adapters they’re impractical.
• MI350P is highlighted as a more accessible PCIe option (144 GB HBM), but still server‑oriented: passive cooling, high power, needs serious airflow or custom cooling.
• Discussion notes the thermal reality of 600W+ accelerators: they behave like space heaters and can significantly heat a room.

AI bubble, demand, and financing

• Debate over whether current AI investment is a bubble.
• One side: trillions in AI capex funded by debt (~$3T cited from an article), with interest possibly exceeding current AI revenue → unsustainable unless AI replaces large amounts of labor or leads to radical societal change.
• Counterpoints: demand is “real and not going away”; prior “bubble pop soon” predictions were wrong; revenues and usage reportedly skyrocketed.
• Some distinguish between real demand and demand dependent on subsidized pricing; concern that demand might drop if cheap capital dries up.

Inference economics and DeepSeek pricing

• MI300X cloud rental at ~$2/hour is discussed as possibly the cheapest way to run DeepSeek V4 Flash at original weights, but detailed math suggests margins are thin or negative versus just using the API.
• Token‑cost modeling shows that cached tokens are extremely cheap to serve if implemented well; DeepSeek’s low prices are partly justified by heavy cache reuse in agentic workloads.
• There is disagreement on whether frontier labs are making significant profit: some argue API inference is profitable with positive gross margins; others doubt it covers full R&D and training costs.
• DeepSeek is assumed to operate near cost and to monetize user data for future training.

Self‑hosting vs API use

• Several argue that given electricity, depreciation, and throughput, self‑hosting is rarely cheaper than API access for individuals.
• Others emphasize non‑cost reasons: privacy, regulatory constraints, IP protection, custom decoding/attention steering, and more stable latency and quality compared to heavily batched commercial endpoints.
• For teams, a single MI300X or small cluster can serve multiple coding agents and batch workloads with strong privacy guarantees.

Performance and optimization

• Reported MI300X throughput for DeepSeek V4 Flash (150–800+ tok/s) is seen as good but still below throughput reported for H800 in DeepSeek’s own DSpark paper (15k tok/s/GPU).
• It’s noted that multi‑GPU, well‑networked setups with wide parallelism achieve superlinear throughput scaling, so single‑GPU numbers aren’t directly comparable.
• Some point out that NVIDIA‑based stacks (e.g., using sglang/dynamo) can achieve much higher token rates per GPU; AMD stacks still have optimization headroom.

Model size, MoE, and context window

• DeepSeek V4 Flash is ~284B total params (MoE) with MXFP4 quantization; other Chinese frontier models are listed in the 1.6T–2.8T range.
• Rumors about US frontier model sizes range from 5T to 100T, with posters agreeing the true numbers are unclear.
• Serving V4 Flash at full spec (1M context, multiple users) appears to exceed what a single 144 GB card can handle; some suggest at least two MI350Ps.
• A 256k context tradeoff on MI300X is viewed as practical; some users report that 1M context is overkill and that quality loss at 256k is modest.

Software stack, data, and prior art

• ROCm is confirmed usable for frontier inference and reportedly used by major labs; some express interest in bypassing higher‑level stacks and talking directly to hardware command buffers.
• Data from user interactions (especially agent traces, codebases, user sentiment) is considered highly valuable for training and RL fine‑tuning, even if noisy.
• Prior work like DwarfStar is mentioned as achieving similar models in less memory, but not yet tuned for MI300X, which explains its absence from the repo’s “prior art” section.