THE ESSENTIALS IN 30 SECONDS

  • Prime Mega MoE extends DeepSeek's forward-only Mega MoE kernel from the DeepGEMM collection with a backward pass that overlaps communication and computation.
  • Prime Intellect reports up to 3.11x faster MoE kernels than the default PyTorch implementation and up to 1.34x faster end-to-end training on 8xH200 nodes.
  • The figures are 'up to' values for specific Qwen3 MoE configurations, and the post does not state a licence for the new code.

The AI lab Prime Intellect published a new GPU kernel on October 29 aimed at one of the most expensive parts of modern language models: mixture-of-experts (MoE) layers, in which each token is routed to a handful of specialised sub-networks.

What was released

The work builds on Mega MoE, a fused kernel that DeepSeek released as part of its open-source DeepGEMM collection. A fused kernel performs several steps in a single GPU invocation; here it routes tokens to experts, runs the expert computation and combines the results without round trips between the GPU and the CPU. DeepSeek's version only implemented the forward pass, which is enough for inference but not for training.

Prime Intellect says it has added a backward pass that overlaps communication between GPUs with computation, so the kernel can now be used to train models. It supports bfloat16, MXFP8 on Nvidia's Blackwell GPUs and blockwise FP8 on Hopper GPUs.

The reported gains

According to the company's benchmarks, run on nodes of eight H200 GPUs, the MoE kernels are up to 3.11 times faster than the default PyTorch-based implementation. In full training runs within its Prime-RL framework, end-to-end throughput rose by up to 34%. The tests used Qwen3-30B-A3B on four nodes and Qwen3-235B-A22B on 16 nodes, in FP8 with sequences of 16,384 tokens. A comparison with TorchTitan appears only as a chart, without figures in the text.

Limits

These are "up to" results from Prime Intellect's own setups, and gains will vary with model shape, routing and hardware. The two FP8 formats each work on only one GPU generation. The post links to DeepSeek's DeepGEMM repository but does not name a licence or point to a separate repository for Prime's backward kernel, so developers will need to check how the code is distributed before adopting it.

Why it matters

Large open models increasingly rely on MoE designs to keep inference costs down, but training them efficiently across many GPUs remains difficult. Tools that let smaller labs train such models faster, building on components published by others, are part of how the open-model ecosystem keeps pace with larger companies.