PyTorch 2.13 brings FlexAttention to Apple Silicon with reported ~12x sparse-attention speedup
Hand-written Metal kernels for sparse attention, native safetensors loading and a memory-saving fused cross-entropy loss headline a release aimed partly at local model development on Macs.
PyTorch 2.13 shipped in the week of July 7-11, 2026 — sources cite July 8 and "around July 11" — with FlexAttention support on Apple Silicon's Metal (MPS) backend as the headline feature, alongside memory optimizations and native safetensors loading. The release contains 3,328 commits from 526 contributors, according to analyses of the release notes by ecorpit and byteiota.
Context
FlexAttention, PyTorch's programmable attention API, previously required CUDA for its best-performing paths. Local inference and fine-tuning of open-weight models on Mac hardware — a segment that grew as machines like the $599 M4 Mac mini became popular entry points for running quantized models — depended on slower fallback attention paths. Bringing compiled, hand-written attention kernels to MPS changes the economics of sparse-attention workloads on Apple hardware.
What changed
Per the release-note analyses:
- FlexAttention on Metal/MPS: hand-written kernels for sparse prefill and decode, with grouped-query attention and captured-buffer support. At 0.8% mask density on a 1×8×32768×64 shape with a 256 window, the MPS path reportedly ran in ~35ms versus ~431ms for scaled dot-product attention — a ~12.3x speedup. Dense attention still favors SDPA.
- nn.LinearCrossEntropyLoss: a fused linear-plus-cross-entropy layer that cuts peak memory by up to 4x on large-vocabulary models.
- Native safetensors support in torch.load, removing a common third-party dependency for loading open-weight checkpoints.
- FSDP2 reduce-scatter overlap and deterministic FlexAttention backward on CUDA.
- Python 3.15 Linux wheels.
The MPS FlexAttention path is explicitly labeled API Unstable, meaning interfaces may change in upcoming releases.
Why it matters
Sparse and sliding-window attention patterns are increasingly common in long-context open-weight models. A 12x speedup on those patterns — even at low mask densities and on consumer hardware — makes on-device experimentation with long-context models meaningfully cheaper, and reduces the dependence of the open-model developer community on rented CUDA capacity for routine work. The memory savings from the fused loss layer address the other common bottleneck on unified-memory Macs.
Details
The 12.3x figure comes from a specific configuration (1×8×32768×64, 256-token window, 0.8% mask density) and should not be generalized to dense attention or shorter sequences, where SDPA remains faster per the same analyses. Both secondary analyses cite the official release notes; the release notes themselves were not independently fetched for this report, and the exact release date is unresolved between July 8 and "around July 11" pending confirmation on pytorch.org.
Limitations and caveats
The performance figures are as reported in secondary analyses of the release notes, not independently benchmarked. The MPS path carries an API Unstable label, so code written against it may break in future point releases. This story rests on two trade/aggregator analyses rather than the primary pytorch.org announcement; treat the exact date and benchmark numbers as medium confidence until the official notes are consulted directly.
Sources
- ecorpit — PyTorch 2.13 FlexAttention Apple Silicon 12x speedup (2026-07-18)
- byteiota — PyTorch 2.13: FlexAttention on Apple Silicon and LLM memory (2026-07-15)
*Update note: This post was last reviewed on 2026-07-22. The exact release date (July 8 vs ~July 11) remains pending confirmation against the official pytorch.org release notes.*
Sources
Drafted with AI assistance from source briefs; reviewed for citation completeness and label accuracy.