Understanding FlashAttention Pt 1: Personal Notes

FlashAttention is an IO-aware algorithm that accelerates transformer attention by reducing data movement through GPU memory hierarchy, using three key techniques: tiling, online softmax, and recomputation. Unlike approximate attention methods, FlashAttention computes the exact same mathematical function as standard scaled dot-product attention but achieves faster wall-clock speeds by dramatically reducing the number of trips to high-bandwidth memory (HBM), even if it requires recomputing some intermediate values. The algorithm's speedup comes not from reducing arithmetic operations but from moving far less data between GPU memory levels.