TabPFN and TabICL vs. tuned XGBoost: the model that doesn't train won 14/14

TabPFN and TabICL, tabular foundation models that make predictions without any training, outperformed carefully tuned XGBoost on all 14 datasets tested from the Grinsztajn benchmark. These models use in-context learning—similar to language models—to process training rows in a single forward pass, eliminating the need for hyperparameter tuning and reversing the typical cost structure where prediction becomes more expensive than fitting. The advantage held up consistently across datasets with up to 32,000 rows, suggesting a significant shift in tabular machine learning practice.