LLM Classification Is Feature Engineering

Using LLMs as direct classifiers is problematic because they lack calibration, struggle to incorporate structured data reliably, and offer limited interpretability—issues stemming from LLMs not being designed as classifiers. The author proposes reframing LLM classification as feature engineering by using the LLM's output as input to traditional ML algorithms like logistic regression, which recovers the benefits of LLM reasoning while enabling proper calibration, threshold control, and interpretability.