Evaluating rule-based vs. machine learning approaches for fraudulent transaction detection

DOI: 10.31673/2412-9070.2025.050346

Abstract

Financial institutions nowadays rely heavily on rule engines, such as thresholds, white/black lists, velocity checks to flag suspicious transactions. Machine learning (ML) models, on the other hand, while promising higher accuracy and adaptability, are being dependent on data characteristics, class imbalance, latency constraints, and interpretability requirements. In this paper, I present a controlled evaluation of a configurable rule-based baseline and several supervised ML models (logistic regression, random forest, gradient boosting) on an imbalanced transaction dataset. I measure detection performance (ROC-AUC, PR-AUC, precision/recall at operating points), operational costs (false-positive rate, alerts per 1k transactions), and engineering trade-offs (inference latency, feature complexity, interpretability). Results show that while rules remain competitive at high-precision, low-recall regimes, ML approaches achieve substantially better recall at comparable precision, especially when coupled with calibrated thresholds and class-imbalance handling. I discuss deployment-oriented considerations and outline a hybrid strategy that combines rules for policy compliance with ML for generalization.

Keywords: fraud detection; financial risk; anomaly detection; rule engine; machine learning; class imbalance; interpretability.