I Trained Six Models for Fraud Detection, and the Best One Isn't in Production
NairaShield is an AI-based fraud detection system built as a final-year project that evolved from a simple classification model into a full decision-support pipeline with regulatory review workflow The system merges two heterogeneous datasets (PaySim for mobile money and IEEE-CIS for card transactions) using schema validation and feature alignment to create a unified training structure Class imbalance was addressed using SMOTE oversampling with a zero-dependency fallback implementation using Euc
Analysis
TL;DR
- NairaShield is an AI-based fraud detection system built as a final-year project that evolved from a simple classification model into a full decision-support pipeline with regulatory review workflow
- The system merges two heterogeneous datasets (PaySim for mobile money and IEEE-CIS for card transactions) using schema validation and feature alignment to create a unified training structure
- Class imbalance was addressed using SMOTE oversampling with a zero-dependency fallback implementation using Euclidean distance interpolation
- Six models were trained and compared: Random Forest, Logistic Regression, XGBoost (baseline and optimized), and LightGBM (baseline and optimized)
- The currently deployed model differs from the best-performing model in testing, reflecting a shift toward practical decision-support over pure accuracy optimization
Why It Matters
This project illustrates the critical gap between academic model selection and production deployment realities in fraud detection systems. It demonstrates how regulatory requirements and supervisory feedback can fundamentally reshape an AI system's architecture, pushing it beyond simple classification into a decision-support pipeline with compliance workflows.
Technical Details
- Dataset Merging: Combined PaySim (mobile money transactions with balance fields like oldbalanceOrg, newbalanceOrig) and IEEE-CIS (card transactions with anonymized features like ProductCD, card1, card2) using Pydantic schema validation to prevent data poisoning from malformed rows
- Feature Engineering: Created unified schema with transaction_id, channel (mapped from ProductCD using product_map dictionary), and source dataset indicator; deployed model uses only three features: transaction amount, channel one-hot encoding, and source dataset one-hot encoding
- Imbalance Handling: Implemented SMOTE with k_neighbors=min(2, minority_samples-1) on training split only, with a hand-written Euclidean distance interpolation fallback for environments without imbalanced-learn dependency
- Model Evaluation: Used consistent evaluation helper across all six models measuring Accuracy, Precision, Recall, AUC-ROC, and AUC-PR with joblib serialization for model persistence
Industry Insight
- Production fraud detection systems often require features available across all data sources rather than optimal features from individual datasets, necessitating careful feature selection based on data availability rather than pure predictive power
- Model selection for deployment should account for regulatory review workflows and decision-support requirements, not just benchmark performance metrics
- Implementing zero-dependency fallbacks for critical ML operations (like SMOTE) ensures system resilience in minimalistic or constrained deployment environments
Disclaimer: The above content is generated by AI and is for reference only.