Engineering Journey: Fine-Tuning LLMs from Laptop to Production
The author built a fully reproducible ML fine-tuning pipeline that evolved from local MLX on Apple Silicon to distributed training on AWS SageMaker, maintaining consistent experiment lineage throughout A git+DVC fingerprinting system (combining git SHA and dvc.lock MD5) became the single source of truth for reproducibility, tagged into MLflow and SageMaker across all three pipeline iterations Moving from MLX local training to SageMaker V2 Pipelines reduced training time from 14 hours to 3.3 hour
Analysis
TL;DR
- The author built a fully reproducible ML fine-tuning pipeline that evolved from local MLX on Apple Silicon to distributed training on AWS SageMaker, maintaining consistent experiment lineage throughout
- A git+DVC fingerprinting system (combining git SHA and dvc.lock MD5) became the single source of truth for reproducibility, tagged into MLflow and SageMaker across all three pipeline iterations
- Moving from MLX local training to SageMaker V2 Pipelines reduced training time from 14 hours to 3.3 hours using spot instances with ~60% cost savings and conditional model promotion logic
- Ray Train integration enabled multi-node data parallelism for 7B models with live TensorBoard monitoring, while keeping training scripts framework-agnostic and portable
- The key architectural insight: SageMaker handles infrastructure provisioning and spot recovery, while Ray handles worker coordination — with zero SageMaker SDK imports in the training code itself
Why It Matters
This article provides a rare, detailed blueprint for teams transitioning from rapid local experimentation to production-grade distributed ML pipelines, demonstrating that reproducibility infrastructure (DVC lineage) should be established from day one rather than retrofitted. The practical bug documentation and cost-performance tradeoff analysis offer actionable guidance for ML engineers building scalable fine-tuning workflows on AWS.
Technical Details
- MLX Local Prototype: Used
mlx_lmfor LoRA fine-tuning of Qwen2.5-1.5B on an M4 Mac Mini, achieving val loss drop from 1.8 to 0.9 on MathInstruct; integrated MLflow for experiment tracking and DVC for data versioning with S3 remote cache - SageMaker V2 Pipeline: Five-step DAG (PrepareData → TrainLoRA → Evaluate → ReadEvalReport → CheckTestLoss) with
ConditionStepfor automated model promotion based on test loss improvement; ran onml.g4dn.xlargespot instances with checkpoint-to-S3 for fault tolerance - SageMaker + Ray Train: Python-orchestrated multi-node data parallelism using
--num-instancesflag with automatic learning rate scaling (Goyal et al. 2017 rule); live Ray Dashboard for per-worker task graphs, OOM events, and gradient sync timing; TensorBoard events streamed to S3 viaTensorBoardOutputConfig - Reproducibility System:
git_sha + dvc.lock_MD5produces a permanent fingerprint (e.g.,e90324a9-dvc3f2a1b7c) logged as MLflow tagdvc.commit, SageMaker pipeline parameterDVCCommitHash, andmlflow.log_input()dataset record — enabling full reconstruction viagit checkout <sha> && dvc checkout - Key Bugs Documented:
processing_classvstokenizerin SFTTrainer (trl version mismatch),KeyError: 'ModelArtifacts'on interrupted SageMaker jobs (resolved with explicit_wait_training()polling),PropertyFileonly works withProcessingStep(required relay step), Ray workercwdmismatch resolved withPath(__file__).parent
Industry Insight
- Start reproducibility infrastructure (DVC lineage, MLflow tagging) from the first experiment — retrofitting it later is significantly more expensive and error-prone than building it in from the start
- The hybrid approach of using SageMaker for infrastructure (provisioning, spot recovery) while keeping training code framework-agnostic (pure Ray + HuggingFace) maximizes portability and reduces vendor lock-in
- Conditional model promotion steps and automated drift monitoring (planned next steps) should be standard in production pipelines to prevent regression shipping and enable proactive retraining
Disclaimer: The above content is generated by AI and is for reference only.