I Deployed My Data Pipeline to AWS. Then Everything That Was "Local" Broke.
Moving a local data pipeline to AWS exposed hidden assumptions built around "everything runs on the same machine," with infrastructure setup being easier than the migration itself Kestra's internal state (flows) is separate from project files, requiring manual re-import of workflows when migrating between environments The Process task runner executes inside Kestra's container, not on the host, causing missing dependency issues like `python3.12-venv` that don't appear in local development The key
Analysis
TL;DR
- Moving a local data pipeline to AWS exposed hidden assumptions built around "everything runs on the same machine," with infrastructure setup being easier than the migration itself
- Kestra's internal state (flows) is separate from project files, requiring manual re-import of workflows when migrating between environments
- The Process task runner executes inside Kestra's container, not on the host, causing missing dependency issues like
python3.12-venvthat don't appear in local development - The key architectural lesson: orchestrators (managers) and workers should be decoupled; Kestra should orchestrate, not perform the actual computation
- Switching to Kestra's Docker task runner allows spinning up isolated worker containers per task, with the Docker socket mounted for container management
Why It Matters
This article illustrates a common pain point for AI practitioners and data engineers: local-to-production migration reveals architectural blind spots that are invisible during development. The lesson about separating orchestration from execution is directly applicable to anyone building ML pipelines, ETL workflows, or data infrastructure that starts locally and scales to cloud environments.
Technical Details
- Infrastructure: AWS EC2 t3.small with Ubuntu 22.04, Elastic IP, security group restricting ports 22/8080/5432 to author IPs, 1GB swap file added to prevent OOM issues
- Storage: Nitro system uses
nvme0n1device naming instead of traditionalxvda, requiringgrowpartandresize2fsfor disk expansion - Orchestration stack: Kestra (workflow orchestration), Postgres (database via Docker), dbt (transformations), WSL2 (local development)
- State management: Kestra flows are stored in its internal database, not as syncable files—requiring manual YAML import post-migration
- Task runner migration: Switched from Kestra's Process runner (executes inside Kestra container) to Docker runner (spins up isolated containers), requiring Docker socket mounting for container daemon access
Industry Insight
- Local development environments create false confidence; always validate pipeline assumptions in a production-like environment early, not after months of local-only development
- When choosing an orchestrator, evaluate whether it conflates management and execution responsibilities—tight coupling between the two creates fragile, hard-to-maintain systems
- Secrets management and state separation (files vs. internal databases) are critical migration considerations that should be addressed in the initial architecture design, not discovered during deployment
Disclaimer: The above content is generated by AI and is for reference only.