Songtradr
“Move everything from BigQuery to Snowflake. And prove nothing got lost on the way.”
The situation
Songtradr, a music-tech company, was moving its data from Google Cloud and BigQuery to Snowflake and dbt. I joined the distributed engineering team remotely, while I was still at university.
Moving data is the easy part. Proving it arrived intact is the job.
What I shipped
- A Python validation framework that generates SQL on the fly to check that migrated data is sane, correct and complete.
- A Snowflake dashboard and reporting table tracking daily transfer performance and migration progress.
- Airflow DAGs, tuned for continuous scheduled transfers, matched to the source system's speed.
- Discrepancies tracked down and resolved, so the team could trust the migrated data.
Stack
Next
→