The situation
Songtradr, a music-tech company, was moving its data from Google Cloud and BigQuery to Snowflake and dbt. I joined the distributed engineering team remotely, while I was still at university.
The job
A pipeline nobody trusts is just a faster way to be wrong.
What I shipped
- 01A migration program in Python and SQL, driven by YAML config, that moved source data into Snowflake and worked with the Google platforms around it, including Cloud Storage.
- 02An automated validation system that generates SQL on the fly to check that migrated data is consistent, correct and complete between Google and Snowflake.
- 03Both automated with Airflow: validation and data moves ran as scheduled DAGs, tuned to the source system's speed, so the migration finished sooner.
- 04Real-time dashboards, built with the BigQuery and Snowflake teams, to watch validation and transfer progress and catch problems early.
- 05Discrepancies tracked down and resolved, so the team could trust the migrated data.
Stack
In their words
What people say
“Moez also demonstrated great initiative by designing an automated validation system to ensure data consistency between Google and Snowflake migrations. This system proved invaluable in maintaining data integrity throughout the migration process.”