The situation
At NACTA, data arrived as Excel files that someone had to open, understand and load by hand, a job that took 6–8 hours each time. Around it sat internal systems that couldn't exchange data directly.
The job
I automated the ingestion and built the layer that lets those systems talk.
What I shipped
- 01Automatic Excel ingestion. Incoming files are detected, classified by their content, routed to the right MongoDB tables and staged, replacing the manual 6–8 hour process.
- 02Failures set aside, not lost. Lookups resolve references across tables; records that fail are isolated with the reason logged for review.
- 03An Apache NiFi data-exchange layer between internal systems that couldn't communicate directly, with Keycloak sign-in and role-based access.
- 04A throughput fix. A NiFi pipeline between FastAPI and RabbitMQ that keeps a high-volume image-processing workflow stable under heavy load.
- 05An audit trail and security visibility. Every exchange logged in PostgreSQL, and a Wazuh SIEM rollout that shows senior staff any unauthorised activity.