NACTA
“A 6–8 hour manual ingestion job, every time. Systems that can't talk to each other. Make it all run on its own.”
The situation
At NACTA, data arrived as Excel files that someone had to open, understand and load by hand, a job that took 6–8 hours each time. Around it sat internal systems that couldn't exchange data directly.
I automated the ingestion and built the layer that lets those systems talk.
What I shipped
- Automatic Excel ingestion. Incoming files are detected, classified by their content, routed to the right MongoDB tables and staged, replacing the manual 6–8 hour process.
- Failures set aside, not lost. Lookups resolve references across tables; records that fail are isolated with the reason logged for review.
- An Apache NiFi data-exchange layer between internal systems that couldn't communicate directly, with Keycloak sign-in and role-based access.
- A throughput fix. A NiFi pipeline between FastAPI and RabbitMQ that keeps a high-volume image-processing workflow stable under heavy load.
- An audit trail and security visibility. Every exchange logged in PostgreSQL, and a Wazuh SIEM rollout that shows senior staff any unauthorised activity.
Stack
Next
→