why the design distributed system
- Worked on maintaining and adding features to an enterprise-wide machine learning platform that is used by various other teams at Capital One to streamline machine learning processes
- Built an internal API that provisioned and created Kubernetes applications for ML jobs. This was done to move away from using AWS EMR to reduce deployment time from 20 minutes to less than 1 minute.
- Migrated the API component of our machine learning platform to a newer AWS account to prevent our subnet running out of IP addresses and reducing clutter
ref: https://blog.pragmaticengineer.com/distributed-architecture-concepts-i-have-learned-while-building-payments-systems/
Written on August 30, 2023