How and why I built A ETL Website using many microservices
About me
I am a Developer who is mostly worked on Ruby on Rails, Node JS and Python..With having strong experience on Cloud PAAS(Heroku, Cloud Foundry), IAAS(AWS) and having some experience on MongoDB and Dynamo DB . I did AWS Solution Architect Associate certification.
The problem I wanted to solve
I always love to work on performance scalability. And for My Web Application, I gave it most of the improved performance for their bulk loading of 60 million data per day.Also the app has ETL functionalit like Extract(csv, Salesforce, Other DB connector, ftp) ..Transform(Mapping) and load into (Database, and other sources)
What is A ETL Website using many microservices ?
I build the ETL application for Robust performance and provide API Services, Extarct from different sources like csv, Other DB, Salesforce, FTP, ...Many others transform the data as User want and Load the data.
Tech stack
Ruby on Rails, Java, Node JS, PostgreSQL, Heroku, AWS (s3, elastic beanstalk, SNS, Code Pipeline),
The process of building A ETL Website using many microservices
- First, we design our database schemas according to the requirements
- Then we design our FDD
- Then configure and Build the applications
- Did deploy the application
- Did CI(for running test and build continuously using ) and CD (Code Pipeline AWS)
Challenges I faced
Timeout issue while integrating different data source
Security issue from data fetching pipeline.
Key learnings
I learn to build a Big Scale site from Scratch
I learned many optimizations on code level and DB level
I learned many AWS services
I learned CD(Continuous Delivery)
I learned lot of security fixes.
Tips and advice
Always keep up working and indulging to the services and tools. And try to give best possible delivery from all top aspects(Security, Performance, Design and Background Process)
Final thoughts and next steps
I am keep supporting this Application for my client..They did not use any Market ETL but they like to use my ETL due to its user friendly ness and performance
