Airflow Case Study: ProofPort
A case study of using Apache Airflow to complete transform the data pipeline at a SaaS startup - in just a few months.
2
2
Ryan Cleary
Mentor
(14 reviews)
US$40.00
For every 15 mins
29
Sessions/Jobs
ABOUT ME
Senior software engineer with 5 years professional experience (including Amazon!)
Senior software engineer with 5 years professional experience (including Amazon!)
- 7 years of programming experience
- 3.5+ years of Python experience, primarily focused on ETL pipelines and web development using Django + React
- I've worked at: 2 Fortune 100 companies (Amazon & ConocoPhillips), a mid-sized company, and a SaaS startup, and started my own Software Consulting firm
- Broad background (7+ years) of programming experience, everything from iOS development to Java to C# / .NET
- Nearly 1 year of volunteer experience teaching people from non-traditional backgrounds how to code
Central Time (US & Canada) (-06:00)
Joined November 2018
EXPERTISE
3 years experience | 4 endorsements
- Nearly 3 years of professional experience in Python, including the following technologies:
Pandas, Apache Airflow, Celery, Django, Unit...
- Nearly 3 years of professional experience in Python, including the following technologies:
Pandas, Apache Airflow, Celery, Django, Unittest, PyTest
- Built a complete production-grade ETL pipeline from scratch in just 3 months for a startup; processes 8000+ records of data each week and generates Slack notifications of what changed, then finally loads the data via a GraphQL API to the production system.
3 years experience | 3 endorsements
- Designed and developed Course Portal where the client could use the admin management screen to build out courses with lessons, videos, ...
- Designed and developed Course Portal where the client could use the admin management screen to build out courses with lessons, videos, file resources, assignments, etc. all without having to contact the developer.
- Developed two internal "portals" using Django, complete with background worker process developed using Celery/Flower.
- Developed full-stack application to track community engagement; includes Leaflet (Map) integration, login, etc. https://github.com/clearyusc/NoPlaceLeftApp
2 years experience
- Designed and implemented two production-grade ETL data pipelines at two different companies
- Ability to deploy Airflow in different wa...
- Designed and implemented two production-grade ETL data pipelines at two different companies
- Ability to deploy Airflow in different ways: PaaS (Astronomer.io), Local, Docker-Compose, Celery
3 years experience
- Built 4+ web applications using the Django framework
- Worked on a complex web app built in C#/.NET
- Worked on an extremely scalable, ...
- Built 4+ web applications using the Django framework
- Worked on a complex web app built in C#/.NET
- Worked on an extremely scalable, complex web application for Amazon built in Java
2 years experience | 1 endorsement
- Built custom course video player UI using React
- Built AWS Lambda function using NodeJS to integrate with a custom Slackbot.
- Small...
- Built custom course video player UI using React
- Built AWS Lambda function using NodeJS to integrate with a custom Slackbot.
- Small full-stack CRUD app written in NodeJS using MongoDB.
3 years experience
- Used Bootstrap to theme a fullstack website that enables teams of users to track community engagement; complete with Leaflet map integr...
- Used Bootstrap to theme a fullstack website that enables teams of users to track community engagement; complete with Leaflet map integration.
- Built an internal UI library for a company using Bootstrap 4.
3 years experience
REVIEWS FROM CLIENTS
(14 reviews)
Glenn Paquette
October 2020
Ryan's the best. He's knowledgeable, patient, and a blast to chat with. A+++ I highly recommend.
KWGD1980
October 2020
Great guy, knows his stuff
Irfan Md
September 2020
Great mentor! Provided quick and useful insights and resources to get started.
Lutaaya Huzaifah Idris
September 2020
Ryan is really a good mentor, he dives and explains the why and the how
Lily Kamas
September 2020
Very helpful, helped me to work through problems, and gave me a ton of resources.
Zack
August 2020
Very helpful
Glenn Paquette
August 2020
Ryan to the rescue! Ryan was able to help me track down a Google API bug in my code and find a workaround to get things working again. AMAZING!
Glenn Paquette
August 2020
Ryan was a big help in troubleshooting my Django api problem! He helped me understand the issue and worked with me to hunt down the source of the bug.
Zack
August 2020
Great mentor
EMPLOYMENTS
Founder & Senior Software Engineer
Praxeis Technology
2020-02-01-Present
- Developing SpencerShadrach.com, a full-stack website with an easy to edit marketing side for the client. Website also includes an inter...
- Developing SpencerShadrach.com, a full-stack website with an easy to edit marketing side for the client. Website also includes an interactive online Course Portal where users can preview courses and purchase them, comment, and it displays user progress. Includes Admin Portal for simple Course creation. Technologies used: Django, Django Rest Framework, Python, JavaScript, React, Bootstrap 4, HTML, CSS, Heroku, Docker, PostgresQL, Stripe.
- Designing Hearth.com, a full-stack website that aims to serve the homeless. Currently working with the client and a designer iterating on product ideas to get a Minimum Viable Product (MVP) out the door as quickly and as cost-effectively as possible. Design was done in Figma. Planning on using Django, Django Rest Framework, JavaScript, React, etc. to build the application out.
HTML/CSS
Python
Django
View more
HTML/CSS
Python
Django
PostgreSQL
Heroku
Bootstrap
Docker
React
JavaScript
View more
Lead Data Developer
ProofPort
2019-07-01-2019-12-01
- Designed and implemented an ETL pipeline that processes over 8000 records every week from various data sources (Import.io, CSVs, PDFs) ...
- Designed and implemented an ETL pipeline that processes over 8000 records every week from various data sources (Import.io, CSVs, PDFs) using Apache Airflow, Python, GraphQL, AWS S3, AWS DynamoDB, and AWS CloudWatch.
- Added automation to the ETL pipeline, reducing the time required to audit the results by 90%.
- Integrated the ETL pipeline with a custom SlackBot function using the Serverless Framework (with an AWS NodeJS Lambda function).
Python
Amazon S3
AWS DynamoDB
View more
Python
Amazon S3
AWS DynamoDB
AWS Lambda
Airflow
View more
Software Developer
Green Mountain Technology
2017-09-01-2018-06-01
- Co-led a team of two that was tasked with redesigning the core architecture of the company's data ingestion and transformation pipeline...
- Co-led a team of two that was tasked with redesigning the core architecture of the company's data ingestion and transformation pipeline, then joined a team of 7 to begin implementing this design
- Designed and implemented an ETL process built on top of Apache Airflow to inject data analytics into dashboard creation and automate a tedious business process, to save hundreds of man- hours of work each year
- Implemented an internal UI pattern library in Bootstrap 4
HTML/CSS
Python
C#
View more
HTML/CSS
Python
C#
Django
Bootstrap
JavaScript
Airflow
View more
PROJECTS
SpencerShadrach.comView Project
Spencer Shadrach
2020
A full-stack website with an easy to edit marketing side for the client. Website also includes an interactive online Course Portal where ...
A full-stack website with an easy to edit marketing side for the client. Website also includes an interactive online Course Portal where users can preview courses and purchase them, comment, and it displays user progress. Includes Admin Portal for simple Course creation. Technologies used: Django, Django Rest Framework, Python, JavaScript, React, Bootstrap 4, HTML, CSS, Heroku, Docker, PostgresQL, Stripe.
HTML/CSS
Python
Django
View more
HTML/CSS
Python
Django
PostgreSQL
Heroku
Stripe
Bootstrap
Docker
JavaScript
View more