How and why I built Custom Health Check Framework for job running in k8s pods

About me

Experienced Software Engineer with excellent problem-solving skills and ability to perform well under pressure and lead a team.

The problem I wanted to solve

Implement a health check framework for client that would monitor a job running in k8s containers and apply custom rules on logs to kill stalled jobs.

What is Custom Health Check Framework for job running in k8s pods?

My client wanted to solve a problem with their current architecture. They had a data processing job that they run in docker container using k8s as orchestrator. As their job is not refined and has bug their container sometimes get stuck in stalled state. They wanted to monitor same and restart the container or kill the job accordingly. With all this they wanted dynamic rules mapping and logging mechanism so developer can study logs later on. Even if the container got evicted.

Tech stack

After researching current tech stack that solves part of the problem, I came up with following.

• Grafana for dashboard and monitoring
• LOKI to gather container logs
• Custom service to take action on k8s pods.
• I decide to go for SpingBoot java service as i am familiar with this tech and have worked on it .

The process of building Custom Health Check Framework for job running in k8s pods

I broke down the problem statement in small and manageable chunks:

• First installed LOKI and Grafana on a test setup as a service. (using helm charts)
• Then I make a sample dashboard . Also implemented some rules to filter logs.
• Then I proposed a design with POC's results that I did on sample data.
• After that I did following tasks for complete implementation:
  • Prepared a Dashboard, with alerts and webhooks configured.
  • Make an Operator-service that listens for webhook calls to perform some action using k8s api
  • Operator service requires some privileges to call k8s api , so did those Challenges
  • then finally packaged it all as a shippable code (with provisioning code for automatic install using helm charts and ansible)
  • submitted for review
  • finally api docs and all the documentation completed.
  • shipped project.

Challenges I faced

• It was my first time playing around with LOKI and Grafana. I was hard in start but got easier with time.
• Coming up with edge cases while preparing design and POC was difficult as expected.

Key learnings

• I learned a great deal of communication skills while discussing solution with stakeholders
• It was due to thorough research that design came up to be useful. I realised its importance

Tips and advice

• It is not recommended to re-invent the wheel. You should use solutions available and come up with fast and impactful solution

Final thoughts and next steps

• No system is perfect, I will work in future to improve the solution i provide to this client.
• It was overall a great project both in terms for learning new tech as well as challenging design.
  Overall a challenging project