So, You Want to DIY Kubernetes Upgrade?

As technology professionals, we all know it’s best to keep our applications and application dependencies up to date. Sure, most of us don’t want to upgrade the first day or week that an update comes out but planning for these updates helps maintain a prudent and proactive approach for your application environment. Your senior technology leader is typically responsible for this environment,. Thus, the article is for anybody that is either responsible for or works with containers and your Kubernetes environment.

It’s important to understand Kubernetes security as it relates to best practices for enterprise deployment in your container environment, and the importance of maintaining that environment and building a culture of ownership and security. This article will focus more on the operational nuts and bolts of a Kubernetes upgrade.

Obviously, there are other third-party companies, including AWS, Google, and Azure, that offer managed Kubernetes environments and the associated upgrades for you. However, if you aren’t using one of these environments, the focus of this article will be on the manual upgrade process itself. You can use these guidelines for on-premises and cloud-based Kubernetes environments.

Kubernetes refresher

While the other goal is to keep this article relevant for the engineer types out there, it is also a bit high-level and broader for the greater audience, to include senior technology roles. So, what is Kubernetes again? It’s a container orchestration platform for working with any container technology, not just Docker containers. While there are other alternatives to manage your containers,

Kubernetes gives you a little more “bang for your buck,'' so-to-speak. Essentially, Kubernetes gives you a means for large deployments, an easy way to scale it, and the robust monitoring that comes with it.

When using both Kubernetes and Docker, specifically, these named products can provide a simpler and easier upgrade path. They do this by permitting you to build containers and new dependencies with virtual ease in your environment. Similar to any application dependencies, security updates and new features are frequently available. Just as important, your Kubernetes infrastructure and underlying nodes must be kept current as well. While you may not be interested in the new features coming out, you should perform these upgrades for the purposes of security.

Items to consider

The goal today isn’t to write a TL;DR or white paper on Kubernetes upgrade paths, but to get you started in the right direction. Of course, there are a ton of resources on this topic, and they can be as long or as short as fits your Kubernetes experience or liking. One great place for all things Kubernetes is GitHub, which is one of the top active projects at any given time.

When it comes to installing a Kubernetes cluster in the cloud or on-premise, you’ll find that there is some great documentation that is straightforward and simple. Additionally, there are other third-party automation tools to make the upgrade process even simpler.

So when upgrades come out, it’s important to proactively plan your upgrades. It would be recommended to run your environment in a HA mode for obvious reasons. If you run a large environment with many clusters you are undoubtedly running HA already. Again, keeping your cluster up to date gives you peace of mind. More importantly, it provides benefits from the new features, bug fixes, and the latest security patches. In order to automate this upgrade process, you’ll need to always be running the latest supported version.

Where to start

There are two main parts that need to be updated: the masters and the nodes. In typical fashion, the masters need to be updated first. Once those are completed, the nodes can then be updated. You can follow your documentation and simply do this upgrade process using the Kubernetes Engine via the administrator center.

As mentioned previously, you’ll want to be running your cluster in HA. This allows for minimal downtime and provides a bit of “wiggle room” if you need to test a few things. What’s convenient about the Kubernetes engine, is that it can automatically upgrade the master as smaller, incremental releases are available.

Although, it typically won’t upgrade on a complete full upgrade version, automatically. The simplest way to upgrade to a new version is by using the upgrade master button within the console of the Kubernetes engine.

We just covered that the masters need to be upgraded before the nodes. Likewise, when managing your HA cluster, it’s important to note that this was just one of two separate tasks within your Kubernetes cluster upgrade process. These specific tasks cannot be performed concurrently, nor can they be overlapped. Furthermore, and just as important, is the “etcd” cluster, which is the distributed backing store of your Kubernetes environment. Typically, you’ll upgrade your etcd cluster last.

From a high-level standpoint, we’ll display the tasks and the order that is followed when upgrading your cluster:

• Go into your first node; upgrade the kubeadm tool only
• Verify, then apply upgrade plan
• Update the kubelet; restart its service
• Apply your upgrade plan to any additional master nodes
• Upgrade kubectl on all the masters
• Upgrade kubeadm on one worker node
• Go into a master node, then drain first worker node
• Upgrade kubelet configuration on worker node
• Upgrade kubelet on worker node, restart its service
• Lastly, restore the worker node
• Depending on your environment or the documentation you follow there may be slight variations to this. Again, it may seem obvious, but it’s important you are following the process step-by-step in the order that is documented.

Final thoughts

While we didn’t go through a detailed upgrade path in a typical HA Kubernetes cluster, we provided some best practices and foundational information that should be followed in an upgrade path. As with any technology environment or infrastructure, these are crucial maintenance processes that you would follow in any typical company environment.

Once you find the right Kubernetes upgrade documentation that best suits your environment, ensure that your team is well-versed in your strategy for these upgrades, whether they are doing the actual upgrade or not. As with anything, communication and a documentation repository go a long way.

Alternatively, you may be inclined to use a managed Kubernetes service with your cloud provider or another third-party to manage all this for you. Depending on your available resources and budget, this can certainly eliminate headaches, downtime, and give you back some of that precious time.