Deploying K8s Persistent Volumes with AWS EBS

Kubernetes is the world’s most popular container orchestrator. It is an open source tool that can help you manage multiple compute nodes and run containerized applications on them. Kubernetes groups together containers that have a similar function or belong to the same application in a unit called pods, and instead of having to micro-manage containers, you can manage and reuse pods.

One of the major challenges in a containerized environment is storage. Containers are ephemeral in nature, which means that when a container shuts down all the data it generated is erased. This is suitable for stateless applications, but for stateful scenarios, such as applications that connect to a database, there is a need to store data persistently. To To solve this problem, Kubernetes provides the concept of a Persistent Volume (PV). A PV can be mounted by a pod, giving it access to permanent, persistent storage. Even if the pod or its containers shut down, storage remains intact.

When running Kubernetes in the Amazon cloud, you will typically manage persistent storage for containers using Amazon Elastic Block Storage (EBS). Let’s see how this works.

What are Amazon EBS Volumes?

Amazon Elastic Compute Cloud (Amazon EC2) offers scalable computing capacity. Amazon EC2 offers persistent storage volumes for data you use via Amazon Elastic Block Store (Amazon EBS), called EBS volumes.

An Amazon EBS volume is a block-level, durable storage device. You can attach this device to your instances. Once you attach a volume to an instance, you use it the same way you use a physical hard drive.

For current-generation volumes linked to current-generation instance types, you may modify the provisioned IOPS capacity, drastically increase size, and modify volume type for live production volumes.

You may use EBS volumes as the principal storage for information that needs regular updates, including storage for a database application or the system drive for an instance. You may also use EBS volumes for throughput-intensive applications that carry out ongoing disk scans. They persist independently from the lifecycle of an EC2 instance.

EC2 pricing models such as on-demand pricing, reserved instances, and spot instances, also apply to Amazon EBS.

Using EBS as Persistent Volumes in Elastic Kubernetes Service

One of the options for running Kubernetes in AWS is the Elastic Kubernetes Service (Amazon EKS). This is a managed service that lets you run a Kubernetes cluster on AWS without having to install and operate the Kubernetes control plane. Cluster nodes can run on Elastic Compute Cloud (EC2) instances or in a serverless fashion using Amazon Fargate.

When running Kubernetes in EKS, you can use the Amazon EBS CSI driver to run EBS volumes as Kubernetes PVs. When the driver is deployed, any application that creates a PV in the cluster, results in Amazon automatically creating EBS volumes in the underlying infrastructure.

The instructions below are adapted from the Amazon EKS documentation.

To follow this tutorial, ensure you have an EKS cluster with version 1.17 or later of Kubernetes, with an IAM OIDC provider. Also install the AWS CLI and kubectl version 1.17 or later on your local machine.

Step 1: Create IAM Policy and Role

First, you need to create an IAM policy to enable the CSI driver to access the Amazon API on behalf of your cluster. Download the IAM policy document and save it in a local path, then create the policy in your AWS account by running this CLI command:

Now create an IAM role and attach the policy to it, using the following command:

Finally, take note of the ARN code of the IAM role you created. You can get it by running the cloudformation describe-stacks command.

Step 2: Deploy the Driver via Helm Chart

To deploy the driver, ensure you have Helm v3 or later. Add the Helm repository for the EBS CSI driver:

Install a release of the driver using the Helm chart. Replace the repository address with the container image address of the cluster. The container image address depends on the Amazon region your cluster is in—see a list of addresses for different regions.

Step 3: Test the EBS CSI Driver

Amazon provides a sample application that uses dynamic provisioning of PVs. You can use it to test that the driver is working. In a real-world deployment, it is recommended to use a container monitoring solution to ensure persistent storage is working correctly.

First, Clone the aws-ebs-csi-driver repository by cloning this AWS repo.

Change to the directory in which you deployed the EBS CSI driver, and switch to the directory /examples/kubernetes/dynamic-provisioning/

The example provides some Kubernetes resources you can use for testing. Create them in your cluster by running:

kubectl apply -f specs/

Now, create a storage class that describes PVs based on EBS volumes:

kubectl describe storageclass ebs-sc

Pods deployed by the preceding commands will now gradually start running as EBS volumes are dynamically provisioned for them. Use this command to see their status change:

kubectl get pods --watch

To see the persistent volumes created on demand for the pods, run this command:

kubectl get pv

You can make sure pods are writing data to the volume by running this command:

kubectl exec -it app -- cat /data/out.txt

Conclusion

In this article I explained the basics of Kubernetes Persistent Volumes and Amazon EBS, and showed how to deploy an EBS driver to your Kubernetes cluster, which allows the cluster to dynamically provision PVs based on EBS volumes. This involves the following steps:

1. Create IAM policy and role—giving the CSI driver permission to access the Amazon API on your behalf.
2. Deploy the driver via Helm chart—deploying a Helm chart to your cluster to create the driver.
3. Testing the EBS CSI Driver—Amazon provides a sample application that lets you create pods, dynamically provision an EBS volume, and see how those pods access the volume as a Kubernetes Persistent Volume.

I hope this will be useful as you take your first steps in Kubernetes storage management.