A Workaround Heroku's Ephemeral File System

Oh oh! my static files go missing when a Dyno restarts

Deploying a Python/Django application to Heroku can be seamless and smooth except when you realize that your generated static files keep disappearing after a dyno restart and then it dawns on you that Heroku has an Ephemeral filesystem , which discards any dynamically generated file on a dyno restart, yeah you saw that right, discards!

For the simple project I was working on, a Python/Django backend and React/Redux frontend where I needed to get my webpack generated files sent to Django staticfiles folder for reference when deployed. This was not the case as Heroku did not persist my file. I apparently did not realize the whole ephemeral thingy yet.

Normally, if you push your project with pre-populated static files everything works well except when it is dynamically generated.

How does Heroku Work by the way?

When you deploy a Heroku application, you build a virtual machine image called a slug using one or more buildpacks. When a virtual machine instance is launched from this slug, it's called a dyno.

Each dyno runs a single process inside your application virtual machine. Heroku does not officially describe how dynos are provisioned, but anecdotal analysis shows many dynos are run on a single Amazon XL EC2 instance, sharing disk, CPU, and memory across all dynos. There are definitely "noisy neighbors" when sharing resources, but direct data or stats are not provided. I can only share my anecdotal experience that this does indeed happen...

stackoverflow source/reference

Further study: All Heroku services are hosted on Amazon's EC2 cloud-computing platform. Interesting huh?

So what did I do after realizing this?

I decided to dynamically generate my file within Heroku's slug compile process, more like make it part of the virtual machine image that will be used to create a dyno (same as an instance), that way I still get access to my generated JS and CSS files irrespective of a restart or not.

How did I achieve this?

Here is how, Heroku buildpack takes the application along with all the dependencies, and the language runtime to produce a slug (think of this as a virtual machine image). The python buildpack for the python project has a post_compile hook which is run by heroku-buildpack-python, in other to get the static file to be part of the slug, you simply take advantage of this post_compile hook to write your custom script according to this git repository example

By creating a proper bin/post_compile bash script in the root directory of your application you can define what should happen before the actual slug compilation begins. Here is a link to my codebase for reference.

and how it looks like before and after compilation from CircleCI console

In my project, I want to achieve 3 things before the actual compilation begins

1. Install nodejs in order for me to run webpack as webpack needs nodejs installed
2. Get all the JS and CSS static files compiled into a specified folder by running webpack (at this point we already have nodeJS installed)
3. Run django collectstatic, this way all the generated JS and CSS file including that of external modules are all collected and placed in the generated Staticfile directory according to my project's settings.py file

With all that done, the post-compile process hands control over to the compile process as shown in the diagram below.

In the end, I have all my generated staticfiles baked into the slug (virtual machine image) and I no longer need to worry about Heroku's ephemeral system.

S3 with Heroku

Note: Ideally if you are dealing with Image Uploads and other Assets it is recommended to use AWS S3 bucket to store your assets as Slug size should not exceed 500mb. Here is a link to read up on using S3 with Heroku

Thanks for reading and feel free to like/share this post.