HTML Optimization & Caching Angular Partials with Python

Premature Optimization

is the root of all evil.

I wanted to start with this reminder because although this is kind of cool, it's not for fun and can cause a number of headaches if you are attempting to debug any HTML after we're done with it.

Just to be clear, this technique is strictly for production use. The only time it should be used in development is to test if this does not break a production site. Otherwise, if everything is bug-free and you're looking to find any way to cut precocious milliseconds off your page/template requests, then you're in the right place.

The Plan

So there are really only a few things we need to do here:

1. Gather our partial template files
2. Minify the HTML
3. Generate a JavaScript file that adds all of our templates into our Angular's $templateCache service.

This way, our generated JavaScript file from step 3 will be loaded once on our initial page load with all of our other JavaScript assets. From then, the $templateCache service will use the minified HTML in that file to handle any requests for partials that your app sends to the server. Although angular asynchronously loads the partial using an xhr request, depending on the server's response time things can seem sluggish at times, especially if the ngView directive makes up the majority of your main template.

Our Tools

I like to write my programming helpers in Python, since every operating system worth its weight comes with Python pre-installed (except for Windows, but I'm not sure this is for Windows users). I never seem to have an issue using Python wherever I find myself.

We can mainly make use of Python's awesome standard library for what we need, except when we're minifying the HTML, because Python's built-in HTML parser, html.parser is notoriously hard to use and prone to error. So, we will just use a third party library instead: htmlmin. It has a light footprint and a small, easy API, and it is just the library we need.

To make sure we can use it in our program, let's install it (hopefully in your project's virtualenv, but this isn't a lecture on project structure, so if it's not there, that won't affect us now. However, I would advise looking into the topic).

$ pip install -U htmlmin

After doing this, minifying out HTML is going to be the easiest part of what we need to do, i.e. we can just use the following lines:

from htmlmin import minify
minified_html = minify(html)

Can't make it easier than that.

Now, onto the rest of what we need.

Generating JavaScript in Python

To generate the JavaScript file that adds our templates to our AngularJS cache, we can just use simple string interpolation techniques.

The funny thing is, Angular has its own interpolation system, and sometimes that can cause conflicts with alternate interpolation engines. The most famous examples are Jinja2 and Django, as they both have conflicts due to sharing their double-curly-brace variable interpolation style of {{ var }} with Angular. Unfortunately, this can confuse Python's standard style of string interpolation, which uses single-curly-braces { var }.

Fortunately for us, Python supports another string interpolation syntax, and it's actually the original way Python did it. It uses the c style "%TYPECODE" % VAR, i.e.:

>>> print 'this text was --> %s added' % ('just')
this was --> just added

So, because our Angular partials will be littered with {{ and }}, we will need to use the older Python interpolation syntax to get around this. Since the current interpolation syntax is recommended, this isn't something you should do regularly without consideration, as it can be a source of odd bugs if done incorrectly, and it also makes code less readable.

With that in mind, the ability Python gives us to use techniques like this to get around weird incompatibilities between systems is one of my favorite things about the language.

Continuing On

Here is the code to interpolate the JavaScript functions that will add the templates to the $templateCache service.

import os
from htmlmin import minify

CACHE_FUNC_TEMPLATE = (
"angular.module('%s').run(['$templateCache',function($templateCache){"
"    $templateCache.put('%s',"
"'%s');\n}]);"
)

cache_template =\
    lambda template,name,module:\
            (CACHE_FUNC_TEMPLATE % (module,name,(minify(open(template,'r').read()))))

Now if this looks like a lot is going on, then you are very observant.

We are accomplishing quite a bit in these few lines. To start with, on the first 2 lines we import the os module, the only part of the standard library we will need, and the minify function from the HTMLMinifier library we installed earlier. The next 3 lines define the string we will use to generate the JavaScript. It has 3 %s's scattered in it, and those are the placeholders for the parts that will change: the Angular module to declare when adding the template to the cache, the name to give the template, and the content of the template.
The next 4 lines define an anonymous function that encapsulates all the string interpolation we need to do.

Gathering Files

Next, we need a function to grab templates for us. To keep things simple, we will only look for files in one directory, but we will have the ability to choose that directory.

def gather_templates(dirname=None):
    rtn = []
    if dirname is None:
        dirname = os.path.realpath(os.getcwd())
    for fle in os.listdir(dirname):
        if fle.endswith('.html'):
            rtn.append(
                (os.path.join(dirname,fle))
            )
    return rtn

The last thing we need is a function that will go through the gathered templates and pass their name and content to our cache_template function from earlier.

def minify_templates(dirname,module):
    '''
        add all templates in :dirname to :modules templates cache
    '''
    html = gather_templates(dirname)
    rtn = ''
    brk = '\n'
    for itm in html:
        if not itm.endswith('.min.html'):
            rtn +=\
                cache_template(
                    itm,
                    os.path.basename(itm),
                    module
                ) + brk
    return rtn

Finishing Up

Finally, we need a function to run it all, as well as our main function. This way we can use it as a command line tool or we can import the do_min function to do it in code, like if we were using fabric.

DEFAULT_OUT_FILE = 'all.min.js'

def do_min(module,dirname=None,outfile=None):
    outfile = outfile or DEFAULT_OUT_FILE
    with open(outfile,'w') as out:
        out.write(minify_templates(dirname or os.getcwd(),module))
    print 'all done'

def main():
    module,outfile = None,None
    args = (len(sys.argv) > 1) and sys.argv[1:]
    if args:
        if len(args) == 1:
            module = args[0]
        elif len(args) == 2:
            module,outfile = args
        do_min(module,None,outfile)
        return
    print 'Usage: %s MODULE [OUTFILE][default: %s]' %\
                    (sys.argv[0],DEFAULT_OUT_FILE)

if __name__ == "__main__":
    main()

Here is an example of running this in a directory with this in the file test.html

<div>
    <span>{{ angularvar }}</span>
</div>

$ Python cache_templates.py test.app test.min.js
$ cat test.min.js

angular.module('test.app').run(['$templateCache',function($templateCache){    $templateCache.put('test.html','<div><span>{{ angularvar }}</span></div>');
}]);

For reference, here is the complete file we have been building

import os
import sys
from htmlmin import minify

CACHE_FUNC_TEMPLATE = (
"angular.module('%s').run(['$templateCache',function($templateCache){"
"    $templateCache.put('%s',"
"'%s');\n}]);"
)

cache_template =\
    lambda template,name,module:\
            (CACHE_FUNC_TEMPLATE % (module,name,(minify(open(template,'r').read()))))


def gather_templates(dirname=None):
    html = []
    if dirname is None:
        dirname = os.path.realpath(os.getcwd())
    for fle in os.listdir(dirname):
        if fle.endswith('.html'):
            html.append(
                (os.path.join(dirname,fle))
            )
    return html

def minify_templates(dirname,module):
    '''
        add all templates in :dirname to :modules templates cache
    '''
    html = gather_templates(dirname)
    rtn = ''
    brk = '\n'
    for itm in html:
        if not itm.endswith('.min.html'):
            rtn +=\
                cache_template(
                    itm,
                    os.path.basename(itm),
                    module
                ) + brk
    return rtn

DEFAULT_OUT_FILE = 'all.min.js'

def do_min(module,dirname=None,outfile=None):
    outfile = outfile or DEFAULT_OUT_FILE
    with open(outfile,'w') as out:
        out.write(minify_templates(dirname or os.getcwd(),module))
    print 'all done'

def main():
    module,outfile = None,None
    args = (len(sys.argv) > 1) and sys.argv[1:]
    if args:
        if len(args) == 1:
            module = args[0]
        elif len(args) == 2:
            module,outfile = args
        do_min(module,None,outfile)
        return
    print 'Usage: %s MODULE [OUTFILE][default: %s]' %\
                    (sys.argv[0],DEFAULT_OUT_FILE)

if __name__ == "__main__":
    main()

Update

I ended up having issues in some partials, when passing the template content to angular. So I tried a few things, and found the best solution was to dump the partials as a large JSON string, then iterating through that string in angular, doing this I was also able to greatly simplify / minimize the actual amount of javascript generated by this script. Here is my updated version of the above script:

import os
import sys
from htmlmin import minify

FUNC_TEMPLATE = (
"var t = %s;"
"angular.module('%s').run(['$templateCache',function($templateCache){"
"    var templates = JSON.parse(t).templates;"
"    angular.forEach(templates,function(val,key){"
"        $templateCache.put(key,val);"
"    });"
"}]);"
)

gather_html = lambda name: {x[0]:minify(x[1]) for x in ( lambda name:
{x:open(os.path.join(name , x),'r').read().decode('utf8') for x in ( lambda name: 
[x for x in os.listdir(name) if x.endswith('.html')])(name)})(name).items()}

make_template_cache = lambda dirname,module: 
FUNC_TEMPLATE  % (repr(json.dumps(dict(templates=gather_html(dirname)))),module)

def do_min(module,dirname=None,outfile=None):
    outfile = outfile or DEFAULT_OUT_FILE
    with open(outfile,'w') as out:
        out.write(make_template_cache(dirname or os.getcwd(),module))
    print 'all done'

def main():
    module,outfile = None,None
    args = (len(sys.argv) > 1) and sys.argv[1:]
    if args:
        if len(args) == 1:
            module = args[0]
        elif len(args) == 2:
            module,outfile = args
        do_min(module,None,outfile)
        return
    print 'Usage: %s MODULE [OUTFILE][default: %s]' %\
                    (sys.argv[0],DEFAULT_OUT_FILE)