Express + Brotli + Webpack 🚀

Let’s compress together and learn how Brotli can help us to increase the performance of our websites. I have implemented it in one of my work projects. So, I just thought to share my experience with everyone.

Basic definitions 📖

Express: Fast, unopinionated, minimalist web framework for Node.js.
Brotli: It is an open source data compression library developed by Jyrki Alakuijala and Zoltán Szabadka. It is based on a modern variant of the LZ77 algorithm, Huffman coding and 2nd order context modeling.
Webpack: It is a module bundler. It takes modules with dependencies and generates static assets representing those modules.

Let’s begin with the real shit 💩 !!!

There were two ways to implement the compression in Express directly (without any web server i.e: nginx etc ):

1. Static building of compressed files with help of Webpack (any other frontend task runner or builder) and serve them on demand as needed by client.
2. Dynamic building of compressed files on the run time (You can use require(‘compression’) ) in express to dynamically compresse files and serve to the client on the fly.
   I have only implemented the static building of files. So let’s talk about that in more detail.

Static compression with Express
In the first step, which is building your bundles, you can include these two plugins compression-webpack-plugin and brotli-webpack-plugin.

const CompressionPlugin = require(‘compression-webpack-plugin’);
const BrotliPlugin = require(‘brotli-webpack-plugin’);
module.exports = {
plugins: [
 new CompressionPlugin({
 asset: ‘[path].gz[query]’,
 algorithm: ‘gzip’,
 test: /\.js$|\.css$|\.html$/,
 threshold: 10240,
 minRatio: 0.8
 }),
 new BrotliPlugin({
 asset: ‘[path].br[query]’,
 test: /\.js$|\.css$|\.html$/,
 threshold: 10240,
 minRatio: 0.8
 })
]
}

These plugins will generate both gzip and brotli file for all of your bundles i.e if bundle name is ‘vendor_d0cfe49e718c1366c661.js’ you will get the vendor_d0cfe49e718c1366c661.js.gzip and vendor_d0cfe49e718c1366c661.js.br in the same directory (let’s assume it is /dist/static/ vendor_d0cfe49e718c1366c661.js.* as of now).

PS: The above code will only generate the .gzip and .br if the minRatio 0.8 is achieved while compressing files. So in case of very small files gzip and br files will not be generated. The reason is that time in compressing and decompressing is costlier than the actual file served without compression.

You may also need to set the public path in webpack output config to ‘/static’. It will specifies the public URL address of the output files when referenced in a browser. Which will help us to filter the URL of the request and serve the the files by express-static-gzip fonly static if URL consist of ‘/static’.

output: {
path: '/dist/static',
filename: ‘[name]_[chunkhash].js’,
chunkFilename: ‘[id].[chunkhash].js’,
publicPath: ‘/static/’,
},

In the second step, which is to serve the right file based on input headers from the client browser. We will use the express-static-gzip

var express = require(“express”);
var expressStaticGzip = require(“express-static-gzip”);
var app = express();
// app.use(express.static(path.join(__dirname))); This was used previously with express.
app.use(“/”, expressStaticGzip(path.join(__dirname), {
 enableBrotli: true
}));

The above code is the default code setting from the ‘express-static-gzip’ but I wanted to serve only the static files from this library. If file doesn’t exist, I wanted to throw error and my code should not go further into other routes. So, I just hacked a bit in the source code and created a new middleware file compression.js.

Below is the hacked code:

var express = require(“express”);
const expressStaticGzip = require(‘compression’); // compression.js gist is available on the github.
var app = express();
 
app.get('*', expressStaticGzip(path.join(__dirname), {
 urlContains: ‘static/’,
 fallthrough: false,
 enableBrotli: true,
}));

There are three parameters that I have used here

1.** urlContains:** It will check if the request original url contains the ‘static/’ in it. Then only serve files through this plugin else ignore the URL.
2. fallthrough: It should be false to throw the error if file you are looking for doesn’t exist in the path.join(__dirname) directory and URL has ‘urlContains’.
3. enableBrotli: It will check if Brotli file is available in the path.join(__dirname) and the appropriate headers is being requested by client then serve the .br file.
Below is the gist of the compression.js. I have hacked from the line 59-65.

const mime = require('mime');
const serveStatic = require('serve-static');
const fileSystem = require('fs');

module.exports = expressStaticGzip;

/**
 * Generates a middleware function to serve static files.
 * It is build on top of the express.static middleware.
 * It extends the express.static middleware with
 * the capability to serve (previously) gziped files. For this
 * it asumes, the gziped files are next to the original files.
 * @param {string} rootFolder: folder to staticly serve files from
 * @param {{enableBrotli:boolean,
 * customCompressions:[{encodingName:string,fileExtension:string}],
 * indexFromEmptyFile:boolean}} options: options to change module behaviour
 * @returns express middleware function
 */
function expressStaticGzip(rootFolder, options) {
  options = options || {};
  if (typeof (options.indexFromEmptyFile) === 'undefined') options.indexFromEmptyFile = true;

    // create a express.static middleware to handle serving files
  const defaultStatic = serveStatic(rootFolder, options);
  const compressions = [];
  const files = {};

    // read compressions from options
  setupCompressions();

    // if at least one compression has been added, lookup files
  if (compressions.length > 0) {
    findAllCompressionFiles(fileSystem, rootFolder);
  }

  return function middleware(req, res, next) {
    changeUrlFromEmptyToIndexHtml(req);

        // get browser's' supported encodings
    const acceptEncoding = req.header('accept-encoding');

        // test if any compression is available
    const matchedFile = files[req.path];
    console.log(req.originalUrl, matchedFile);
    if (matchedFile) {
        // as long as there is any compression available for this
        // file, add the Vary Header (used for caching proxies)
      res.setHeader('Vary', 'Accept-Encoding');

                // use the first matching compression to serve a compresed file
      const compression =
            findAvailableCompressionForFile(matchedFile.compressions, acceptEncoding);
      if (compression) {
        convertToCompressedRequest(req, res, compression);
      }
    }

      // allways call the default static file provider
    defaultStatic(req, res, (err) => {
      if (err && (req.originalUrl.indexOf(options.urlContains) > -1)) {
        console.log('Hola', req.originalUrl, err);
        return res.status(404).json({ error: `No file found with ${req.originalUrl}` });
      }
      return next();
    });
  };

    /**
     * Reads the options into a list of available compressions.
     */
  function setupCompressions() {
        // register all provided compressions
    if (options.customCompressions && options.customCompressions.length > 0) {
      for (let i = 0; i < options.customCompressions.length; i += 1) {
        const customCompression = options.customCompressions[i];
        registerCompression(customCompression.encodingName, customCompression.fileExtension);
      }
    }

        // enable brotli compression
    if (options.enableBrotli) {
      registerCompression('br', 'br');
    }

        // gzip compression is enabled by default
    registerCompression('gzip', 'gz');
  }

    /**
     * Changes the url and adds required headers to serve a compressed file.
     * @param {Object} req
     * @param {Object} res
     */
  function convertToCompressedRequest(req, res, compression) {
    const type = mime.lookup(req.path);
    const charset = mime.charsets.lookup(type);
    let search = req.url.split('?').splice(1).join('?');

    if (search !== '') {
      search = `?${search}`;
    }

    req.url = req.path + compression.fileExtension + search;
    res.setHeader('Content-Encoding', compression.encodingName);
    res.setHeader('Content-Type', type + (charset ? `; charset=${charset}` : ''));
  }

    /**
     * In case it's enabled in the options and the
     * requested url does not request a specific file, "index.html" will be appended.
     * @param {Object} req
     */
  function changeUrlFromEmptyToIndexHtml(req) {
    if (options.indexFromEmptyFile && req.url.endsWith('/')) {
      req.url += 'index.html';
    }
  }

    /**
     * Searches for the first matching compression available from the given compressions.
     * @param {[Compression]} compressionList
     * @param {string} acceptedEncoding
     * @returns
     */
  function findAvailableCompressionForFile(compressionList, acceptedEncoding) {
    if (acceptedEncoding) {
      for (let i = 0; i < compressionList.length; i += 1) {
        if (acceptedEncoding.indexOf(compressionList[i].encodingName) >= 0) {
          return compressionList[i];
        }
      }
    }
    return null;
  }

    /**
     * Picks all files into the matching compression's file list. Search is done recursively!
     * @param {Object} fs: node.fs
     * @param {string} folderPath
     */
  function findAllCompressionFiles(fs, folderPath) {
    const filesMain = fs.readdirSync(folderPath);
        // iterate all files in the current folder
    for (let i = 0; i < filesMain.length; i += 1) {
      const filePath = `${folderPath}/${filesMain[i]}`;
      const stats = fs.statSync(filePath);
      if (stats.isDirectory()) {
                // recursively search folders and append the matching files
        findAllCompressionFiles(fs, filePath);
      } else {
        addAllMatchingCompressionsToFile(filesMain[i], filePath);
      }
    }
  }

    /**
     * Takes a filename and checks if there is any compression type matching the file extension.
     * Adds all matching compressions to the file.
     * @param {string} fileName
     * @param {string} fillFilePath
     */
  function addAllMatchingCompressionsToFile(fileName, fullFilePath) {
    for (let i = 0; i < compressions.length; i += 1) {
      if (fileName.endsWith(compressions[i].fileExtension)) {
        addCompressionToFile(fullFilePath, compressions[i]);
        return;
      }
    }
  }

    /**
     * Adds the compression to the file's list of available compressions
     * @param {string} filePath
     * @param {Compression} compression
     */
  function addCompressionToFile(filePath, compression) {
    const srcFilePath = filePath.replace(compression.fileExtension, '').replace(rootFolder, '');
    const existingFile = files[srcFilePath];
    if (!existingFile) {
      files[srcFilePath] = { compressions: [compression] };
    } else {
      existingFile.compressions.push(compression);
    }
  }

    /**
     * Registers a new compression to the module.
     * @param {string} encodingName
     * @param {string} fileExtension
     */
  function registerCompression(encodingName, fileExtension) {
    if (!findCompressionByName(encodingName)) {
      compressions.push(new Compression(encodingName, fileExtension));
    }
  }

    /**
     * Constructor
     * @param {string} encodingName
     * @param {string} fileExtension
     * @returns {encodingName:string, fileExtension:string,files:[Object]}
     */
  function Compression(encodingName, fileExtension) {
    this.encodingName = encodingName;
    this.fileExtension = `.${fileExtension}`;
  }

    /**
     * Compression lookup by name.
     * @param {string} encodingName
     * @returns {Compression}
     */
  function findCompressionByName(encodingName) {
    for (let i = 0; i < compressions.length; i += 1) {
      if (compressions[i].encodingName === encodingName) { return compressions[i]; }
    }
    return null;
  }
}

Analysis of the results:

Let’s compare the performance of the website with Brotli or GZip or just uncompressed minified files.

In God we Trust, all others bring data. -W. Edwards Deming

Let’s dig ⛏ into the analysis and find the real numbers. Since this is my work place website which we used for internal purpose. I can’t share the real screenshot here but below are the real numbers that I have collected while testing the website on different type of compressions.

1. Brotli is ~8% ( ( 7.24–6.67 ) / 7.24) efficient than GZIP and 65.7% ( ( 19.48–6.67 ) / 19.48) efficient than Uncompressed file. If browser will not be able to serve the Brotli we have a fallback for gzip which is 62% ( ( 19.48–7.24 ) / 19.48) efficient than uncompressed file. So here we have Win Win situation.
2. Now let’s analyse the size. Brotli is (( 586–458)/586)~21.85% efficient than GZIP and it is (( 2.51024–458)/2.51024)~82.1% efficient than uncompressed files. So lot of bandwidth can be saved by using Brotli compression.
   Some data for Slow 3G network:

Thank you all for reading so far. If you like it and want me to write more. Please let me know.