Surveillance For Everybody

D.J. Abrie, 2019-08-26

TL;DR

Leverage open-source deep learning models and cloud tools to democratise surveillance and intelligence at scale.

Introduction

Lately, there has been a lot of media publicity about the seemingly limitless power of facial recognition, digital surveillance and tracking by large tech companies and government agencies (Facebook, American NSA & British GCHQ).

Facial Recognition is of particular interest to a wider audience as it’s intrinsically coupled to persons’ biographical information. A system that gives you access and searches the personal details (names, occupations, relations, criminal history, etc.) of individuals in your social or professional circles can be useful to many people for different reasons.

Applications at a personal level can range from home security systems that notify you when a stranger is on your property, to community-based surveillance (i.e. personal identification) systems that track movement in the streets of a suburb. Commercial applications for small businesses can include surveillance of the typical in-store shopper and so lend aid to a company’s market research.

To the individual or small business it might seem like scalable, industry grade, face recognition and data retrieval system is beyond reach, but that is not true.

Cloud to the Rescue

Any individual or small business might not be able to employ a Facebook-size army of software engineers to develop a face detection system. However, hiring a single freelance software engineer is feasible.

With the advent of cloud technology, the discerning software engineer has many tools at his disposal in his proverbial tool-belt. Just like any good tradesman, the software engineer has to know which tool to employ in which context. If he does this right, he can leverage the full power of thousands of computers, petabytes of data storage, machine learning technology, as well as high speed networks to his client’s advantage.

Cloud providers such as Google Cloud and Amazon Web Services have thousands of engineers working day and night to provide and maintain this infrastructure and tooling. So even though the freelancer - or small group of freelancers - are limited in manpower, they can leverage the work of many others.

Deep Learning: Science Vs. Application

Large cloud providers like AWS and Google Cloud also provide machine learning services such as face detection. Face Recognition is not that common, but there are some great organisations such as OpenFace, who have made high performance deep learning face recognition models freely available.

The software engineer is therefore not always required to be an expert in data science or machine learning in order to leverage the power of these models. The already designed and trained open-source models can be employed on the bespoke dataset for impressive results.

With a bit of work, these custom models can be converted to run on Google’s Cloud Functions, which offers a scalable and maintenance-free solution to run these models on large workloads.

The Components of our Skynet

In this article, we will detail the system-level design of a scalable and powerful face recognition and storage/retrieval system that can be built by a single software engineer in a matter of weeks. We will use the tools in the Google Cloud together with open source machine learning models. Because the models are freely available and Google Cloud has a free tier on many of their products, our software will not cost us much (if anything!) while our face recognition throughput is small.

Hardware

We assume some hardware is in place to acquire a feed of images containing faces into a Google Cloud Storage bucket. This can be done with modern CCTV camera systems by configuring the DVR to upload images to the cloud. Alternatively, one might use a fleet of RasberryPis each with their own webcam or PiCam to the same end. See his example on recognising faces using a RasberryPi with PiCamor on how to recognise faces using your webcam.

Cloud Components

We will use a cloud storage bucket to store our images. Storing the images in the cloud makes it possible to access it from a cloud function.

We can have as many hardware devices saving images to this bucket as we want and we don’t have to worry about running out of memory as the cloud provider takes care of this.

This service has a free tier which offers the 5 TB of data storage for free.

In terms of ML models, we will use a TensorFlow implementation of MTCNNfor high-quality face detection. Then we will use OpenFace’s implementation of Googles’ FaceNet model for face recognition. The OpenFace model is a PyTorch implementation which has a system dependency on DLib - a C++ based deep learning framework - which is available to Google Cloud functions. We will therefore convert this model to a TensorFlow model. TensorFlow is a deep learning framework developed by Google and is available to cloud functions.

We will use Google Cloud Functions to run our OpenFace Machine Deep learning models. These functions can run JavaScript and Python and are easy to deploy and Google handles the scaling in the background to make sure that the function can handle a large amount. This service has a free tier which offers the first 2 million cloud function calls (per month) for free.

Google BigQuery is a powerful analytical database which is great for analysing data over large datasets. We will use this database to store our facial embeddings - a set of numbers that uniquely characterises a face. We will also store meta-data with the embeddings such as names, geolocation, timestamps, etc.We can then use SQL to query this data to find the insights we want. BigQuery charges it’s users based on the amount of data they query and the first 1 TB of data queries per month is free. Since our facial embeddings will be around 512 bytes each, we can query about 244 million facial embeddings for free each month!

Together with BigQuery, Google also offer some data visualisation software (called Data Studio) for free. We can use this to produce some regular reports on our data. There is also a useful tool for visualising geographical location data on a map, which we can use if the images contain GPS location metadata.

System Architecture

Putting all the components together will take some skill and thorough testing. The high-level design for this system is shown in the diagram below.

The process works as follows:

1. Generate images with your hardware and save to a Google Cloud Storage bucket

2. Setup a cloud function to trigger on new images in the bucket
   a. Find faces in the image with the MTCNN model
   b. Transform faces to have eyes and mouth in a standard location
   c. For each face, extract the FaceNet facial embedding using the converted OpenFace model

3. Store the embedding and any additional metadata back to a file in Cloud Storage

4. Load the embedding and metadata into BigQuery

5. Draw up reports from your data in Google Data Studio

Possible Extensions

r information to faces

• Automatically cluster face embeddings to link faces to the same personal identity
• Extract image metadata (such as geolocation) in the cloud function and add it to the BigQuery record
• Run some more advanced models against the images such as age and gender detection.
  

Conclusion

By implementing the above design, the software engineer can leverage a set of tools to implement a customisable application of face recognition together with scalable processing, data storage and data retrieval. With the free tiers, anybody can get started for only the cost of the hardware.

There are many exciting applications for this technology

• Protect your kids by matching people in your social circles with public criminal or sex offender registers
• Help retailers do market research with automated profiling of their clientele
• Send yourself alerts when strangers (unidentified faces) show up in your back yard.

A world of possibilities await!