Chatbots: How to Make a Bot for Messenger From Scratch (Part 1)

Table of Contents

Getting Started

Ever wondered how to make a bot? This walkthrough will guide you through the process of developing your own chatbot for Messenger from scratch — no convenience libraries or bot building platforms. The tools we will use to make the chatbot include:

1. Node.js 6.x
2. Express 4.x framework
3. HTTP
4. Redis
5. EC2
6. MySQL? MongoDB?
7. Ngrok

Prerequisites:

• Experience with JavaScript and Node.js
• A desire to work with HTTP and REST APIs
• A desire to build a chatbot on Facebook’s Messenger Platform.
• A Facebook app
• ngrok
• Node.js
• An account with API.ai
• An AWS account
• Github or Bitbucket account with knowledge on how to use GIT for code management.

The best way to get started with building a chatbot is to create an app on the Facebook Developer platform. This guide should get your Facebook app ready, but in order to test it, you will need to develop a webhook. This will be the main entry point for our bot. All messages and data from Facebook will come to our app via this HTTP endpoint. In order to test webhooks, we need a server to forward live webhooks to our localhost. For live testing, we will use a service called Ngrok.

Let’s begin by building our Node.js endpoint. First, you need to install Node.js. You can download it at nodejs.org. Once you’ve installed Node.js and NPM, we will create a new folder for you project — we will call this folder /base in the rest of post.

Inside the /base folder, create a new file called “package.json.” Copy and paste the following content into that file and save it in the /base folder.

{
  "name": "Messenger-Bot-Architecture-and-Design",
  "version": "0.0.1",
  "private": true,
  "scripts": {
    "start": "node app.js"
  },
  "dependencies": {
  }
}

Now, open a terminal and type the following commands.

> npm install --save express
> npm install --save body-parser
> npm install --save multer

Open a code editor and create a new file called “app.js.” This will be the Node.js server for our chatbot. Add the following code to your app.js file:

var express        = require('express'),
    bodyParser     = require('body-parser'),
    http           = require('http'),
    app            = express();


// set port
app.set('port', process.env.PORT || 8080);

// create a health check endpoint
app.get('/health', function(req, res) {
  res.send('okay');
});

// start the server
http.createServer(app).listen(app.get('port'), function(){
  console.log('Express server listening on port ' + app.get('port'));
});

Open a terminal and execute the following command.

npm start

Open a browser and go to http://localhost:8080/health — you should see the word “okay” printed to the browser window.

Open another terminal window so you have two open inside the /base folder. In Terminal 1, start your Node.js server with this command:

npm start

In Terminal 2, execute this command, but replace “<mycustomdomain>” with any subdomain that you would like.

ngrok http 8080

Ngrok will print out information about your account and server with a domain that you can then send requests to.

Now, when you open the domain http://xxx.ngrok.io/health you should see the word “okay” printed to your screen.

So, now we have a working server that is exposed to a public domain. We now need a new endpoint that accepts POST requests.

Let’s add the code to our server and modify your server so it looks like the following code:

var express        = require('express'),
    bodyParser     = require('body-parser'),
    http           = require('http'),
    app            = express();

app.use(bodyParser.json());

// set port
app.set('port', process.env.PORT || 8080);

// create a health check endpoint
app.get('/health', function(req, res) {
  res.send('okay');
});

app.post('/fb',  function(req,res){
  res.send(req.body)
})

// start the server
http.createServer(app).listen(app.get('port'), function(){
  console.log('Express server listening on port ' + app.get('port'));
});

Accepting Webhooks

In order for Facebook to verify our server, we need to create another GET endpoint that validates a token send from the Facebook Messenger webhook configuration screen.

Add this code to your server:

app.get('/fb', function(req, res) {
  if (req.query['hub.verify_token'] === 'abc') {
     res.send(req.query['hub.challenge']);
   } else {
     res.send('Error, wrong validation token');
   }
});

You should use a stronger token than “abc” (as shown above) — I chose “abc” simply for demonstration purposes. When you click “Verify and Save”, your webhook should be accepted by Facebook.

You should now see this in your Facebook app:

In the Webhooks section, select a page from the drop down, then click to subscribe to your messenger events for that page. You should see something similar to the following in the webhooks section of your Facebook app configuration:

Now, in the “Token Generation” section, select the page you just subscribed to, and copy the token Facebook generates.

Create a new variable in your server to save the token.

Var token = “xxxxxxx”

The server will need this token to send messages to Facebook on behalf of the users sending messages to your chatbot. Now, we should be able to send messages to our page and see the messages appear on our server’s console because we added a line console.log(req.body) to our webhook endpoint. When you view the log, you should see something like this:

{ object: 'page',
  entry: 
   [ { id: 'xxx',
       time: 1489404379192,
       messaging: [Object] } ] }

We would have to add a JSON.stringify to view the entire object from Facebook. Modify the console.log line in your POST endpoint to use JSON.stringify. The POST endpoint should look like the following code:

app.post('/fb', function(req, res){
  console.log(JSON.stringify(req.body))
  res.send(req.body)
})

Now, when you send a message to your bot, you should see something similar to this JSON printed in your server’s console. This is the req.body in the server code.

{
  "object": "page",
  "entry": [
    {
      "id": "137101496360273",
      "time": 1489404570380,
      "messaging": [
        {
          "sender": {
            "id": "xyz"
          },
          "recipient": {
            "id": "xyz"
          },
          "timestamp": 1489404570341,
          "message": {
            "mid": "mid.1489404570341:ce017e2f89",
            "seq": 72281,
            "text": "Hi"
          }
        }
      ]
    }
  ]
}

You will notice that the “entry” key of the object is an Array. You will also see that the entry[0].messaging is an Array. So, in order to get to our text and sender, we must use the following JSON notation.

Sender:
req.body.entry[0].messaging[0].sender.id

Text:
req.body.entry[0].messaging[0].message.text

Sending Message Replies

In order for our chatbot to reply to the user, we need to send an HTTP POST request with the page’s Access Token to Facebook. To do this, I will use the Node.js “request” module. Stop your server and execute the following command in your terminal:.

npm install --save request

Now we can add it to our server. Update your server code to look like so:

var express        = require('express'),
    bodyParser     = require('body-parser'),
    http           = require('http'),
    server         = require('request'),
    app            = express(),
    token 	= 'xyz';

app.use(bodyParser.json({}));
app.post('/fb', function(req, res){
  console.log(JSON.stringify(req.body))
  res.send(req.body)
})
app.get('/fb', function(req, res) {
  if (req.query['hub.verify_token'] === 'abc') {
     res.send(req.query['hub.challenge']);
   } else {
     res.send('Error, wrong validation token');
   }
});

// create a health check endpoint
app.get('/health', function(req, res) {
  res.send('okay');
});

// set port
app.set('port', process.env.PORT || 8080);
// start the server
http.createServer(app).listen(app.get('port'), function(){
  console.log('Express server listening on port ' + app.get('port'));
});

Now, we need to write an asynchronous function to handle incoming messages. It is important for this to be async in order for our chatbot to reply to the incoming webhook instantly with a 200 response. This concept is important for a Messenger bot to have smooth and responsive functionalities. Add the following code to your server:

app.messageHandler = function(j, cb) {
  setTimeout(function(){
    cb(true);
  }, 10000)
}

This is a simple test function to demonstrate the async behavior of Node.js. Now, modify your POST endpoint to look like this.

app.post('/fb', function(req, res){
  console.log(JSON.stringify(req.body))
  app.messageHandler(req.body, function(result){
    console.log("Async Handled: " + result)
  })
  res.send(req.body)
})

Now, when you send a message to your bot from Messenger, you can monitor the server console and see that the message arrives. 10 seconds later, the server should print “Async Handled: true.”

Now let’s echo the message back to the user. Modify the “messageHandler” function to look like the following:

app.messageHandler = function(j, cb) {
  var data = {
    "recipient":{
    	"id":j.entry[0].messaging[0].sender.id
    },
    "message":{
    	"text":j.entry[0].messaging[0].message.text
    }
  };
  var reqObj = {
    url: 'https://graph.facebook.com/v2.6/me/messages',
    qs: {access_token:token},
    method: 'POST',
    json: data
  };
  console.log(JSON.stringify(reqObj))
  request(reqObj, function(error, response, body) {
    if (error) {
      console.log('Error sending message: ', JSON.stringify(error));
      cb(false)
    } else if (response.body.error) {
      console.log("API Error: " + JSON.stringify(response.body.error));
      cb(false)
    } else{
      cb(true)
    }
  });
}

When you message the bot, it should reply you with the same message you sent. Our simple echo server is now complete.

Natural Language Processing

The next step to having a functional chatbot is for it to understand customers. The best way to understand a message is to use Natural Language Understanding (NLU). A common tool for NLU is Google’s API.ai. I will use API.ai to build a simple Q&A bot for this walkthrough.

Create a new agent on API.ai:

Now create a new Intent for your agent:

Name your intent “AreYouARobot” and then in the “User Says” section, add the following phrases:

Are you a robot?
Are you a bot?

Now, in the “Speech Response” section of the Intent, add the following text:

Yes, I am a Messenger Bot.

Once you’ve saved your Intent, you can test it in the right panel of the API.ai interface. When you send the question “Are you a robot?” to your bot, it should reply, “Yes, I am a Messenger bot.”

When you click on “Show JSON,” you should see the following.

You can see from this JSON document that we will get our API.ai speech response from the following JSON notation: JSON.result.fulfillment.speech

We need to tie this NLU into our server. To do this, we need another asynchronous function that sends the incoming messages to API.ai. Add the following function to your server:

app.speechHandler = function(text, id, cb) {
  var reqObj = {
    url: 'https://api.api.ai/v1/query?v=20150910',
    headers: {
      "Content-Type":"application/json",
      "Authorization":"Bearer 4485bc23469d4607b19a3d9d2d24b112"
    },
    method: 'POST',
    json: {
      "query":text,
      "lang":"en",
      "sessionId":id
    }
  };
  request(reqObj, function(error, response, body) {
    if (error) {
      console.log('Error sending message: ', JSON.stringify(error));
      cb(false)
    } else {
      console.log(JSON.stringify(body))
      cb(body.result.fulfillment.speech);
    }
  });
}

This function will send the Async HTTP request to the API.ai endpoint. The API.ai endpoint will return the JSON we need to reply to any incoming message. Now, we need to modify our async messageHandler function to the following code.

app.messageHandler = function(text, id, cb) {
  var data = {
    "recipient":{
    	"id":id
    },
    "message":{
    	"text":text
    }
  };
  var reqObj = {
    url: 'https://graph.facebook.com/v2.6/me/messages',
    qs: {access_token:token},
    method: 'POST',
    json: data
  };
  console.log(JSON.stringify(reqObj))
  request(reqObj, function(error, response, body) {
    if (error) {
      console.log('Error sending message: ', JSON.stringify(error));
      cb(false)
    } else if (response.body.error) {
      console.log("API Error: " + JSON.stringify(response.body.error));
      cb(false)
    } else{
      cb(true)
    }
  });
}

Finally, we need to modify our POST endpoint to use our speech handler function with our message handler function. Modify your POST endpoint to look like the following code:

app.post('/fb', function(req, res){
  var id = req.body.entry[0].messaging[0].sender.id;
  var text = req.body.entry[0].messaging[0].message.text;
  console.log(JSON.stringify(req.body))
  app.speechHandler(text, id, function(speech){
    app.messageHandler(speech, id, function(result){
      console.log("Async Handled: " + result)
    })
  })
  res.send(req.body)
})

Now, restart your server and send the message “Are you a robot?” to your chatbot in Facebook Messenger. The bot should reply with the message, “Yes, I am a messenger bot.”

Your bot is now integrated with API.ai. You will notice that if you send the bot any message it is unfamiliar with, it will not understand how to handle it. To expand the knowledge of your bot, you can turn on “Domains” in the API.ai dashboard.

By using domains, you can easily expand the knowledge of your chatbot. Now, add your code to a GIT repository, and push you code to GitHub for safe keeping.

Context & Session Management

Our current chatbot is very simple and easy to develop. Q&A bots like this do not require any session retention. However, if you want to make a bot that people might actually use, you will most likely have to require the retention of the session and some memory of the context. API.ai has a “Context” feature for their Intents, which allows the chatbot to maintain the thread of the conversation. The best use case for this feature is the concept of gathering some information from the user. To demonstrate this, we will add a 3 step question to our bot. When the user says, “Save my task” the bot will respond with the following sequence of questions:

What subject is it for?
When is it due?

The user will be prompted to answer the questions. When the questions are resolved, the chatbot will tell the user the homework was saved.

Create a new “Entity” on API.ai that looks like the following image:

We will create an intent that leverages this Entity to understand what the message is actually trying to say. Create an intent on API.ai that looks like the following image:

Add another parameter for the due date. Your Intent parameters should look like the following image:

Finally, add a speech response that says “Your $subject homework was saved for due.” our new intent is complete! API.ai will automatically use the Subject and Due Date in the response because we used the “” variable in our Speech Response text. Since our server is already integrated, and we are passing the “Sender ID” from Facebook to API.ai as the “Session ID,” API.ai can already manage the context and session for us. Give your bot a try by saying “Save my homework” to it on Messenger.

Deploying to AWS

The first thing to do is to launch a new Ubuntu image on the EC2 cloud. Login to AWS, navigate to the EC2 console, and launch a new instance.

Select the Ubuntu 16.04 LTS 64bit image from the list of AMIs.

Select the t2.Micro instance type, then go to “Configure Instance Details.” The instance details should be fine on the default settings. Click “Add Storage” then select the magnetic type of storage.

Using an SSD will boot faster, however, using magnetic storage will save money in the long run. Since we are deploying on distributed infrastructure, we won’t need a file system. When deploying scalable infrastructure, you might not want to save files directly on the server. For file storage, you will need to use a service like S3.

Move on to adding tags, and give your server a “Name” tag. After you add the Name, move on to the security group configuration. Create your security group configuration like so:

Now, click “review and launch” then click “Launch.” Create a new key pair, give it a name, and download it into your /base folder. After you download the key pair, launch your instance. Go back to the EC2 dashboard, and select your newly created instance. You will see that it has a Public IPv4 address. This is the address we will use to SSH into the Ubuntu machine.

Open a new terminal in the /base folder and type the following command

sudo chmod 400 <your key>.pem

This will secure the key file for SSH. You can now execute the following command to log in.

ssh -i <your key>.pem ubuntu@<your ip address>

This will log you into the Ubuntu machine. The first thing to do is to install Node.js. Run this command on the ubuntu machine:

curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
sudo apt-get install -y nodejs

Now, you can use GIT to install your code by pulling from your repository on Github or Bitbucket. For this tutorial, I’m going to upload a ZIP file using SCP. Create a ZIP archive of your /base folder then add the ZIP file to your /base folder. Now run the following command:

scp -i <your key>.pem ./base.zip ubuntu@<your ip>:base.zip

You should see the file upload in the Terminal.

Now SSH back into your server.

ssh -i <your key>.pem ubuntu@<your ip address>

Run the following commands.

unzip apt-get unzip
unzip base.zip
cd base
sudo rm <your key>.pem
sudo rm -rf node_modules

We delete all the node_modules and then rebuild them using the native Ubuntu system. Otherwise, we will likely experience errors. Run the following commands.

npm install
npm start

The other vital thing we need is an SSL certificate. Facebook only allows bots to run over SSL encrypted connections. We can get an SSL certificate for free using Let’s Encrypt Certbot.
Run the following command:

sudo apt-get install letsencrypt

Now type this command.

wget https://dl.eff.org/certbot-auto
chmod a+x ./certbot-auto

We also need a domain name. You can get a free domain name here.

Point your domain’s A record for @ and WWW to the IPv4 address of your EC2 instance.

We need to enable SSL on port 443 for our server. Click on the security group of your EC2 instance from the dashboard.

Then add the SSL rule like this.

Now run this command.

./certbot-auto certonly --standalone --email aaron@nimblestack.io -d <your domain> -d www.<your domain>

This command will output the path to your Key and your new Certificate. You can find the path to the certificate in the “Important Information” that Certbot prints out after the certificate is generated. Be sure to read it carefully. Now modify your server code to use SSL. The final server code should look like this.

var express        = require('express'),
    bodyParser     = require('body-parser'),
    https           = require('https'),
    request        = require('request'),
    app            = express(),
    fs             = require('fs'),
    token          = 'XXXXXXXXXXXXXXXXXXXXXXXXXX',
    sslOpts        = {
      "key":fs.readFileSync("/etc/letsencrypt/keys/0000_key-certbot.pem"),
      "cert":fs.readFileSync('/etc/letsencrypt/live/<your domain>/fullchain.pem')
    }
// accept JSON bodies.
app.use(bodyParser.json({}));

// accept incoming messages
app.post('/fb', function(req, res){
  var id = req.body.entry[0].messaging[0].sender.id;
  var text = req.body.entry[0].messaging[0].message.text;
  console.log(JSON.stringify(req.body))
  app.speechHandler(text, id, function(speech){
    app.messageHandler(speech, id, function(result){
      console.log("Async Handled: " + result)
    })
  })
  res.send(req.body)
})
app.messageHandler = function(text, id, cb) {
  var data = {
    "recipient":{
    	"id":id
    },
    "message":{
    	"text":text
    }
  };
  var reqObj = {
    url: 'https://graph.facebook.com/v2.6/me/messages',
    qs: {access_token:token},
    method: 'POST',
    json: data
  };
  console.log(JSON.stringify(reqObj))
  request(reqObj, function(error, response, body) {
    if (error) {
      console.log('Error sending message: ', JSON.stringify(error));
      cb(false)
    } else if (response.body.error) {
      console.log("API Error: " + JSON.stringify(response.body.error));
      cb(false)
    } else{
      cb(true)
    }
  });
}
app.speechHandler = function(text, id, cb) {
  var reqObj = {
    url: 'https://api.api.ai/v1/query?v=20150910',
    headers: {
      "Content-Type":"application/json",
      "Authorization":"Bearer XXXXXXXXXXXXX"
    },
    method: 'POST',
    json: {
      "query":text,
      "lang":"en",
      "sessionId":id
    }
  };
  request(reqObj, function(error, response, body) {
    if (error) {
      console.log('Error sending message: ', JSON.stringify(error));
      cb(false)
    } else {
      console.log(JSON.stringify(body))
      cb(body.result.fulfillment.speech);
    }
  });
}

// verify token to subscribe
app.get('/fb', function(req, res) {
  if (req.query['hub.verify_token'] === 'abc') {
     res.send(req.query['hub.challenge']);
   } else {
     res.send('Error, wrong validation token');
   }
});

// create a health check endpoint
app.get('/health', function(req, res) {
  res.send('okay');
});

// set port
app.set('port', process.env.PORT || 443);

// start the server
https.createServer(sslOpts, app).listen(app.get('port'), function(){
  console.log('Express server listening on port ' + app.get('port'));
});

In order to keep our app running at all times, we need to use a process manager. In this walkthrough, we will use PM2. Run the following command.

sudo npm install -g pm2

Then run:

sudo pm2 start app.js

PM2 will make sure your app is always online. To check the status and view the log files, you can run this command:

sudo pm2 show app

Now that we have a server running on our IP address on port 443, we can verify that it works by opening our domain name under HTTPS.

https://<your domain>/health 

You should see “Okay” printed on to the browser screen.

Now go back to your Facebook Messenger App, and point your webhook’s endpoint at the domain name of your server:

https://<your domain>/fb

This will then redirect all Messenger Bot traffic to your server running live on AWS.

Conclusion

Setting up your local environment for bot development is not easy. Now that you’ve set up a basic bot environment, you will be able to reuse this setup for other chatbots. You could update your API.ai configuration and deploy it in another Facebook app to create new chatbots. Unfortunately, that would be too much to cover in one blog post. With that said, I am working on at least one follow-up post that will walk through adding a database, configuring EC2 for scale, and adding complexity to the NLU with API.ai.

Part 2 of this tutorial, which deals with saving context and keeping track of data, can be found here.