How and why I built a real-time state synchronisation service

Realtime apps are everywhere. From instant messaging apps to document collaboration software and games. You may have built applications with it using open source library such as SignalR, SocketIO or some other commercial products. I’ve used SignalR and SocketIO, and most recently building a Chat SDK with SocketIO in the last ProductHunt hackathon.

The problem I wanted to solve

I and some colleagues have had to design state logic around real-time engines like SocketIO and SignalR in the past. Most of the time it's a project that needs a real-time dashboard. These implementations had similar logic or architecture and it felt like there was a repetitive effort to always designing from scratch. I wanted a way I can:

1. Add real-time state synchronization to any app with ease.
2. Avoid designing from scratch around SocketIO rooms and events, and handling what happens when a client disconnects and reconnects.
3. An SDK that I can integrate into mobile and web apps.
4. A service that handles the infrastructure concerns.

What I built

So I built Hamoni Sync, a real-time state synchronization as a service.

It provides SDK to add real-time state synchronization to a web or mobile. I made just the web SDK at the moment. It persists your application state in the cloud and can serve as the source of truth for your application. It re-synchronizes with the server when a client reconnects.

You can use it to synchronize your application state in real-time. You can build a real-time dashboard, manage game leaderboard or any other game state, and real-time data visualization or chart.

Tech stack

In building this I used ReactJS, NodeJS, RollUp, SocketIO, CouchDB, AWS Lambda, AWS API Gateway, Amazon Cognito, AWS Amplify, AWS ElasticBeanstalk, Gitbook, Git.

1. Source Control: I'm using Git and GitLab. I use GitLab because the free plan allows me to have private repositories.

2. Database: I needed a NoSQL and chose CouchDB because it has a conflict resolution strategy that I want to later use to handle automatic conflict resolution for this project. I'm using a managed CouchDB instance from IBM Cloudant because I don't want to manage the database myself.

3. Backend: There are a couple of open-source real-time engines and SocketIO seems to be more popular. I chose SocketIO because I'm familiar with SocketIO and it has client libraries for Swift and Java which I want to use in building SDKs in those languages so my solution can work on mobile. I'm also using AWS Lambda and API gateway to build APIs used by the dashboard. I assumed I'll have different scaling requirement for the real-time system and the API so I didn't want to have them on the same server. I separated them and used a serverless architecture because to save cost.

4. Frontend: I'm using React with Material Design components from MDB Bootstrap. I thought of using Preact but decided on React because AWS Amplify has an integration package for React and I need to use it to manage authentication logic for the web app. It may as well work with Preact but I didn't want to invest the time in trying to see if it works.

5. Authentication: I'm using Amazon Cognito and AWS Amplify. Amazon Cognito for the user account management and Amplify as a library to handle authentication in the React app. It was a breeze using these services.

6. Documentation: I'm building a service with an SDK so it needs to have good documentation. I asked the folks in WeAllJS slack group and they gave me options. One of them went ahead and made a list of tool to consider. I had to choose between Gitbook, Sphinx, and Slate. Sphinx required me installing tools I don't use and I didn't what to learn the reStructured text format at the moment. Gitbook and Slate support markdown which I'm familiar with. I used Gitbook because I like how it's displayed and allows me to apply themes. I think in my next project I'll try slate because it is easy to use and setup.

7. Cloud Provider: I'm using AWS because I have some credits left on my account. I'm also using IBM Cloudant as a managed database provider.

8. Hosting: I purchased the domain from get.tech and hosting the website, documentation, and dashboard using Netlify.

9. SDK Packaging: I wrote my code in ES6 and used Babel to transpile to ES5. I used RollUp to bundle it for use in browsers. I think the RollUp documentation makes it easy to get started with and it's best suited for library authors to package their library. I don't remember it being this easy for me the first time I tried Webpack.

The process of building

I started working on the project on the 1st of March with the goal of finishing everything and launch on March 23rd. That is a total of 20 days because I don't work on Sundays.

I started out with the dashboard. Bootstrapped the frontend with create-react-app and a Material design React library. Added Authentication with Amazon Cognito and AWS Amplify. Then I created the Lambda functions exposed via API gateway for creating and viewing applications.

Finishing that I moved to work on the JavaScript SDK and backend. I coded the first feature which allowed to create what I called a value primitive. It took about 8 hours to build that which then made it easier to make the rest of the target features. As at March 10, I already have the intended features working.

With the main features working, the next thing was to create a sample application and documentation to guide people on how to use the product. It took me a couple of days to finish the initial draft of the documentation. By March 17, I had finished the documentation and made 2 sample application. Writing code seemed to be the easy part. Writing documentation took slightly longer than I planned. With the progress at this point, what was left was deploying to production. I bought a domain and deployed the website and dashboard using Netlify. Then I finished off with deploying to AWS and publishing to npm.

Challenges I faced

The hard part of this journey was finding the right words to convey the intent of the project through the documentation and website. Also it was hard learning and using the AWS services I used for this project. AWS Cognito and Amplify weren't hard to use but the others were. Maybe the documentation was hard for me to follow.

Key learnings

The key learning was knowing how to use various AWS services. I also had to learn to use RollUp to package the JavaScript module and how to deploy to npm. If I would do anything differently it'll be using a different React Component for Material Design and probably using Slate for documentation.

Final thoughts and next steps

The journey has been fun and I've learned a lot. I'm looking forward to seeing how this project takes off. I will continue working on it because I have other features planned for it, for example, allow users see the current number of connections for an app. I am hoping to get feedback and see how people use it and what features they'd like to see added or removed.

Try it and give your app a real-time boost! Feel free to get in touch and let me know what you think.

Checkout this blog post to learn how to build a real-time voting app with it in JavaScript

Links

• Website 🚀
• Documentation
• Quickstart guide