How I Built a Tracking Tool that Monitors Websites While Moonlighting From My Day Job

If you want to build a web application from scratch, you might think that you need:

• a solid knowledge of software development and latest technologies
• previous experiences building something from scratch
• a lot time

My story of building Custodee, a tool to monitor competitors’ websites, will hopefully convince you otherwise. Before building Custodee, my IT knowledge was mostly about high-level strategy and architecture, and I had limited experience with hands-on coding. At that time, I was also working full-time as a consultant (50+ hours per week) and travelling extensively for my job. I didn’t have much time on my hands for a side project, to say the least.

Despite these challenges, I successfully launched my project on Product Hunt, and got 600+ users, including my first paid users.

I like to see Custodee as living proof that you don’t need a degree in computer science or prior experience in order to build something valuable. It’s never too late to start coding.

With Custodee, my goal was to learn by doing and to generate revenue by solving a real problem. I hope that my story will motivate you to start working on your own side project and raise your confidence that you can succeed, even if you’ve never coded before.

My background

My background is in the music industry, where I spent 5 years working on various projects, including a classical production that was nominated for a Grammy Award. I also used to play in a metal band that toured internationally. So much for software development. Anyhow, at some point I felt the need for a safety net — in case I ever wanted to do something other than music.

That’s why I decided to get a degree in business administration and information technology. Quickly, I realized that I really liked technology. I found it both exciting and easy to learn, and it became clear that this path could work out well for me. While getting my degree, I worked in the marketing department of an eCommerce company. This is where I started experimenting with automation as a way to solve problems more efficiently. One particular experiment set the stage for Custodee.

We wanted to know when our competitors ran out of stock, so we could quickly start high conversion Google AdWords campaigns to capture customers from them. My colleagues had created a spreadsheet with 800 websites that had to be monitored. My assignment was to do this manually, by clicking all 800 links. Urgh. I didn’t want to do this. Instead, I read about ways of automating the process. I ended up building a very simple crawler in Google Sheets. I wrote a simple JavaScript function that, for each of these 800 websites, would open the right webpage, go to a specific section of that particular webpage, and extract the information required. I was done in 20 minutes when I could have spent weeks going through every website. As you’ll soon see, this experience would turn out to be a key event for Custodee.

The idea: Monitoring competitors’ websites

Right after graduation, I joined a consulting company doing high-level strategy consulting on IT infrastructure and architecture for big insurance companies, corporations, and government projects. As a consultant, I had to travel a lot, jumping on planes early in the morning and flying back in the afternoon. The upside was that I had a lot of time to read. One particular website, Indiehackers, made me fall in love with the idea of building side projects on your own to solve real problems (and generate revenue). I started wondering: “Which problem can I solve?”

This is when I went back to the problem that I experienced in my previous job: monitoring competitors’ websites. I knew that this was an actual problem that really needed to be solved.

What I built: Custodee

Custodee is a competitor tracking tool that monitors website content and visual changes, and keeps a record of these changes. Features include daily automated website screenshots, keyword alerts, email notifications on website changes, plus excel and image exports. I’m also currently working on a Dropbox integration for the screenshots, and a feature that will allow users to paste JS code which will be executed before screenshots are taken.

Custodee can be used to monitor competitor products in order to adjust marketing campaigns. Specifically, companies can increase their sales when their competitors raise their prices, optimize Google Adwords conversions (by picking a strategic time for the campaign), track SEO with screenshots, and archive web page changes. Although Custodee was initially conceived for companies to monitor their competitors’ websites, there are other creative use cases. For instance, I’ve seen people using Custodee to record viewcount increases for specific YouTube videos over a period of time, or to monitor engagement on online forums. Funnily, I actually use Custodee to monitor comments on Custodee’s Product Hunt page and other platforms so I don’t need to check in everyday.

Technologies used

Another crucial question was, “How can I do it?” At this point, I only had a high-level understanding of programming. I had only written very basic JavaScript, so my knowledge of modern development tools and technologies was limited.

Everything I had learned at university was Java-based, which I hated. I felt like I had to write a thousand lines of code just to say “Hello World.” This technology was holding me back from GSD — getting shit done...quickly!

For this project, I decided to go with JavaScript because of two main reasons. First, I had used JavaScript before on the front-end and in building my basic crawler in Google Sheets. Second, I could use JavaScript on both the client-side and the server-side, which was quite convenient.

To pick specific technologies, I first researched current trends. I looked at popular development websites and tried to get a sense of what everyone was discussing. I could see a hype around Node.js, which naturally drew me to it. Since I was going to work with technologies I’d never used before, I decided to pick ones that were amongst the most sought after by companies. The rationale was the following: even if the project failed, I’d still be able to “sell” the new knowledge I had acquired to clients.

To build Custodee, I ended up using the following:

• Node.js, MongoDB, PhantomJS, AngularJS, AWS
• Third-party APIs → Dropbox, Drive for Premium

Here’s what Custodee’s architecture looks like:

Notes:

• There can be many more back-end servers, depending on the traffic (hence the +n).
• This is the reason why there’s the AWS ELB (load balancer) — it routes the traffic to the back-end servers, sharing the load between them.
• The website is on the front-end server with Node.js and AngularJS.
• The Node.js application on the front-end server has two purposes:
  • Importing and saving all the images as a REST API (the images are sent from the back-end servers)
  • Pushing images of premium users to their Dropbox

Building Custodee

In order to build Custodee, I had to set a realistic deadline for myself. I already knew that I couldn’t build the MVP in two weeks. Not only because I didn’t have much experience beyond the architecture side of programming, but also because I was super busy with my full-time job.

Throughout the building process, I worked on everything at the same time — switching between the following:

• Creating the front-end / design with AngularJS
• Building the models for the MongoDB database
• Writing the Node.js back-end to get results from the crawler
• Optimizing the backend to use minimal CPU and RAM so I can go with small EC2 Servers

Before getting started, I took notes of everything the app would require. I knew I wanted to create a simple, user-friendly interface. So I started by looking for HTML templates for the front-end. I did not want to design everything from scratch on my own because I’m not a designer and there are people out there than can do this better than me, and I wanted to focus on solving the problem that my app would solve. I still ended up switching and refactoring templates three times. That wasn’t the only thing I had to rewrite. When building the crawler part on Node.js, I also had to rewrite it three times, mainly because my initial approach wasn’t the best resource-wise — it used too much RAM and CPU.

There were more hurdles I had to get past. Getting familiar with MongoDB took some time. Querying the database with MongoDB was completely new to me and quite different from the SQL technologies I had used before. Setting up the servers on AWS took some extra stress and time as well. Still, most of the work was spent on making the back-end with Node and PhantomJS.

The first version of Custodee had no screenshots. I then thought that it might be very useful to include screenshots, instead of simply monitoring the data. In order to add screenshots, I had to work with a headless browser. I decided to go with PhantomJS, because it allowed me to get full page screenshots. After implementing this, I thought it would also be great to single out the changes (instead of just letting users compare screenshots by themselves.) In order to do this, I used an NPM package: resemble.

Overall, I have to say that my process of building Custodee wasn’t very lean. I first thought of all the features that users might need, and then just went and built them. That probably caused me to spend extra time on the project.Then again, I learned so much, so this was best return ever. I ended up removing many features later on in order to make Custodee simpler.

Technical challenges & learnings

Global variables

Very early on, I learned about global variables in Node.js the hard way. In JavaScript, if you forget to declare a variable properly (even just once!), it will go global. This happened to me with the user ID variable. Some friends were testing Custodee, and every time one of them logged in, all the user IDs were overwritten with the latest user ID. So everybody was becoming the same user after refreshing their page. That was my first (funny!) mistake. It was fixed quickly, but we all had a good laugh.

Memory leak

I also wasn’t familiar with memory leaks in Node.js. I was wondering why my server was becoming really slow all the time. That problem was related to PhantomJS. PhantomJS isn’t very new, and it’s also quite slow and eats up a lot of server resources. There was a memory leak, and I couldn’t figure out where. I had to come up with a way of restarting the server every 2 hours so that it wouldn’t become too slow. I had to write a lot of code to optimize this to make it more efficient, so as to not waste too much CPU. Optimizing the whole thing to lower server costs took 2-3 weeks. I know there are other substitute technologies (like Headless Chrome), but using PhantomJS is still the best way to capture real full-screen screenshots without faking them.

Undefined variables

Undefined variables aren't so much of a problem in Java, which I’m currently using in my freelancing projects, but in Node.js, in the rare scenario where you haven’t implemented tests, if there’s an undefined variable, your Node.js application crashes. This is actually a little stupid. If there’s something undefined in your project, your Node.js application will go down. Although it isn’t a huge deal because your server will restart in seconds, your user sessions will all be killed, and you’ll have to log in again. All this is to say that if you use Node.js, make sure to catch everything. If you’re working full-time on a project with a team and proper testing, you should be fine. But if you want to develop an MVP really quick like I did, this becomes really important.

AWS bills

I made the mistake of setting up AWS without reading the documentation properly. As a result, I got a pretty expensive invoice from Amazon the following month. Some advice: use something cheaper, or read the AWS documentation thoroughly. Also, keep in mind that your architecture might change. Mine has changed since the first version of Custodee because of the high number of users that have started using Custodee. At the very beginning, I only had one front-end / DB server, and one back-end server for PhantomJS.

Showing my code

Since I was totally new to all of these technologies, I relied extensively on Stack Overflow. I also talked to people who built Node.js modules. At some point, I even showed my code to two friends who had experience in building enterprise level applications, but this turned out not to be particularly helpful. They laughed at me, and simply told me, “Nobody can read this.” Although they were probably right, I didn’t think it made sense for me to write enterprise-level code. If you want to develop a product on your own, which perhaps nobody will be using after 2 years, that’s just a waste of time. Nobody will pay you because your source code is super clean and full of comments. You’ll be better off building your project quickly, getting it out fast, and improving the next version after getting feedback from users.

Naturally, different settings call for different working methods. I work differently in my consulting projects. When working with clients, everything needs to be built in a very clean way and in the style of the company (e.g. reusing their models, abstract methods, comments etc.). Working with multiple scrum teams on a big enterprise project is very different from building an MVP very fast as a side project.

In short, I used quite a few quick and dirty tricks when building Custodee. But that was inevitable since I was working 50+ hours per week. Again, the goal was to come up with something fast and to learn by doing. I know many developers who agree with this mindset but keep working on the same side projects, which they try to build like other things they build on their day job. The issue with this approach is that it takes forever, and it also often fails to get something out at all because developers will get stuck in the process.

Launch & feedback

I first spent 50-100 euros on Google Adwords, just to get users and quick feedback. At the same time, I implemented a script from Tamboo, a service which records user behavior on your website and shows you what they are doing. This helped me check if my MVP was working. It also showed me how users used Custodee, which features they liked, and which ones they didn’t use. I was then able to optimize the UX accordingly. After that, I moved on to other projects. I continued using Custodee myself and saw that a few users (10-12) were using it, but nothing was happening beyond that.

Three months later, I started considering getting rid of Custodee. I was going through a phase where I didn’t like it anymore. I’m not sure why, but I just thought Custodee wasn’t good at all. At that time, I had started talking with Clifford Oravec from Tamboo, who also wrote The Epic Guide to Bootstrapping a Startup by Yourself. I had listened to his podcast interview on Indiehackers, and contacted him via email after reading his post, which I found super inspiring. We started chatting via email and Twitter, and I remember telling him, “I think Custodee sucks, I’ll just put it offline and start working on something else instead.” He advised not to do that. He told me to get it out to people instead, and wait a few months before throwing it away. Listening to him, I changed my mind and decided to launch Custodee on Product Hunt to get more feedback. I wanted to know if there was anyone who would actually use Custodee. If not, I could always put it offline at a later stage, and move on to another project.

I was not at all expecting the success and positive response Custodee found on Product Hunt. People liked it. I got 600+ users and even my first paying customer. I noticed that many accounts were from companies that had started using Custodee. Interestingly, it happened a few time that someone from a company would sign up, and shortly after (a few days, and sometimes even just minutes later) more employees from that company would sign up. I found it really cool. It meant that Custodee was helpful for them, and they were referring it to each other at the office.

There were also people writing about Custodee. The most unexpected article was published on a Korean blog. I also found posts on French and English tech blogs.

After the launch, every time I was working on my computer, I kept the customer support chat open. Each time a new user would sign in, I’d chat with them. This helped me better understand what my users were looking for. I’d also check where these users came from and look at how they were using the product. I would think about the features they didn’t seem to know or like and how Custodee could be improved.

Conclusion

Building Custodee was wild ride, to say the least. I’m very happy because I managed to fulfill the goals that I set for myself with this project. I learned how to code by building a side project. The best part is that Custodee is actually solving an existing problem and generating revenue. Even if it doesn’t end up being a huge success (or if it fails), what matters to me is that I learned by building it from scratch. As a result, I’ve been able to increase my profile as a developer, the value that I can provide as a freelancer, and therefore my rate as well.

Right now, I’m still putting a lot of energy into Custodee. I’m on a short break, and I’m still learning a lot - not just about coding, but about how to develop useful features for people. I’m also learning more about getting people to pay for a SaaS and how to run such a product. My plan is to add a few more features and make Custodee stable, so that I can move on to my next project. I already have a lot of ideas (about 50!), which I keep in an Excel file. Spoiler: I’ll be launching another website on Product Hunt in 2-3 weeks from now.

I’ve quit my job as consultant, and I’ve started working on my own, focusing on IT architecture and development. Thanks to very positive references from my previous consulting jobs, I can now live off 3-4 freelancing 3-4 days per week and spend more time on my own projects.

Key learnings and advice

Go and do something, stop reading, stop talking, do it now

Stop reading The Lean Startup,interviews of entrepreneurs and blog posts, and stop going to conferences. All of these things are very helpful, motivating, and inspiring, but this alone will not get you anywhere. Starting is the best thing you can do. At some point, you’ll have to act, and better now than later. Start now.

You don’t necessarily need to think about scaling at first

My friends laughed at my code because it was unreadable, and said that Custodee wasn’t scalable. I just thought: “I don’t have any employees. Why should I refactor everything just so my friends can read it?” Caring too much about building something scalable at the beginning is a waste of time. It is not a guarantee that anybody will use your app/service anyway. The best thing to do is to get your product out to users as soon as possible.

Focus

I wasted a lot of time building features which no one ended up using. Before the Product Hunt launch, I decided to throw away all the features that less than 80% of users used. I even scraped features that had required the most development time. Focusing your feature offering is important because if you flood your app or website with tons of features, nobody will understand your product. You’ll just be distracting users from your product’s real value.

Plan for both failure and success

I wasn’t expecting so many upvotes on Product Hunt. I had tried to cost-optimize everything, and as a consequence my servers were really small. This turned out to be a problem. A couple days after the launch, there was a lot of traffic on Custodee. Therefore, I had to rewrite some code on the back-end. I also had to set up new servers. Bottom line: if you launch on Product Hunt (or somewhere else), don’t just plan for failure, plan for success. Try to also think about what you’ll do in case you do get traffic and people actually use your product.

Monetizing

While it’s not necessary to monetize from day 1, you should have a clear path towards monetization and try to reach it as quickly as possible. I had to change the free plan into a 2-week test phase because more and more users started to sign up a daily basis. Sticking with my original plan would have increased my server costs dramatically.

Don’t give up, keep going, and get your product out to people - even if you think it sucks

Right before the Product Hunt launch, I was that close to throwing Custodee away. I’m glad I decided to get it out to more people instead. I would have missed out on so much learning and feedback had I decided to kill Custodee and not launch on Product Hunt.

You can learn everything really quickly

Many people wouldn’t think that you can learn coding in such a short time and build an app like Custodee, let alone do all of this in addition to your day job. Here’s what you need to know: if you really want to do build something, you can. Nowadays, information is everywhere. You don’t have to go to university in order to learn how to code. The best thing you can do is to make your learning experience a practical one. Build something to address a problem that you want to solve. This will make you think about the problem in a different way, and seek out creative solutions. You’ll also have more fun, and your learning experience will only be even better.

That’s it from me. I would really love to hear any feedback you have, even if you think that Custodee and my approach towards learning and building fast are the worst thing ever.

Please let me know what you think in the comment section 👇 or on Twitter.

Don’t forget to check out the project on Product Hunt or try it out on Custodee.com.

I will soon officially launch more projects. Stay tuned for the latest news about clubpal.io and soundcollab.com