Programming For Personal Benefits

See original article here, on Hackernoon.

Disclaimer: This post doesn’t aim to focus on any ethical aspects regarding programming. It doesn’t aim to be educational nor inspirational. This is merely an exemplary story of how programming can be a useful skill for simplifying daily life, along with a display of gratitude I have for my line of work.

At the university where I received my bachelor’s degree, the internal web system has considerable performance problems which peak during the course registration periods. In the beginning of each academic semester, students get up early and try to log in to the registration system. As the system can’t handle all students at once, only a portion of fortunate students get to log in and register to their courses.

For an average student, the case is that the course they would like to enroll in is already full by the time they are able to log in. This happens mostly because either the course’s capacity is quite limited, the student isn’t allowed to take the course on that day because of their major degree, or people who have already taken the course are saving a spot for their friends who weren’t able to log in yet. Only when the number of login attempts decrease does the system function as expected.

This is probably a standard issue for most schools with an underperforming registration system. The problem, however, is that not being able to register to a course can get in the way of students who are trying to register for higher level courses, or even worse, those who are trying to graduate. These people end up having to enroll in summer courses, or extend their education by at least another term.

A couple of nights ago, I received a message from my sister, who’s a freshman at the same university. She informed me that a student had created an application that follows how many spots are available for a course and sends an e-mail to students who would like to be notified of any openings. It would not solve the issue of not being able to log in. However, when put into practice, people would be instantly notified of the available spots and possibly register to their courses, allowing them to finish their education on time. All the developer needed to make this happen was a list of the course registration numbers, along with the student’s e-mail address.
Certainly a pretty nice thing to do for your fellow students.
…
Right? Naaaah.
Apparently, he requested 50 Turkish Liras (50 TRY is roughly 10$ as of the time this article is written) for each course the student wants to follow. For instance, if a student wants to follow the seat availability of a Psychology course, a Mathematics course and the Applied Mathematics section that is a requisite to the Mathematics course, (s)he needs to pay 30$ to the developer, which is a considerable amount of money in the current Turkish economy.
Even worse, there is no guarantee that there will be an opening in that course. If no one drops the course, tough luck. ¯\(ツ)/¯

Although I respect the time and effort put into creating the application, as well as the decision to offer this as a profitable service, I feel that charging 10$ per course is nothing but exploiting people who don’t really have a choice but to pay the fee just to be able to register to some courses.

Needless to say, I was pretty irritated when my sister asked me if I could help her pay the developer for 3+ courses’ worth. I was having a calm Saturday evening; I figured I would take a look at how one would go about building such an application: how hard could it be, really?

There were a couple of key things I would have to look into:

1. Understanding how to get the course seating information from the course registration website
2. Being able to check for the availability information periodically (i.e. every x minutes)
3. Sending informative e-mails to students, regarding a course’s availability

As I had used the same registration system for several years during my bachelors, I was already acquainted with where to find the information I needed: the total course capacity, number of people that are already registered, and the number of remaining available seats. After jumping around a couple of links, it turned out to be manageable with simple web scraping.

The only challenge regarding this was to investigate how the HTML for the course pages was laid out. After getting my hands dirty for a while, I was able to scrape the necessary information using Node.js, along with the npm packages cheerio and request-promise. This process was a little ugly, but I was able to summon my strength before giving into the Impostor Syndrome, thanks to these wise words from the Spider-man franchise:

With awful HTML comes awful document queries.
Ben Parker

After I was able to receive the necessary information, I had to make sure it could be handled periodically. Registering to courses with high demand is a race against the clock, so the more frequently we checked to see if there is an opening, the better.

A quick Google search on how I could do that led me to the concept of a cron job. Simply put, it is a utility used to schedule desired commands to run at specific times/intervals. I’ll list a couple of resources at the end of the article, to help learn more about cron jobs and delicacies in writing the commands to run them.

To add the script as a cron job, in my Ubuntu terminal I ran

crontab -e

…and I added the following lines so the script runs every minute (note that # is for commenting a line)

# Uncomment the line below before registration starts, so the script can run every minute
# */1 * * * * node ~/Desktop/Code/robin/crawl.js

The crawl.js script basically loads a list of CRN’s, along with e-mail addresses to notify.

The list of Course Registration Numbers (CRNs)

After crawling the corresponding course’s website, the number of total seats, taken seats and available seats are stored for further comparisons. In the next minute, the script checks whether there is any difference compared to the previous minute. If there are no changes, the student is not notified (or rather spammed).
The last step I had to take was to see how I could send e-mails from my node script. I previously had exposure to node-mailer, so after creating a Gmail account to send the e-mails from, I went ahead and used the package as can be seen in its documentation.

Forming the e-mail object that will be passed onto nodemailer

With the addition of a simple caching mechanism so that I would only be sending the e-mails if there was an actual change from the last minute it was checked, the whole thing took me about 4–5 hours to build.

On Monday morning, the moment of truth had arrived. I had asked for a list of e-mail addresses and course registration numbers prior to the official start time of the course registration period. One minute before the beginning, I uncommented the line that initiated the periodical check.

By the following day, thanks to the e-mail notifications, my sister was able to register to 5 courses, enabling her to not only register to courses which serve as core prerequisites to her major degree, but also to change her schedule as she pleased. Based on this successful execution, I felt further motivated to take the next step for this application, which is making it public and sharing it with the students for free.

This is part of what I truly love about being a software developer. I encountered something that I thought was problematic, or that could be handled better. I did some research, put something very basic together, and watched it do its job.

In just about 3 years of professional experience, I found the opportunity to undertake many toy projects alongside my full-time job(s). Most of the toy projects I tackled came out of immediate necessity, just as in this story. Most of them weren’t even too challenging; everything boiled down to doing research and putting in the effort to pick up good practices along the way.

Through building these projects I’ve learned different technologies and design patterns, but the quality I appreciate the most that I’ve developed is investigating deeper and asking better questions. Whether that makes me a better software developer or not is arguable, however I feel that these qualities affect my daily life for the better.

I enjoy the fact that I can practice my craft out of curiosity or just for fun, and have it return positive value for me or others.

I love failing at what I do. I’ve mostly made my peace with the fact that the more I fail, the less I do.

I love that I can summon my determination just out of a spark of curiosity, and I love that I can make a living out of it.

These and many other motivating factors help me confidently encourage people to get started with programming.

I love my job.

Thank you for reading my (first) article!
As promised, below are some links I found useful when looking into cron.
https://cron-job.org/en/ A service for serving free cron job executions
A good article to learn more about the concept can be found here.
https://crontab.guru/ A website helping to visualize the output of your cron command