How I learned Parallel Computing

About me

I'm Pasindu Tennage, a Computer Science and Engineering graduate from University of Moratuwa, Sri Lanka. Having excelled in academics while achieving the highest GPA of 4.13 / 4.2 in the Engineering faculty, I'm currently working as a full time research assistant in the university of Moratuwa.

Why I wanted to learn Parallel Computing

Parallel Computing has been one of my key interests since my undergraduate period. I have followed 3 courses on this domain; concurrent programming, distributed programming and many core / multi core programming. As a theoretical computer science enthusiastic, I was really impressed in the underlying theoretical and practical applications of parallel computing.

How I approached learning Parallel Computing

Unlike other Computer Science domains, where you can get into coding without having much of a theoretical background, parallel computing needs a very sound knowledge on the underlying theories. I first took three courses that are related to parallel computing domain during my Bsc degree; concurrent programming, distributed programming and many core/ multi core programming. Then I focused more on the semester projects and assignments to get the practical implications of this domain. After getting familiarized with the basics, I focused on doing big projects using the knowledge that I had gained.

Challenges I faced

Initially it took some time to get familiarized with the parallel systems design. In my opinion, one should change the way he looks at a problem when dealing with parallel systems design. Though it seems pretty straightforward to convert the serial program in to parallel program in some instances, it is not the case when it comes to more advanced problems. So it is necessarily to acquire the right method of designing parallel systems.

Unlike in serial programming, parallel programs are very hard to debug. In a multi threaded environment, where several threads run in parallel, it is a tedious task to isolate a single thread and trace back its execution.

Key takeaways

Under parallel programming I got familiarized with 3 novel technologies; pthreads, OpenMP and CUDA. Though they are different when it comes to implementation, the underlying concepts are almost similar. Hence having a sound theoretical background helped me a lot in adapting to these technologies.

I built a distributed file querying system that operated in peer to peer architecture. The project involves WebServices and UDP. I came up with a library called relyUDP which can be used to make UDP reliable.

Tips and advice

Fist you need to get a thorough understanding of the concepts in parallel computing. This should include knowledge about threads, synchronization, message passing, exception handling.

Create a sample project for each new technology you come across. Try out all the possibilities that are mentioned in the documentation and get your self familiarized.

Get your self familiarized with a debugger. GCC has a command line based debugger for debugging multi threaded applications.

Use tools like Helgrind and Valgrind to identify memory leaks, inconsistent memory accesses and race conditions.

Final thoughts and next steps

After getting familiarized with the basic knowledge and technologies, now its time to build a big system. Think about a good idea that has both learning value as well as financial value. Lets do it!!