6 Useful Python Libraries Recommended by #1 Stack Overflow Answerer

Codementor Python expert and Stack Overflow legend Martijn Pieters sat down with us during Office Hours and shared some of the less-well known libraries that he likes to use for web development or daily tasks.

The text below is a summary done by the Codementor team and may vary from the original video and if you see any issues, please let us know!

Requests: HTTP for Humans

I am a huge fan of the requests library. If you have to do anything with http requests to other servers, this library goes straight past your urllib that comes with Python, and install requests. It is a fantastic tool and really has a much more intuitive API, which makes handling web requests much easier.

Beautiful Soup

Another great tool for web scraping is Beautiful Soup. When you’re web scraping and looking at web pages to extract information from, you should check this tool out. Beautiful Soup is a great html parsing library that makes and sends off the html “soup”.

Robobrowser

I recently came across a new library that marries the aforementioned two tools—Robobrowser. It is based on the requests library and beautiful soup to replace an older tool called mechanize, which does similar things. Namely, the Robobrowser lets you simulate a browser that goes out to the web, fetches html pages, lets you fill in forms, and submits those. Therefore, if you have to crawl a web page that requires forms or follow links to get to the data you want to get to and extract, take a look at Robobrowser, which can be also found on GitHub. Though it’s a relatively young project, I like the way the API works and the tool it uses.

Autobahn

On the server side of web development, Autobahn has piqued my interest. This tool makes programming web sockets easy, which is very helpful to me as a web developer. It’s also interesting as it makes use of Python 3.4’s new library.

Flake 8 Lint

For daily development, I like to use flake8 lint, which is based on Flake8. This Sublime Text plugin combines Pep8, which is for the Python style guide testing, together with Pyflakes, which detects common errors in your code (e.g. imports you forgot to add or imports you have too many of). Flake8 makes writing code much easier for me, and it integrates quite nicely with modern IDEs. I use sublime text and I have a plugin that puts flake8 error messages right at my fingertips. This is not a new project, but it is certainly something I use every day.

Scrapy

I haven’t actually come across a project where I use Scrapy, but to my knowledge Scrapy is excellent at extracting patterns for specific data. For example, if you want to scrape the latest football scores of Britain’s major leagues, you’d have to go to multiple sites and pages to extract the same information. Scrapy is good for such a task, as it can automate searching and walk through a site to extract very structured data you can specify through expressions. I’d love to have a project for me to use Scrapy that but I haven’t had the chance yet.