A Comprehensive Guide to Handling Exceptions in Python

No one wants their code throwing errors, but exceptions in Python can have a whole variety of use cases and are critically important to writing good code.
This article describes some examples of what to do and not do when writing exceptions. Hopefully, it’ll instill in you a deeper understanding of exception handling broadly as well provide some useful Python tips.

Do Write Lots of Exceptions

• AttemptedToAccessCoreDatabasePriorToDatabaseTransactionInitiatedException
• JavascriptMillisecondTimeFormatBufferOverflowException
• CannotDeleteUserAccountThatDoesntExistException

Exceptions, exceptions, exceptions everywhere. These names look challenging to read and are long to type, but are they really so scary?
These exceptions provide the most valuable function exceptionally well — they’re specific, informative, and to the point.
OK, the last one could probably be renamed to CannotDeleteNonExistentUser, sure, but the point is they’re clear. You know where to go to find the issue, and you know exactly what to try catch if you’re OK with attempting to delete a user that’s already been deleted.
There was once a ti
me before Atom, Visual Studio, and IntelliJ, in the land of Vi and Nano, where variable name length actually affected developer productivity.
I learnt to code in C, using Vim, with no autocomplete. And, yes, I used to name things int num or char* s1. Sure, I was lazy. Sure, I was a young student. But also, I had to type all three of those characters in num. These days, I’ll be lucky to type more than two before my IDE realises what I’m trying to do and finishes the job.

Long names no longer slow down development, but they do significantly benefit anyone debugging or reading the code. That’s right — exceptions aren’t just for debugging or handling. They also help people understand what’s going on. Take the following example.

Confusing API responses being translated into a clear exception

Some providers just want to watch the whole world burn.
We could definitely refactor this to make the response codes an enum or a similarly expressive format, but there’s no confusion in this code that whatever those random response codes are, they mean that we don’t have 2-factor authentication enabled and we need to configure this setting in our example provider to get this to work.

Rule of thumb: Whenever you have information about a specific, replicable edge case, use it.

But why bother with exceptions at all? We use asserts in testing all the time, and they do a great job. Why not just use them instead of exceptions?

Don’t ‘assert’ Unless You’re in a Test

assert and raise appear to function similarly. Both stop the control flow, both can terminate the program, and both can log/print a message explaining why.

At first, it might seem tempting to use asserts to confirm everything is in a valid state, but this is considered bad practice in enterprise Python development. There are several reasons why, and they could be an article of their own.

But to keep it short: You can customise exception handling and exception details, and exception raises will never be “optimised” out of your code.

Rule of thumb: You should only assert impossible conditions, such as asserting that a value you’ve just squared isn’t negative (assuming you’re not modelling quantum physics or something equally wild). If there’s even a remote possibility the assertion might fail, then it should be replaced by an exception.

A classic example where people often make this mistake is when dealing with third-party providers.

Say you are calling a Yahoo Weather API within your app, and you’ve decided to add in an assert that response is not None. You chose an assert because Yahoo always returns the weather (though the predictions might not always be correct).
But what happens when Yahoo’s Weather API experiences a disruption? Suddenly, all of your services are failing, and all you have to work with is AssertionError: response is None.

Now I’m sure Yahoo Weather is probably very reliable, and you probably have line numbers on your logs. But that’s no reason not to spend the one minute of extra development time and create a WeatherProviderUnresponsiveException (“naming things is hard … [give this writer a break]” — Phil Karlton).

OK, so we should use lots of exceptions, but which exceptions should we use? Before we dive into that, we need to understand a crucial concept in exception handling (and programming in general).

Quick Tangent — What’s Type Inheritance?

Let’s start with the basics here because there’s a lot to unpack.

Firstly, inheritance. When a class in Python inherits from another class, it takes on all the methods and attributes, but it also takes on the type of the parent class — that is, in the world of Python, I am myself, but I’m also my father and my mother. Even if you override methods or attributes of parent classes, you’ll still keep the type inheritance. Let’s look at some code to explain.

Just one note before diving in: Classes aren’t exactly types, but they kind of are. OK, now that you’re more confused than ever, let’s look at some code:

A script to explain type inheritance

We’ve defined a few custom types here, all inheriting from the inbuilt dict class. Take RedDict, for example, which has the type RedDict but also dict.

isinstance(red_map, dict) # True
isinstance(red_map, RedDict) # True

However, the reverse isn’t true — in that RedDict isn’t a PurpleDict.

isinstance(red_map, PurpleDict) # False

Run the above snippet to see the results for all the classes above.

You can objectify anything you want in Python since everything is an object

You can objectify anything you want in Python since everything is an object
OK, now let’s get back to where we were …

try:
segway_about_types()
catch OnATangentError, Exception as e:
smooth_transition(e)

Don’t Catch ‘Exception as e’

Catching all exceptions and throwing them away is the second most effective way to have bug-free code (the first is to remove all code). So surely that means it’s good, right?

Catching every exception in the program to make it bug free

The issue with catching Exception is related to type inheritance (hence, the interlude) because we won’t just be catching all custom exceptions in your application but a whole heap of Python inbuilt exceptions as well, including some you might not want to be throwing away.

Python inbuilt exceptions have a complex inheritance structure. Here’s the latest list from the docs, where each indentation means inheritance.

Don’t you wish Python3 docs had pretty diagrams?

This hierarchy is for a good reason so you can use type inheritance to be clever about how you catch exceptions (see the next point).

What we just learnt about type inheritance tells us this means a TypeError is also an Exception (its parent), so TypeErrors will get caught when we catch Exception. This is probably fine.

But what about ModuleNotFoundError? Do you really want your program to keep running if it’s missing a dependency entirely? What about MemoryError? Surely, you don’t want to turn your back when Python is suffocating in your memory card.

Not only will you be catching all these wild and wonderful inbuilt exceptions but also every custom exception (yes … except those that derive from BaseException rather than Exception).

Is that really what you wanted to do? Perhaps, the solution is to catch multiple, specific custom exceptions (which you do through tuples) like so:

catch (FileNotFoundError, IsADirectoryError, PermissionError) as e:

This will safely catch a FileNotFoundError but won’t catch a more dangerous OSError, such as ChildProcessError.

Of course, there are scenarios where catching all exceptions is what you want to do, but they’re few and far between. It’s also important to note that how you handle the exception is important here. If you catch all exceptions but then raise that exception again or use logger.exception(), this isn’t an issue.

Some examples when you might want to catch all exceptions:
-In pulling from a queue and handling one message at a time, you might use logger.exception() to log the issue without breaking the flow
-As part of chaos-engineering practices at the service level, particularly for async services, you might catch all exceptions, safely close everything, and then raise or log the exceptions
-Web scraping or crawling links is a dirty task, and often throws all kinds of errors — in some cases, this requires very broad exception handling.

Rule of thumb: Catch as specific of an error as you can. Don’t catch Exception as e unless you know exactly what you’re doing.

I did mention the inbuilt exception hierarchy is useful. Let’s take a look at how exceptions can work for us, not against us.

Do Use Inbuilt Exceptions When It’s Sensible

This may sound like a counterargument to the earlier do about writing lots of exceptions, but sometimes it makes sense to use a simple inbuilt Python exception like ValueError or TypeError.

Let’s say you’re making a car that can be either electric-powered, petrol-powered, or hybrid, and you want to pass two booleans to create a car that specifies the engine type.

A python electric car class, with two default boolean variables raising a ValueError

This is a perfect example where ValueError is suitable. You can’t define the rule that at least one of electric or petrols must be true in your function definition (RIP overriding functions), so you’ll need to check this manually.

Note: This is a contrived example that I’ve created to prove a point — don’t use boolean flags like this.

This is a great opportunity to use the inbuilt ValueError, as many programs may try catch this error and use it. For example, a program to automatically run parametric tests might run through every combination of true, false, and catch and skip any that return ValueErrors.

However, I like to go one step further. We can use type inheritance in our favour to add readability to our code by inheriting our exceptions from a more specific exception like ValueError. That way, any program that catches ValueError will also catch our exception, but we can add custom code and names.

The same example as above but with a custom exception defined

Must … write … more … exceptions.

Rule of thumb: Always inherit from as specific of an inbuild exception as you can. If you don’t have any additional information to add, just use the inbuilt exception.

For the full list of inbuilt Python exceptions and their hierarchy, see the official docs.

Don’t Put Sensitive Data in Exception Messages

I want a secure system, so it makes sense I raise CommonPasswordException(f"password: {password} is too common"). (Unsure what that f means? Check out my article on it).

It’s a good exception — it’s clear what the exception means, I’ve provided adequate information with it, and it’s specific enough it could be safely wrapped in a try-catch and handled differently if, for example, we were more relaxed with admin passwords than users.

The issue here comes down to sensitive data. I’ve used an unfair example here, so I hope it’s blindingly obvious to you that this isn’t a good exception. This exception will be spewing raw-text passwords through your logs, in responses, into your monitoring software, and, perhaps, into unsavoury hands.

Exceptions for internal use (see note at the bottom about client-facing exceptions) can contain technical details, such as user_ids or specific data that caused the crash, but it’s important to remember when an exception occurs, these messages will be spread far and wide, through logging, reporting, and monitoring software — and, if you’re not careful, possibly to your users.

Of course, this all comes down to good software design, but in a world where regulation around personal data is constantly getting stricter, you can never be too careful, and it’s almost always possible to provide valuable error messages without compromising customer privacy.

Rule of thumb: Don’t use sensitive information in exception messages.

It’s not only customer privacy you need to worry about, though. Bad actors are everywhere.

trying to secure your customers data, protect your servers from hackers all while building features on time

Let’s say you run a website where users supply a large number, and you calculate the factors of this number. Let’s call your website FactoriseMe.com.

You’re about to raise your seed investment, but a competitor pops up in the market, NumbersWithinNumbers.com. You’ve tested their product, and while it works, it’s not as fast as yours. And the customer experience is far worse.

One of your developers notices that for very large numbers, your back-end service struggles to compute the factors — in fact, for very, very large numbers, the service spends the full 180 seconds number crunching and just times out.

You decide to continue to improve your customer experience by responding with a nice error for the customer: “Calculator timed out while computing factors. Sorry, these ones are hard.”

This is great. Now the customer knows why the website failed to find any factors. You walk into your investor to give a demo, and suddenly your website is down. What happened?

After looking through the logs, you notice that between 9 a.m. and 10 a.m., you received 1,000 requests with huge numbers from an IP just down the road, not far from the NumbersWithinNumbers HQ. These requests overloaded your back-end services and crashed your website. How?

You revealed your weakness. Never reveal your weakness. Or at least never reveal weaknesses in the internal workings of your software to users. Hackers, competitors, trolls, and the internet at large is full of people who want to break what you’ve built. Yes, you should give your users feedback about what’s going on, but never, ever tell them about the internal workings of your software.

Rule of thumb: Tell the user about what they can do, not what happened.

You’ll see this is common in many applications: “Please try again later.” “If this happens again please contact support.” “Unknown error — we’re sorry, we’re looking into it.”

But this whole thing is getting off topic now, User-focused exceptions are a whole new can of worms. Just remember to take a leaf out of the knight from Monty Python’s book, and pretend that everything is OK.

Don’t Be Inconsistent With the Strictness of Your Code

All these dos and don’t’s are great in theory, but in the real world, you have to deal with other teams writing services with all kinds of exceptions.

Say Team A is the innovation team — they’re mavericks, untamable by your by-the-books VP of engineering. They decide their code has a radical exception policy of throwing exceptions at the slightest change (in the next point, you’ll see why this isn’t so radical at all). In Team B, on the other hand, you inherit all exceptions from a TeamBException and use them scarcely and only to denote serious issues.

Eventually, you have to work on a project using a library built by Team A and find that you can’t so much as rely on a simple print function without all sorts of strange exceptions shutting down your service. You’re on a deadline, and you realise the only way you can finish your task is to wrap all interfaces with the library with broad try-catch Exception as eclauses and hope for the best. Not ideal.

This is why it’s crucial that your whole codebase contains consistent exception use and handling. Your VP of engineering and tech leads (and everyone) should be pushing for consistent principles and practices when it comes to exception handling because inconsistencies can quickly build to become significant technical debt.

Rule of thumb: Be an evangelist for good exception handling within your company.
If your company has a policy on exceptions, follow it. If it doesn’t, why not write one based on this article?

Do Catch Lots of Specific Exceptions

Python is a beautiful language — it’s incredibly popular, has a ton of great resources for learning, has libraries for everything, and can do anything from machine learning to hardware to web development.

Perhaps this is why you chose to learn Python, but for many, we chose Python because we believe in the values of Python. If this is sounding culty, that’s because it is. Go to your terminal, and run Python. Then type import this. You’ll see the manifesto of Python, the 19 (or 20) aphorisms of the Python language, as written by its creator, Tim Peters.

The 19 aphorisms in the manifesto of Python - "The Zen of Python" by Tim Peters

This does have value beyond entertainment, as it provides an insight into the mission behind Python and gives us tips on how it was intended to be used. The important line in this case is:

“Errors should never pass silently. Unless explicitly silenced.”

I hope this has been made clear in the above points. I read this as: “Don’t catch all exceptions unless you really mean to, and catch explicit exceptions often.”
Another Pythonic idiom in the community is:

“Ask for forgiveness, not permission.”

This is where Python differs dramatically to the traditional paradigms of C, Java, and other traditional languages. The Python way to handle processes with exceptions is to just go for it and handle the exceptions if they come up.

In C, you might write lines and lines of code to check for all preconditions before writing to a file — such as, does the file exist, does the process have write permissions to the file, etc. It’d be bad practice in C to disregard all these cases and just write directly to the file.

In Python, it’s quite the opposite. In fact, it’s considerably more efficient in some cases to try first and handle exceptions later. In the above example, while it’s entirely possible the file permissions are wrong (or some other issue), it’s an edge case. A majority of the time, you’ll be writing to a file that exists and you have permission to access, so why waste valuable processing power checking for file permissions?

The Pythonic way to handle this is to wrap the write in a try-catch and catch only the specific exceptions/edge cases that are likely to come up — and to handle them appropriately. In most cases, the write will succeed, and any logic in the catch block will be missed entirely. In the case where something does go wrong, we still handle it.

Check out this Stack Overflow thread for a few more examples.

Rule of thumb: Don’t spend excessive amounts of time checking for preconditions when you can predict and catch specific exceptions instead (i.e., ask for forgiveness, not permission).

But What About Client-Facing Exceptions?

Client exceptions are great — but very different.

Part 2 of this guide will cover how client-facing exceptions differ from internal exceptions and will include some tips on architecting a simple and effective exception structure. Follow me on codementor, medium or subscribe to my substack to stay tuned for part 2.