Generator is a powerful feature in Python and helps you write better and organized code.

WHY I use them

Processing sequential data means cleaning, filtering, transforming data for the next process, or storing into a database. A list comes handy to implement such actions.

List a typical and general purpose data structure in Python. I use list often, but it’s not a brilliant choice for volatile data size. For example, a list may have 5, 100, 10,000 or 5,00,0000 items or even more.

def scale_to_log10(till_number):

 log10_number_seq = []

 for num in range(1, till_number):

     log10_number_seq.append(math.log10(num))

 return log10_number_seq

The code above is just nice and will give the expected output, If at all .

If the argument ‘till_number’, passed into the range function, is high. Which may cause it to create a list in the memory, which is by loading data into it. And if your program is lucky enough to be running in an eternal memory space, then everything will be fine. Now imagine this, the size of the list generated by the function range is 4GB and the memory size is 2GB, ouch. This will make the process sluggish and may crash with out-of-memory error, and they will assume you are not a good developer .

As a good, if not the great, developer we need to make sure that the function we are writing works as expected without abusing physical resources named random access memory (RAM) and central processing unit (CPU).

HOW I use them

To avoid the worst-case scenario of process crashing and overloading the memory, other processes could use, I use generators.

Here is the snippet.

For Python2:

def scale_to_log10(till_number):

    for num in xrange(1, till_number):

        yield math.log10(num)

For Python3:

def scale_to_log10(till_number):

    for num in range(1, till_number):

        yield math.log10(num)

Notice that the range generator function in python3 is like the range in python2.

Variation between the two methods is clear. The method carries out generator uses yield statement instead of the return statement and further uses the xrange function instead of the range function. This is the case with lazy loading or lazy processing. It stores only the processing item into memory, not the full list.

In python3, the range function is same as the xrange function in python2. which means, it returns generator object in python3 and list in python2.

In Conclusion

Consider using generators when dealing with potentially large sequential data. Generators are efficient when dealing with large sequential stream of data so we can use them as building blocks of functions that help design a better structure.