To understand how to loop without a for loop, we’ll need to discover what makes for loops tick.

We’re about to learn how for loops work in Python. Along the way we’ll need to learn about iterables, iterators, and the iterator protocol. Let’s loop. ➿

Looping with indexes: a failed attempt

We might initially try to remove our for loops by using a traditional looping idiom from the world of C: looping with indexes.

1
2
3
4
5

colors = ["red", "green", "blue", "purple"]
i = 0
while i < len(colors):
    print(colors[i])
    i += 1

This works on lists, but it fails on sets:

1
2
3
4
5
6
7
8
9

>>> colors = {"red", "green", "blue", "purple"}
>>> i = 0
>>> while i < len(colors):
...     print(colors[i])
...     i += 1
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
TypeError: 'set' object does not support indexing

This approach only works on sequences, which are data types that have indexes from 0 to one less than their length. Lists, strings, and tuples are sequences. Dictionaries, sets, and many other iterables are not sequences.

We’ve been instructed to implement a looping construct that works on all iterables, not just sequences.

Iterables: what are they?

In the Python world, an iterable is any object that you can loop over with a for loop.

Iterables are not always indexable, they don’t always have lengths, and they’re not always finite.

Here’s an infinite iterable which provides every multiple of 5 as you loop over it:

1
2

from itertools import count
multiples_of_five = count(step=5)

When we were using for loops, we could have looped over the beginning of this iterable like this:

1
2
3
4

for n in multiples_of_five:
    if n > 100:
        break
    print(n)

If we removed the break condition from that for loop, it would go on printing forever.

So iterables can be infinitely long: which means that we can’t always convert an iterable to a list (or any other sequence) before we loop over it. We need to somehow ask our iterable for each item of our iterable individually, the same way our for loop works.

Iterables & Iterators

Okay we’ve defined iterable, but how do iterables actually work in Python?

All iterables can be passed to the built-in iter function to get an iterator from them.

1
2
3
4
5
6

>>> iter(['some', 'list'])
<list_iterator object at 0x7f227ad51128>
>>> iter({'some', 'set'})
<set_iterator object at 0x7f227ad32b40>
>>> iter('some string')
<str_iterator object at 0x7f227ad51240>

That’s an interesting fact but… what’s an iterator?

Iterators have exactly one job: return the “next” item in our iterable. They’re sort of like tally counters, but they don’t have a reset button and instead of giving the next number they give the next item in our iterable.

You can get an iterator from any iterable:

1

>>> iterator = iter('hi')

And iterators can be passed to next to get their next item:

1
2
3
4
5
6
7
8

>>> next(iterator)
'h'
>>> next(iterator)
'i'
>>> next(iterator)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

So iterators can be passed to the built-in next function to get the next item from them and if there is no next item (because we reached the end), a StopIteration exception will be raised.

Iterators are also iterables

So calling iter on an iterable gives us an iterator. And calling next on an iterator gives us the next item or raises a StopIteration exception if there aren’t any more items.

There’s actually a bit more to it than that though. You can pass iterators to the built-in iter function to get themselves back. That means that iterators are also iterables.

1
2
3
4

>>> iterator = iter('hi')
>>> iterator2 = iter(iterator)
>>> iterator is iterator2
True

That fact leads to some interesting consequences that we don’t have time to go into right now. We’ll save that discussion for a future learning adventure…

The Iterator Protocol

The iterator protocol is a fancy term meaning “how iterables actually work in Python”.

Let’s redefine iterables from Python’s perspective.

Iterables:

1. Can be passed to the iter function to get an iterator for them.
2. There is no 2. That’s really all that’s needed to be an iterable.

Iterators:

1. Can be passed to the next function which gives their next item or raises StopIteration
2. Return themselves when passed to the iter function.

The inverse of these statements should also hold true. Which means:

1. Anything that can be passed to iter without an error is an iterable.
2. Anything that can be passed to next without an error (except for StopIteration) is an iterator.
3. Anything that returns itself when passed to iter is an iterator.

Looping with iterators

With what we’ve learned about iterables and iterators, we should now be able to recreate a for loop without actually using a for loop.

This while loop manually loops over some iterable, printing out each item as it goes:

1
2
3
4
5
6
7
8
9

def print_each(iterable):
    iterator = iter(iterable)
    while True:
        try:
            item = next(iterator)
        except StopIteration:
            break  # Iterator exhausted: stop the loop
        else:
            print(item)

We can call this function with any iterable and it will loop over it:

1
2
3
4

>>> print_each({1, 2, 3})
1
2
3

The above function is essentially the same as this one which uses a for loop:

1
2
3

def print_each(iterable):
    for item in iterable:
        print(item)

This for loop is automatically doing what we were doing manually: calling iter to get an iterator and then calling next over and over until a StopIteration exception is raised.

The iterator protocol is used by for loops, tuple unpacking, and all built-in functions that work on generic iterables. Using the iterator protocol (either manually or automatically) is the only universal way to loop over any iterable in Python.

For loops: more complex than they seem

We’re now ready to complete the very silly task our interviewer assigned to us. We’ll remove all for loops from our code by manually using iter and next to loop over iterables. What did we learn in exploring this task?

Everything you can loop over is an iterable. Looping over iterables works via getting an iterator from an iterable and then repeatedly asking the iterator for the next item.

The way iterators and iterables work is called the iterator protocol. List comprehensions, tuple unpacking, for loops, and all other forms of iteration rely on the iterator protocol.

I’ll explore iterators more in future articles. For now know that iterators are hiding behind the scenes of all iteration in Python.

We’ll be discussing how to make code with this shape a little more descriptive:

1
2
3
4
5

all_good = True
for item in iterable:
    if not condition(item):
        all_good = False
        break

An Example: Primality

Here’s a function that checks whether a given number is prime by trying to divide it by all numbers below it:

1
2
3
4
5

def is_prime(candidate):
    for n in range(2, candidate):
        if candidate % n == 0:
            return False
    return True

Note: a square root makes this faster and our code breaks below 2 but we’ll ignore those issues here

This function:

1. loops from 2 to the given number
2. returns False as soon as a divisor is found
3. returns True if no divisor was found

This primality check is asking “do any numbers evenly divide the candidate number”.

Note that this function returns as soon as it finds a divisor, so it only iterates all the way through the number range when the candidate number is prime.

Let’s take a look at how we can rewrite this function using all.

What’s all?

Python has a built-in function all that returns True if all items are truthy

1
2
3
4
5
6
7
8

>>> all(['hello, 'there'])
True
>>> all(['hello, 'there', ''])
False
>>> all([1, 2, 3])
True
>>> all([0, 1, 2, 3])
False

You can think of truthy as meaning non-empty or non-zero (Python chat on truthiness). For our purposes, we’ll treat it as pretty much the same as True.

The all built-in function is equivalent to this:

1
2
3
4
5

def all(iterable):
    for element in iterable:
        if not element:
            return False
    return True

Notice the similarity between all and our is_prime function? Our is_prime function is similar, but they’re not quite the same structure.

The all function checks for the truthiness of element, but we need something a little more than that: we need to check a condition on each element (whether it’s a divsior).

Using all

Our original is_prime function looks like this:

1
2
3
4
5

def is_prime(candidate):
    for n in range(2, candidate):
        if candidate % n == 0:
            return False
    return True

If we want to use all in this function, we need an iterable (like a list) to pass to all.

If we wanted to be really silly, we could make such a list of boolean values like this:

1
2
3
4
5
6
7
8

def is_prime(candidate):
  divisibility = []
    for n in range(2, candidate):
        if candidate % n == 0:
            divisibility.append(False)
        else:
            divisibility.append(True)
  return all(divisibility)

We could simplify this function like this:

1
2
3
4
5

def is_prime(candidate):
  divisibility = []
    for n in range(2, candidate):
      divisibility.append(candidate % n != 0)
  return all(divisibility)

I know this is probably doesn’t seem like progress, but bear with me for a few more steps…

List comprehensions

If you’re familiar with list comprehensions, this code structure might look a little familiar. We’re creating one iterable from another which is exactly what list comprehensions are good for.

Let’s copy-paste our way into a list comprehension (see my article on how to write list comprehensions):

1
2
3
4
5
6

def is_prime(candidate):
  divisibility = [
      candidate % n != 0
      for n in range(2, candidate)
  ]
  return all(divisibility)

That’s quite a bit shorter, but there’s a problem: we’re building up an entire list just to loop over it once!

This is less efficient than our original approach, which only looped all the way when candidate was prime.

Let’s fix this inefficiency by turning our list comprehension into a generator expression.

Generator expressions

A generator expression is like a list comprehension, but instead of making a list it makes a generator (Python chat on generators).

A generator is a lazy iterable: generators don’t compute the items they contain until you loop over them. We’ll see what that means in a moment.

We can turn our list comprehension into a generator expression by changing the brackets to parentheses:

1
2
3
4
5
6

def is_prime(candidate):
  divisibility = (
      candidate % n != 0
      for n in range(2, candidate)
  )
  return all(divisibility)

Now our code doesn’t create a list to loop over. Instead it provides us with a generator that allows us to compute the divisibility of each number one-by-one.

We can make this code even more readable by putting that generator expression inside the function call (notice that we can drop the second set of parentheses):

1
2
3
4
5

def is_prime(candidate):
  return all(
      candidate % n != 0
      for n in range(2, candidate)
    )

Note that because our generator is lazy, we stop computing divisibilities as soon as our all function finds a divisible number. So we end up calculating candidate % n != 0 only as many times as we did in our original function.

Recap

So we started with a for loop, an if statement, a return statement for stopping once we find a divisor, and a return statement for the case where our number had no divisors (when it’s prime).

1
2
3
4
5

def is_prime(candidate):
    for n in range(2, candidate):
        if candidate % n == 0:
            return False
    return True

We turned all that into a generator expression passed to the all function.

1
2
3
4
5

def is_prime(candidate):
    return all(
        candidate % n != 0
        for n in range(2, candidate)
    )

I prefer this second approach (a generator expression with all) because I find it more descriptive.

We’re checking to see whether “all numbers in a range are not divisors of our candidate number”. That sounds quite a bit more like English to me than “loop over all numbers in a range and return False if a divisor is found otherwise return True”.

If you don’t find the behavior of all intuitive, you might find it easier to understand (and more English-like) when used with if:

1
2
3
4

if all(condition(item) for item in iterable):
    message = "All good"
else:
    message = "Bad value found"

You can always reformat your code to use an if statement if you find it more readable.

any or all

We’ve been working with the all function, but I haven’t mentioned it’s counterpart: the any function. Let’s take a look at how all and any compare.

These two expressions:

1
2
3
4
5
6
7
8

all_good = all(
    condition(x)
    for x in things
)
some_bad = not all(
    condition(x)
    for x in things
)

Are equivalent to these two expressions (because of DeMorgan’s Laws):

1
2
3
4
5
6
7
8

all_good = not any(
    not condition(x)
    for x in things
)
some_bad = any(
    not condition(x)
    for x in things
)

So this code:

1
2
3
4
5

def is_prime(candidate):
    return all(
        candidate % n != 0
        for n in range(2, candidate)
    )

Is feature-identical to this code:

1
2
3
4
5

def is_prime(candidate):
    return not any(
        candidate % n == 0
        for n in range(2, candidate)
    )

Both of them stop as soon as they find a divisor.

I find the use of all more readable here, but I wanted to mention that any would work just as well.

Cheat sheet for refactoring with any and all

All that explanation above was valuable, but how can we use this new knowledge to refactor our own code? Here’s a cheat sheet for you.

Anytime you see code like this:

1
2
3
4
5

all_good = True
for item in iterable:
    if not condition(item):
        all_good = False
        break

You can replace that code with this:

1
2
3
4

all_good = all(
    condition(item)
    for item in iterable
)

Anytime you see code like this:

1
2
3
4
5

any_good = False
for item in iterable:
    if condition(item):
        any_good = True
        break

You can replace it with this:

1
2
3
4

any_good = any(
    condition(item)
    for item in iterable
)

Note that break is used in the code above because we’re not returning from a function. Using return (like we did in is_prime) is another way to stop our loop early.

Python’s any and all functions were made for use with generator expressions (discussion here and here). You can use any and all without generator expressions, but I don’t find a need for that as often.

Quick note: any(item == 'something' for item in iterable) is the same as 'something' in iterable. Don’t use all/any for checking containment, use in.

Conclusion: code style in a process

As you discover new Python idioms and new language features are invented, your code style will evolve. Your preferred code style may never stop evolving. Code style is not concrete: it’s a process.

I hope I’ve inspired you to embrace the use of any/all with generator expressions for improved readability and code clarity.

Have a question about code style? Have a thought about any, all, and generator expressions? Please tweet me, email me, or comment below. 😄

I have been holding live webcasts every week for almost 2 months now. I started this trend after my regular expressions webinar in March. I soon came up with a name and made a website for these weekly python chat events. Now there’s also a Twitter account and a Facebook page.

Guest speakers and other experimentation

I’ve really enjoyed holding these events. The audience participation has been great: ample questions and plenty of helpful chat adding on to the discussion and occasionally correcting my mistakes.

I’ve been experimenting with the chat format by bringing in guest speakers the last couple weeks and I plan to introduce more general topics occasionally in the future.

Web chats about PyCon, live from PyCon

Next week I’ll be continuing my experimentation by hosting two Weekly Python Chat events live from PyCon.

The first chat next week will be during the first day of PyCon. I will likely be in the hallway accompanied by a couple other Python friends. We’ll answer your questions about what there is to do at PyCon, how it’s different from other cons, and why we go.

The second chat will be during the first day of the sprints. We’ll chat about who the sprints are for, how new contributors can get involved with the sprints, and what makes the sprints rewarding.

I want to convince you to join me at PyCon 2017

If you’ve never attended the sprints, sign up for the second chat to ask your questions and state your concerns. Hopefully I can convince you to stay for the sprints next time.

If you’ve never been to PyCon, sign up for both chats and voice your questions and concerns in each. I will address your questions and concerns, even if you can’t make the live event. Both chats will be recorded and you can re-watch them afterward.

Chat 1: Live from Day 1 of PyCon

Monday May 30, 2016 at 3:30pm PDT

We’ll discuss what PyCon is all about.

Sign up here to attend the PyCon Day 1 live chat

Chat 2: Live from the PyCon sprints

Thursday June 2, 2016 at 11:00am PDT

We’ll chat about how the sprints work.

Sign up here to attend the PyCon Sprints live chat

Get in touch

Have questions? Want to share your PyCon experiences during the live chat? Going to PyCon next week and want to meet up? Contact me!

In this article I’ll compare Python’s for loops to those of other languages and discuss the usual ways we solve common problems with for loops in Python.

For loops in other languages

Before we look at Python’s loops, let’s take a look at a for loop in JavaScript:

1
2
3
4

var colors = ["red", "green", "blue", "purple"];
for (var i = 0; i < colors.length; i++) {
    console.log(colors[i]);
}

This JavaScript loop looks nearly identical in C/C++ and Java.

In this loop we:

1. Set a counter variable i to 0
2. Check if the counter is less than the array length
3. Execute the code in the loop or exit the loop if the counter is too high
4. Increment the counter variable by 1

Looping in Python

Now let’s talk about loops in Python. First we’ll look at two slightly more familiar looping methods and then we’ll look at the idiomatic way to loop in Python.

while

If we wanted to mimic the behavior of our traditional C-style for loop in Python, we could use a while loop:

1
2
3
4
5

colors = ["red", "green", "blue", "purple"]
i = 0
while i < len(colors):
    print(colors[i])
    i += 1

This involves the same 4 steps as the for loops in other languages (note that we’re setting, checking, and incrementing i) but it’s not quite as compact.

This method of looping in Python is very uncommon.

range of length

I often see new Python programmers attempt to recreate traditional for loops in a slightly more creative fashion in Python:

1
2
3

colors = ["red", "green", "blue", "purple"]
for i in range(len(colors)):
    print(colors[i])

This first creates a range corresponding to the indexes in our list (0 to len(colors) - 1). We can loop over this range using Python’s for-in loop (really a foreach).

This provides us with the index of each item in our colors list, which is the same way that C-style for loops work. To get the actual color, we use colors[i].

for-in: the usual way

Both the while loop and range-of-len methods rely on looping over indexes. But we don’t actually care about the indexes: we’re only using these indexes for the purpose of retrieving elements from our list.

Because we don’t actually care about the indexes in our loop, there is a much simpler method of looping we can use:

1
2
3

colors = ["red", "green", "blue", "purple"]
for color in colors:
    print(color)

So instead of retrieving the item indexes and looking up each element, we can just loop over our list using a plain for-in loop.

The other two methods we discussed are sometimes referred to as anti-patterns because they are programming patterns which are widely considered unidiomatic.

What if we need indexes?

What if we actually need the indexes? For example, let’s say we’re printing out president names along with their numbers (based on list indexes).

range of length

We could use range(len(our_list)) and then lookup the index like before:

1
2
3

presidents = ["Washington", "Adams", "Jefferson", "Madison", "Monroe", "Adams", "Jackson"]
for i in range(len(presidents)):
    print("President {}: {}".format(i + 1, presidents[i]))

But there’s a more idiomatic way to accomplish this task: use the enumerate function.

enumerate

Python’s built-in enumerate function allows us to loop over a list and retrieve both the index and the value of each item in the list:

1
2
3

presidents = ["Washington", "Adams", "Jefferson", "Madison", "Monroe", "Adams", "Jackson"]
for num, name in enumerate(presidents, start=1):
    print("President {}: {}".format(num, name))

The enumerate function gives us an iterable where each element is a tuple that contains the index of the item and the original item value.

This function is meant for solving the task of:

1. Accessing each item in a list (or another iterable)
2. Also getting the index of each item accessed

So whenever we need item indexes while looping, we should think of enumerate.

Note: the start=1 option to enumerate here is optional. If we didn’t specify this, we’d start counting at 0 by default.

What if we need to loop over multiple things?

Often when we use list indexes, it’s to look something up in another list.

enumerate

For example, here we’re looping over two lists at the same time using indexes to look up corresponding elements:

1
2
3
4
5

colors = ["red", "green", "blue", "purple"]
ratios = [0.2, 0.3, 0.1, 0.4]
for i, color in enumerate(colors):
    ratio = ratios[i]
    print("{}% {}".format(ratio * 100, color))

Note that we only need the index in this scenario because we’re using it to lookup elements at the same index in our second list. What we really want is to loop over two lists simultaneously: the indexes just provide a means to do that.

zip

We don’t actually care about the index when looping here. Our real goal is to loop over two lists at once. This need is common enough that there’s a special built-in function just for this.

Python’s zip function allows us to loop over multiple lists at the same time:

1
2
3
4

colors = ["red", "green", "blue", "purple"]
ratios = [0.2, 0.3, 0.1, 0.4]
for color, ratio in zip(colors, ratios):
    print("{}% {}".format(ratio * 100, color))

The zip function takes multiple lists and returns an iterable that provides a tuple of the corresponding elements of each list as we loop over it.

Note that zip with different size lists will stop after the shortest list runs out of items. You may want to look into itertools.zip_longest if you need different behavior. Also note that zip in Python 2 returns a list but zip in Python 3 returns a lazy iterable. In Python 2, itertools.izip is equivalent to the newer Python 3 zip function.

Looping cheat sheet

Here’s a very short looping cheat sheet that might help you remember the preferred construct for each of these three looping scenarios.

Loop over a single list with a regular for-in:

1
2

for n in numbers:
    print(n)

Loop over multiple lists at the same time with zip:

1
2

for header, rows in zip(headers, columns):
    print("{}: {}".format(header, ", ".join(rows)))

Loop over a list while keeping track of indexes with enumerate:

1
2

for num, line in enumerate(lines):
    print("{0:03d}: {}".format(num, line))

In Summary

If you find yourself tempted to use range(len(my_list)) or a loop counter, think about whether you can reframe your problem to allow usage of zip or enumerate (or a combination of the two).

In fact, if you find yourself reaching for enumerate, think about whether you actually need indexes at all. It’s quite rare to need indexes in Python.

1. If you need to loop over multiple lists at the same time, use zip
2. If you only need to loop over a single list just use a for-in loop
3. If you need to loop over a list and you need item indexes, use enumerate

If you find yourself struggling to figure out the best way to loop, try using the cheat sheet above.

For more a more detailed explanation of the fundamentals of looping in Python, see Ned Batchelder’s Loop Like a Native presentation.

Thanks Steven Kryskalla and Diane Chen for proof-reading this post.

Update: If you’re interested in learning how to make your own Python objects that can be looped over, you may want to watch the operator overloading chat I held on April 30, 2016.

Happy looping!

Having trouble making your regular expressions readable?

I’m doing a 90 minute webinar to show you how to write readable regular expressions.

You can sign up here.

May 2015 Update: the webinar was recorded and you can watch it by signing up at the same link. The second half of this webinar event will be held this Saturday May 7 at 9am Pacific Time and is also accessible from the same link.

What’s a webinar?

Is this like a seminar? Aren’t seminars boring? Seminars can be boring but I’m hoping this webinar will be fun.

“Webinar” doesn’t sound cool, but I prefer it over “Wwworkshop”.

This will basically be like an online workshop. I’ll demonstrate some concepts through live coding and explanations and we’ll take a couple breaks to work through exercises together. There will be a chat room so we can discuss the concepts and share our answers for each of the exercises.

I do plan to do more of these so if you have a better suggestion for what to call this kind of thing than “webinar”, I’m all ears.

What will we learn?

We’ll learn about using regular expressions for validating text and for searching within text. We’ll cover the basic regular expression syntax and how to use use regular expressions in Python in particular.

Most importantly, we will discuss how to make your regular expressions readable.

I will be reviewing the basics of regular expressions, so if you’ve never used regular expressions before you should be able to follow along.

Will there be any follow-up to this?

Yes! I’m planning a second half to this webinar which will also be 90 minutes long. I’ll announce the date and time to attendees of part 1.

There is more to regular expressions than validation and searching.

Here are some things we will not cover in part 1, but which I hope to go over in part 2:

• Substitutions
• Data normalization
• Greediness
• Lookahead/lookbehind

When is this?

March 26 at 9am Pacific Time.

That’s:

• 11am in Chicago
• 12pm in New York
• 4pm in London
• 5pm in Berlin
• 6pm in Kyiv
• 7pm in Moscow
• 9:30pm in Bengaluru
• midnight in Perth

Find out what time that is in your time zone

How much does this cost?

$500.

Just kidding.

It’s free.

Where can I sign up?

You can sign up at regex.eventbrite.com.

I set the registration cap at 100 and as of this writing, 49 tickets have been claimed.

I did a similar event (on list comprehensions) for PyLadies Remote in the past and it went well.

I plan to do more of these kinds of events in the future. I’ll announce events to my email list first.

There are multiple ways to solve this problem: some are awkward, some are inaccurate, and most require multiple lines of code.

Let’s walk through the different ways of solving this problem and discuss which is the most Pythonic.

Our Problem

Before we can discuss solutions, we need to clearly define our problem.

Our code has two dictionaries: user and defaults. We want to merge these two dictionaries into a new dictionary called context.

We have some requirements:

1. user values should override defaults values in cases of duplicate keys
2. keys in defaults and user may be any valid keys
3. the values in defaults and user can be anything
4. defaults and user should not change during the creation of context
5. updates made to context should never alter defaults or user

Note: In 5, we’re focused on updates to the dictionary, not contained objects. For concerns about mutability of nested objects, we should look into copy.deepcopy.

So we want something like this:

1
2
3
4
5

>>> user = {'name': "Trey", 'website': "http://treyhunner.com"}
>>> defaults = {'name': "Anonymous User", 'page_name': "Profile Page"}
>>> context = merge_dicts(defaults, user)  # magical merge function
>>> context
{'website': 'http://treyhunner.com', 'name': 'Trey', 'page_name': 'Profile Page'}

We’ll also consider whether a solution is Pythonic. This is a very subjective and often illusory measure. Here are a few of the particular criteria we will use:

• The solution should be concise but not terse
• The solution should be readable but not overly verbose
• The solution should be one line if possible so it can be written inline if needed
• The solution should not be needlessly inefficient

Possible Solutions

Now that we’ve defined our problem, let’s discuss some possible solutions.

We’re going to walk through a number of methods for merging dictionaries and discuss which of these methods is the most accurate and which is the most idiomatic.

Multiple update

Here’s one of the simplest ways to merge our dictionaries:

1
2
3

context = {}
context.update(defaults)
context.update(user)

Here we’re making an empty dictionary and using the update method to add items from each of the other dictionaries. Notice that we’re adding defaults first so that any common keys in user will override those in defaults.

All five of our requirements were met so this is accurate. This solution takes three lines of code and cannot be performed inline, but it’s pretty clear.

Score:

• Accurate: yes
• Idiomatic: fairly, but it would be nicer if it could be inlined

Copy and update

Alternatively, we could copy defaults and update the copy with user.

1
2

context = defaults.copy()
context.update(user)

This solution is only slightly different from the previous one.

For this particular problem, I prefer this solution of copying the defaults dictionary to make it clear that defaults represents default values.

Score:

• Accurate: yes
• Idiomatic: yes

Dictionary constructor

We could also pass our dictionary to the dict constructor which will also copy the dictionary for us:

1
2

context = dict(defaults)
context.update(user)

This solution is very similar to the previous one, but it’s a little bit less explicit.

Score:

• Accurate: yes
• Idiomatic: somewhat, though I’d prefer the first two solutions over this

Keyword arguments hack

You may have seen this clever answer before, possibly on StackOverflow:

1

context = dict(defaults, **user)

This is just one line of code. That’s kind of cool. However, this solution is a little hard to understand.

Beyond readability, there’s an even bigger problem: this solution is wrong.

The keys must be strings. In Python 2 (with the CPython interpreter) we can get away with non-strings as keys, but don’t be fooled: this is a hack that only works by accident in Python 2 using the standard CPython runtime.

Score:

• Accurate: no. Requirement 2 is not met (keys may be any valid key)
• Idiomatic: no. This is a hack.

Dictionary comprehension

Just because we can, let’s try doing this with a dictionary comprehension:

1

context = {k: v for d in [defaults, user] for k, v in d.items()}

This works, but this is a little hard to read.

If we have an unknown number of dictionaries this might be a good idea, but we’d probably want to break our comprehension over multiple lines to make it more readable. In our case of two dictionaries, this doubly-nested comprehension is a little much.

Score:

• Accurate: yes
• Idiomatic: arguably not

Concatenate items

What if we get a list of items from each dictionary, concatenate them, and then create a new dictionary from that?

1

context = dict(list(defaults.items()) + list(user.items()))

This actually works. We know that the user keys will win out over defaults because those keys come at the end of our concatenated list.

In Python 2 we actually don’t need the list conversions, but we’re working in Python 3 here (you are on Python 3, right?).

Score:

• Accurate: yes
• Idiomatic: not particularly, there’s a bit of repetition

Union items

In Python 3, items is a dict_items object, which is a quirky object that supports union operations.

1

context = dict(defaults.items() | user.items())

That’s kind of interesting. But this is not accurate.

Requirement 1 (user should “win” over defaults) fails because the union of two dict_items objects is a set of key-value pairs and sets are unordered so duplicate keys may resolve in an unpredictable way.

Requirement 3 (the values can be anything) fails because sets require their items to be hashable so both the keys and values in our key-value tuples must be hashable.

Side note: I’m not sure why the union operation is even allowed on dict_items objects. What is this good for?

Score:

• Accurate: no, requirements 1 and 3 fail
• Idiomatic: no

Chain items

So far the most idiomatic way we’ve seen to perform this merge in a single line of code involves creating two lists of items, concatenating them, and forming a dictionary.

We can join our items together more succinctly with itertools.chain:

1
2

from itertools import chain
context = dict(chain(defaults.items(), user.items()))

This works well and may be more efficient than creating two unnecessary lists.

Score:

• Accurate: yes
• Idiomatic: fairly, but those items calls seem slightly redundant

ChainMap

A ChainMap allows us to create a new dictionary without even looping over our initial dictionaries (well sort of, we’ll discuss this):

1
2

from collections import ChainMap
context = ChainMap({}, user, defaults)

A ChainMap groups dictionaries together into a proxy object (a “view”); lookups query each provided dictionary until a match is found.

This code raises a few questions.

Why did we put user before defaults?

We ordered our arguments this way to ensure requirement 1 was met. The dictionaries are searched in order, so user returns matches before defaults.

Why is there an empty dictionary before user?

This is for requirement 5. Changes to ChainMap objects affect the first dictionary provided and we don’t want user to change so we provided an empty dictionary first.

Does this actually give us a dictionary?

A ChainMap object is not a dictionary but it is a dictionary-like mapping. We may be okay with this if our code practices duck typing, but we’ll need to inspect the features of ChainMap to be sure. Among other features, ChainMap objects are coupled to their underlying dictionaries and they handle removing items in an interesting way.

Score:

• Accurate: possibly, we’ll need to consider our use cases
• Idiomatic: yes if we decide this suits our use case

Dictionary from ChainMap

If we really want a dictionary, we could convert our ChainMap to a dictionary:

1

context = dict(ChainMap(user, defaults))

It’s a little odd that user must come before defaults in this code whereas this order was flipped in most of our other solutions. Outside of that oddity, this code is fairly simple and should be clear enough for our purposes.

Score:

• Accurate: yes
• Idiomatic: yes

Dictionary concatenation

What if we simply concatenate our dictionaries?

1

context = defaults + user

This is cool, but it isn’t valid. This was discussed in a python-ideas thread last year.

Some of the concerns brought up in this thread include:

• Maybe | makes more sense than + because dictionaries are like sets
• For duplicate keys, should the left-hand side or right-hand side win?
• Should there be an updated built-in instead (kind of like sorted)?

Score:

• Accurate: no. This doesn’t work.
• Idiomatic: no. This doesn’t work.

Dictionary unpacking

If you’re using Python 3.5, thanks to PEP 448, there’s a new way to merge dictionaries:

1

context = {**defaults, **user}

This is simple and Pythonic. There are quite a few symbols, but it’s fairly clear that the output is a dictionary at least.

This is functionally equivalent to our very first solution where we made an empty dictionary and populated it with all items from defaults and user in turn. All of our requirements are met and this is likely the simplest solution we’ll ever get.

Score:

• Accurate: yes
• Idiomatic: yes

Summary

There are a number of ways to combine multiple dictionaries, but there are few elegant ways to do this with just one line of code.

If you’re using Python 3.5, this is the one obvious way to solve this problem:

1

context = {**defaults, **user}

If you are not yet using Python 3.5, you’ll need to review the solutions above to determine which is the most appropriate for your needs.

Note: For those of you particularly concerned with performance, I also measured the performance of these different dictionary merging methods.

If you’re interested in deep-merging this dictionary (merging a dictionary of dictionaries for example), check out this deep merging technique from Mahmoud Hashemi.

Update: If you’re interested in learning more about the new features of * and ** in Python 3.5 and their history you may want to watch the Packing & Unpacking Operators chat I held on April 23, 2016.

I teach Python for a living. If you like my teaching style and your team is interested in Python training, please contact me!

List comprehensions in Python are great, but mastering them can be tricky because they don’t solve a new problem: they just provide a new syntax to solve an existing problem.

Let’s learn what list comprehensions are and how to identify when to use them.

Update: I held a 1 hour video chat about list comprehensions which extends the material in this article. If you want more after reading this post, check out the recording.

What are list comprehensions?

List comprehensions are a tool for transforming one list (any iterable actually) into another list. During this transformation, elements can be conditionally included in the new list and each element can be transformed as needed.

If you’re familiar with functional programming, you can think of list comprehensions as syntactic sugar for a filter followed by a map:

1
2

>>> doubled_odds = map(lambda n: n * 2, filter(lambda n: n % 2 == 1, numbers))
>>> doubled_odds = [n * 2 for n in numbers if n % 2 == 1]

If you’re not familiar with functional programming, don’t worry: I’ll explain using for loops.

From loops to comprehensions

Every list comprehension can be rewritten as a for loop but not every for loop can be rewritten as a list comprehension.

The key to understanding when to use list comprehensions is to practice identifying problems that smell like list comprehensions.

If you can rewrite your code to look just like this for loop, you can also rewrite it as a list comprehension:

1
2
3
4

new_things = []
for ITEM in old_things:
    if condition_based_on(ITEM):
        new_things.append("something with " + ITEM)

You can rewrite the above for loop as a list comprehension like this:

1

new_things = ["something with " + ITEM for ITEM in old_things if condition_based_on(ITEM)]

List Comprehensions: The Animated Movie™

That’s great, but how did we do that?

We copy-pasted our way from a for loop to a list comprehension.

Here’s the order we copy-paste in:

1. Copy the variable assignment for our new empty list (line 3)
2. Copy the expression that we’ve been append-ing into this new list (line 6)
3. Copy the for loop line, excluding the final : (line 4)
4. Copy the if statement line, also without the : (line 5)

We’ve now copied our way from this:

1
2
3
4
5
6

numbers = [1, 2, 3, 4, 5]

doubled_odds = []
for n in numbers:
    if n % 2 == 1:
        doubled_odds.append(n * 2)

To this:

1
2
3

numbers = [1, 2, 3, 4, 5]

doubled_odds = [n * 2 for n in numbers if n % 2 == 1]

List Comprehensions: Now in Color

Let’s use colors to highlight what’s going on.

doubled_odds = []
for n in numbers:
    if n % 2 == 1:
        doubled_odds.append(n * 2)

doubled_odds = [n * 2 for n in numbers if n % 2 == 1]

We copy-paste from a for loop into a list comprehension by:

1. Copying the variable assignment for our new empty list
2. Copying the expression that we’ve been append-ing into this new list
3. Copying the for loop line, excluding the final :
4. Copying the if statement line, also without the :

Unconditional Comprehensions

But what about comprehensions that don’t have a conditional clause (that if SOMETHING part at the end)? These loop-and-append for loops are even simpler than the loop-and-conditionally-append ones we’ve already covered.

A for loop that doesn’t have an if statement:

doubled_numbers = []
for n in numbers:
    doubled_numbers.append(n * 2)

That same code written as a comprehension:

doubled_numbers = [n * 2 for n in numbers]

Here’s the transformation animated:

We can copy-paste our way from a simple loop-and-append for loop by:

1. Copying the variable assignment for our new empty list (line 3)
2. Copying the expression that we’ve been append-ing into this new list (line 5)
3. Copying the for loop line, excluding the final : (line 4)

Nested Loops

What about list comprehensions with nested looping?… 😦

Here’s a for loop that flattens a matrix (a list of lists):

flattened = []
for row in matrix:
    for n in row:
        flattened.append(n)

Here’s a list comprehension that does the same thing:

flattened = [n for row in matrix for n in row]

Nested loops in list comprehensions do not read like English prose.

Note: My brain wants to write this list comprehension as:

flattened = [n for n in row for row in matrix]

But that’s not right! I’ve mistakenly flipped the for loops here. The correct version is the one above.

When working with nested loops in list comprehensions remember that the for clauses remain in the same order as in our original for loops.

Other Comprehensions

This same principle applies to set comprehensions and dictionary comprehensions.

Code that creates a set of all the first letters in a sequence of words:

first_letters = set()
for w in words:
    first_letters.add(w[0])

That same code written as a set comprehension:

first_letters = {w[0] for w in words}

Code that makes a new dictionary by swapping the keys and values of the original one:

flipped = {}
for key, value in original.items():
    flipped[value] = key

That same code written as a dictionary comprehension:

flipped = {value: key for key, value in original.items()}

Readability Counts

Did you find the above list comprehensions hard to read? I often find longer list comprehensions very difficult to read when they’re written on one line.

Remember that Python allows line breaks between brackets and braces.

List comprehension

Before

1

doubled_odds = [n * 2 for n in numbers if n % 2 == 1]

After

1
2
3
4
5

doubled_odds = [
    n * 2
    for n in numbers
    if n % 2 == 1
]

Nested loops in list comprehension

Before

1

flattened = [n for row in matrix for n in row]

After

1
2
3
4
5

flattened = [
    n
    for row in matrix
    for n in row
]

Dictionary comprehension

Before

1

flipped = {value: key for key, value in original.items()}

After

1
2
3
4

flipped = {
    value: key
    for key, value in original.items()
}

Note that we are not adding line breaks arbitrarily: we’re breaking between each of the lines of code we copy-pasted to make these comprehension. Our line breaks occur where color changes occur in the colorized versions.

Learn with me

I did a class on list comprehensions with PyLadies Remote recently.

If you’d like to watch me walk through an explanation of any of the above topics, check out the video:

1. list comprehensions
2. generator expressions
3. set comprehensions
4. dictionary comprehensions

Summary

When struggling to write a comprehension, don’t panic. Start with a for loop first and copy-paste your way into a comprehension.

Any for loop that looks like this:

new_things = []
for ITEM in old_things:
    if condition_based_on(ITEM):
        new_things.append("something with " + ITEM)

Can be rewritten into a list comprehension like this:

new_things = ["something with " + ITEM for ITEM in old_things if condition_based_on(ITEM)]

If you can nudge a for loop until it looks like the ones above, you can rewrite it as a list comprehension.

This article was based on my Intro to Python class. If you’re interested in chatting about my Python training services, drop me a line.

Let’s look at different techniques for counting the number of times things appear in a list. While analyzing these techniques, we will only be looking at code style. We’ll worry about performance later.

We will need some historical context to understand these different techniques. Fortunately we live in the __future__ and we have a time machine. Let’s jump in our DeLorean and head to 1997.

if Statement

It’s January 1, 1997 and we’re using Python 1.4. We have a list of colors and we’d love to know how many times each color occurs in this list. Let’s use a dictionary!

1
2
3
4
5
6
7

colors = ["brown", "red", "green", "yellow", "yellow", "brown", "brown", "black"]
color_counts = {}
for c in colors:
    if color_counts.has_key(c):
        color_counts[c] = color_counts[c] + 1
    else:
        color_counts[c] = 1

Note: we’re not using += because augmented assignment won’t be added until Python 2.0 and we’re not using the c in color_counts idiom because that won’t be invented until Python 2.2!

After running this we’ll see that our color_counts dictionary now contains the counts of each color in our list:

1
2

>>> color_counts
{'brown': 3, 'yellow': 2, 'green': 1, 'black': 1, 'red': 1}

That was pretty simple. We just looped through each color, checked if it was in the dictionary, added the color if it wasn’t, and incremented the count if it was.

We could also write this as:

1
2
3
4
5
6

colors = ["brown", "red", "green", "yellow", "yellow", "brown", "brown", "black"]
color_counts = {}
for c in colors:
    if not color_counts.has_key(c):
        color_counts[c] = 0
    color_counts[c] = color_counts[c] + 1

This might be a little slower on sparse lists (lists with lots of non-repeating colors) because it executes two statements instead of one, but we’re not worried about performance, we’re worried about code style. After some thought, we decide to stick with this new version.

try Block

It’s January 2, 1997 and we’re still using Python 1.4. We woke up this morning with a sudden realization: our code is practicing “Look Before You Leap” (LBYL) when we should be practicing “Easier to Ask Forgiveness, Than Permission” (EAFP) because EAFP is more Pythonic. Let’s refactor our code to use a try-except block:

1
2
3
4
5
6
7

colors = ["brown", "red", "green", "yellow", "yellow", "brown", "brown", "black"]
color_counts = {}
for c in colors:
    try:
        color_counts[c] = color_counts[c] + 1
    except KeyError:
        color_counts[c] = 1

Now our code attempts to increment the count for each color and if the color isn’t in the dictionary, a KeyError will be raised and we will instead set the color count to 1 for the color.

get Method

It’s January 1, 1998 and we’ve upgraded to Python 1.5. We’ve decided to refactor our code to use the new get method on dictionaries:

1
2
3
4

colors = ["brown", "red", "green", "yellow", "yellow", "brown", "brown", "black"]
color_counts = {}
for c in colors:
    color_counts[c] = color_counts.get(c, 0) + 1

Now our code loops through each color, gets the current count for the color from the dictionary, defaulting this count to 0, adds 1 to the count, and sets the dictionary key to this new value.

It’s cool that this is all one line of code, but we’re not entirely sure if this is more Pythonic. We decide this might be too clever so we revert this change.

setdefault

It’s January 1, 2001 and we’re now using Python 2.0! We’ve heard that dictionaries have a setdefault method now and we decide to refactor our code to use this new method. We also decide to use the new += augmented assignment operator:

1
2
3
4
5

colors = ["brown", "red", "green", "yellow", "yellow", "brown", "brown", "black"]
color_counts = {}
for c in colors:
    color_counts.setdefault(c, 0)
    color_counts[c] += 1

The setdefault method is being called on every loop, regardless of whether it’s needed, but this does seem a little more readable. We decide that this is more Pythonic than our previous solutions and commit our change.

fromkeys

It’s January 1, 2004 and we’re using Python 2.3. We’ve heard about a new fromkeys class method on dictionaries for constructing dictionaries from a list of keys. We refactor our code to use this new method:

1
2
3
4

colors = ["brown", "red", "green", "yellow", "yellow", "brown", "brown", "black"]
color_counts = dict.fromkeys(colors, 0)
for c in colors:
    color_counts[c] += 1

This creates a new dictionary using our colors as keys, with all values set to 0 initially. This allows us to increment each key without worrying whether it has been set. We’ve removed the need for any checking or exception handling which seems like an improvement. We decide to keep this change.

Comprehension and set

It’s January 1, 2005 and we’re using Python 2.4. We realize that we could solve our counting problem using sets (released in Python 2.3 and made into a built-in in 2.4) and list comprehensions (released in Python 2.0). After further thought, we remember that generator expressions were also just released in Python 2.4 and we decide to use one of those instead of a list comprehension:

1
2

colors = ["brown", "red", "green", "yellow", "yellow", "brown", "brown", "black"]
color_counts = dict((c, colors.count(c)) for c in set(colors))

Note: we didn’t use a dictionary comprehension because those won’t be invented until Python 2.7.

This works. It’s one line of code. But is it Pythonic?

We remember the Zen of Python, which started in a python-list email thread and was snuck into Python 2.2.1. We type import this at our REPL:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Our code is more complex (O(n²) instead of O(n)), less beautiful, and less readable. That change was a fun experiment, but this one-line solution is less Pythonic than what we already had. We decide to revert this change.

defaultdict

It’s January 1, 2007 and we’re using Python 2.5. We’ve just found out that defaultdict is in the standard library now. This should allow us to set 0 as the default value in our dictionary. Let’s refactor our code to count using a defaultdict instead:

1
2
3
4
5

from collections import defaultdict
colors = ["brown", "red", "green", "yellow", "yellow", "brown", "brown", "black"]
color_counts = defaultdict(int)
for c in colors:
    color_counts[c] += 1

That for loop is so simple now! This is almost certainly more Pythonic.

We realize that our color_counts variable does act differently, however it does inherit from dict and supports all the same mapping functionality.

1
2

>>> color_counts
defaultdict(<type 'int'>, {'brown': 3, 'yellow': 2, 'green': 1, 'black': 1, 'red': 1})

Instead of converting color_counts to a dict, we’ll assume the rest of our code practices duck typing and leave this dict-like object as-is.

Counter

It’s January 1, 2011 and we’re using Python 2.7. We’ve been told that our defaultdict code is no longer the most Pythonic way to count colors. A Counter class was included in the standard library in Python 2.7 and it does all of the work for us!

1
2
3

from collections import Counter
colors = ["brown", "red", "green", "yellow", "yellow", "brown", "brown", "black"]
color_counts = Counter(colors)

Could this get any simpler? This must be the most Pythonic way.

Like defaultdict, this returns a dict-like object (a dict subclass actually), which should be good enough for our purposes, so we’ll stick with it.

1
2

>>> color_counts
Counter({'brown': 3, 'yellow': 2, 'green': 1, 'black': 1, 'red': 1})

After thought: Performance

Notice that we didn’t focus on efficiency for these solutions. Most of these solutions have the same time complexity (O(n) in big O notation) but runtimes could vary based on the Python implementation.

While performance isn’t our main concern, I did measure the run-times on CPython 3.5.0. It’s interesting to see how each implementation changes in relative efficiency based on the density of color names in the list.

Conclusion

Per the Zen of Python, “there should be one– and preferably only one– obvious way to do it”. This is an aspirational message. There isn’t always one obvious way to do it. The “obvious” way can vary by time, need, and level of expertise.

“Pythonic” is a relative term.

Related Resources

• import this and the Zen of Python: Zen of Python trivia borrowed from this post
• Permission or Forgiveness: Alex Martelli discusses Grace Hopper’s EAFP
• Python How To: Group and Count with Dictionaries: while writing this post, I discovered this related article

Credits

Thanks to Brian Schrader and David Lord for proof-reading this post and Micah Denbraver for actually testing out these solutions on the correct versions of Python.

While chatting with other users of patch, I realized that everyone seems to have their own favorite way to use it. In this post I will discuss the ways I use patch.

Decorator

patch can be used as a method decorator:

1
2
3
4
5
6
7
8

from mock import patch

class MyModelTest:
    @patch('mylib.utils.other_func')
    def test_some_func(self, other_func):
        other_func.return_value = "MY STRING"
        assert some_func("my string") == "MY STRING"
        other_func.assert_called_once_with("my string")

or as a class decorator:

1
2
3
4
5
6
7
8

from mock import patch

@patch('mylib.utils.other_func')
class MyModelTest:
    def test_some_func(self, other_func):
        other_func.return_value = "MY STRING"
        assert some_func("my string") == "MY STRING"
        other_func.assert_called_once_with("my string")

I use patch as a decorator when I have a function I want patched during my whole test. I tend not to use patch as a class decorator and I’ll explain why below.

Decorator example

Context Manager

patch can be used as a context manager:

1
2
3
4
5
6
7
8

from mock import patch

class MyModelTest:
    def test_some_func(self):
        other_func.return_value = "MY STRING"
        with patch('mylib.utils.other_func') as other_func:
            assert some_func("my string") == "MY STRING"
        other_func.assert_called_once_with("my string")

I prefer to use patch as a context manager when I want to patch a function for only part of a test. I do not use patch as a context manager when I want a function patched for an entire test.

Context manager example

Manually using start and stop

patch can also be used to manually patch/unpatch using start and stop methods:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

from mock import patch

class MyModelTest:

    def setUp(self):
        self.other_func_patch = patch('mylib.utils.other_func')
        self.other_func = self.other_func_patch.start()
        self.other_func.return_value = "MY STRING"

    def tearDown(self):
        self.other_func_patch.stop()

    def test_some_func(self):
        assert some_func("my string") == "MY STRING"
        self.other_func.assert_called_once_with("my string")

I prefer to use patch using start/stop when I need a function to be patched for every function in a test class.

This is probably the most common way I use patch in my tests. I often group my tests into test classes where each method is focused around testing the same function. Therefore I will usually want the same functions/objects patched for every test method.

I noted above that I prefer not to use class decorators to solve this problem. Instead, I prefer to use test class attributes to store references to patched functions instead of accepting patch arguments on every test method with decorators. I find this more DRY.

Warning: One of the primary benefits of the decorator/context manager forms of patch is that they handle clean up for you. Whenever you call start to setup your patch object, always remember to call stop to clean it up. Otherwise you’ll have a monkey patched function/object for the rest of your running program.

start and stop example

Summary

Patch can be used:

1. as a method or class decorator
2. as a context manager
3. using start and stop methods

I prefer my tests to be readable, DRY, and easy to modify. I tend to use start/stop methods for that reason, but I also frequently use patch method decorators and sometimes use patch context managers. It’s useful to know the different flavors of patch because your favorite flavor may not always be the most suitable one for the problem at hand.

Did I miss a flavor? Want to let me know which flavor you prefer and why? Please comment below.

In this post I will discuss a method for customizing the HTML generated by Django form fields, with the specific goal of adding custom CSS classes to Django form fields.

Here’s a Django form definition:

1
2
3
4
5

from django import forms

class AuthenticationForm(forms.Form):
    username = forms.CharField(max_length=254)
    password = forms.CharField(widget=forms.PasswordInput)

Here’s the form used in a template:

{{ form.as_p }}

The Problem

We’re using Bootstrap and we want to add an input-lg CSS class onto our username field to make it really big.

The Solution(s)

There are many ways to solve this problem. I will discuss some solutions I dislike before I discuss my preferred solution.

Using a form widget attribute

We could add a class attribute to our Django form field:

1
2
3
4
5
6
7
8

from django import forms

class AuthenticationForm(forms.Form):
    username = forms.CharField(
        max_length=254,
        widget=forms.TextInput(attrs={'class': "input-lg"}),
    )
    password = forms.CharField(widget=forms.PasswordInput)

I dislike this approach because it requires including presentation rules in our back-end code. This class attribute is used exclusively by our CSS and/or JavaScript and should therefore live in Django templates, not in Python code.

Using django-floppyforms

If we’re using django-floppyforms we could include logic in our floppyforms/attrs.html template to add specific classes based on a context variable (example). Here’s an example:

{% for name, value in attrs.items %} {{ name }}{% if value != True %}="{{ value }}{% if name == "class" %} {{ extra_classes }}{% endif %}"{% endfor %}

This should work but it’s ugly and in general I do not enjoy maintaining heavy logic in my templates.

Using django-widget-tweaks

I prefer to solve this problem with django-widget-tweaks.

The django-widget-tweaks library provides two solutions to this problem:

1. add_class template filter
2. render_field template tag.

The add_class template filter

Mikhail Korobov originally created the django-widget-tweaks library in 2011. It started as a series of template filters for modifying form field attributes from your Django templates.

Here’s an example usage of the add_class filter for adding a CSS class to our form field:

{% load widget_tweaks %}
<p>
    {{ form.username|add_class:"input-lg" }}
    {{ form.username.errors }}
</p>
<p>
    {{ form.password }}
    {{ form.password.errors }}
</p>

I find this solution both easy to read and easy to maintain.

The render_field template tag

I discovered django-widget-tweaks shortly after Mikhail created it. I appreciated his solution for this problem, but I wanted a more HTML-like syntax for my form field customizations. I created the render_field template tag to satisfy that desire.

With the render_field tag you can add attributes to form fields with a much more HTML-like syntax:

{% load widget_tweaks %}
<p>
    {% render_field form.username class+="input-lg" %}
    {{ form.username.errors }}
</p>
<p>
    {% render_field form.password %}
    {{ form.password.errors }}
</p>

As a bonus, with render_field we can also set a CSS class for erroneous and required form fields. See the documentation for more details.

Conclusion

I have not had a chance to use django-floppyforms yet, but I expect that django-widget-tweaks and django-floppyforms would integrate well together.

I am on the lookout for new solutions to this problem, but django-widget-tweaks has served me well so far. I have used it for three years now it remains one of my go-to libraries for new Django projects.

How do you add CSS classes do your Django form fields? If you have another solution please leave a comment below.

Django 1.7 uses the migrations sub-package in your app for database migrations and South relies on the same package. Unfortunately, you can’t store both packages in the same place. At first glance, it seems we cannot support both Django 1.7 and previous versions of Django using South. However, as I explain below, we can support both at once.

Assessing your options

In order to support both Django 1.7 and Django 1.6 with South we can rename the migrations package and instruct users to reference the new package in their settings module. We can do this with the MIGRATION_MODULES or SOUTH_MIGRATION_MODULES settings. There are three options:

1. Move existing migrations directory to south_migrations and create Django 1.7 migrations in migrations package
2. Create new Django 1.7 migrations package in django_migrations directory and leave existing South migrations package
3. Move existing migrations directory to south_migrations and create Django 1.7 migrations in django_migrations directory

The first option requires existing users either switch to Django 1.7 or update their settings module before upgrading to the new version of your app. The second option requires all Django 1.7 users to customize their settings module to properly install your app. The third option requires everyone (both Django 1.7 and South users) to update their settings module.

Out of those options I prefer the first one. When you eventually drop support for South, you will probably want your Django 1.7 migrations to live in the migrations directory. If you don’t force that switch now, you would eventually need to break backwards-compatibility or maintain two duplicate migrations directories.

So our plan is to move the South migrations to south_migrations and create Django 1.7 migrations. An example with the django-email-log app:

1
2
3

$ git mv email_log/migrations email_log/south_migrations
$ python manage.py makemigrations email_log
$ git add email_log/migrations

Breaking South support

If you move migrations to south_migrations and make a Django 1.7 migrations package, what happens to existing users with South?

Your new app upgrade will break backwards compatibility for South users and you want to make sure they know they need to make a change immediately after upgrading. Users should see a loud and clear error message instructing them what they need to do. This can be done by hijacking their use of the migrate command with South.

Existing users will run migrate when upgrading your app. If they don’t migrate immediately, they will when they notice a problem and realize they need to run migrate. Upon migrating, we want to show a clear error message telling the user what to do.

Failing loudly and with a clear error message

When South looks for app migrations it will import our migrations package. Our migrations package contains Django 1.7 migrations, which South won’t understand. So we want to make sure that if our migrations package is imported either Django 1.7 is installed or a proper error message is displayed. Upon importing this package, we can check for the presence of the new django.db.migrations module and if not found we will raise an exception with a descriptive error message.

For example, this is the code I plan to add to the email_log/migrations/__init__.py file for django-email-log to add Django 1.7 support:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

"""
Django migrations for email_log app

This package does not contain South migrations.  South migrations can be found
in the ``south_migrations`` package.
"""

SOUTH_ERROR_MESSAGE = """\n
For South support, customize the SOUTH_MIGRATION_MODULES setting like so:

    SOUTH_MIGRATION_MODULES = {
        'email_log': 'email_log.south_migrations',
    }
"""

# Ensure the user is not using Django 1.6 or below with South
try:
    from django.db import migrations  # noqa
except ImportError:
    from django.core.exceptions import ImproperlyConfigured
    raise ImproperlyConfigured(SOUTH_ERROR_MESSAGE)

Now when we run migrate with Django 1.6 and South, we’ll see the following exception raised:

1
2
3
4
5
6
7

django.core.exceptions.ImproperlyConfigured:

For South support, customize the SOUTH_MIGRATION_MODULES setting like so:

    SOUTH_MIGRATION_MODULES = {
        'email_log': 'email_log.south_migrations',
    }

Conclusion

This breaks backwards compatibility, but our users should immediately understand what has broken and how to fix it. Remember to upgrade the major number of your package version to note this backwards-incompatible change.

I would love to hear your thoughts about this approach in the comments below. Let me know if you have other ideas about how to handle supporting Django 1.7 migrations and South at the same time.

You can find the tutorials at http://bit.ly/pysd-tdd . The tutorials are provided under a CC BY-SA license so you can reuse and modify them for your own purposes.

Tutorial markdown files and working source code may be found on Github. We plan to improve and extend these tutorials for a future workshop. If you have ideas for improvements/additions or if you notice a bug, please submit an issue or open a pull request.

Visual testing frameworks

Visual testing tools compare screenshots to ensure tested webpages look pixel perfect. Capturing webpage screenshots requires a full-featured web browser to render CSS and execute JavaScript. All three of the visual testing tools I found rely on Selenium or PhantomJS for rendering.

PhantomCSS

PhantomCSS uses PhantomJS for screenshot differencing. PhantomCSS won’t integrate directly with the Django live server or your Python test suite, so if you want to run a visual integration test, you’d need to manually start and stop the test server between tests. I might eventually try out PhantomCSS for CSS unit tests, but I wanted to visually test my full website so I needed integration with the Django live server.

Django-casper

Django-casper uses Django live server tests to execute CasperJS test files (which use PhantomJS) to compare screenshots. Each test requires an additional Python test which references a JavaScript file that executes the navigation and screenshotting code. I found this approach messy and difficult to setup.

Needle

The needle Python library uses Selenium to navigate your website and screenshot rendered pages. Unfortunately needle has poor test coverage, a seemingly failing test suite, and no change log. Despite these shortcomings, I went with needle for my visual integration tests because it got the job done.

Django and Needle

I used the following mixin to integrate the Django live server with needle. I used PhantomJS, but Firefox or another Selenium web driver should work as well.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

from django.test import LiveServerTestCase
from needle.cases import NeedleTestCase
from selenium.webdriver import PhantomJS


class DjangoNeedleTestCase(NeedleTestCase, LiveServerTestCase):

    """Needle test case for use with Django live server"""

    driver = PhantomJS

    @classmethod
    def get_web_driver(cls):
        return type('NeedleWebDriver', (NeedleWebDriverMixin, cls.driver), {})()

Unfortunately the above code only works with the version of needle on Github. The PyPI version does not yet include the NeedleWebDriverMixin (which I contributed recently for Django support). I have created an issue suggesting a new PyPI release be made to resolve this problem.

Room for improvement

Currently I only run my visual tests manually. Visual tests are very brittle and occasionally they just break without any changes. If I manage to stabilize my visual tests so that they pass consistently on different platforms, I may run them during continuous integration.

Do you have another solution for visual integration testing? Let me know in the comments.

JSON Encoding using the json library

With the current Python json library, using an extensible JSON encoder in your generic application may require some/all of the following:

• Allowing specification of custom encoder class by client applications
• Overriding the default JSON encoder class (or a client-specified one) for any further extensions
• Passing JSON encoder classes into other serialization libraries used by your application

Combining JSON encoders

If you need to compose two custom JSON encoders specified in two different packages, you may need to:

• Use multiple inheritance and hope the encoders play nicely together
• Duplicate code from one of the packages and create a new serializer with single inheritance
• Monkey patch one or both of the libraries

JSON encoder using single-dispatch generic functions

I created a wrapper around the json library to make a JSON encoder using single-dispatch generic functions. Here’s how to use it:

As you can see, it’s fairly easy to extend the encoder to understand serialization rules for new data types.

The impementation is fairly simple, albeit a hack:

This code is intended as a proof-of-concept to demonstrate the power of single-dispatch generic functions. Feel free to use it however you like.

Related Links

• What single-dispatch generic functios mean for you
• Python 3.4 single dispatch, a step into generic functions

I got the idea from a StackOverflow answer and I decided to make a real application out of it. All emails are stored in a single model which can easily be viewed, searched, sorted, and filtered from the admin site.

I used test-driven development when making the app and I baked in Python 3 support from the beginning. I found the process of TDD for a standalone Python package fairly easy and enjoyable.

I recently attempted to revive django-simple-history. I added tests, put it on PyPI, and made it easier to use with newer versions of Django. I moved my fork of the project to git and Github, added Travis and Coveralls support for continuous integration and code coverage tracking, and noted future features on the issue tracker.

Soon after I started writing tests for the project I received feature requests, issues, pull requests, and emails with words of encouragement. I appreciate all of the help I’ve had while reviving the project. I plan to remain responsive to the suggestions for my fork of the code. If you’d like to help out with the project please feel free to submit an issue, make a pull request, or comment on the code commits on the Github page.

Since discovering DuckDuckGo’s random word generation, I’ve hoped that someone would make a DuckDuckGo plugin to generate random names also.

I didn’t want to write my own DuckDuckGo plugin yet so I made a Python-powered command line tool instead using 1990 Census data.

The program is called names and can be found on Github and PyPI.

It’s really simple

It’s basically just one file currently that’s about 40 lines long. There is only one feature available from the command line currently: generate a single random full name. There’s a few more features if importing as a Python package: generate random last name or generate random first name (with or without specifying gender), generate random full name (also without or without gender).

The random name picker relies on the cumulative frequencies listed in the included Census data files. Here’s the steps that are taken: 1. A random floating point number is chosen between 0.0 and 90.0 2. Name file lines are iterated through until a cumulative frequency is found that is less than the randomly generated number 3. The name on that line is chosen and returned (or printed out)

Examples

Here’s how you use it from the command line:

$ names
Kara Lopes

Here’s how you use it as a Python package:

>>> import names
>>> names.get_full_name()
u'Patricia Halford'
>>> names.get_full_name(gender='male')
u'Patrick Keating'
>>> names.get_first_name()
'Bernard'
>>> names.get_first_name(gender='female')
'Christina'
>>> names.get_last_name()
'Szczepanek'

Recently I had a need to store a snapshot of every state of particular model instance in a Django application. I basically needed version control for the rows in my database tables. When searching for applications that provided this feature, which I call model history, I found many different approaches but few good comparisons of them.  In an attempt to fill that void, I’m going to detail some of my findings here.

django-reversion

The django-reversion application was started in 2008 by Dave Hall. Reversion uses only one table to store data for all version-tracked models. Each version of a model adds a new row to this table, using a JSON object representing the model state. Models can be reverted to previous versions from the admin interface.  This single-table version structure makes django-reversion very easy to install and to uninstall, but it also creates problems when model fields are changed.

django-revisions

The django-revisions application was created by Stijn Debrouwere in 2010 because the existing Django model history applications at the time were abandoned or suffered from fundamental design problems. Revisions uses a model history method called same-table versioning (design details outlined here). Same-table versioning adds a few fields to each version-tracked model which allows it to record the most recent version of each model as well as old versions in the original model table. Model changes are simplified because they change all versions at once and no new tables need to be added to use revisions (just new fields on existing tables). The only problem I found with revisions was that it does not currently support database-level uniqueness constraints. Adding unique=True to a model field or a unique_together Meta attribute will result in an error. Currently uniqueness constraints must be specified in a separate way for Revisions to honor them when saving models.

django-simple-history

The django-simple-history application was based on code originally written by Marty Alchin, author of Pro Django. Marty Alchin posted AuditTrail on the Django trac wiki in 2007 and later revised and republished the code in his book Pro Django in 2008, renaming it to HistoricalRecords. Corey Bertram created django-simple-history from this code and put it online in 2010.

Simple History works by creating a separate “historical” model for each model that requires an audit trail and storing a snapshot of each changed model instance in that historical model. For example, a Book model would have a HistoricalBook created from it that would store a new HistoricalBook instance every time a Book instance was changed. Collisions are avoided by disabling uniqueness constraints and model schema changes are accepted by automatically changing historical models as well. This method comes at the cost of creating an extra table in the database for each model that needs history.

My conclusions

When testing these three applications myself, I immediately eliminated django-reversion because I needed to allow easy model schema changes for my project. I found that both django-revisions and django-simple-history worked well with schema migrations through South (which I use on everything). Django-revisions worked better for data migrations in South (due to only needing to change one model), but the uniqueness constraint problems with django-revisions would have been problematic for some of my models. So eventually I settled on django-simple-history.

Revealing the primary key of an entity is often not desirable. An astute visitor of the website mentioned above may be able to guess information from the URL such as how many book reviews are available on the website or how old specific reviews are.

The code snippet below demonstrates one way to use a unique but cryptic identifier for an object without needing to change the way primary keys are generated. There are two notable extensions to the basic Django Model in the below code:

1. The encrypted_pk and encrypted_id model properties return an AES-encrypted version of the primary key as a 13 character base-36 string.
2. The get method of the default manager can be queried with an encrypted primary key by using the keyword argument encrypted_pk.

Feel free to use this code however you want.