late.am

Flask-PyMongo, Back from the Dead

2018-06-21T09:00:00-04:00

Long ago, when I worked at MongoDB I created Flask-PyMongo to make it easy for programmers using Flask to use the database. Fast forward almost 8 years, during which time I wasn't a consistent user of either Flask or MongoDB, and Flask-PyMongo has fallen into disrepair.

MongoDB, PyMongo, and Flask have moved on, and Flask-PyMongo hasn't been kept up to date. There are more than twice as many forks as pull requests, a GitHub ratio I'm not proud of. Fotunately, the future for Flask-PyMongo is bright.

False Starts and New Beginnings

At PyCon US 2017, I first had the idea to restore Flask-PyMongo. PyCon always has this effect on me, but sadly the effect is often short lived. In 2017, I got as far as mentioning plans for a 2.0 release, but did not go into any detail, nor begin to make any progress toward that goal.

This year at PyCon 2018, I once again had the urge to work on Flask-PyMongo. I had the same conversation with Jesse, decided, again, to jump from 0.x to 2.0, and even came up with the same technical plan as the previous year (about which more below). All without realizing that I had been down this road once before.

I can now confidently say that Flask-PyMongo 2.0 is (soon to be) a real thing, and it will set the stage for easier maintainability into the future and a better experience for users and contributors. Flask-PyMongo 2.0 will be released in early July, and pre-release versions are available today.

What's Changing

Flask-PyMongo 2.0 is not backwards compatible!

A lot of the historical problems with Flask-PyMongo have cenetered on the confusing and difficult configuration system. Originally, I envisioned that users would want configuration abstracted from PyMongo itself, and created a system where you could set Flask configurations for MONGO_HOST, MONGO_PORT, and MONGO_DBNAME, and be off to the races. For a while this worked, and many users seemed to like it. Unfortunately, there are quite a lot of configuration options for PyMongo, so the list of configurations grew. Worse, PyMongo and MongoDB are under active development, and grow and lose features over time. Attempts to make Flask-PyMongo version-agnostic added tremendous complexity to the configuration system, and evidently frustrated many users over the years.

In any event, it turns out that there's a better way to configure PyMongo -- with MongoDB URIs. Most hosted PyMongo services already provide configuration information in exactly this format. Going forward in 2.0, MongoDB URIs are the preferred configuration method for Flask-PyMongo. Flask-PyMongo will only look for or respect a single Flask configuration variable, MONGO_URI.

If you prefer, you may also pass positional and keyword arguments directly to Flask-PyMongo, which will be passed through to the underlying PyMongo MongoClient object.

Flask-PyMongo no longer supports configurating multiple instances via Flask configuration. If you wish to use multiple Flask-PyMongo instances, you must configure at least some of them using a URI or direct argument passing.

Flask-PyMongo 2.0 also clarifies the support policy for versions of Flask, PyMongo, MongoDB, and Python that are supported. For Flask and PyMongo, it supports "recent" versions -- those versions with releases in the preceding 3 years (give or take). For MongoDB, we follow the MongoDB server support policy, and support versions that are not end-of-lifed. For Python, we support 2.7 for as long as it is supported by the CPython core maintainers; and the most recent 3 versions of the 3.x series. For an exact list of supported versions and combinations, see the build matrix.

What You Should Do

If you are a Flask-PyMongo user and you are using the 0.x series, you should immediately pin a particular version. Flask-PyMongo 2.0 is not backwards compatible, so you should take steps to ensure that you don't accidentally break your application.

If you are already using a URI for Flask-PyMongo configuration, or if that is an easy change for you, I would appreciate if you could upgrade, test compatibility, and report any issues on GitHub. You can install Flask-PyMongo 2.0 pre-releases with pip install --pre flask-pymongo. You may also want to follow the general discussion and release notices in issue #110.

I also hope for Flask-PyMongo to be a place that supports Flask and MongoDB with more than just connection assistance. Please suggest ideas and propose contributions!

Now Seeking PyGotham Program Committee Members

2018-05-13T12:11:00-04:00

marc thiele

Last year’s PyGotham had a great program committee of four organizers and eight volunteers. Unfortunately due to the constrained timeline of the 2017 conference and everyone’s individual schedules, we had only a few chances to meet, and never as the full group. The major consequence of this is that the final talk decisions were effectively made by a smaller group, mostly conference staff. We failed, by way of logistics, in our mission to make PyGotham as community focused as possible. This year we aim to do better.

Changes to the Program Process

As a guiding principal for programming the conference, we want the organizers to act more like coaches than players in selecting the talks. We retain the right to change the lineup, but we want most of the calls to be made by the volunteer committee members.

To that end, this year we’re going to experiment with a different process than in previous years. We will form sub-committees from among the volunteers, and these subcommittees will each program a mini conference of talks about a particular topic area. The final topic breakdown will be guided by the set of talks we get before the close of our CFP later this month, but we expect committees on Data Science & Machine Learning, Web Programming, Testing, and so on.

Each subcommittee will be given a sub-set of talks from the total list as a suggested starting point. These talks will be initially ordered by the rank order of public votes (2018 talk voting will open a few days after the CFP closes), but picking the top N won’t necessarily make the best conference. Each subcommittee should feel free to nominate talks from other topic areas’ lists if there’s a strong reason to (or if we have made a mistake in categorization).

The job of the subcommittee will be to apply a human touch and to arrange the talks, to the extent possible, into “mini-tracks” of 2 or 3 talks. We will in all likelihood pick a few more talks for each subcommittee than can actually fit into the final conference — we expect there to be some cases of speakers being selected more than once, and we will need some speakers to be on a wait-list in case anyone drops out between now and October. Still, it is my hope that we can arrange these mini tracks into the final PyGotham schedule more or less as-is.

All of the review by the subcommittees will be anonymous, so after all the committees have produced their mini-tracks, the organizers will de-anonymize the talk list to ensure that we haven’t selected multiple talks by a single speaker.

Getting Involved

Here’s where we need help from you: so far, the program committee doesn’t exist! If you think you’ll have a few hours to spare over the first half of June, and you want to help shape PyGotham, then we want to hear from you. Drop us a line at program@pygotham.org, and let us know if there’s a particular topic area you’re interested in helping to review. Thanks in advance for your help!

Diversity at PyGotham

2018-04-27T10:45:00-04:00

This year, for the first time, PyGotham is asking everyone who has submitted a talk to our Call for Proposals to fill out a brief demographic survey to help support our efforts to ensure that PyGotham is representative of the Python community we believe in, welcoming to all our conference attendees, and helps to broaden and advance the conversation on diversity and representation that the wider software development community is undertaking.

Saaleha Bamjee

Why is Diversity Important to Us?

Without any outside influence, experience has shown us that tech conference speakers tend to predominantly come from a single background: male career programmers. There are exceptions, but this is the pattern we see most often at PyGotham and other tech conferences.

At PyGotham we want to do more than reflect the current demographics of professional developers. We want to hear from scientists and designers, from teachers and students, from community organizers and political activists, from everyone. Anyone with an interest in technology, be it academic, recreational, or professional, has something interesting to give a talk about. Often, we are too close to our work to notice what is unique, misunderstood, or confusing about it, but with a little effort, great topics can be found everywhere.

In order to tap into that variety of topics beyond those of interest just to professional programmers, we have to engage with speakers outside of that group. This is, in a word, “diversity.”

The Diversity Survey

The organizers debated taking this step back and forth. Might collecting this information frighten away speakers from under-represented groups, exactly those we most want to attract? Would it spark controversy and enrage majority members who nevertheless take every opportunity to cast themselves as victims? Would anyone even bother to fill it out?

The argument that won the day, ultimately, is that in order to improve the diversity of speakers at PyGotham, we first need to know something about what kind of diversity we have. Last year, I eyeballed the list of speakers and guesstimated that about 23% of talks had a female speaker (some talks had multiple speakers, so we were probably a bit lower as a share of speakers in general). This is comparable to women’s representation numbers at a variety of large tech companies, but as they are all pursuing internal improvements to their diversity and inclusion programs, we too want to aim higher.

Of course, I don’t mean to simplify all of “diversity” or “inclusion” to just the share of women speakers. It is the only measure on which we have any data at all, even as unreliable as name-game guessing. It’s from this position that we decided to launch the survey this year.

What Information We Request

We will be asking about gender identity, ethnicity, level of speaking experience, amount of programming experience, and age.

How We Will Use the Results

PyGotham’s program selection process this year will be similar to last year’s: we will begin with a round of public, anonymous voting beginning shortly after the CFP ends in late May. During public voting in 2017, 116 people cast over 12,000 votes for the nearly 200 proposals we had. (I hope we can count on the community to show a similar amount of engagement this year!)

The public talk voting is anonymous. Demographic information about the author of each proposal will not be available to public voters nor PyGotham organizers during this time. To do so would risk introducing bias, and the purpose of public voting is to take the pulse of likely attendees with regards to their level of interest in the submissions we receive.

After public voting, a smaller committee of PyGotham organizers and volunteers from the community will work together to select the 60 talks that will make it into the final conference. The vast majority of this process is also anonymous, and will be conducted without regard to the demographic survey’s results.

After we have a very strong candidate talk list, we must de-anonymize the results to ensure that a) no individual speaker has too many talk slots (this is unfair both to that speaker and to others who, by definition, don’t get those slots), and b) to ensure that we have the proper balance of speaker backgrounds. We will use the survey results to ensure that among the major axes, and for the information that we have collected, that our final speaker lineup represents at least as much diversity as is present in our overall submission pool.

The often frustrating reality of putting together a conference schedule is that the committee is often forced to choose a small number of talks out of a pool of excellent proposals. Voting helps us break the tie in one dimension — how can we produce a conference that the attendees will enjoy and learn from — and the demographic survey results will help us break ties in another dimension, to favor having a more diverse rather than less diverse speaker lineup.

Further up the Pipeline

Even if every submitter fills out the survey, by the time we are making individual talk decisions, it is often too late to do much to move the needle on diversity. We can ensure that our speaker lineup is representative of the submitter pool, but what if even that pool doesn’t live up to our expectations for what PyGotham could be?

Building on the success of last year’s Proposal Brainstorming workshop, we are holding at least 4 similar events this year, with one more event scheduled on May 2 with NYC PyLadies. At all of these events, attendees get a chance to work with one another to suss out what is great and talk-worthy about each of their experience.

We are also planning to proactively reach out to a number of NYC-area tech meetups and societies representing women, people of color, and other under-represented groups, in order to solicit their members to participate in our call for proposals.

PyGotham Talk Voting Retrospective

2017-09-25T12:20:00-04:00

PyGotham 2017 is less than two weeks away! (Do you have your ticket yet? Use code "IVOTED" for 25% off!) As we ramp up for the final conference, I wanted to take a moment to share some thoughts on our experiment with public talk voting this year.

Håkan Forss

First, Some Facts and Stats

In some ways, our job on the program committee was very easy: about 150 people submitted close to 200 different talks. Having such a wide pool of amazing talks ensured we could make PyGotham a conference that we're all looking forward to. But the abundance of great talks forced us to make painful choices: saying "no" to most of these talks was the hardest thing the committee had to do this year.

I would also be remiss if I didn't thank the 116 people who collectively cast over 12,000 votes during the public voting round of our process this year.

After public voting, there was another round of closed voting. Using the feedback from the public voting round, we created batches of talks about similar topics, and chose finalists from among each of those batches. Seven volunteers, along with several of the organizers, donated another ten or so hours of their time for this round of voting.

Many thanks to all of you who took time to submit a talk or vote to help us shape the schedule this year! We couldn't have done it without you.

What Worked Well

Public voting was a huge success. I never would have imagined we would have so many people cast so many votes. I expected we might get a weak signal of interest from a handful of die-hards, but instead we got a very strong and consistent ranking across all the talks, with each talk receiving about 62 votes -- each vote for a talk was positive, negative, or neutral. We had a good sample size and a lot of confidence that the vote results reflect the interests of our community. (I am not a statistician, and these statements have not been independently evaluated for mathematical rigor.)

In fact, all of our requests for help were met with overwhelming responses. We had to turn away volunteers from the batch voting round and program committee membership, since we wanted to keep the group small enough to work well together in meetings -- we try to follow the two pizza rule.

What Could Have Been Better

First, we were -- or at least I felt like we were -- behind schedule throughout much of the talk selection process. Prospective speakers who have submitted to our CFP have an expectation about when we will get back to them with final decisions, and we didn't meet that schedule. In part, this is because the voting process takes time -- with 195 proposals to read and consider, and in order to gather feedback from as wide an audience as possible, we have to allow both voting rounds to go on for a fairly long time. We should have been more transparent about the schedule, and we should have designed a schedule with announcements to speakers much earlier than this year.

The batch voting round didn't provide as clear a signal about which talks to accept into the final schedule as we had hoped. With so many excellent proposals, and so few slots to fill, consensus was elusive. Some of the organizers and I had to scramble at the last minute to make the process efficient and productive, and we ended up creating a straw man proposal for the final list, then soliciting feedback on this specific proposal from the committee members. This was very efficient, but I fear it may not have solicited as much input as another process.

Finally, on diversity: out of 60 talk slots, 14 of them, 23 percent, have a woman speaker, according to the very scientific "names that sound to Dan like they are women" methodology (and caveat all the obvious flaws inherent in this kind of counting). 23 percent is a bit higher than last year's PyGotham, but it's not as high as we'd like. I won't attempt here to quantify our speaker lineup according to other axes of diversity, other than to say that I'm sure we have room to improve there, as well.

Two main factors contributed to us falling short of our own goals: we failed to reach out early enough to some groups that specifically work with underrepresented groups, and we didn't assemble a list and individually contact speakers from underrepresented groups to ask them to submit to our CFP. My co-chair, A. Jesse Jiryu Davis, and I are both first-time organizers, and we definitely underestimated the amount of work and time that simply putting a conference together would take. This is an area he and I both care about, and aim to make major improvements in next year.

I would also like to have had a better gauge and guide for diversity than the "does this name sound like an X" method. I hope next year to be able to include at least one demographic question on our CFP, "are you a member of an underrepresented group in technology?", which we can use to better assess the results of our diversity and outreach efforts.

See You In A Few Weeks

One of the most rewarding aspects of this process, which can be demanding at times, is all of the people I've gotten to talk to, email with, and meet as an organizer of PyGotham. Though it sometimes felt like it was against all odds, we have a full schedule of amazing talks and speakers, and I'm looking forward to putting faces with a lot of these names in under two weeks. See you at the conference!

PyGotham Talk Voting is Open!

2017-07-20T10:00:00-04:00

For the first time, the PyGotham program committee is looking for you, our potential attendees, speakers, and community, to help us shape the conference by voting on the 195 talk proposals we've received. We're going to hold open voting until August 7th, after which the Program Committee will use the votes to inform our final selections for the conference.

How You Can Help

We want PyGotham to reflect the interests and desires of our community, so we ask that you share your time to help review the talk submissions. We're asking a single, simple question for each talk: would you like to see this talk in the final PyGotham schedule? You can give each talk either a +1 ("I would definitely like to see this talk"), a 0 ("I have no preference on this talk"), or a -1 ("I do not think this talk should be in PyGotham").

You can sign up for an account and begin voting at vote.pygotham.org. The talk review web site will present you with talks in random order, omitting the ones you have already voted on. For each talk, you will see this form:

Be sure to click "Save Vote" to make sure your vote is recorded. Once you do, a button will appear to jump to the next proposal.

Many thanks to Ned Jackson Lovely for sharing progcom, the US PyCon talk voting app, which we are using (with light adaptation) for PyGotham. Thanks Ned!

Submit to PyGotham's CFP

2017-06-07T09:02:00-04:00

I'm honored to be an organizer on the program committee for PyGotham 2017 this year, and I encourage you all to submit a talk to our little conference. PyGotham will be held October 6, 7, and 8 in New York City, and the Call for Proposals is open until July 18.

PyGotham attendees are diverse, come from varied backgrounds and skill levels, and have lives and interests beyond Python programming. Accordingly, the topics at PyGotham often vary a bit more widely than other programming language conferences. In the past, we have hosted talks about subjects from detecting sarcasm in audio files of speech, to open source stenography; from game programming to what we can learn about code review from J.R.R. Tolkien (sort of).

The threads connecting all these talks together are Python and New York, and the people interested and involved in both of those. If you're a member of either of these communities, if you think the PyGotham audience would like to hear your talk, then we want to see your proposal

The Importance of Outlines

PyGotham's CFP, like many other Python conferences, explicitly suggests you draft an outline of your talk as part of the submission. This has obvious benefits to those of us on the program committee -- we can get a clearer vision of what your talk will actually be like -- so we ask for it. But you may not be interested in why the outline is a benefit to us on the program committee, so let me tell you why it is a boon to you:

The outline is where you really solidify what your talk is about. You may formulate your talk beginning from a library you want to explore, a programming trick you like, or a pithy title — each of us has a different creative process. But you won't really know if your idea can stand on its own until you've fleshed out what you want to say about it. Brandon Rhodes summarizes this nicely:

By the time I finish a proposal, I feel about halfway done writing the talk. That feeling is always hilariously inaccurate, but it hopefully signals the point at which I have gotten enough detail into the proposal for the committee to imagine what the talk will be like.

(From Example PyCon talk proposals)

Once you have an outline written, your abstract, longer description, and title practically write themselves, so I recommend you start with the outline first.
The next major benefit is structure. Some talks suffer from a lack of coherent organization, which makes it difficult for the audience to follow along. At the stage of an outline, you won't be too attached to particular phrasing, slide ordering, or example code, so you're more free to rearrange topics within your talk to be most comprehensible.

I like to structure my talks a bit like a five-paragraph essay, a technique I learned in high school. First, I'm going to introduce a problem, topic, or thesis; next, I will provide supporting evidence, and develop it for most of the duration of the talk; and finally, I return to the original problem statement or thesis and form a conclusion or suggest next steps.

I want my audience to have the experience that I have watching a documentary, rather than reading reference documentation. You should learn something, and you should be engaged throughout; but you shouldn't have to keep notes or refer back to previous pages (or slides). The moment when your ideas are complete enough to write down, but not so set in your brain that you're attached is the right time to organize this structure; that time is when you can write your outline.
Finally, your outline also answers an important question: do I have enough here to talk about for 25 minutes? Do I have more than enough, and should I request a 40 minute slot? Timing talks is tricky, especially if you haven't spoken much before, but it's hopeless to try to guess from a vague idea whether that can sustain a proper talk slot. You'll feel better about your proposal knowing how long each section will take, and using that framework to guide writing your actual talk once you're selected.

A lot of credit for the above belongs to Celia La, Sarah Guido, Jason Myers, and Jon Banafato, who joined me on a panel at the NYC Python meetup last week to discuss the PyGotham CFP and conference talk submissions in general.

Resources for the CFP

A few specific resources were mentioned during the panel, as well:

Write An Excellent Programming Blog by A. Jesse Jiryu Davis describes five common blog post archetypes. Not all are appropriate for a conference talk, but understanding these can help shape your idea indevelopment. Jesse is my co-chair on the PyGotham program committee.
Jesse also has an article about getting talks into PyCon with a lot of great advice.
Finally, although I have not yet read it all (it's long!), Hyneck Schlawack recently published a piece on how he prepares and gives conference talks.

Ask Me Anything

I've promised in several tweets to be available if you have any questions about PyGotham or the CFP, or if you want help brainstorming an idea. The best way to get in touch with me is on Twitter, so drop me a line.

PyGotham's Call for Proposals is open until July 18, so don't delay!

Introducing Tox-Docker

2017-05-21T10:05:00-04:00

Today I released Tox-Docker to GitHub and the Python Package Index. Tox-Docker is a plugin for Tox, which, you guessed it, manages one or more Docker containers during test runs.

Why Tox-Docker?

Tox-Docker began its life because I needed to test some code that uses the PostgreSQL JSONB column type. In another life, I might have done this by instructing Jenkins to first install Postgres, start the server, create a user, create a database, and so on. There's even a small chance that this would work well, some of the time -- so long as tests didn't fail, the build didn't die unexpectedly without cleaning up, multiple tests didn't run at once, and so on. In fact, I've done exactly this sort of hackery a few times in the past already. It is dirty, and often requires manual cleanup after failures.

So, when confronted with the need to write tests that talk to a real Postgres instance, rather than reaching into my old toolbox I was determined to find a better solution. Docker can run multiple instances of Postgres at the same time in isolation from one another, which obviates the need for mutual exclusion of builds. Docker containers are lightweight to start, and easy to clean up (you can delete them all at once with a single command), so when the tests are done, we can simply remove it and move on.

There was still the question of how to manage the lifecycle of the container, though, which is where Tox comes in. Tox is a test automation tool that standardizes an interface between your machine (or your continuous integration environment) and your test code. Like Docker, Tox encourages isolation, by creating a clean virtualenv for each test run, free of old package installs, custom hacks, and so on. Tox already has a well-defined set of steps it runs to build your package, install dependencies, start tests, and gather results. Happily, it allows plugins to hook into this sequence to add custom behavior.

How Tox-Docker Works

Tox's plugins implement callback hooks to participate in the test workflow. For Tox-Docker, we use the pre-test and post-test hooks, which set up and tear down our Docker environment, respectively. Importantly, the post-test hook runs regardless of whether the tests passed, failed, or errored, ensuring that we'll have an opportunity to clean up any Docker containers we started during the pre-test hook. Finally, Tox plugins can also hook into the configuration system, so that projects using Tox-Docker can specify what Docker containers they require.

The simplest use of Tox-Docker is to specify the Docker image or images, including version tags, that are required during test runs. For instance, if your project requires Postgres, you might add this to your tox.ini:

[testenv]
docker = postgres:9.6

With Tox-Docker installed, the next time you run tox, you will see something like the following:

py27 docker: pull 'postgres:9.6'
py27 docker: run 'postgres:9.6'
py27 runtests: PYTHONHASHSEED='944551639'
py27 runtests: commands[0] | py.test test_your_project.py
============================= test session starts ==============================
platform linux2 -- Python 2.7.12, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /home/travis/build/you/your-project, inifile:
collected 3 items
test_your_project.py ...
=========================== 3 passed in 0.02 seconds ===========================
py27 docker: remove '72e2ffea02' (forced)

Tox-Docker has picked up your configuration, and pulled and started a PostgreSQL container, which it shuts down after the tests finish. This is equivalent to running docker pull, docker run, and docker rm yourself, but without the manual hassle.

Challenges and Helpers

Not every Dockerized component can be started and be expected to "just work". Most services will require or allow for some amount of configuration, and your tests will need some information back out of Docker to know how to use the services. In particular, we need:

A way to pass settings into Docker containers, in cases where the defaults are not sufficient
A way to inform the tests how to communicate with the service inside the container, specifically, what ports are exposed
A way to delay beginning the test run until the container has started and the application within it is ready to work

Tox-Docker lets you specify environment variables with the dockerenv setting in the tox.ini file:

[testenv]
docker = postgres:9.6
dockerenv =
    POSTGRES_USER=user_name
    POSTGRES_DB=database_name

Tox-Docker takes these variables and passes them to Docker as it launches the container, just as you might do manually with the --env flag. These variables are also made available in the environment that tests run in, so they can be used to construct a connection string, for instance.

Additionally, Tox-Docker interrogates the container just after it's started, to get the list of exposed TCP or UDP ports. For each port, Tox-Docker constructs an environment variable named after the container and exposed port, whose value is the host-side port number that Docker has mapped the exposed port to. Postgres listens on TCP port 5432 within the container, which might be mapped to port 32187 on your host system. In this case, an environment variable POSTGRES_5432_TCP will be set with value "32187".

Tests can use these environment variables to parameterize connections to the Dockerized services, rather than having to hard-code knowledge of the environment.

Finally, in order to avoid false-negative test failures or errors, Tox-Docker waits until it can connect to each of the ports exposed by the Docker container. This is not a perfect way to determine that the service inside is actually ready, but Docker provides no way for a service inside the container to signal to the outside world that it's finished starting up. In practice, I hope that this heuristic is good enough.

Future Work

The most obvious next step for Tox-Docker is to support Docker Compose, the Docker tool that lets you launch a cluster of interconnected containers in a single command. For the projects I am working with, I haven't yet had need of Docker Compose, but for projects of a certain level of complexity this will be preferable to attempting to manually manage this in tox.ini.

Installation and Feedback

Tox-Docker is available in the Python Package Index for installation via pip install tox-docker. Contributions, suggestions, questions, and feedback are welcome via GitHub.

Delete Your Dead Code!

2016-04-28T00:00:00-04:00

A few days ago, Ned Batchelder's post on deleting code made the rounds on HN, even though it was originally written in 2002. Here I want to echo a few of Ned's points, and take a stronger stance than he did: delete code as soon as you know you don't need it any more, no questions asked. I'll also offer some tips from the trenches for how to identify candidate dead code.

This is the second in an ongoing series on eating your vegetables in software engineering, on good, healthy practices for a happy and successful codebase. I don't (yet) know how long the series will be, so please stay tuned!

What Is Dead May Never Die

This heading isn't just an oh-so-clever and timely pop culture reference. Dead code, that is, code that can't possibly be executed by your program, is a real hindrance to the maintainability of your codebase. How many times have you gone to add what seemed like a simple feature or improvement, only to be stymied by the complexity of the code you have to work around and within? How much nicer would your life be if the practice of adding a feature or fixing a bug was as easy as you actually thought it would be during sprint planning?

Each time you want to make a change, you must consider how it interacts with each of the existing features, quirks, known bugs, and limitations of all the code that surrounds it. By having less code surrounding the feature you want to add, there's less to consider and less that can go wrong. Dead code is especially pernicious, because it looks like you need to consider interactions with it, but, since it's dead, it's merely a distraction. It can't possibly benefit you since it's never called.

The fact that dead code might never actually die is an existential threat to your ability to work with a given codebase. In the limit, if code that isn't called is never culled, the size of your application will grow forever. Before you know it, what might only be a few thousand lines of actual functionality is surrounded by orders of magnitude more code that, by definition, does nothing of value.

It's Got to Go

Ned (Batchelder, not Stark) was a little more nuanced and diplomatic than I'm going to be here:

Let's say you have a great class, and it has many methods. One day you discover that you no longer are calling a particular method. Do you leave it in or take it out?

There's no single answer to the question, because it depends on the class and the method. The answer depends on whether you think the method might be called again in the future.

From http://nedbatchelder.com/text/deleting-code.html.

I say: scorch the earth and leave no code alive. The best code is code that you don't even have.

For those less audacious than I, remember that version control has your back in case you ever need that code again.

That said, I've never experienced a need to add something back that I have previously deleted, at least not in the literal sense of adding back in, line for line, verbatim, a section of code I'd previously deleted.

Of course I'm not talking about things like reverting wrong-headed commits here -- we're all human, and I make as many mistakes as the next person. What I mean is, I've never deleted a feature, shipped to production, then come back weeks, or months later and thought to myself, "boy howdy, that code I wrote a year or more ago was probably pretty good, so let's put it back now." Codebases live and evolve with time, so the old code probably doesn't fit with the new ideas, techniques, frameworks, and styles in use today. I might refer back to the old version for a refresher, especially if it's particularly subtle, but I've never brought code back in, wholesale.

So, do yourself -- and your team -- a favor, and delete dead code as soon as you notice it.

How Did We Get Here?

Ned's post goes into great detail on how and why dead code happens -- perhaps the person making a change didn't think the code would be gone forever, and so commented it out or conditionally compiled it. Perhaps the person making a change didn't know enough to know that the code was actually dead (about which more later).

I'll add another hypothesis to the list: we might all just be lazy. It's definitely easier not to do something (i.e. to leave the code as-is) than to do something (delete it).

Laziness is, after all, one of the three great virtues of a programmer. But the Laziness that Larry Wall was talking about isn't this kind, but another kind: "The quality that makes you go to great effort to reduce overall energy expenditure." Viewed this way, deleting dead code is an act of capital-L Laziness -- doing something that's easy now to prevent yourself from having to do something hard later. We could all stand to develop more of this kind of Laziness, what I like to think of as "disciplined laziness," in our day-to-day habits.

How Do We Get Out Of Here?

I spend most of my time programming in Python, where, unfortunately, IDEs can't usually correctly analyze a complete codebase and identify never-called code automatically. But, with a combination of discipline and some run-time tooling, we can attack this problem from two sides.

For simple cases, a better sense of situational awareness can help identify and remove dead code while you're making changes. Imagine you're working on a particular function, and you notice that a branch of an if/else phrase can't be executed based on the valid values of the surrounding code. I call this "dead code in the small," and this is quite easy to reason about and remove, but it does require a bit more effort than one might ordinarily expend.

Until you develop the habit of noticing this during the course of your ordinary programming routine, you can add a step to your pre-commit checklist: review the code around your changes for any now-dead code. This could happen just before you submit the code to your co-workers for review (you do do code review, right?) so that they don't have to repeat that process while reading through your changes.

Another kind of dead code happens when you remove the last usage of a class or function from within the code you're changing, without realizing that it's the last place that uses it. This is "dead code in the large," and is harder to discover in the course of ordinary programming unless you're lucky enough to have eidetic memory or know the codebase like the back of your hand.

This is where run-time tooling can help us out. At Magnetic, we're using Ned's (yes, the same Ned) coverage.py package to help inform our decisions about dead code. Ordinarily coverage is used during testing to ensure that your test cases appropriately exercise the code under test, but we also use it within our code "running as normal" to get insight into what is or isn't used:

import coverage

cov = coverage.Coverage(
    data_file="/path/to/my_program.coverage",
    auto_data=True,
    cover_pylib=False,
    branch=True,
    source=["/path/to/my/program"],
)
cov.start()

# ... do some stuff ...

cov.stop()
cov.save()

This sets up a Coverage object with a few options that make the report more usable. First, we tell coverage where to save its data (we'll use that later to produce a nice HTML report of what is and isn't used), and ask it to automatically load and append to that file with auto_data=True. Next we ask it not to bother calculating coverage over the standard library or in installed packages -- that's not our code, so we'd expect that a lot of what's in there might not be used by us. It's not dead code that we need to maintain, so we can safely ignore it. We ask it to compute branch coverage (whether the true and false conditions of each if statement are hit). And finally, we point it out our sources, so that it can link its knowledge of what is or isn't called back to the source code for report computation.

After our program runs, we can compute the HTML coverage report like:

$ COVERAGE_FILE=/path/to/my_program.coverage coverage html -d /path/to/output

Which generates a report like:

(A complete example coverage HTML coverage report is available as part of the coverage.py docs.)

The lines highlighted in red are lines that were never hit during the recorded execution of the program -- these are candidate lines (and, by extension, methods) for dead code elimination.

I'll leave you with three warnings about using this approach to finding and removing dead code:

Be careful when reviewing the results of a coverage run -- the fact that a line or function wasn't executed during a single run of the program doesn't mean they're necessarily dead or unreachable in general. You must still check the codebase to determine whether they're completely dead in your application.
Computing coverage means your program needs to do more work, so it will become slower when run in this mode. I wouldn't recommend running this all the time in production, but in a staging environment or in targeted scenarios you'll probably be OK. As always, if performance is an important concern, you should definitely measure what impact coverage has before you run it.
Finally, dont trust code coverage reports from testing runs to find dead code. Some code might be dead, save for the tests that excercise it; and some code might be alive, but untested!

Parting Thoughts

To you, dear reader, I must apologize. I left out an important part of Ned's blog post when I quoted him earlier. He says:

There's no single answer to the question, because it depends on the class and the method. [...] A coarse answer could be: if the class is part of the framework, then leave it, if it is part of the application, then remove it.

If you're an author of a library or framework, rather than an application, then the question of dead code becomes in some ways harder and in other ways easier. In essence, you can't ever remove a part of your public API (except during major version bumps). Essentially, all of your public API is live code, even if you, yourself, don't use it. But behind the public interface, dead code can still happen, and it should still be removed.

Delete your dead code!

Demystifying Logistic Regression

2016-04-22T00:00:00-04:00

For our hackathon this week, I, along with several co-workers, decided to re-implement Vowpal Wabbit (aka "VW") in Go as a chance to learn more about how logistic regression, a common machine learning approach, works, and to gain some practical programming experience with Go.

Though our hackathon project focused on learning Go, in this post I want to spotlight logistic regression, which is far simpler in practice than I had previously thought. I'll use a very simple (perhaps simplistic?) implementation in pure Python to explain how to train and use a logistic regression model.

Predicting True or False

Logistic regression is a statistical learning technique useful for predicting the probability of a binary outcome -- true or false. As we've previously written, we use logistic regression with VW to predict the likelihood of a user clicking on a particular ad in a particular context, a true or false outcome.

Logistic regression, like many machine learning systems, learns from a "training set" of previous events, and tries to build predictions for new events it hasn't seen before. For instance, we train our click prediction model using several weeks of ad views (impressions) and ad clicks, and then use that to make predictions about new events to determine how likely the user is to click on an impression.

Each data point in the training set is further broken down into "features," attributes of the event that we want the model to learn from. For online advertising, common features include the size of the ad, the site on which the ad is being shown, the location of the user viewing the ad, and so on.

Logistic regression, or more accurately, Stochastic Gradient Descent, the algorithm that trains a logistic regression model, computes a weight to go along with each feature. The prediction is the sum of the products of each feature's value and each feature's weight, passed through the logistic function to "squash" the answer into a number in the range [0.0, 1.0].

The standard logistic function. Public Domain image from Wikipedia by Qef.

Applying a logistic regression model is therefore relatively easy, it's a simple loop to write. What surprised me most during the hackathon project is that training a logistic regression model, coming up with the values for the weights we multiply by the features, is also surprisingly concise and straightforward.

An Example

Let's consider a click data set. Each row in the table is an event we've (hypothetically) recorded, either an impression which did not get a click, or an impression plus click, along with a series of attributes about the event that we'll use to build our regression model.

Size	Site	Browser	State	EventType
160x600	wunderground.com	IE	AZ	impression
160x600	ebay.com	IE	IA	click
728x90	msn.com	IE	MS	impression
160x600	ebay.com	Chrome	FL	impression
728x90	popularscience.tv	Chrome	CA	impression
320x50	weather.com	Chrome	NB	impression
300x250	latimes.com	Firefox	CA	impression
728x90	popularscience.tv	IE	OR	impression
300x250	msn.com	Chrome	MI	impression
728x90	dictionary.com	Chrome	NY	impression
300x250	ebay.com	Safari	OH	impression
320x50	msn.com	Chrome	FL	click
728x90	dictionary.com	Chrome	NJ	impression
728x90	urbanspoon.com	Firefox	OR	impression
728x90	msn.com	Chrome	FL	impression
728x90	latimes.com	Chrome	AZ	impression
728x90	realtor.com	Chrome	CA	impression
728x90	deviantart.com	Chrome	DC	impression
728x90	weather.com	IE	OH	click
728x90	msn.com	IE	AR	impression
728x90	msn.com	IE	MN	impression
300x250	wunderground.com	Chrome	MI	impression
300x250	dictionary.com	IE	OK	impression
300x250	popularscience.tv	Chrome	IN	impression
300x250	dictionary.com	Chrome	MA	impression
300x250	weather.com	Chrome	VT	impression
320x50	ebay.com	Chrome	OH	click
300x600	popularscience.tv	IE	NJ	impression
728x90	weather.com	IE	WA	impression
300x250	weather.com	IE	MO	impression

These 30 data points will form our training data set. The model will learn from these examples what impact the attributes -- size, site, browser, and US state -- have on the likelihood of the user clicking on an ad that we show them.

In reality, to build a good predictive model you need many more events than the 30 that I am showing here -- we use tens of millions in our production modeling pipelines in order to get good performance and coverage of the variety of values that we see in practice.

To convert this data set to work with logistic regression, which computes a binary outcome, we'll treat each click as a target value of 1.0, also known as a "positive" event, and each impression as 0.0, or a "negative" event.

We also need to account for the fact that each of our features is a string, but logistic regression needs features that have values that it can multiply by its weight to compute the sum-of-products. Our features don't have numeric values, just some set of possibilities that each attribute can take on. For instance, the size attribute is one of three ad sizes we work with, 300x250, 728x90, or 160x600. Features that can take on some set of discrete values are called "categorical" features.

We could assign a numeric value to each of the possibilities, for instance use the value 1 when the size is 300x250, 2 when the size is 728x90, and 3 when the size is 160x600. This would work mechanically, but won't produce the outcome that we want. Since the weight of the feature "size" is multiplied by the value of the feature in each event, this would mean that we've decided, arbitrarily, that 160x600 is three times more "clicky" than 300x250, which the data set may not bear out. Moreover, for some attributes, like site, we have many many possible values, and determining what numeric value to assign to each of the possiblities is tedious, and will further amplify the problem as described here.

The proper solution to this problem is as effective as it is elegant: rather than having 1 feature, "site" with values like "ebay.com" or "msn.com", we create a much larger set of features, one for each site that we have seen in our training or evaluation data set. For a given training data example, if the "site" column has value "msn.com", we say that the feature "site:msn.com" has value 1, and that all other features in the "site:..." family have value 0. Essentially, we pivot the data by all the values of each categorical feature. Of course, the total number of features in our training set becomes very large, but, as we will see, SGD can accomodate this gracefully.

Stochastic Gradient Descent In Action

At the start of the hackathon, we wrote some very simple Python code to serve as a reference for how logistic regression should work. Python reads almost like pseudocode, so thats what we'll show here, too.

We decided to model examples as Python dictionaries, with a top-level field indicating the actual outcome (1.0 for clicks, 0.0 for impressions), and a nested dictionary of features. Here's what one looks like:

{
    "actual": 1.0,
    "features": {
        "site": "ebay.com",
        "size": "160x600",
        "browser": "IE",
        "state": "IA",
    }
}

Given an example and a dictionary of weights, we can predict the outcome straighforwardly:

def predict(weights, example):
    score = 0.0
    for feature in extract_features(example):
        score += weights.get(feature, 0.0)
    score += weights.get("bias", 0.0)
    return 1.0 / (1 + math.exp(-score))

We initialize the score to 0, then for each feature in the example, we get the weight from the dictionary. If the weight doesn't exist (because the model hasn't been trained on any examples containing this feature), then we assume 0 as the weight, which leaves the score as-is.

Additionally, we add in a "bias" weight, which you can think of as a feature that every example has. The "bias" weight helps SGD correct for an overall skew in our training data set one way or another. It's possible to leave this out, but it's generally helpful to keep it.

The expression at the end of this function is the logistic function, as we saw above, which squashes the result into the range 0 to 1. We do this because we are trying to predict a binary outcome, so we want our prediction to be strictly between these two boundary values. If you do the math, you'll see that for an empty model -- one with nothing in the weights dictionary -- we will predict 0.5 for any input example.

Generating the features we use in SGD from our example is similarly straightforward:

def extract_features(example):
    return example["features"].iteritems()

Since our features are categorical, we can take advantage of Python's dictionary to generate (key, value) pairs as the actual features. Though all our examples have values for each of the four attributes we're considering, they each have different values -- thus the tuples generated by extract_features will be different for each example. This is our conversion from attributes with a set of possible values, to features which carry the implicit value 1.

Finally, we have enough code in place to see how stochastic gradient descent training actually works:

def train_on(examples, learning_rate=0.1):
    weights = {}
    for example in examples:
        prediction = predict(weights, example)
        actual = example["actual"]
        gradient = (actual - prediction)

        for feature in extract_features(example):
            current_weight = weights.get(feature, 0.0)
            weights[feature] = current_weight + learning_rate * gradient

        current_weight = weights.get("bias", 0.0)
        weights["bias"] = current_weight + learning_rate * gradient

    return weights

For each example in the training set, we first compute the prediction, and, along with the actual outcome, we compute the gradient. The gradient is a number between -1 and 1, and helps us adjust the weight for each feature, as we'll see in a moment.

Next, for each feature in the example, we update the weight for that feature by adding or subtracting a small amount from the current weight, the learning_rate. Here we can now see how the gradient works in action. Consider the case of a positive example (a click). The actual value will be 1.0, so if our prediction is high (close to 1.0), we'll adjust the weights up by a small amount; if the prediction is low, the gradient will be large, and we'll adjust the weights up by a correspondingly large amount. On the other hand, consider when the actual outcome is 0.0 for a negative event. If the prediction is high, the gradient will be close to -1, so we'll lower the weight by a relatively large amount; if the prediction is close to 0, we'll lower the weight by a relatively small amount.

For the first example, or more generally, the first time a given feature is seen, the feature weight is 0. But, since each example has a different set of features, over time the weights will diverge from one another, and converge on values that tend to minimize the difference between the actual and predicted value for each example.

Real-World Considerations

What I've shown here is a very basic implementation of logistic regression and stochastic gradient descent. It is enough to begin making some predictions from your data set, but probably not enough to use in a production system as is. But I've taken up enough of your time with a relatively long post, so the following features, then, are left as an excerise to the reader to implement:

The train_on function assumes that one pass through the training data is sufficient to learn all that we can or need to. In practice, often several passes are necessary. Different strategies can be used to determine when the model has "learned enough," but most rely on having a separate testing data set, composed of real examples the model will not train on. The error on these examples is critical to understanding how the model will perform in the wild.
As it is written here, we only support categorical features. But what if the data set has real-valued features, like the historical CTR of the site or the user viewing the impression? To support this example, we need to multiply the weight by the feature value in predict, and multiply the gradient by the feature value in train_on.
This code doesn't perform any kind of regularization, a technique used to help combat over-fitting of the model to the training data. Generally speaking, regularization nudges the weights of all features closer to 0 every iteration, which over time has the effect of reducing the importance to the final prediction of infrequently-seen features.

Should Sense

2015-11-05T00:00:00-05:00

I've just finished reading Dan Pupius' Medium article on Software Engineer Traits, where he lays out four characteristics of great software engineers. I agree with his assessment, and want to add one more quality to the discussion -- something I call "should sense".

In a nutshell, "should sense" is that thing which drives a great developer to dig deeper, to not be satisfied with the answers she or he has at hand. It manifests as:

Should it take this long to read ten gigabytes of data off disk?
Should we have to reach past the public API of this library to get it to do what we want?
Should it require fifty lines of hairy code to solve this problem?
Should this test be failing one in a hundred times?

I don't mean things like "should I add another branch to this long if/else block," or "should I extract this method to share it with the other place that's using it." Those are all questions that good developers ought to be able to answer, and that inexperienced ones can learn how to answer. Refactoring is both a skill and a sense of judgement in its own right (possibly one that deserves its own post). But that's not the kind of "should" I'm talking about here.

What I mean is a gut feel for when something is wrong with a working program. Should sense is associated with, and maybe drives, good habits like questioning assumptions, performance testing, diving deeper, and developing expertise with frameworks and tools. Or perhaps it's that engineers who have those good habits have or are on the road to building their should sense.

It's possible that Dan meant some of these things when he enumerated his four traits. In particular, it seems likely that some portion of "Curiosity" and "Awareness" are touching on some of the ideas I'm mentioning here, even if he doesn't spell it out explicitly.

Jesse Davis, who was kind enough to review an early draft of this post, pointed out that should sense is a lot like what Ira Glass' describes in his essay Taste. He desbribes how artists and creative workers start out with great taste, a sense of what they want to create, but that it takes a lot of hard-won experience to grow their skills to the point where their own work lives up to their taste, to what they know it could be.

In technology, I think the opposite happens: it's fairly easy to get started with programming (especially today in the age of a seemingly infinite number of online tutorials, coding schools, and MOOCs), and at first, the world seems your oyster, as you can make the machines do ever more impressive things. What I think we don't learn until we've fought the hard battles, probably many times, is the technical equivalent of good taste: knowing what your code should look like, and how it should behave.

Click Prediction with Vowpal Wabbit

2015-06-01T00:00:00-04:00

My co-worker Sam Steingold and I have a new article at the Magnetic engineering blog about how we predict clicks in real-time online advertising.

Optimize Python with Closures

2015-05-07T00:00:00-04:00

Magnetic's real-time bidding system, written in pure Python, needs to keep up with a tremendous volume of incoming requests. On an ordinary weekday, our application handles about 300,000 requests per second at peak volumes, and responds in under 10 milliseconds. It should be obvious that at this scale optimizing the performance of the hottest sections of our code is of utmost importance. This is the story of the evolution of one such hot section over several performance-improving revisions.

Real Time Bidding

Real Time Bidding, or RTB, is a technique by which many internet ads delivered. When you visit a website using RTB, a request is sent to dozens or hundreds of "bidders" which have to quickly decide whether they want to show you an ad, and if so, how much they would like to pay. RTB bidders have up to 100 milliseconds to make this decision -- any slower and you won't win the auction no matter how much you bid -- so performance is key.

At Magnetic, we do much of our targeting in real time during the bid request. We use a combination of filters to qualify ad campaigns for the particulars of the bid request. Here we'll consider just one of the several types of filters we use, one which checks that the content category of the page where an ad will be shown against a set of categories that we want to target.

Of course, different campaigns have different targeting criteria, and thus different filter configurations. Additionally, each campaign may use a different subset of filters. On average, we end up calling about 150 filters per bid request, or 4.5 million filters per second total at peak times, so ensuring maximal performance is essential.

First Approach: Classes

The obvious way to implement such a filter is to use a class to store the configuration, thus allowing multiple instances of the filter for different campaigns. At bidding time, we look up the set of instances which represent the filters for each of the campaigns the user was targeted for, and call each filter to determine if the user is eligible for the campaign. Such a class might look like:

class PageCategoryFilter(object):
    def __init__(self, config):
        self.mode = config["mode"]
        self.categories = config["categories"]

    def filter(self, bid_request):
        if self.mode == "whitelist":
            return bool(
                bid_request["categories"] & self.categories
            )
        else:
            return bool(
                self.categories and not
                bid_request["categories"] & self.categories
            )

This code would pass most code reviews, and is a fairly straightforward implementation of the idea we've thus far described only in prose. (We pass in a dictionary of configuration, rather than direct arguments, since we have different types of filters and want all of them to expose the same interface to the code that sets them up and calls them.)

Unfortunately, though perhaps unexpectedly given the topic of this post, there is a performance problem with this approach, especially for a block of code that will be called millions of times per second.

The Bound Method Problem

Superficially, accessing the categories attribute and filter method appear to be doing about the same amount of work -- both access the attribute of an instance. Unfortunately, looks can be deceiving:

>>> a_filter = PageCategoryFilter(dict(mode="whitelist", categories=["foo", "bar", "baz"]))
>>> a_filter.categories
frozenset(['baz', 'foo', 'bar'])
>>> a_filter.filter
<bound method PageCategoryFilter.filter of <PageCategoryFilter object at 0x107f13910>>

As expected, categories returns the attribute's value, but accessing filter returns a bound method object. What is a bound method? It's the magic that allows Python to insert the self argument when you call the method.

Specifically, an instance's methods are access via a descriptor, a Python feature that allows some code to be executed to satisfy the results of an attribute access expression. When Python executes the definition of a class, it wraps each function in a descriptor whose job is to supply the self argument. Later, when you access the attribute for the method, Python calls __get__ on the descriptor, and supplies the instance as an argument. This allows the method descriptor to rewrite the call to the underlying function to include the self argument. For more, see Christ Beaumont's excellent Python Descriptors Demystified, or watch a short video version of it.

From a performance perspective, the key difference between ordinary attribute access and method calls is that each time you call a method, or rather each time you access the attribute which refers to a method, the Python VM must execute some additional code to create the bound method. It must do more work in order to provide the self argument. None of this extra work is ordinarily visible -- it doesn't show up in stack traces, for instance -- but it does take some small amount time to execute, and if you call methods often enough, that can add up.

Second Approach: Functions

If bound methods are a problem, perhaps we can arrange for all the pertinent arguments just to be passed in to an ordinary function. This should avoid the overhead of supplying the self attribute:

def page_category_filter(bid_request, config):
    if config["mode"] == "whitelist":
        return bool(
            bid_request["categories"] & config["categories"]
        )
    else:
        return bool(
            config["categories"] and not
            bid_request["categories"] & config["categories"]
        )

As we'll see in benchmarks, this code slightly outperforms the class-based implementation by avoiding the need for bound method calls. But is this as fast as we can make the filter go?

The Dictionary Access Problem

In either branch of this function, we do three dictionary accesses: one for "mode", one for the "categories" item in the bid request dictionary; and one for the "categories" item in the configuration dictionary. Since none of these accesses is repeated in any individual call of this function, it makes no sense to assign the result of the dictionary accesses to a local variable and thus "cache" the result.

"But wait," you say, "aren't dictionaries fast?"

Ordinarily, and algorithmically, yes, dictionaries are quite fast. However, hidden behind those square brackets is quite a bit of work: Python must hash the key, apply a bit-mask to the hash, and look for the item in an array that represents the storage for the dictionary. Edge cases can create collisions which take even longer to resolve. For a more thorough exploration of how dictionaries are implemented in Python, see Brandon Rhodes' talk The Mighty Dictionary.

In practice, for frequently-called code the cost of dictionary item access, however small, is amplified and adds up.

Final Approach: Closures

So if even dictionaries are too slow, how can we resolve this performance crisis? We can turn the "variables" stored in the dictionary into local variables (sort of), which are the fastest to access since they are stored in an array and referenced by pre-computed offset. Accessing a local variable in Python is nearly instantaneous, or as close to it as we can get writing Python code.

A closure is a function which uses variables defined in its enclosing scope -- a nested function. Python recognizes that the (inner) function uses variables that aren't in its argument list, and makes those variables available to the inner function. Such an inner function is said to "close over" the variables it uses from the outer scope, hence "closure".

Because the number and names of closures is known at compile time, the VM can use an array to track the closed-over variables, in the same way as it uses an array to track local variables and constants. For more on how Python builds and executes a function, see Exploring Python Code Objects.

We translate our page category filter into a closure by creating a factory function which returns another function, the closure itself. From the perspective of the calling code, this looks and feels a lot like creating an instance of a class.

def make_page_category_filter(config):
    categories = config["categories"]
    mode = config["mode"]
    def page_category_filter(bid_request):
        if mode == "whitelist":
            return bool(bid_request["categories"] & categories)
        else:
            return bool(
                categories and not
                bid_request["categories"] & categories
            )
    return page_category_filter

As we'll see, this implementation outperforms either of the others, because we avoid two of the three dictionary lookups, and replaces them with fast access to the closed-over variables categories and mode. Additionally, callers don't suffer from the bound method problem, since page_category_filter is just a regular function with no magically inserted arguments.

The Proof is in the Timing

Each of the three methods shown here is "fast" in the sense that filtering any single bid request takes very little actual time -- less than a microsecond on my machine in all cases. However, since this code (and other code like it) is called inside of what amounts to very tight loops, even tiny differences in the run time of a single invocation amount to a lot overall.

To fully exercise the code paths, we'll create four filter configurations: a whitelist with some allowed categories, a blacklist with some forbidden categories, an empty whitelist, and an empty blacklist. We'll call each of them an equal number of times in a straightforward timing loop. Here's an example using the first implementation:

filters = [
    PageCategoryFilter(dict(mode="whitelist", categories=["foo", "bar", "baz"])),
    PageCategoryFilter(dict(mode="blacklist", categories=["foo", "bar", "baz"])),
    PageCategoryFilter(dict(mode="whitelist", categories=[])),
    PageCategoryFilter(dict(mode="blacklist", categories=[])),
]

bid_request = {"categories": set("bat")}
start = time.time()
for _ in xrange(N):
    for a_filter in filters:
        a_filter.filter(bid_request)
end = time.time()

Averaged over 15 million calls to each filter in CPython 2.7.9 on my 2013 MacBook Pro, 2.6GHz Core i7, I get:

Approach	Per-Call Time	Speedup
Class	0.4082 us	--
Function	0.3744 us	8.2%
Closure	0.3213 us	21.3%

Feel free to check out the benchmark script.

What about PyPy?

At Magnetic, we've switched from running all of our performance-sensitive Python code from CPython to PyPy, so let's consider whether these code optimizations are still appropriate. I used the same benchmark setup as above, but with PyPy 2.5.1 and a warmup run for each implementation to allow PyPy time to JIT:

Approach	Per-Call Time	Speedup
Class	0.0432 us	--
Function	0.0554 us	-28.3%
Closure	0.0431 us	0.1%

PyPy is incredibly fast, around an order of magnitude faster than CPython in this benchmark. Because PyPy performs more advanced optimizations than CPython, including many optimizations for classes and methods, the timings for the class vs. closure implementations are a statistical tie.

Curiously, the function implementation is actually slower than the class approach in PyPy. PyPy is able to optimize code using classes more than code using dictionaries, because it is able to specialize code for each class in your program. Dictionaries, on the other hand, are a generic mapping data structure, and PyPy is not able to create specialized machine code for one particular "shape" of dictionary or another. Alex Gaynor covers this distinction in greater depth in his talk Fast Python, Slow Python from PyCon 2014.

So what have we learned?

The Zen of Python implores us to implement our code in the "one -- and preferably only one -- obvious way", which is likely one of the first two approaches presented here (depending on whether you favor functions or classes). Indeed, this is still good advice, as code is read far more often than it is written. However, there are cases where less obvious ways have distinct benefits. Or maybe I'm just Dutch.

We also must always remember our Knuth: "premature optimization is the root of all evil." At Magnetic, we pursued this optimization only because it made sense for our application, where the filters are called very frequently, and only after measurement showed that the real-time filters accounted for a large amount of our per-request processing time.

Many thanks to A. Jesse Jiryu Davis for reviewing a draft of this article.

Good Test, Bad Test

2015-04-20T00:00:00-04:00

A good test suite is a developer's best friend -- it tells you what your code does and what it's supposed to do. It's your second set of eyes as you're working, and your safety net before you go to production.

By contrast, a bad test suite stands in the way of progress -- whenever you make a small change, suddenly fifty tests are failing, and it's not clear how or why the cases are related to your change.

But how do you know if your test suite is "good"? At PyCon US 2015 I gave a talk that addressed three common myths about testing, which lead down a path of painful testing.

The talk, and this article, are heavily opinionated. As with all opinions, it's up to you, the reader, to decide whether they're applicable to your particular situation or not. In other words, you still need to use your judgement.

What's a "Good" Test?

I'll define a good test (and by extension a good test suite) as one that has the following virtues:

Fast. If a test is slow, developers won't run it locally. This will lead to broken CI builds, and more time spent fixing the code that broke the test afterwards, more context switching, lack of flow, and all the concomitant problems.
Selective. If a bug is introduced into the code under test, only one or a few test cases should fail. If all cases start failing, then it will require a too much time to figure out exactly what is wrong. Your tests should point you in the direction of what's wrong.
Repeatable. Tests should always give the same result if the code being tested hasn't changed. Any reliance in a test on timing, randomness, or external state (databases, the date or time, etc) will lead to wasted time spent investigating spurious failures.
Reliable. Your test should pass when the code is working, and fail when the code has a bug. Perhaps this is self-evident, but it bears repeating -- a test should not give false positives or false negatives.
Helpful. In particular, test error messages should be helpful. Seeing "user_has_logged_in() was False after user login" tells you exactly what's wrong, whereas "AssertionError: False is not True" requires you to figure it out for yourself.

As we'll see, the myths, and the mistakes they cause developers to make while writing tests, come from good intentions, but can lead to violations of one or more of the above principles.

Myth 1: 100% Coverage

This myth is pernicious primarily because of how common it is. Perhaps a team adopts code coverage monitoring after a few bouts with bugs that could easily have been prevented with tests, and there's a land rush to improve coverage. More tests must be better, right?

Beware: danger lurks in too-eagerly chasing full coverage. The truth of the matter is, there are some tests that simply shouldn't be written: they add too much complexity to your codebase, and provide very little value.

For example, consider tests that exercise parts of your application that interact with some external resource, not under your control, like a third party API your application works with:

There's a reason we draw it as a cloud... (source, modified)

Notice that thing in the middle. There's a reason we draw it as a cloud: we don't really know what happens in there. Anything could happen in there. You might get a very slow connection; you might get corrupted bytes back that don't make any sense; you might simply shout into the void, never to hear back from the third party; or you might be on a plane, with no access to the internet. Since you can't control it, you're totally subject to the whims of the internet and of the third party.

I call tests that rely on third parties in this way "Mutually-Assured Destruction" tests. Ordinarily, the external service is operating normally and your tests pass. But occasionally, something outside of your control is broken, and suddenly your test suite is reporting a bug that doesn't exist in your code. Either both sides work, or both sides fail, hence "MAD".

Perhaps this is controversial, but I recommend that you not write tests for this kind of code in the first place. It's failures don't tell you anything interesting or actionable about your codebase.

Instead, try to minimize the amount of MAD code, and make it as dumb as possible. Don't put business logic that needs thorough testing in-line with third party integrations, and make sure your integrations can report errors back to you. Be sure to test your exception reporting service, and keep notification rules up to date. If you depend on an external service for critical functionality, you can't ensure that it will always be available and correct; but you can be sure that you're notified when it goes down, and have a plan in place to mitigate the impact.

Myth 2: Assert About Everything

Assertions are how tests tell us whether our code is working or not. Without assertions, tests literally wouldn't do anything useful. So, if we add more assertions, we'll increase our confidence that our test does what we want it to do, right?

Too many assertions, and especially the wrong kinds of assertions create a maintenance hazard in your test suite. Let's illustrate the point with an example, a function which parses a log line from a web server in a hypothetical pipe-separated format, returning a dictionary:

def test_it_parses_log_lines(self):
    line = "2015-03-11T20:09:25|GET /foo?bar=baz|..."

    parsed_dict = parse_line(line)

It is tempting to use the familiar assertEqual with a dictionary whose contents are identical to the parsed result we expect:

def test_it_parses_log_lines(self):
    line = "2015-03-11T20:09:25|GET /foo?bar=baz|..."

    parsed_dict = parse_line(line)

    self.assertEqual({
        "date": datetime(2015, 3, 11, 20, 9, 25),
        "method": "GET",
        "path": "/foo",
        "query": "bar=baz",
    }, parsed_dict)

This will certainly work, assuming the parse function is implemented correctly. But what kind of error messages will we get from this test? Take a look:

AssertionError: {'date': datetime.datetime(2015, 3, 11, 20, 9, 25), 'path': '/foo', 'method': 'G [truncated]... != {'date': datetime.datetime(2015, 3, 11, 20, 9, 25), 'path': '/foo?', 'method': ' [truncated]...

Can you see what the error is? No? Neither can I. Fortunately, we can fix this fairly easily:

def test_it_parses_log_lines(self):
    line = "2015-03-11T20:09:25|GET /foo?bar=baz|..."

    parsed_dict = parse_line(line)

    self.assertEqual(
        datetime(2015, 3, 11, 20, 9, 25),
        parsed_dict["date"],
    )
    self.assertEqual("GET", parsed_dict["method"])
    self.assertEqual("/foo", parsed_dict["path"])
    self.assertEqual("bar=baz", parsed_dict["query"])

Now when our test fails, we can see exactly what's going on:

AssertionError: '/foo' != '/foo?'

We fix the bug, and move on with our lives. Eventually, we'll expand the functionality of the parser, and add more test cases. Inevitably, these assertions will be copied and pasted, with some modification:

def test_it_parses_get_request_log_lines(self):
    # ...
    self.assertEqual("GET", parsed["method"])
    self.assertEqual("/foo", parsed["path"])
    self.assertEqual("bar=baz", parsed["query"])

def test_it_parses_post_request_log_lines(self):
    # ...
    self.assertEqual("POST", parsed["method"])
    self.assertEqual("/foo", parsed["path"])
    self.assertEqual("bar=baz", parsed["query"])

def test_it_parses_log_lines_with_oof_requests(self):
    # ...
    self.assertEqual("GET", parsed["method"])
    self.assertEqual("/oof", parsed["path"])
    self.assertEqual("bar=baz", parsed["query"])

Notice that only one assertion is actually changing in each test, but we're repeating most of the others. This means that if we break one feature -- say request path parsing -- we'll get failures from most of our test cases. In order to figure out what's actually broken, we'll have to dig through each failure and think hard about each test case to figure out what to fix.

You should ask yourself, for each test case, "What would I do when this test case fails?" We can minimize the effort required to transate a test failure into a fix by following one simple rule: make only one assertion per test case. Yup, only one.

Let's rewrite this test suite using the new rule:

def test_it_parses_request_method(self):
    # ...
    self.assertEqual("GET", parsed["method"])

def test_it_parses_request_path(self):
    # ...
    self.assertEqual("/foo", parsed["path"])

def test_it_parses_query_string(self):
    # ...
    self.assertEqual("bar=baz", parsed["query"])

def test_it_gives_none_when_no_query_string(self):
    # ...
    self.assertIsNone(parsed["query"])

Now, when we break request path parsing, we'll only get one test case failure, and the test case name and failure traceback will point us directly at what we need to fix: the request path parsing portion of our code.

When you adopt this style of test design, you'll also find that the way you write your test cases changes. Before, we had test cases representing some special cases we expect in the real world: an "oof" request, a POST request, etc.

Now, our test cases reflect the orthogonal behaviors of the code under test: it parses the request method, path, query string, etc. Not only are the test failures more instructive, but our test cases now serve as better documentation: scanning the names tells us what the parse_line function actually does (and, by extension, what it doesn't do).

Myth 3: Mock Makes Tests Better

Mock is a powerful library for simulating and replacing Python objects in tests. You can safely monkey-patch, check what calls were made against your mocks, raise exceptions, and more.

Common reasons for using mock are to reduce dependence on external, possibly slow resources, like databases, to isolate testing to one layer of your code, or to control "inputs" to a method or function that come from some nested call. Mock is certainly capable of doing all of these things.

But with great power comes great responsibility -- used unwisely, mock can wreak more havoc on your testing than the thing you were trying to replace would have. Take, as an example, a suite of tests on a web application with login functionality:

class DBClient(object):
    def authenticate(self, username):
        """Returns True if the user is authenticated.
        """
        # WARNING: Don't use this code in production!
        return True

class MyApp(object):
    def login(self, username):
        if not self.db.authenticate(self.params["username"]):
            return Response(status=401)
        # do more stuff ...
        return Response(status=200)

Since we want to test the behavior of the login method under two scenarios -- when the user is authenticated, and when the user is not. We might use mock to control the return value of the internal call to authenticate:

def test_it_returns_200_on_success(self):
    db = mock.Mock(spec=DBClient)
    db.authenticate.return_value = True

    app = MyApp(db)
    response = app.login("dcrosta")

    self.assertEqual(200, response.status)

def test_it_returns_401_on_failure(self):
    db = mock.Mock(spec=DBClient)
    db.authenticate.return_value = False

    app = MyApp(db)
    response = app.login("dcrosta")

    self.assertEqual(401, response.status)

Now we'll add a new feature to our application -- user permissions. For simplicity, let's have the authenticate method return not only whether the user is authenticated, but what roles in our application the user has:

class DBClient(object):
    def authenticate(self, username):
        """Returns a tuple of (authenticated, list_of_roles).
        """
        # WARNING: Still don't use this code in production!
        return (True, ["admin"])

If we stop here, we've introduced a bug into our code: we haven't updated login to handle the new return value from authenticate. But our test suite isn't telling us that, because the test mocks still supply the old format of return value (just a boolean).

In the worst case, you might not notice this bug until it has been in production for some time, since this particular error won't raise any exceptions (a non-empty tuple is considered truthful in Python).

Whenever you write a test, you have to consider: what would cause this test case to pass when it ought to fail? With mock, this can happen whenever the object you're replacing has changed.

The solution for cases like this is to use a "verified double" -- a replacement object that closely, hopefully identically, emulates the behavior of the thing you're replacing. Here's a double for the DBClient class we were previously mocking:

class StubDBClient(object):
    def __init__(self):
        self.users = {}

    def register(self, username, roles):
        self.users[username] = roles

    def authenticate(self, username):
        if username not in self.users:
            return (False, [])
        return (True, self.users[username])

Had we been using this double all along, the test cases might have looked like this:

def test_login_returns_200_on_success(self):
    db = StubDBClient()
    db.register("dcrosta")

    app = MyApp(db)
    response = app.login("dcrosta")

    self.assertEqual(200, response.status)

def test_login_returns_401_on_failure(self):
    db = StubDBClient()

    app = MyApp(db)
    response = app.login("dcrosta")

    self.assertEqual(401, response.status)

Not only are the tests more readable -- they demonstrate the full set of actions that have to take place to achieve the result being verified -- but the test cases now demonstrate that we've got a bug in our code until we update the login method to understand the new return value from authenticate.

Since the double itself isn't a one-liner, you're likely to share the double across all test cases that use this functionality. Now you only hvae one place in your tests to update when you change the interface of the real object. Once you change it, your test suite will tell you all the other areas you need to revisit. Once the tests pass, your refactor is complete.

Final Thoughts

Too often, testing is an afterthought. The exciting work that we all crave is to build the actual working code. But no code is created in an eternally perfect and pristine state. We will eventually have to modify it to add or remove functionality, to address performance concerns, to work with new frameworks, etc. Tests are our safety net when we perform these operations on a codebase: they tell us what we've broken, and what we have to fix.

It's important to acknowledge that tests can smell just as well as real code. We deal with code smell in a test suite the same as we would in any other type of code -- with thought and care. A well designed test suite, one which lives up to the virtues of good testing, will be your best friend in a long-lived codebase.

Crickets

2015-03-31T00:00:00-04:00

It's been real quiet around here. Like, tumbleweed-rolling-by quiet.

Lessonslearnedlastnight

I've been lazy. I'm going to change that. We've started a tech blog at work, and it's gotten me back in the mood to blog. I'll be cross-posting a few articles, and also posting some original work here as time permits.

I've also updated the blog to use Pelican, which might have broken your feeds, in case anyone reads that way. My apologies if you suddenly get a slew of duplicates.

How to Rebuild a Virtualenv in Heroku

2012-09-22T00:00:00-04:00

Today I managed to get myself a little bit stuck with Heroku's otherwise seamless Python support. In an earlier revision of my app, I had used -e to install an editable version of a package (the version of the package I needed had not yet been released to PyPI, so I pointed pip at Github). Since then, the version I need has been released.

So, update requirements.txt, then git push heroku master, and all is well, right? Not so fast...

Here's what actually happened:

Traceback (most recent call last):
  File "/tmp/.../pip-1.1-py2.7.egg/pip/basecommand.py"
    status = self.run(options, args)
  File "/tmp/.../pip-1.1-py2.7.egg/pip/commands/install.py"
    requirement_set.install(install_options, global_options)
  File "/tmp/.../pip-1.1-py2.7.egg/pip/req.py"
    requirement.uninstall(auto_confirm=True)
  File "/tmp/.../pip-1.1-py2.7.egg/pip/req.py"
    assert (link_pointer == dist.location), ('Egg-link %s does '
        'not match installed location of %s (at %s)') %
        (link_pointer, self.name, dist.location)
AssertionError: Egg-link ../../../../src/package does not match
installed location of package (at /tmp/.../src/package)

It turns out this is a known bug in Pip, and will hopefully soon be fixed. Unfortunately, I don't think I can use pip to install an editable version of pip which fixes the bug in which pip can't uninstall editable versions of packages on Heroku. (Sorry for that sentence).

So I did what any good internet citizen would do: tweet first, google later. Google turned up a patch to the Heroku Python buildpack that was merged several months ago, and appeared to do the right thing -- sadly, there was no apparent way to get the CLEAN_VIRTUALENV environment variable to the compile phase of my app.

Fortunately, at this point my friend Andy saw the tweet, and provided the critical hint: the user-env-compile Heroku labs feature. This gave me a way to ask the buildpack compiler to create a new virtualenv for me. On my next git push heroku master, I got a nice message: "CLEAN_VIRTUALENV set, rebuilding virtualenv" -- however, the same bug as before when installing requirements :'(

I reviewed the patch more closely, then traced through the buildpack file, and found that the VIRTUALENV_DIRS variable that the compiler uses to clean up the virtualenv is never set, so nothing was being deleted, and despite the message, my virtualenv was not being cleared and recreated. However, since the compiler is just a shell script, the same trick that got CLEAN_VIRTUALENV to the compiler could also be used to set VIRTUALENV_DIRS. (And I've submitted a patch so that in the future this won't be necessary.)

So, to put it all together, here's what you need to do:

$ heroku labs:add user-env-compile
$ heroku config:set CLEAN_VIRTUALENV=yes
$ heroku config:set VIRTUALENV_DIRS=.heroku/venv
$ git push heroku master
$ heroku config:unset CLEAN_VIRTUALENV
$ heroku config:unset VIRTUALENV_DIRS
$ heroku labs:disable user-env-compile

Hope this saves someone some time!

A Modest Proposal for Tech Conferences

2012-07-28T00:00:00-04:00

After more difficulty than expected, I'm at PyOhio, and once again I found myself not taking notes during today's sessions, but madly scrambling to open browser tabs to each of the open source projects, blog posts, and other online resources the presenters mentioned. It's a rush to keep up, particularly with laundry-list-of-links slides!

The sessions at PyOhio (and most other conferences I attend) are recorded and posted online, but I never seem to make the time to look for the links after the fact -- and you can't click a link in a video of a projected slide anyway.

So this has gotten me thinking: since presenters typically submit talks through web forms, why not also ask them for a listing of the resources that are a part of their presentation, and preserve this in the online talk notes? A few words on the subject of the link would be nice as well, but with not that much cleverness, this could be extracted from the page. Knowing that this will be available after the fact frees presenters to move fluidly through their presentations, and allows attendees to focus on the content rather than trying to type URLs as quickly as possible.

Perhaps not everyone will be willing to take the time to fill out what, to them, surely seems like a busy-work form, but with the right application of social pressure, and after the expectation has been set that speakers will do this, I suspect that a majority would be willing to make the effort.

What the heck is an xrange?

2012-06-18T00:00:00-04:00

Pop quiz: how would you implement Python's xrange (known in Python 3.x as range) in Python and without making a list or any other sequence type?

If you answered "with a generator," you'd be wrong! And it's likely because you've only ever used an xrange in code like:

for i in xrange(10):
    # do something 10 times

In this case (where the xrange is only used as an "argument" to a for loop), a generator would probably suffice. But, in fact, xrange is implemented as an object, not a function or generator. Why? Let's find out.

The Sequence Protocol

The range builtin returns a list, which (as the de facto sequence type) implements all the sequence protocol methods. xrange was intended to replace range (and did in Python 3.x), so it, too, must support all that a sequence supports.

So, what does a sequence support? Sequences have a finite, ordered bunch of elements, which means they have a length (they support len), they support indexed access (with the [] operator), and most importantly, they can be iterated -- in forward order with a for loop, and in reverse order with the help of the reversed builtin. These behaviors are all implemented by way of "dunder" methods, and typically accessed using language syntax (as with for ... in) or builtins (as with len).

Sequences also support two instance methods: index, which returns the 0-based position in the sequence of the first occurrence of the argument, and count, which returns the number of occurrences of the argument.

Perhaps surprisingly, we can actually implement all of this functionality with constant-time and constant-space operations, using only the bounds of the xrange.

Building an `xrange`

As a quick refresher, both range and xrange take between 1 and 3 arguments. A single argument defines the length of the sequence (with an implicit starting value of 0). Two arguments define the starting point and ending point (actually, the second argument is the next integer after the last one that will be a part of the sequence). A third argument defines a step value, the number between each of the elements of the sequence. All arguments must be integers.

With that in hand, we can begin building our implementation of xrange:

class xrange(Sequence):
    """Pure-Python implementation of an ``xrange`` (aka ``range``
    in Python 3) object. See `the CPython documentation
    <http://docs.python.org/py3k/library/functions.html#range>`_
    for details.
    """

    def __init__(self, *args):
        if len(args) == 1:
            start, stop, step = 0, args[0], 1
        elif len(args) == 2:
            start, stop, step = args[0], args[1], 1
        elif len(args) == 3:
            start, stop, step = args
        else:
            raise TypeError('xrange() requires 1-3 int arguments')

        try:
            start, stop, step = int(start), int(stop), int(step)
        except ValueError:
            raise TypeError('an integer is required')


        if step == 0:
            raise ValueError('xrange() arg 3 must not be zero')
        elif step < 0:
            stop = min(stop, start)
        else:
            stop = max(stop, start)

        self._start = start
        self._stop = stop
        self._step = step
        self._len = (stop - start) // step + bool((stop - start) % step)

Update: thanks to sltkr for a correction to the _len calculation on Hacker News.

Most of this is error checking and adjustment of values for impossible sequences (like a sequence starting at 0 and going to 10 stepping by -1; in these cases, we collapse the xrange to a zero-length sequence with the same start value).

Importantly, since the entire sequence is "known" at initialization time, and is immutable, I precompute the length of the sequence, which is useful in the implementation of several of the methods, as we will see.

With our computed _start, _stop, _step, and _len attributes in hand, we can implement equality, representation, and length trivially:

def __repr__(self):
    if self._start == 0 and self._step == 1:
        return 'xrange(%d)' % self._stop
    elif self._step == 1:
        return 'xrange(%d, %d)' % (self._start, self._stop)
    return 'xrange(%d, %d, %d)' % (self._start, self._stop, self._step)

def __eq__(self, other):
    return isinstance(other, xrange) and \
           self._start == other._start and \
           self._stop == other._stop and \
           self._step == other._step

def __len__(self):
    return self._len

Nothing really surprising here.

Next we'll implement index, as it serves as a basis for several other methods.

To find the index of an element in an immaterial sequence, we have to do a little math. First, we compute the difference between the element we're looking for, and the start value for the sequence. Next, we compute the quotient and remainder using the divmod builtin. If the remainder is 0, then the element may be a member of the sequence (it falls on one of the values that might be produced by adding some number of _steps to the _start). If the quotient is between 0 (inclusive) and the sequence's length (exclusive), then this is definitely a member, and we can return its index; if not, we raise ValueError to indicate it was not found:

def index(self, value):
    """Return the 0-based position of integer `value` in
    the sequence this xrange represents."""
    diff = value - self._start
    quotient, remainder = divmod(diff, self._step)
    if remainder == 0 and 0 <= quotient < self._len:
        return abs(quotient)
    raise ValueError('%r is not in range' % value)

Note that we're safe from ZeroDivisionError in the divmod call, since we've previously ensured that _step is not zero.

With this in hand, we can trivially implement __contains__ (the dunder method that is called by x in foo syntax for checking membership):

def __contains__(self, value):
    """Return ``True`` if the integer `value` occurs in
    the sequence this xrange represents."""
    try:
        self.index(value)
        return True
    except:
        return False

Implementing count is trivial, as well, since an element can only occur zero or one times in an xrange:

def count(self, value):
    """Return the number of ocurrences of integer `value`
    in the sequence this xrange represents."""
    return int(value in self)

Element access with the [] operator (implemented with the __getitem__ dunder method) is relatively easy, except for two complications: we have to support negative indexes, for sequence access from the end; and we have to support slice access.

The former is easy, as we can just add the length of the sequence to the index we're attempting to access, and proceed with bounds-checking as usual. Since the behavior of slices of sequences is somewhat subtle, we implement it in a separate method for clarity:

def __getitem__(self, index):
    """Return the element at position ``index`` in the sequence
    this xrange represents, or raise :class:`IndexError` if the
    position is out of range."""
    if isinstance(index, slice):
        return self.__getitem_slice(index)
    if index < 0:
        # negative indexes access from the end
        index = self._len + index
    if index < 0 or index >= self._len:
        raise IndexError('xrange object index out of range')
    return self._start + index * self._step

def __getitem_slice(self, slce):
    """Return an xrange which represents the requested slce
    of the sequence represented by this xrange.
    """
    start, stop, step = slce.start, slce.stop, slce.step
    if step == 0:
        raise ValueError('slice step cannot be 0')

    start = start or self._start
    stop = stop or self._stop
    if start < 0:
        start = max(0, start + self._len)
    if stop < 0:
        stop = max(start, stop + self._len)

    if step is None or step > 0:
        return xrange(start, stop, step or 1)
    else:
        rv = reversed(self)
        rv._step = step
        return rv

Finally, we come to iteration, which is (I suspect) what most people use xrange objects for. Remember that xrange objects act like any other sequence -- you can have multiple, independent iterators iterating the same underlying sequence, so we have to implement a separate object for the iterator. The implementation of the iterator is about as simple as you might expect: it tracks the last value it returned, and adds the _step value to that value, then returns it for each element it iterates:

class xrangeiterator(Iterator):
    """An iterator for an :class:`xrange`.
    """

    def __init__(self, xrangeobj):
        self._xrange = xrangeobj

        # Intialize the "last outputted value" to the value
        # just before the first value; this simplifies next()
        self._last = self._xrange._start - self._xrange._step
        self._count = 0

    def __iter__(self):
        """An iterator is already an iterator, so return ``self``.
        """
        return self

    def next(self):
        """Return the next element in the sequence represented
        by the xrange we are iterating, or raise StopIteration
        if we have passed the end of the sequence."""
        self._last += self._xrange._step
        self._count += 1
        if self._count > self._xrange._len:
            raise StopIteration()
        return self._last

Put it all together

I've published the code for the xrange object, along with a comprehensive test suite, to my GitHub.

Closing Thoughts

Python already has xrange (or, in Python 3, range), as a built-in, implemented efficiently in C, so re-implementing it in Python is largely a learning exercise, rather than code which you should actually use.

However, this implementation is superior to CPython's implementation in two ways:

I have backported support for slicing, index, and count from Python 3 to Python 2-compatible code; if these features are important and you need to support multiple versions of Python, this may be of use to you.
CPython 2.x's implementation of xrange does not implement __contains__, so the expression 1000 in xrange(1001) actually has to scan all 1000 elements in the xrange before it can determine if 1000 is a member of the sequence. This was corrected in CPython 3, but very long lists in CPython 2.x will have __contains__ performance characteristics more similar to lists than to 2.7 xranges. This implementation uses a constant-time algorithm to compute membership, so it may actually be faster than the CPython 2.x implementation for very large sequences.

The exec Statement and A Python Mystery

2012-04-30T00:00:00-04:00

A few weeks ago I examined Python code objects using the dis module. In that post, I showed several examples of executing code objects created at runtime using the exec statement; here we'll explore compile()'s compliment, exec, how to invoke it, and some of the quirks of using it.

Note that, like my post on code objects, the information here is only relevant to Python 2.x, specifically, Python 2.7. Unlike that post, at least one technique presented here will not work in Python 3.x -- in 3.x, exec is a function, and modifications to locals will not propagate to the calling scope. (Thanks to comex for pointing this out on Hacker News.)

A Refresher: `compile()`

Before we dive in to exec, recall that compile() converts a string of Python source code into a code object, using the same machinery as the usual Python compilation process. It takes three arguments: the source code to compile, the filename of the source code (for which it is customary to use "" for source code not originating in a file), and the "mode" of the compilation (for which "exec" is the most useful mode, at least for these sorts of experiments).

Make It Go

By itself, a code object is not terribly useful. Sure, it encapsulates all the information needed to execute some Python code, but it is just a noun, with no way to run itself.

>>> code_object = compile("""
... print "Hello, world"
... """, '<string>', 'exec')
>>> code_object()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'code' object is not callable

This isn't terribly useful. Of course we can use the dis module to inspect the bytecode, and its attributes to find out what variables it uses, the source file from which this code object was created, etc, but what we really want is to run it. Enter the exec statement:

>>> exec code_object
Hello, world

That's more like it!

Playing With `exec`

Imagine that we have a function which accepts a code object and an argument x:

>>> def exec_code_and_return_x(code_object, x):
...     exec code_object
...     return x
...

You might expect this function to return the value of x without modification -- after all, no code in the function actually manipulates x in any way. However (depending on the code object), you can get some very unexpected outcomes:

>>> my_code_object = compile("""
... x = x + 1
... """, '<string>', 'exec')
>>> exec_code_and_return_x(my_code_object, 1):
2

But you won't always get the unexpected results you expect:

>>> my_code_object = compile("""
... del x
... """, '<string>', 'exec')
>>> exec_code_and_return_x(my_code_object, 1)
1

So what's going on here?

On `exec` and Scoping

Without any additional instructions, exec uses the current global and local namespaces to execute your code. This means that (as we saw above), code objects being exec'd can modify variables in scope when it is exec'd (as well, obviously, as any globals the code object happens to manipulate).

You can also customize (to a certain extent) the scope given to the code object by using the in "arguments" to exec:

>>> code_globals = {}
>>> code_locals = {}
>>> code_object = compile("""
... x = 1
... """, '<string>', 'exec')
>>> exec code_object in code_globals, code_locals
>>> print code_locals
{'x': 1}

Note that builtins, as their name suggests, are always available, even when execing with a empty scopes:

>>> code_globals = {}
>>> code_locals = {}
>>> code_object = compile("""
... print unicode("Hello, world")
... """, '<string>', 'exec')
>>> exec code_object in code_globals, code_locals
Hello, world
>>> code_globals.keys()
['__builtins__']

Solving Our Python Mystery

Returning to our mystery example from before:

>>> my_code_object = compile("""
... del x
... """, '<string>', 'exec')
>>> exec_code_and_return_x(my_code_object, 1)
1

Why can't the code object delete the local variable x? Specifically, why is the above code different from this:

>>> def delete_local_then_return_it():
...     x = 1
...     del x
...     return x
...
>>> delete_local_then_return_it()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in delete_local_then_return_it
UnboundLocalError: local variable 'x' referenced before assignment

What we're seeing here is the result of a Python performance optimization: rather than using a dictionary (or some other mapping object) for function locals, and Python bytecode instructions reference positions within this array. As a consequence of this optimization, dynamic code (such as that returned by compile(), or invocations of exec directly on a string) cannot modify the size of the locals array, since it is fixed at compile time.

Moreover, the implementation of exec, that is, of the EXEC_STMT opcode, arranges for a new Python stack frame to be pushed, which means that it has its own array of locals (those used by the code object being exec'd):

>>> import inspect
>>> def get_stack_depth():
...     frame = inspect.currentframe()
...     # begin depth at -1 to account for the extra
...     # frame created when calling get_stack-depth
...     depth = -1
...     while frame:
...             depth += 1
...             frame = frame.f_back
...     return depth
... 
>>> def show_stack_depths():
...     print 'stack depth in function', get_stack_depth()
...     exec compile("""
...     print "stack depth in exec", get_stack_depth()
...     """, '<string>', 'exec')
... 
>>> show_stack_depths()
stack depth in function 2
stack depth in exec 3

Code objects can, of course, modify the local variables on their own stack frame:

>>> def show_code_object_locals():
...     code_globals = {}
...     code_locals = {'x': 1}
...     exec compile("""del x""", '<string>', 'exec') in code_globals, code_locals
...     return code_locals
... 
>>> show_code_object_locals()
{}

Warnings and Gotchas

Using exec to run code dynamically at runtime is fun, but it is not without its costs.

First, I would be remiss if I did not strongly urge you never to use exec; Python provides no way (and, in fact, there is no general way) to ensure that the code you're attempting to execute will do no harm to your system. Think of this like Python's version of cross-site scripting attacks. If you are ever executing untrusted code submitted by end-users of your system, you're doing it wrong.

Secondly, executing code in this fashion is significantly slower than simply calling regular Python code, as the latter code path is heavily optimized in the interpreter, whereas calling dynamic code is not.

Finally, if you do need to use exec, you may find that it is more convenient to pass a string or file object as the first argument -- however, beware that this will incur the cost of compiling the source code each time the exec statement is encountered. If you are repeatedly execing the same piece of code, you will see significant performance improvement by first compile()ing and saving the resulting code object for re-use.

A Python "Cast Constructor"

2012-04-05T00:00:00-04:00

Very occasionally, I write code where I'm given an object of some class (usually from a library call), but I wish to use additional methods on that class as though they had been defined there. In some languages (Objective-C shines in this regard with categories), you can do this very naturally. In Python, most people probably resort to monkey patching to accomplish this.

This is Probably An Anti-Pattern

I want to preface this with an admission that I have no idea if this will generally work (please comment if you know of a reason why it won't!), and that I suspect that this is a horrible idea. And yet, here we go:

import copy

class Base(object):
    def __init__(self):
        self.base_init = True

    def foo(self):
        return 'base foo'

    def bar(self):
        return 'base bar'

class Child(Base):
    def __new__(cls, other):
        if isinstance(other, Base):
            other = copy.copy(other)
            other.__class__ = Child
            return other
        return object.__new__(cls)

    def __init__(self, other):
        self.child_init = True

    def bar(self):
        return 'child bar'


b = Base()
assert b.base_init == True
assert b.foo() == 'base foo'
assert b.bar() == 'base bar'
assert b.__class__ == Base

c = Child(b)
assert c.base_init == True
assert c.child_init == True
assert c.foo() == 'base foo'
assert c.bar() == 'child bar'
assert c.__class__ == Child
assert b.__class__ == Base

Have I made some glaring mistake? Is there something obviously wrong with this, other than the abuse of the language inherent in trying to do such a thing?

Edit: Thanks to Jesse Davis for the suggestion to use copy.copy. Also see the Stackoverflow question which inspired this post.

Exploring Python Code Objects

2012-03-26T00:00:00-04:00

Inspired by David Beazley's Keynote at PyCon, I've been digging around in code objects in Python lately. I don't have a particular axe to grind, nor some particular task to solve (yet?), so consider this post just some notes and ramblings that might be of interest (and my apologies if not).

Disclaimer: This post is about CPython version 2.7, though much of it is also likely true for other CPython versions (including 3.x). I make no claims to its accuracy or applicability to PyPy, Jython, IronPython, etc.

Step 0: What?

So first of all, what is a code object? Many people (particularly Python haters) claim that Python is an interpreted language, but all your Python code is actually compiled before it is ever executed. This goes even for code you write interactively in the Python shell. CPython implements a virtual machine that executes a stack-based bytecode. At runtime, executable things (functions, methods, modules, class bodies, lambdas, statements, expressions, etc) are all executed as bytecode by the Python virtual machine.

Code objects, then, are Python objects which represent some piece of bytecode, along with all that it needs to execute: a declaration of the expected argument types and counts, a list (not dictionary! more about which later) of locals, information about the source code from which the bytecode was generated (for debugging and printing stack traces), etc -- oh, and also (perhaps obviously), the bytecode itself, as a str (or, in Python3, bytes).

Though code objects represent some piece of executable code, they are not, by themselves, directly callable. To execute a code object, you must use the exec keyword or eval() function.

Step 1: Make some Code

Most of the time, you won't encounter code objects in ordinary Python programming. When you do, there's a very good chance that they were created and are managed for you by Python, without any special attention. In some cases, you might want to create code objects yourself, like in this post where we'll be experimenting with them:

>>> code_str = """
... print "Hello, world"
... """
>>> code_obj = compile(code_str, '<string>', 'exec')
>>> code_obj
<code object <module> at 0x1054c74b0, file "<string>", line 2>

Woohoo, your first code object!

The first argument to compile() is the string of Python code to be compiled, which should be obvious. The second defines the "filename" of the piece of code (here, as is conventional, we use <string> to indicate code attained from the interactive shell). The third is the type of compilation, which most often will be exec as you see here. The other choices for mode are eval, which is used for strings containing only a single expression, or single, in which the generated code object is expected to contain a single statement, whose return value is printed if it is not None (like in the interactive shell).

When using eval mode, if the code contains statements (as our example above does, it contains a print statement), compilation will fail with a syntax error:

>>> code_str = """
... print "Hello, world"
... """
>>> code_obj = compile(code_str, '<string>', 'eval')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 2
    print "Hello, world"
        ^
SyntaxError: invalid syntax

When using single, only a single statement is processed; multiple statements (or non-statements) will be ignored:

>>> code_str = """
... print "Hello, world"
... print "Goodbye, world"
... """
>>> code_obj = compile(code_str, '<string>', 'single')
>>> exec code_obj
Hello, world

What happened to my "goodbye"?

For the rest of the post, we'll stick with exec, which is the type of compilation Python does for you when importing modules.

Step 2: Open 'er Up

Let's go back to our first example, and have a look inside the code object to see what we have:

>>> code_str = """
... print "Hello, world"
... """
>>> code_obj = compile(code_str, '<string>', 'exec')
>>> dir(code_obj)
# dunder attributes excluded for readability
['co_argcount', 'co_cellvars', 'co_code', 'co_consts', 'co_filename',
 'co_firstlineno', 'co_flags', 'co_freevars', 'co_lnotab', 'co_name',
 'co_names', 'co_nlocals', 'co_stacksize', 'co_varnames']

These attributes are documented in the inspect module, but I'll highlight a few cool ones here:

First, we can see where our second argument to compile() ended up:

>>> code_obj.co_filename
'<string>'

And, perhaps surprisingly, our code represents an anonymous module (code compiled with exec mode is always treated as module-level code, though, of course, it can contain function or class definitions, or any other valid Python):

>>> code_obj.co_name
'<module>'

And, as we expect, a code object representing a Python module (that's effectively what our code string was -- a series of statements at the top-most level, that is, not indented at all) takes no arguments:

>>> code_obj.co_argcount
0
>>> code_obj.co_varnames
()

If we were to take a piece of code from a function which does have arguments, we'd see them here:

>>> def foo(x, y):
...     print x, y
... 
>>> foo.func_code
<code object foo at 0x1054b9830, file "<stdin>", line 1>
>>> foo.func_code.co_varnames
('x', 'y')
>>> foo.func_code.co_argcount
2

If you're curious, you can also see the raw bytecode that will be processed by the Python virtual machine:

>>> code_obj.co_code
'd\x00\x00GHd\x01\x00S'

I don't recommend trying to learn to read that directly, there's an easier way (hint: see the next section).

Finally, we have one constant object within scope, the string "Hello, world", which is printed by our code:

>>> code_obj.co_consts
('Hello, world', None)

Wait. Where's that None coming from?

A Detour into Code Disassembly

We can see exactly what's going on in our code object with the dis module, which disassembles code objects into a human readable series of bytecode instructions:

>>> dis.dis(code_obj)
  2           0 LOAD_CONST           0 ('Hello, world')
              3 PRINT_ITEM
              4 PRINT_NEWLINE
              5 LOAD_CONST           1 (None)
              8 RETURN_VALUE

Reading disassembled Python code requires a bit of experience, so let me walk you through it. The LOAD_CONST instruction reads a value from the co_consts tuple, and pushes it onto the top of the stack. The PRINT_ITEM instruction pops the top of the stack, and prints the string representation. PRINT_NEWLINE should be pretty self-explanatory.

Next we see the mysterious None. It turns out this is a bit of a quirk of the implementation details of the CPython virtual machine. Since function calls in Python (including "hidden" function calls, like those behind an import statement) are implemented with function calls in C in the Python virtual machine, modules actually have a return value -- this indicates to the Python virtual machine that execution of the module has completed, and control can be returned to the calling scope (i.e. the module in which the import statement appeared). I won't embarrass myself by trying to explain this further -- if you are interested, see Larry Hastingss PyCon presentation Stepping through CPython around 44:22 -- that video covers Python 3.x, but Python 2.7 does the same thing. If you're interested in this sort of implementation detail, then you should definitely watch the entirety of this video, and David Beazley's keynote as well.

Step 3: Interesting Internals

Many of the features we've looked at are clearly useful for a running Python virtual machine, but what about the human side of the story? What if we want to interactively debug code (using pdb or a similar tool), or get helpful, readable tracebacks from exceptions?

It turns out, code objects support this as well. As we've already seen, code objects indicate from which file they were generated, and this will obviously help in looking up source code; they also indicate the line number on which the source code for this code object begins:

>>> code_obj.co_firstlineno
2

And the mysterious co_lnotab attribute. To illustrate its purpose, we'll need a larger code snippet:

>>> code_str = """
... x = 1
... y = 2
... print x + y
... """
>>> code_obj = compile(code_str, '<string>', 'exec')
>>> code_obj.co_lnotab
'\x06\x01\x06\x01'

Hm, so what are we to make of this? Perhaps the dis module can help here again:

>>> dis.dis(code_obj)
  2           0 LOAD_CONST           0 (1)
              3 STORE_NAME           0 (x)

  3           6 LOAD_CONST           1 (2)
              9 STORE_NAME           1 (y)

  4          12 LOAD_NAME            0 (x)
             15 LOAD_NAME            1 (y)
             18 BINARY_ADD
             19 PRINT_ITEM
             20 PRINT_NEWLINE
             21 LOAD_CONST           2 (None)
             24 RETURN_VALUE

At the far left of (some) lines, is the line number of the Python source from which this code object was created (notice that 2 here corresponds to the value of code_obj.co_firstlineno). The next column is the offset into the code of the bytecode instruction, 0 bytes for the first instruction, 3 bytes for the second, and so on. The third column is the instruction name itself, and the fourth is the argument to the instruction, if any, along with the value of the argument in parentheses.

Now we can put this together with the co_lnotab (which stands for "line number table", by the way) to see how Python makes sense of the code objects' relation to their original source code:

>>> code_obj.co_lnotab
'\x06\x01\x06\x01'

After a little tinkering and trial and error, I realized that this is a series of pairs of bytes: the first is a length offset into the bytecode (6 bytes, which advances us to the second LOAD_CONST as seen in our disassembly), followed by a number of source lines of code that the skipped instructions appeared on.

We can confirm this theory by slightly modifying our source code, recompiling, and examining the co_lnotab attribute of the resulting code object:

>>> code_str2 = """
... x = 1
... 
... y = 2
... print x + y
... """
>>> code_obj2 = compile(code_str2, '<string>', 'exec')
>>> code_obj2.co_lnotab
'\x06\x02\x06\x01'

We've moved the second assignment down one line, so we see that in the second byte of co_lnotab, we are incrementing the "current line number" by two instead of by one.

We can also verify that the bytecode resulting from these two (slightly) different source codes is identical:

>>> code_obj2.co_code == code_obj.co_code
True

Since both the bytecode offset and line number offset at single (unsigned) bytes, one might wonder what happens if you have, say, 257 (or more) blank lines between statements in a Python source file? Let's see:

>>> thousand_blanks = '\n' * 1000
>>> code_str = """
... x = 1
... """ + thousand_blanks + """
... y = 2
... print x + y
... """
>>> code_obj = compile(code_str, '<string>', 'exec')
>>> code_obj.co_lnotab
'\x06\xff\x00\xff\x00\xff\x00\xec\x06\x01'

Since both the bytecode offsets and line offsets are, well, offsets, having large empty spaces just means that some of the interleaved offsets are 0-length offsets. Here we have a 6-byte offset into bytecode, followed by a 255-line offset into the source code, then a 0-byte offset into the bytecode, another 255 lines of source, another 0 bytes into the bytecode, yet another 255 lines of source, one more 0-byte offset into bytecode, and a final 236 lines of offset into the source code (then the usual, expected 6 bytes of bytecode and 1 line of source code for the final print statement). Neat!

Wrapping Up

I started by apologizing for the rambling nature of this post, but I hope it's been interesting. Stay tuned for an examination of the nature of exec and its usage in Keystone in the near future.

Time to Answer @PythonQuestions

2012-01-04T00:00:00-05:00

What's your new year's resolution? One of mine is to be more active in helping and mentoring new Pythonistas, which is why I wrote the twitter bot behind @PythonQuestions. Inspired by the @MongoQuestion twitter bot, @PythonQuestions tweets new Stack Overflow questions tagged "Python".

How it Works

@PythonQuestions is actually a (relatively) generic "tweet questions from Stack Overflow" bot, composed of three components:

The Stack Overflow polling script, which is designed to be run from a cron job (or another means of periodic scheduling)
The database of questions (MongoDB of course), which for now is used only to remember which questions have already been tweeted.
The Twitter update script, also designed to be run from a cron job.
(There's also a bare-bones Flask web application which serves redirects for the pyq.io link-shortening domain. The web app currently redirects requests to http://pyq.io/ to the Stack Overflow "Questions tagged Python" page.)

Polling Stack Overflow

Stack Overflow doesn't (to my knowledge) have a Push API, so @PythonQuestions polls the /questions API periodically (right now, every 5 minutes). It queries using fromdate sorted by creation to find recent questions. For @PythonQuestions, it filters questions to only those containing the "python" tag, though this is configurable.

A few details of these questions are then upserted into MongoDB using the Stack Overflow question ID as _id, which ensures that the database only holds one copy of any given question, even if it appears several times while polling.

Tweeting About It

The twitter script consults the questions database periodically to find questions which have not yet been tweeted about, then uses the Flask-Tweepy extension (which adapts the Tweepy module for use with Flask) to update the status of the @PythonQuestions Twitter user.

Tweepy makes sending Tweets fairly painless, but Twitter itself sure doesn't. In a future blog post, I'll cover the steps needed to set up an application account, configure it properly, configure your Tweepy appropriately, and get up and running. I've done this once or twice before, so I'm reasonably familiar with the steps, but they're not well or obviously documented on Twitter's Developer site.

Putting it Together

Using a web application framework for what is essentially a series of scripts may seem an odd choice. Aside from the fact that I hope to expand the scope of the web application, using Flask and its extensions made development painless--the whole application was a matter of a few hours to write, test, and launch. I'm using Flask-PyMongo, Flask-Tweepy (as previously mentioned), and Flask-Script.

Flask-Script loads the Flask application object (and thus the configuration), and offers a few decorators to declare settings for argparse. Writing a command-line script then becomes:

from pythonquestions import app
from flask.ext.script import Manager
manager = Manager(app)

@manager.command
@manager.option('-s', '--since')
def stackoverflow(since='1d'):
    # implement the script here

Critically, this setup lets me share one set of configuration (API keys, usernames, database configuration, etc) for both the cron scripts and the web application.

Lend a Hand

In his 2011 in Review: The Python Portion, Jesse Noller called on us to be welcoming to new or inexperienced users, and to mentor them to a point where they can become contributors to, rather than passive members of, the Python community. In some small way, I hope @PythonQuestions can be a part of that.

Here's how I think that can happen:

Follow @PythonQuestions on Twitter, and start answering questions. Stack Overflow's game mechanics will give you a jolt of dopamine that's the perfect remedy for caffeine wearing off in the afternoon.
Fork the python-questions project on GitHub and help make it better. I'm especially interested in ideas about what could make the web app portion of it more useful than simply a list of questions (which already exists in better and more useful form at Stack Overflow), or in integrations with other Q&A services or sites frequented by Pythonistas.

And more broadly, echoing Jesse's thoughts:

Be active, constructive, and engaged with less-experienced Pythonistas in your community and online. The goal is to teach, not just to answer specific questions, and certainly not to ridicule or insult (even unintentionally) a novice programmer. Take the time to link to documentation, or explain detail that may not be obvious at first blush.

Hitchhiker's Guide to Python

2011-12-29T00:00:00-05:00

I first heard about The Hitchhiker's Guide to Python at PyCodeConf a few months ago. It's a fantastic idea: open source, community-driven documentation on how to do Python right: everything from how to learn Python, to how to write idiomatic code, to how to distribute your projects, to surveys of best-of-breed open source projects and libraries you can build projects and applications on top of. Many many thanks to Kenneth Reitz for creating and maintaining the project, which is hosted at GitHub.

At this time, the Hitchhiker's guide is a little rough around the edges: many sections are only outlined, and need content written; other sections may not even exist yet. We can safely consider it a first draft, or, if you prefer, an alpha.

This sort of undertaking is effectively impossible for one person to maintain--one person can't possibly know of every project, library, and idiom. Moreover, it's unfair as a user of the Guide to demand that one person must do all the work.

Therefore, a call to action: in order to make the Python community great, everyone should fork the Hitchhiker's Guide today, find (or add) a section of interest, and submit a pull request. If you're lucky, Kenneth will give you a sparkly cake.

And a pledge: time allowing (for, sadly, contributing to Python documentation is not my day job), I will make one contribution to the Hitchhiker's Guide per week until it reaches completion, or until there's nothing left to which I can conscientiously contribute (I won't attempt to document things which I know too little about).

Finally, since this is a community effort, I want to give a shout-out to all those who've contributed to the Guide so far:

Kenneth Reitz (follow on Twitter & GitHub)
Aaron Weinberger (follow on GitHub)
Kamil Kisiel (follow on Twitter & GitHub)
Seyi Ogunyemi (follow on Twitter & GitHub)
Alex Gaynor (follow on Twitter & Github)
nstielau (follow on GitHub)
Adam Brenecki (follow on Twitter & GitHub)
Nanda Kishore (follow on GitHub)
Donald Stufft (follow on Twitter and GitHub)
Brent O'Connor (follow on Twitter & GitHub)
Dalton Barreto (follow on Twitter & GitHub)
Daniel Schauenberg (follow on Twitter & GitHub)

Thanks to everyone who's helping to make Python a better place!

(To the folks above, if you'd like me to add or correct any of the contact information, please leave a comment.)

Customize Virtualenvwrapper for Fun and Profit

2011-12-22T00:00:00-05:00

Yaniv Aknin (whose blog I found through Planet Python) recently blogged about his ZSH and virtualenv setup. As I was reading it, I recognized that he's solving some of the same problems I only recently solved myself (though through a very different approach than he took), so I figured I would share it, too.

What is `virtualenv`?

virtualenv is, in my opinion, really the only way to do sane Python development. As its name implies, virtualenv creates a virtual Python environment, and in so doing solves a number of problems quite elegantly:

You are working on more than one project, each of which requires some of the same dependencies, but at different versions. You aren't using distribute's pkg_resources, so you can't know that the right version will be used by the right project (nor even which version will be used when imported).
Alternately, you are working on more than one project, each of which requires different dependencies, but your dependencies are poorly-behaved and trample one another's namespaces.
You want to be sure that your development and deployment environments are as identical as possible with regards to package makeup.
You have grown tired of typing sudo pip install ... to install packages.
You have grown tired of errors about PYTHON_EGG_CACHE when using zipped eggs (though I would be remiss if I didn't note: don't use zipped eggs, they are a stupid and awful idea, which could be the subject of a different blog post)

virtualenv solves all of these in one fell swoop by creating what is essentially a clone of your standard python installation into a new location, optionally including any system-installed site-packages, configures a few environment variables, and sets everything to be owned and usable by your unprivileged user.

After creating the virtual environment, you then activate it by sourcing the bin/activate script inside the virtual environment. Now when you type python at your command prompt, you are activating the virtualenv's Python executable rather than the system executable, and all Python scripts you run (importantly, pip) will use this executable as well. Your virtual environment has its own site-packages directory, and package installs done while the environment is activated will install into the virtual environment as well.

When you're done, simply type deactivate (or close your shell) to undo everything that the activate script did.

And what about `virtualenvwrapper`?

virtualenv isn't without its deficiencies, though. It's up to you to manage your own virtual environments and to remember where each one's activate script lies.

Many people adopt the convention that you create a directory named env or .env containing the virtual environment within the project root folder of the project for which the environment is meant. However, this clutters your checkout root, and you might mistakenly check your virtual environment in to source control.

Alternately you can have a centralized location for all your environments, but then you have to remember two paths for each project, rather than one, and you have to remember it each time you want to open a new shell or terminal tab for a given project.

virtualenvwrapper is a well-thought-out set of shell function wrappers for virtualenv which make life using it much better.

virtualenvwrapper stores all your environments in one place (by default in ~/.virtualenvs, or wherever the $WORKON_HOME environment variable points), but alleviates the pain of that approach by providing a suite of tools to manipulate and interrogate the virtual environments you have.

There is mkvirtualenv which supersedes the virtualenv command, and its new counterpart rmvirtualenv; lsvirtualenv which lists all the virtualenvwrapper-managed environments; cdvirtualenv and cdsitepackages which change directory to the virtual environment's root or site-packages directories, respectively; and my favorite, workon, which activates an environment by name.

But I want more!

virtualenvwrapper also has a full suite of hook scripts (both "global" hooks that are used regardless of which environment you are working on, and environment-specific versions) to customize aspects of its behavior. I'll mention two here which I've found incredibly useful:

First is the postmkvirtualenv hook, which runs after any invocation of mkvirtualenv. I have mine set up to create a directory named after the virtual envrionment as a subdirectory of the current working directory, if it doesn't already exist, and then cd into it:

# ~/.virtualenvs/postmkvirtualenv
parent="$PWD"
envname="$(basename $VIRTUAL_ENV)"
envdir="$parent/$envname"

if [ ! -e $envdir ]; then
    mkdir -p $envdir
fi

cd $envdir

The $VIRTUAL_ENV variable points to the directory of the newly-created virtual environment, something like ~/.virtualenvs/foobar, and is set because before the postmkvirtualenv hook is run, virtualenvwrapper has already activated the new environment for us.

Secondly, I use the postmkvirtualenv hook to write a postactivate hook into the newly created virtual environment, so that on subsequent activations with workon, I also get this "cd to the right place" behavior:

# also in ~/.virtualenvs/postmkvirtualenv
postactivate=$VIRTUAL_ENV/bin/postactivate

echo "cd $envdir"             >> $postactivate

With these two hooks configured, I can now forget about all paths, either to the virtual environment itself, or to my project's source root; all I need to remember is the name I used when I created it, and type workon that_name, and I'll be brought to the right place. And if I happen to forget the name, I've always got lsvirtualenv to help me remember.

My virtualenvwrapper hooks (as well as my ZSH setup, vim configuration, and a few other bits of random customization) are in my dotfiles repository -- feel free to fork it if you think it'll be useful for you!

Truncating HTML with Python

2011-12-02T00:00:00-05:00

On the 10gen events pages we show a table of sessions for our MongoDB Day conferences, some of which are quite crowded. In order to try to keep the table relatively sanely laid out, we truncate the session descriptions to about 150 characters, and add a "Read More" link which expands to show the full description.

Our initial approach did this truncation on the client side, using Javascript, but we found that this was difficult to get right, particularly when the description spanned several paragraphs or other block-level tags. So I wrote a quick HTMLParser sub-class to do the truncation on the server side.

import re
from HTMLParser import HTMLParser

whitespace = re.compile('(\w+)')

class HTMLAbbrev(HTMLParser):

    def __init__(self, maxlength, *args, **kwargs):
        HTMLParser.__init__(self, *args, **kwargs)
        self.stack = []
        self.maxlength = maxlength
        self.length = 0
        self.done = False
        self.out = []

    def emit(self, thing, count=False):
        if count:
            self.length += len(thing)
        if self.length < self.maxlength:
            self.out.append(thing)
        elif not self.done:
            # trim trailing whitespace
            self.out[-1] = self.out[-1].rstrip()

            # close out tags on the stack
            for tag in reversed(self.stack):
                self.out.append('</%s>' % tag)
            self.done = True

    def handle_starttag(self, tag, attrs):
        self.stack.append(tag)
        attrs = ' '.join('%s="%s"' % (k, v) for k, v in attrs)
        self.emit('<%s%s>' % (tag, (' ' + attrs).rstrip()))

    def handle_endtag(self, tag):
        if tag == self.stack[-1]:
            self.emit('</%s>' % tag)
            del self.stack[-1]
        else:
            raise Exception(
                'end tag %r does not match stack: %r' % (tag, self.stack))

    def handle_startendtag(self, tag, attrs):
        self.stack.append(tag)
        attrs = ' '.join('%s="%s"' % (k, v) for k, v in attrs)
        self.emit('<%s%s/>' % (tag, (' ' + attrs).rstrip()))

    def handle_data(self, data):
        for word in whitespace.split(data):
            self.emit(word, count=True)

    def handle_entityref(self, name):
        self.emit('&%s;' % name)

    def handle_charref(self, name):
        return self.handle_entityref('#%s' % name)

    def close(self):
        return ''.join(self.out)

(You can download the source from this gist)

HTMLAbbrev attempts to truncate to a target number of visible (i.e. in the browser) characters --- in fact, it will truncate at the first word break after the target number of characters --- and maintains a tag stack so that the emitted HTML is correctly closed, to prevent breaking layout in the browser. It doesn't count HTML tag contents towards the length, so links and other markup won't be counted towards the total.

We have a call to HTMLAbbrev wrapped into a Jinja template filter, like so:

@filter
def htmlabbrev(value, maxlen=150):
    parser = HTMLAbbrev(maxlen)
    parser.feed(value)
    return parser.close()

Which we then call from our templates:

<div class="abbrev">
  {{session.description|htmlabbrev(120)}}
  <span class="ellipsis">...</span>
  <span class="more clickable">Read More</span>
</div>
<div class="full">
  {{session.description}}
  <span class="less clickable">Hide</span>
</div>

CSS hides the <div class="full"> element by default, and javascript toggles between displaying the two.

Keystone: A Simple Python Web Framework

2011-11-27T00:00:00-05:00

After a conversation with my friend and co-worker Jesse Davis last week about Python web frameworks, I had an idea: what if there were a Python web framework that combined some of the simplicity of workflow and deployment of PHP with the readability and embodiment of best practices of Python? This should be a framework targeted towards folks who want to (or need to) learn web development, but don't have the background, interest, or time to learn one of the heavy-weight frameworks (like Pyramid or Django) or middle-weight frameworks (like Flask or CherryPy). This framework should let you get started immediately, and let you move smoothly from static sites to dynamic sites, let you learn best practices of both Python and web programming, and should not stand in your way when it comes time to go live or scale up.

Well, it turns out something almost like this already exists, Aspen. Aspen has a number of very nice features: view code and template code combined in one file, but without being fully mixed in like PHP; leveraging the filesystem for URL mapping; and a relatively simple set-it-up-and-try-it-out story. It's got a few things I really don't like, too: the use of ASCII page breaks (^L characters) to separate Python from templates; lots of configuration; and about 5,300 lines of code.

So, inspired by what's great about Aspen, and determined not to duplicate what's not, I give you Keystone!

Hang on, another Python web framework?

Yes. It's OK. There are a lot of them. Choice is good. Let's move on.

Keystone's Main Idea

Keystone's views combine Python and (Jinja) template code into the same file, which must end with extension .ks. To avoid the tag-and-code soup that most people start out with in PHP, Keystone separates the code and template with four hyphens. No Python is allowed below the line, and no literal template code above it:

import random
name = random.choice(['Pal', 'Friend', 'Buddy'])
----
<!doctype html>
<html>
  <head>
    <title>Greetings from Keystone</title>
  </head>
  <body>
    <h1>Hello, {{name}}</h1>
  </body>
</html>

This does pretty much exactly what you'd expect: on each request, the Python code is executed and randomly assigns one of "Pal", "Friend", or "Buddy" to the name variable; that variable is then injected into the template's rendering context, and the template renders the HTML to be returned to the user.

You can use all of Python in Keystone views, import standard, third party, or custom modules, etc. All in-scope variables at the end of execution are made available to the template for rendering (i.e. there's something like an implicit return locals() at the end of the view code).

Keystone also injects a number of variables into your Python code: request which contains method, cookies, headers, and other helpful attributes; (response) headers for controlling the HTTP response; set_cookie and delete_cookie; and a few others.

Mapping URLs

The URL of a Keystone view is its path, relative to the application root. In other words, GET / will execute $APP/index.ks; GET /foo will execute $APP/foo.ks; GET /foo/bar/ will execute $APP/foo/bar/index.ks; etc.

Keystone also allows for parameterized paths, by prefixing the name of any directory or .ks file within the application root with "%". So GET /account/dcrosta would execute $APP/account/%username.ks, and within that view, the variable username would have the value "dcrosta".

Running Keystone

Keystone comes with a keystone command-line script which runs a development web server, including in-browser traceback display and inspection.

When it comes time to deploy a Keystone application, the keystone script can generate boilerplate code for you automatically in several of the popular Platform-as-a-Service providers (currently Heroku, DotCloud, or ep.io, and possibly more in the future) or a stock WSGI script.

Ready to Begin?

Check out Keystone's documentation for beginners and experts alike or source repository, and dive in. I'd love to hear feedback on all of the above, so please leave a comment.

Ensuring Write-Your-Own-Reads Consistency in MongoDB

2011-11-18T00:00:00-05:00

In a question on StackOverflow a few days ago, a user was asking how to ensure that a document hasn't changed between when a client read the document and wrote to it. If user A reads the document and makes some changes (through a web form, for instance), the change should be accepted if and only if no other user B has updated the document since when user A read the document.

MongoDB doesn't support transactions, and even if it did, they wouldn't help in this case. An assumption underlying the question is that the time between any given user's reads and writes is long -- otherwise explicit/pessimistic locking would be the simplest solution -- in which case holding a transaction open on a traditional database server would be prohibitively costly (in terms of resource usage and performance).

The solution is to leverage MongoDB's atomic update semantics with an optimistic concurrency solution. This comprises four basic steps: 1. read a document; 2. modify the document (i.e. present it to the user in a web form); 3. validate that the document hasn't changed; 4. commit or abandon the user's update. For anyone who's used a source code version control system before, these steps should be familiar (i.e. pull, work locally, commit, and push for git users).

How to Version Your Documents

In order to do steps 1 and 3 efficiently, add a version field to the document. In most cases, this can be an always-increasing number, which MongoDB allows us to easily modify using $inc. When we create a document for the first time, add a field version and set its value to 0 (or 1 if you prefer):

{ _id: ObjectId(...),
  title: "Ensuring Write-Your-Own-Reads Consistency in MongoDB",
  tags: ["mongodb"],
  body: "In a question on StackOverflow...",
  version: 0
}

Next we load the document and display a form to the user, being sure to hold on to the version in a hidden field. Once the form is submitted, we calculate the changes the user made (or simply prepare to overwrite the document with the new values), but we issue our update using the previous version number:

result = db.blogposts.update(
  {'_id': ObjectId(...), 'version': 0},
  {'$set': {
    # key-value pairs from the form,
    # excluding the "version" field
    },
  '$inc': {
    'version': 1}
  },
  safe=True)

By using a safe=True write, PyMongo (and the other MongoDB drivers) will batch a call to getLastError with our update. The result of getLastError will indicate how many documents were updated -- in our case, since we are not using a multi-update, we expect that number to be 1 in the happy case (i.e. no one else updated the document in the meantime) or 0 if a conflict was detected (i.e. the version is no longer 0). The getLastError output will look like this:

{'updatedExisting': True, 'connectionId': 112, 'ok': 1.0, 'err': None, 'n': 1}

The n field in the getLastError output is the one we want to look at -- if it's 1, then the document was updated and will now have a version number of 1; if it's 0, then there was a conflicting update.

What to do in the case of conflicting updates is up to the application, MongoDB doesn't handle this for you at all (in fact, MongoDB isn't even aware of the fact that you're trying to do optimistic locking at all). A few common solutions are:

Show the user the form again, with a message indicating that another user has updated the document, and asking the user to handle the error
If possible, handle the conflict automatically in your application (this might require having tracked all the values of the previous version of the form, in order to detect which values have changed and what the delta is)
Throw up your arms, admit that programming is hard, and go shopping.

OK, maybe not the last one :)

Web A/B Testing with Dabble

2011-11-04T00:00:00-04:00

Thanks to a series of recent posts on the SvN blog, I've been thinking more about my little Python A/B testing framework, Dabble.

I built Dabble to A/B test (sometimes also called "split test") features on 10gen.com. Following the advice of a blog post I've since lost track of, Dabble configures A/B test parameters entirely in code, follows procedure for independent testing, and generally works without much of a hassle.

Dabble from 10,000 Feet

Aside from a little necessary configuration, Dabble is composed of two user-facing classes: ABTest and ABParameter. The former represents the thing you're testing -- for instance, the design of a signup page or a call-to-action form; the latter is where you actually implement the variants which make up the tested features.

Here's an example, using a class-based Python framework:

class Signup(page):
    path = '/signup'

    abest = ABTest('signup_button', ['red', 'green'], ['show', 'signup'])
    color = ABParameter('signup_button', ['#FF0000', '#00FF00'])

    def get(self):
        # do some stuff
        self.abtest.record('show')
        return render('signup.html, color=self.color)

    def post(self):
        # validate the submission
        # save the user's information
        self.abtest.record('signup')

And in your template:

<form action="/signup" method="post">
  <!-- form goes here -->
  <input type="submit" style="background-color:{{color}}"/>
</form>

That's it.

Wait, what?

Yup, that's it. No if/else, no ugly duplication of code across templates. Just one template variable.

A/B testing is really that simple. Ruby's got 7 Minute Abs, and now Python's got 30 second Dabbles... or something... I guess I'll have to work on that.

Behind the Scenes

What magic makes this possible? First, let's take a look at the parent class for ABTest and ABParameter, aptly named AB:

class AB(object):
    # these are set by the configure() function
    _id_provider = None
    _storage = None

    # track the number of alternatives for each
    # named test; helps prevent errors where some
    # parameters have more alts than others
    __n_per_test = {}

    def __init__(self, test_name, alternatives):
        if test_name not in AB.__n_per_test:
            AB.__n_per_test[test_name] = len(alternatives)
        if len(alternatives) != AB.__n_per_test[test_name]:
            raise Exception('Wrong number of alternatives')

        self.test_name = test_name
        self.alternatives = alternatives

    @property
    def identity(self):
        return hash(self._id_provider.get_identity())

    @property
    def alternative(self):
        alternative = self._storage.get_alternative(
            self.identity, self.test_name)

        if alternative is None:
            alternative = random.randrange(
                len(self.alternatives))
            self._storage.set_alternative(
                self.identity, self.test_name, alternative)

        return alternative

The alternative property here does most of the heavy lifting, it consults with the storage object (which, in 10gen's case, is backed by MongoDB, of course) to find out if the current user already has been assigned an alternative; if not, it creates and stores one for the user. The identity property consults an IdentityProvider to identify the user; this might be implemented with cookies for sites that don't have logins, or might simply be a user ID for sites that do.

Next up is ABParameter:

class ABParameter(AB):
    # a descriptor object which can be used to vary parameters
    # in a class definition according to A/B testing rules.
    #
    # each viewer who views the given class will always be
    # consistently shown the Nth choice from among the
    # alternatives, even between different attributes in the
    # class, so long as the name is the same between them

    def __get__(self, instance, owner):
        return self.alternatives[self.alternative]

Python will call __get__ with a reference to the instance ("self") and the owning class (in our example, the Signup class), and return the value returned by this function when the attribute to which this descriptor is bound is accessed.

The storage backends (currently MongoDB and filesystem are supported) are responsible for storing results from the tests, test configurations, and user-to-alternative assignments. The code there is not particularly exciting, so I won't show it here.

Collecting Results

Dabble's storage object has a report method which summarizes the recorded results in a dictionary like:

{
    'test_name': 'signup_button',
    'results': [
        {
            'alternative': 'red',
            'funnel': [{
                'stage': ('show', 'signup'),
                'attempted': 1813,
                'converted': 18
            }],
        },
        {
            'alternative': 'green',
            'funnel': [{
                'stage': ('show', 'signup'),
                'attempted': 1838,
                'converted': 32
            }]
        }
    ]
}

Each alternative's results has a funnel element, which shows the number of users who attempted (i.e. the first step was recorded) and converted (i.e. the second step was recorded) for each successive pair of steps as defined in the ABTest. Here we can see that about the same number of people attempted the show-signup stage for each alternative, but nearly twice as many people signed up with the green button.

What's Next?

Here are a few areas where I'd like to improve Dabble:

Support non class-based views web frameworks (Django, Flask, etc)
Build IdentityProvider support for common web frameworks
DRY reporting (currently the two backends duplicate much of the same functionality in the report method)
Support other back-ends for storing results

You can help! fork the repository on GitHub, use it in your application, report or fix bugs, contribute improvements, etc.

On Job "Requirements"

2011-10-19T00:00:00-04:00

If you look at pretty much any job posting, you'll see some variation on this (in this case, for a hypothetical web programming position):

Requirements:

BSc in Computer Science or equivalent

2-5 years web development experience

Expert in Java, SQL, HTML, CSS, Javascript

Strong communicator

Able to multi-task

The numbers, degrees, and technologies will vary from one job to another, but the basic form is almost always there.

But what's the point of this section? I've never gotten a job for which I met all the "requirements," and I've never cared if the candidate I'm hiring doesn't meet them either. Is it just out of habit?

As an occasional job seeker, I've learned to ignore this, like web ads. It's filler with no relevance to my job search. I care about what a company can offer me: is the work exciting? Does the product have social value? Will I get paid well? Work with exceptional people who are excited about what they do? And do those people have lives outside of work, and interesting stories and experiences to share?

If you're looking for a job and reading this, I recommend you do the same. If a job looks interesting and you think you deserve a company's consideration, don't be scared off by a list of "requirements." If you're a good fit, you'll get an interview and fair treatment. To turn away because of a bullet point or two is a shame and a waste.

On the flip side, when screening and interviewing, I don't want to miss a strong candidate because they can't check a box in the HR application. Even if I did care about them, these "requirements" are weak signals, at best. I'm much more impressed by your history and your work: do you spend time contributing to open source projects? Have you done anything impressive at previous jobs, or on school projects? Does your resume, cover letter, and online presence show potential for an upwards trajectory in your career? Ultimately, do I think you can make a positive impact on the product, culture, and success of the company?

A corollary, to all those writing job requisitions: you have only a few hundred words to make an impression on candidates. If you want the best, most excited, most motivated candidates, you'll have to work hard to impress them. Tell them why they should care about your company, and what you do. Tell them what will make them excited to get up and go to work every morning. Tell them, honestly, what you expect of them -- not a list of credentials, but a sampling of what they'll do and how you expect them to do it.

With this advice in mind, I present a revised "requirements" section:

Why we'll love you:

You live and breathe web development. You've hit all the bumps in the road, and learned from each encounter what to do and what not to do. You've used enough frameworks, libraries, and platforms to have opinions, and you'll stand up for them.

You scoff at static sites, and haven't built one in years. You know that content belongs in a database. Bonus points if you've used MongoDB.

You read blogs, watch videos, attend meetups, and go to conferences, because you're a geek who never stops learning and improving your skillset.

Better, no?

Never Use Source Control GUIs!

2011-09-27T00:00:00-04:00

Just received this IM from my friend, Nick:

11:48:58 AM Nick: kjsahfdlkjashdflkajshdflkjh
11:49:02 AM Nick: it was the git gui itself
11:49:09 AM Nick: it periodically refreshes
11:49:14 AM Nick: with a forked git process

Version control GUIs are uniformly evil. They entice you with a multitude of colors, and 1-pixel lines; their siren song is one-click commits and graphical 3-way merges.

But, like Odysseus, if we have any hope of prevailing, we must lash ourselves to the mast and steer past this temptation.

The real issue, of course, isn't the GUI per se, but the fact that a developer has virtually no idea what commands the GUI is issuing. I claim that any GUI which sufficiently exposes the underlying tool's features to know with confidence what clicking any button will do fails at being any more intuitive than the command line tool itself; and any tool which does not must by necessity contain hidden traps that an unwary developer will inevitably hit. Do not use GUIs for source control.

In 5 years, something like 90% of the source control mishaps I've seen can be attributed to GUIs hiding complexity of what they're doing. Git can be a beast to learn, for sure; but at least once you develop an intuition, you can be reasonably safe in performing even advanced operations at the command line tool, in a way I've never felt (or felt from co-workers) about GUI tools.

There is one place in which I think a GUI far outstrips the command line: read-only history and patch/diff viewing. I use GitX for this, though there may be others which are as good or better. (And, true to form, I always launch it from the command line with gitx.)

Professor, a MongoDB Profile Viewer

2011-09-22T00:00:00-04:00

I presented Professor, my MongoDB profile viewer and analyzer, this past Tuesday at the New York MongoDB User Group. Professor aims to use the new features in the MongoDB 2.0 profiler to make profile information intelligible and actionable. It's open-source (BSD-licensed) and written in Python and Flask.

10,000 Feet

The primary design goal of Professor is to give you at-a-glance information about the performance of your queries (and, ultimately, updates and commands, though these are not yet supported). It presents a dashboard-style interface showing the worst-performing queries first, with aggregate timing information and a mini-histogram:

Here you can see that there are 3 collections in this example database, and one query that we've seen for each. The "skeleton" of the query is essentially the query without any values. Skeletons preserve the structure of the query and allow Professor to group same-structured queries together, to present their information in aggregate, even if the queries used different values.

Immediately to the right of the skeleton is a log-scale histogram of the query times: the leftmost bucket represents queries that ran in less than 1ms, the next queries that ran less than 2ms, then 4ms, 8ms, etc. When using professor, you want to see histograms like the one for "indexdocs," which shows that all or nearly all queries executed in less than a millisecond.

Zooming In

You can click on the collection name to see only queries for that collection (along with some collection-specific information: number of documents, average size, total storage size, and the indexes that exist on the collection), or you can click on the query skeleton to see all queries collected which match that skeleton, and the time each one took to execute:

(Here you can see more clearly how the actual queries translate to the skeleton above.)

So why are these queries taking so long? (Yes, even 3 milliseconds counts as "long"!) This page gives you all the information you need. (Hint: take a look at the number of documents and at the indexes).

Under the Hood

Professor attempts to limit its impact on your running database -- after all, one of the situations where you might need Professor's help is when your database is performing badly (although you should have investigated your slow queries far sooner!). Rather than repeatedly querying the system.profile collection, Professor makes a local copy (preferably into a separate mongod instance running on another server), and uses that database to serve all the web pages.

Every time an update command is issued (either by clicking the "update now" link on the dashboard, or by the profess command line script), Professor connects to the target database, queries only for system.profile documents newer than the last time it looked, processes them (this is when Professor generates the skeleton) and inserts them into its local store.

Professor's Future

Professor is a work in progress. Here is a laundry list of features I hope to add, in no particular order:

Support for other operation types (as I've already mentioned)
Use update modifiers ($inc and friends) to generate aggregate statistics during profile intake, rather than on page generation
Use AJAX or simple page reloads to turn the static web view into a periodically-updating live view (like the UNIX top command)
Show other fields on the query list (nscanned, ntoreturn, nreturned, scanAndOrder, etc)
Ability to replay a query with explain() to get more detailed information, such as what index, if any, was used to service a query

Or, nudge nudge, you could fork Professor on GitHub and help me work on some of these yourself!

The new Profiler in MongoDB 2.0

2011-09-20T00:00:00-04:00

One of my favorite features in MongoDB 2.0 is the finer-grained output of the database profiler. In earlier versions, the bulk of the profile information was contained within a (structured, parseable) string, but as of 2.0 the fields have been broken out and made queryable.

Back in time

In production versions of MongoDB up to and including 1.8, database profiler output looked like this:

{"ts" : "Thu Jan 29 2009 15:19:45 GMT-0500 (EST)" ,
 "info" : "query test.foo ntoreturn:0 reslen:102 nscanned:2 query: {} nreturned:2 bytes:86" ,
 "millis" : 0}

That info string has a lot of useful information in it, but how can I get at it? It's log parsing all over again. The profiler documentation even suggests querying this collection with un-anchored regular expressions and server-side javascript, which can be painfully slow if you want to do anything automated with the profile information. Want to filter by date? Yeah, good luck.

Today

In 2.0, the profiler output looks like this (for the same fictional query as above):

{"ts": ISODate("2009-01-29T15:19:45Z"),
 "op": "query",
 "ns": "test.foo",
 "query": {
     "$query": {}
 },
 "responseLength": 102,
 "nscanned": 2,
 "nreturned": 2,
 "millis": 0
}

Much nicer. Now we have lots of data to get our hands on; we can filter by ns to find only queries against certain collections, sort by ts which is a proper, orderable date object; we can consider only queries slower or faster than a certain time, and we can even reach into the query itself and only look at operations touching a certain field, or containing a certain target value. Cool!

The profiler also contains entries on updates, inserts, removes, commands (like Map-Reduce, group, count, etc).

Get Started with the Profiler

If you haven't yet, read the documentation on the profiler. Even though it still references the old formats, it has a lot of useful information and warnings about using the profiler (hint: it has small, but non-zero, overhead on performance -- use with caution!).

Next fire up the profiler on your database:

test> db.runCommand({profile: 1, slowms: 5})

This tells MongoDB to log all operations that took 5 or more milliseconds to the system.profile collection. It's a capped collection which is initially set to 1MB. This is enough to capture around 4,000 small operations (for some reasonable value of "small").

To make the profile collection larger, just drop it and explicitly create it:

test> db.runCommand({profile: 0})
test> db.system.profile.drop()
true
test> db.createCollection("system.profile", {capped: true, size: 100 * 1024 * 1024})

{ "ok" : 1 } test> db.runCommand({profile: 1, slowms: 5})

And you're back up with a 100MB profile (which holds a lot of operations).

Wishlist

The profiler in MongoDB 2.0 has come a long way, but I'd like to see a few features added:

logging of nScannedObjects, as in the explain() output, which will help diagnose queries that are hitting the collection too hard
logging of the explain plan, or at least index, used to serve a query
filtering options for entries going into the profile, like a whitelist/blacklist approach by collection name, op type, etc.

Coming Soon

I've written a web-based profile analyzer, named "Professor", that I'm presenting at tonight's MongoDB User Group. A post about it should be up in the next few days.

Hello, Plog

2011-09-16T00:00:00-04:00

I've wanted to start writing a blog for the past few months. By which I mean, writing blog posts. But why focus on writing great content when you can write great software instead?

Allow me to introduce Plog, the software running late.am. Read on to see how Flask, Mongoengine, and MongoDB can work together to create something beautiful.

Data Model

The data model for Plog is about as simple as you'd expect:

class Post(db.Document):
    pubdate = db.DateTimeField(required=True)
    updated = db.DateTimeField()
    published = db.BooleanField(default=True)

    title = db.StringField(required=True)
    slug = db.StringField(required=True, unique=True)

    blurb = db.StringField(required=True)
    body = db.StringField(required=False)

    tags = db.ListField(db.StringField())

    views = db.IntField()

    _words = db.ListField(db.StringField())

    meta = {
        'allow_inheritance': False,
        'indexes': [
            {'fields': ['published', 'slug', 'pubdate']},
            {'fields': ['published', '_words', 'pubdate']},
        ],

    }

Setting allow_inheritance to False instructs Mongoengine not to add two bookkeeping fields _cls and _types to the documents. In cases where many types of documents share a collection, Mongoengine uses these fields to filter results only to the type (or subtypes) of document corresponding to the Python class you are querying for.

The first index will be used on virtually every page in the site. Individual post pages will always be queried using published and slug, and sorted by pubdate; the homepage and archive pages don't query by slug, but do sort by (and, in the case of archive pages, filter by) pubdate.

Search

The second index includes the _words field, which is automatically generated by Plog when a post is saved. Indexes on arrays in MongoDB are called "multi-key" indexes, since each element of the array is given an entry in the index. This allows efficient queries into array fields, looking for one or more of the values.

Plog's search is implemented using the $all operator against _words, which returns documents whose _words array contains each of the search terms:

posts = Post.objects(published=True, _words__all=query).order_by('-pubdate')

Plog doesn't support complex search use cases like stemming (although that would be easy to add using nltk or a similar tool), boolean queries, phrase searches, etc.

Atom Feed

Flask has built-in support for generating Atom XML feeds from (or, to be more accurate, Werkzeug, the WSGI library beneath Flask, does)

feed = AtomFeed(
    title='late.am',
    feed_url=url_for('feed', _external=True),
    author={'name': 'Dan Crosta', 'email': 'dcrosta@late.am'},
    icon=url_for('static', filename='mug.png', _external=True),
    generator=('plog', 'https://github.com/dcrosta/plog', '0.1'),
)

posts = Post.objects(published=True).order_by('-pubdate')
for post in posts[:20]:
    feed.add(
        title=post.title,
        content=markdown(post.blurb + '\n' + post.body),
        content_type='html',
        author={'name': 'Dan Crosta', 'email': 'dcrosta@late.am'},
        url=url_for('post', slug=post.slug, _external=True),
        id=url_for('permalink', post_id=post.pk, _external=True),
        published=post.pubdate,
        updated=post.updated)

Other Goodies

As a web developer, obviously, I dislike writing HTML. That's why Plog uses Markdown powered by the markdown2 module. markdown2 supports code syntax highlighting using pygments, which was an added bonus.