dmalcolm

Autodetecting reference-counting bugs: the video!

2012-03-23T22:39:37Z

I gave a talk at PyCon 2012 about my Python plugin for GCC - how this lowers the barrier for entry to potential GCC hackers, and how I've been using this to find reference-counting errors in Python C extension modules.

A video of the talk can be seen here:
http://pyvideo.org/video/648/static-analysis-of-python-extension-modules-using
or on YouTube here.

The slides are here.

We had a mini-sprint after the talk, and another later on in the main sprints, covering these topics:

getting the plugin to build on OS X (using MacPorts' build of gcc-4.6.1): this works, but needed some compat patching around some of the differences between glibc and OS X's libc (also case-sensitivity of filenames).
improving the static analysis engine: currently it takes the simplistic approach of trying to generate all traces of execution through the function in a big tree, which suffers from exponentially explosions. I'm hoping the insides can be reworked to implement an iterative solver that can handle loops more robustly (data flow equations)
improving the HTML error reports that the analyzer generates. More on this to follow!

I've also finally succumbed to the inevitable and joined github: you can fork the plugin here (this is just a clone I keep synchronized with the Fedora Hosted repository)

Adding a spellchecking pass to GCC, using Python

2011-09-30T18:16:10Z

LWN recently posted an interesting article on writing GCC plugins, showing how to add a spellchecking pass to the compiler.

I thought this was a neat example, so I had a go at porting it from C to use my Python plugin for GCC

The result is about 35 lines of Python code (including comments), which you can see here

I hope the code is easy to read. One other thing this demonstrates is how by extending the compiler in Python you have access to the whole of the Python packaging ecosystem - I was able to use the "enchant" spellchecking library (originally from the AbiWord project) with about 4 lines of code.

Automatically detecting reference-count bugs in Python extension modules

2011-07-15T21:32:30Z

[ For the tl;dr version, scroll down to see the pretty screenshots :) ]

I've been working on a static analysis tool to automatically detect reference-count bugs in C Python extension code.

(see my earlier posts on verifying calls to the PyArg_ParseTuple API here and here)

Mismanaging reference counts can lead to the python process leaking memory (and other resources), for when an object becomes immortal, or segfaulting, when an object is cleaned up when things still refer to it.

My "cpychecker" code is still an early prototype (don't expect to use it on arbitrary C code yet), but here's an example of some of the things it's already capable of:

Can you see the reference-counting error in this (contrived) code fragment?

    22	PyObject *
    23	refcount_demo(PyObject *self, PyObject *args)
    24	{
    25	    PyObject *list;
    26	    PyObject *item;
    27	    list = PyList_New(1);
    28	    if (!list)
    29	        return NULL;
    30	
    31	    item = PyLong_FromLong(42);
    32	    if (!item)
    33	        return NULL;
    34	
    35	    PyList_SetItem(list, 0, item);
    36	    return list;
    37	}
    38	
    39	static PyMethodDef test_methods[] = {
    40	    {"refcount_demo",  refcount_demo, METH_VARARGS, NULL},
    41	    {NULL, NULL, 0, NULL} /* Sentinel */
    42	};

Compiling like this:

  [david@fedora-15 gcc-plugin]$ ./gcc-with-python cpychecker.py -I/usr/include/python2.7 refcount-demo.c

the checker adds this output to gcc's:

refcount-demo.c: In function ‘refcount_demo’:
refcount-demo.c:37:1: error: ob_refcnt of PyListObject is 1 too high
refcount-demo.c:27:10: note: PyListObject allocated at:     list = PyList_New(1);
refcount-demo.c:27:10: note: when PyList_New() succeeds at:     list = PyList_New(1);
refcount-demo.c:28:8: note: when taking False path at:     if (!list)
refcount-demo.c:31:10: note: reaching:     item = PyLong_FromLong(42);
refcount-demo.c:31:10: note: when PyLong_FromLong() fails at:     item = PyLong_FromLong(42);
refcount-demo.c:32:8: note: when taking True path at:     if (!item)
refcount-demo.c:33:9: note: reaching:         return NULL;
refcount-demo.c:37:1: note: when returning

which can be navigated in any IDE that can parse GCC's output messages (works for me in emacs).

This demonstrates a particular path of execution that has a bug.

I found the textual output a bit heavy on the eye, so I've hacked up the plugin script so it can render graphical HTML visualizations of the errors that it finds.

Here's that same report, in HTML form:

The report shows the control flow through the function: lines that get executed are written in bold and outlined in blue, with arrows connecting them, and additional annotations in italics. (I'm not so good at HTML/CSS, so help here would be most welcome!).

I used the jsplumb JavaScript library to add lines to the HTML to link together elements. This uses the newish <canvas> element, so the control-flow lines may only appear in recent browsers. It works for me in Chromium 12 and Firefox 4. You can see the HTML report itself here:
http://fedorapeople.org/~dmalcolm/blog/2011-07-15/refcount_demo-refcount-errors.html

(Currently it's hardcoded to generate the reports, but I'll probably add something like a -fdump-html command-line option to the gcc-with-cpychecker harness).

Here are some more examples:

Detecting the all-too-common: "return Py_None;" bug:

As HTML: http://fedorapeople.org/~dmalcolm/blog/2011-07-15/losing_refcnt_of_none-refcount-errors.html

Another (very contrived) reference leak:

As HTML: http://fedorapeople.org/~dmalcolm/blog/2011-07-15/object_leak-refcount-errors.html

Detecting a stray Py_INCREF that makes the reference count too high, or segfaults python, depending on what happened earlier:

As HTML: http://fedorapeople.org/~dmalcolm/blog/2011-07-15/too_many_increfs-refcount-errors.html

This is still an experimental prototype, so it's not yet ready for general purpose use, but I'm frantically working on it, and I hope it will be ready in time for inclusion in Fedora 16.

The checker is Free Software (licensed under GPLv3 or later), and if you want to get involved, go to https://fedorahosted.org/pipermail/gcc-python-plugin/ (as I said above, I could really use some help with HTML and CSS! The checker is written in Python itself, if you're interested in hacking on the code).

(Thanks to Red Hat for allowing me to spend a substantial proportion of my $DAYJOB on this)

Verifying the more awkward parts of the CPython API

2011-06-25T00:45:13Z

I've been running my Python extension module static analyser over CPython itself (the latest in the 2.7 hg branch, specifically).

I'm pleased to say that the project's mailing list received the first patch to the checker from someone other than me (Thanks Tom!) - He's been running it over the sources of gdb (which embeds python).

These make for good torture tests for the analyser, and I'm pleased with how far it survived. Some bugs do remain - in the checker, that is.

There are quite a few places where CPython calls PyArg_ with a code expecting a "const char*", but receives a "char*'. I think the ones in CPython are all false positives, so I think we're going to need to make that configurable.

Perhaps the most fiddly part of the checking this API is the "O&" conversion code - I wasn't able to handle this in my previous Coccinelle-based approach to this problem.

Here's an example:

    68	
    69	extern int convert_to_ssize(PyObject *, Py_ssize_t *);
    70	
    71	PyObject *
    72	buggy_converter(PyObject *self, PyObject *args)
    73	{
    74	    int i;
    75	
    76	    if (!PyArg_ParseTuple(args, "O&", convert_to_ssize, &i)) {
    77	        return NULL;
    78	    }
    79	
    80	    Py_RETURN_NONE;
    81	}
    82

The idea is that you're meant to supply a conversion callback, which can extract a value back to the next argument.

The above example has a bug (can you see it?)

After a fair amount of coding today, the checker is now able to detect it:

[david@fedora-15 gcc-python]$ ./gcc-with-python cpychecker.py $(python-config --cflags) demo.c

demo.c: In function ‘buggy_converter’:
demo.c:76:26: error: Mismatching type in call to PyArg_ParseTuple with format code "O&" [-fpermissive]
  argument 4 ("&i") had type
    "int *" (pointing to 32 bits)
  but was expecting
    "Py_ssize_t *" (pointing to 64 bits) (from second argument of "int (*fn) (struct PyObject *, Py_ssize_t *)")
  for format code "O&"

Notice how it used the type of the callback to figure out what the type of the next argument must be.

I've also reformatted the error messages slightly, adding newlines and indentation to try to make them easier to grok.

Hopefully we'll shake out the rest of the bugs soon, and then on to reference-count checking...

If's still rather rough around the edges, but if you want to try running it on your extension module, then come and join us.

Static analysis of CPython extensions, using a new GCC plugin

2011-06-23T21:50:55Z

I've been looking at ways to improve the quality of Python extensions written in C.

CPython provides a great C API that makes it easy to relatively easy to integrate C and C++ libraries with Python code. We use it extensively within Fedora - for example, Fedora's installation program is written in Python.

But you do have to be write such code carefully:

you have to correctly keep track of reference counts in your objects. If you get this wrong, you can segfault the interpreter, or introduce a memory leak.

some APIs use a format string, with C variable-length arguments (see e.g PyArg_ParseTuple and its variants). If the C compiler doesn't know the rules, it can't enforce type-safety. This can lead to people accidentally writing architecture-specific code (more on this below)

like any API, function calls can fail. This seems to be a universal rule of computer programming: it's tricky to correctly handle all the errors that can occur - bugs tend to lurk in the error-handling cases

I want to make it easier for people to write correct Python extension code, so I've been looking at static analysis.

None of the existing tools seemed to do exactly what I wanted, and given that all of my work is done with GCC, I wanted a solution that was well integrated with GCC. I also wanted to be able to use Python itself to work on the tool. (I attempted some of this a while back with Coccinelle, but I use GCC, so I wanted to embed the checking directly into GCC).

So I've written a GCC plugin that embeds Python within that compiler. This means that it's now possible to write new C and C++ compilation passes in Python, and use Python packages for things like syntax-highlighting, visualization, and so on.

That's the theory, anyway. The code is still fairly new, so I've only wrapped a small subset of GCC's types and APIs.

I've started using this to write a static analyser for CPython extension code.

Here's an example of what it can do so far...

Given this fragment of C code:

    24	
    25	PyObject *
    26	socket_htons(PyObject *self, PyObject *args)
    27	{
    28	    unsigned long x1, x2;
    29	
    30	    if (!PyArg_ParseTuple(args, "i:htons", &x1)) {
    31	        return NULL;
    32	    }
    33	    x2 = (int)htons((short)x1);
    34	    return PyInt_FromLong(x2);
    35	}
    36

there's a bug: at line 30, the "i" code to PyArg_ParseTuple signifies an "int", but it's being passed an "unsigned long" from line 28 (via a pointer) to write back its result to. This will break badly on a big-endian 64-bit CPU.

First of all, we can use the Python support in the compiler to visualize the code:

[david@fedora-15 gcc-python]$ ./gcc-with-python show-ssa.py -I/usr/include/python2.7 demo.c

Here's the output. This visualization shows the basic blocks of code, with source code on the left, interleaved with GCC's internal representation on the right:

(If you're wondering what the "PHI<>" functions mean in the above, this is actually showing the SSA representation after some of GCC's analysis and optimizations passes have already happened).

Given that this is Python, it's really easy to write new visualizations.

I've also written the first new compiler warnings using the Python plugin.

Here's the output from compiling that C code using my "cpychecker.py" script to add new warnings:

[david@fedora-15 gcc-python]$ ./gcc-with-python cpychecker.py $(python-config --cflags) demo.c 
demo.c: In function ‘socket_htons’:
demo.c:30:26: error: Mismatching type in call to PyArg_ParseTuple with format code "i:htons" [-fpermissive]
  argument 3 ("&x1") had type "long unsigned int *" (pointing to 64 bits)
  but was expecting "int *" (pointing to 32 bits) for format code "i"

I've tried to make the new error message readable, containing as much information as possible.

Any ideas on how to improve this?

I'm now working frantically on implementing reference-count checking :)

I hope that I'll be able to get this into a working state in time for Fedora 16: I'd like to run all of the C Python extension code in the Fedora distribution through a checker, but I need to do a lot of polishing before it's ready!

The code is free software (GPLv3 or later), and you can grab it from this git repository:

http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=summary

I'm using this Trac instance for bug tracking:

https://fedorahosted.org/gcc-python-plugin/

Anyone got ideas for other uses for this? Visualizations of code? New compiler warnings? Remember, this thing's built on top of GCC, so (in theory) it can handle anything that GCC can handle e.g. C++ templates, Java, Fortran, and so on.

If you want to get involved, or want more information, there's a mailing list here:

https://fedorahosted.org/mailman/listinfo/gcc-python-plugin

Thanks to Red Hat for supporting the development of this software! (and for general awesomeness); thanks also to Read the Docs for providing a nifty hosting service for free software API documentation.

PyCon US talks on memory usage and gdb

2011-03-13T06:20:48Z

Various people have been asking for the slides to my PyCon US talks, so here goes.

"Dude, Where's My RAM?" - A deep dive into how Python uses memory

Slides can be found here in ODP format (2.7M) and here as a PDF (2.5M)

To answer some of the questions asked about the gdb-heap tool:

I've used it with both Python 2.6 and Python 2.7 as the Python running inside gdb
I've used to analyze the memory usage of Python 2.4, 2.6, 2.7, 3.1 and 3.2
If anyone wants to work on extending this (e.g. adding PyPy support), I'm attending the sprints until Thursday

Using Python to debug C and C++ code (using gdb)
The slides can be seen here

(BTW, these were built using slippy, which I like a lot. I first tried using rst and rst2s5 but ran into lots of aspect ratio issues when trying it out on a projector, so I fixed up the s5 html into slippy's html. Thanks to Ned Batchelder for showing me slippy)

FUDcon talk: Different Species of Python

2011-01-30T01:46:50Z

I'm at FUDCon in Tempe, Arizona. Today I gave a talk entitled "Different Species of Python", comparing the various implementations of the Python language, and how we might better support them within Fedora.

I hope this will be of interest to both the folks on the Planet Fedora and the Planet Python aggregators.

I ran overtime a bit: lots of interesting questions and discussion on the internals of the two CPython implementations, so I didn't get to cover Jython, IronPython and PyPy as much as I might have done, or the packaging issues, but hopefully a useful time was had by all.

I've uploaded a copy of my slides to my Fedora People page, as PDF here and as ODF here - enjoy!

UPDATE:
Tim Flink heroically transcribed the talk and questions for those on IRC. A transcript can be found here

Python 3 support for GTK via PyGI: proof-of-concept

2010-04-20T23:47:10Z

I spent much of last week taking part in a GTK/Python hackfest.

My particular interest is in Python 3 support, so I spent some time helping John Ehresman clean up pygobject, and some time with John Palmieri on PyGI

I'm somewhat new to gobject-introspection, so I created an ASCII-art diagram to try to help figure out how it works:
http://live.gnome.org/GObjectIntrospection/Architecture

In the old GTK approach to binding native libraries for use by other language runtimes (such as Python's), a .defs file provided metadata on the API, which had to be kept in-sync with the code. An example can be seen here. For Python, this was used to generate .c code which could then be compiled into an extension module. A problem with this approach is that you typically need to write numerous .overrides files to handle the awkward cases, and these are specific to Python.

This means that for every N libraries (gtk, atk, gstreamer, dbus, telepathy, etc) and M runtimes (python 2, python 3, javascript, java, etc) you potentially need to handle N*M bindings, and each one could involve specialcasing.

In the new approach, "gobject-introspection" defines a simple textual format for source-code comments, containing similar information to a .defs file, but (I hope) rich enough to handle more of the special cases. This is scraped from the source into an XML file (e.g. Foo.gir), then compiled into an efficient binary format (e.g. Foo.typelib) which can be mapped into memory at runtime using a library (libgirepository.so).

The idea is that each language runtime can define a bridge, which calls into libgirepositrory, mapping between the language and the various libraries, using libffi, dynamically lazy-creating the glue code.

My hope is that we now need to handle only N + M cases, and so hopefully we have less of a combinatorial explosion, and potentially, faster start-up times and less RAM usage.

I've now got a py3k branch of PyGI in a Fedora git repository here.

(the tracking bug is https://bugzilla.gnome.org/show_bug.cgi?id=615872)

This code is able to be compiled against both Python 2.6 and Python 3.1 (in each case using the py3k branch of pygobject), and all tests pass with both versions of Python.

I can now write something like this:

from gi.repository import Gtk
w = Gtk.Window()
w.set_title('\u6587\u5b57\u5316\u3051') # 3 Kanji and a hiragana: "Mojibake"
b = Gtk.Button()
b.set_label('I am a GtkButton created by Python 3 via PyGI')
w.add(b)
w.show_all()
Gtk.main()

and have Python 3 dynamically load the Gtk typelib, dynamically load the underlying C libraries, and generate the machine code glue to call into the GTK library and create a Gtk Window. In this case a unicode string is correctly marshalled from Python 3, converted into UTF 8 internally within PyGI, and used to invoke gtk_window_set_title dynamically.

Early days yet (still seeing far too many errors when spelunking around inside the API), but at least it works well enough for a screenshot!

Thanks to the other hackfest attendees for answering my many dumb questions about PyGI, to Red Hat for paying me to work on Python 3 porting, and for sponsoring our food at the event, to One Laptop per Child for letting us use a room in their office as hacking space, and to Canonical for buying us coffee.

What variability exists within proposed updates to the Fedora package collection?

2010-03-09T18:53:43Z

There's been a lot of discussion, and alas, some bad feeling, I think, about trying to balance updates versus testing in Fedora.

I believe there are many areas where we can mitigate risk for the users of Fedora without imposing extra work on package maintainers.

I don't think "one size fits all" - I believe that one of the problems we face is that no package or update is alike, and that discussion tends to lump things together without recognizing those differences.

In the hope that it's helpful, I've tried to gather some of the variables that I think are meaningful in the context of "how likely is it that a proposed update might break something?" (and there's some of my opinion in here too)

Built-in test suite
There's great variability here between different src rpms:

Does the upstream code have a test suite?
Do we run it during %check ?
If something started failing, do we actually know about it? (e.g. does an error here kill the build? is there a list of known good/known bad tests?)

I think that a package that's passed hundreds of selftests during the build and not failed any is doing better than one that has no built-in test suite, and should be in some way privileged in our update system. (It's possible to auto-detect the presence of a %check section as well)

External test suite
How much testing does the package get via autoqa? How much testing has this proposed update had via an automated system?

Manual testing
Yes, having a human actually try the software is often good, and finds different types of problem to those that can be found via automated testing. Having said that, I feel we have far too little automated testing in Fedora today, and that the current way we do manual testing has flaws: people test for the bugs they wanted to see fixed, and report whether they are fixed or not. From a formal coverage perspective, we've no idea if they're hitting the important use-cases for other users of the package. But presumably we do hit a lot of coverage with the current approach.

Can multiple versions be installed?
The kernel gets to install multiple copies of itself, and this is the ultimate escape hatch for when a kernel update hoses your system: you still have a known-good version installed (I hope) and get to reboot with that.

To what extent are other packages allowed to do that? (or would the maintainers want to?) Would extending it be a way to mitigate risk on those packages that want to rebase more often?

Meaningful test coverage
With my former professional QA hat on I think the ideal for software testing is:

The "functional" side: to have a set of personas describing who will be using the software, what they will be using it to do, and to use that to come up with a set of test cases that cover that functionality
The "non-functional" side: to know about the types of flaw expected based on the technology (e.g. I expect buffer overruns in C), and to use this to come up with appropriate test cases to prevent these

This should give an idea of what test coverage is appropriate, and you can _then_ think about automating them.

So I think that a package that has some test cases on the wiki is "in better shape" for doing updates than one that doesn't, and I hope that's a lightweight way of guiding testing. I hope there's a way of streamlining this within our processes so that we do smarter testing without needing extra work for package maintainers. (I don't expect anyone wants to adopt IEEE 829 in Fedora QA; see p133-136 of "Lessons Learned in Software Testing", Kaner et al (2002) for excellent arguments for not using it; a great book, BTW).

Lines of code overall
Some packages are small, some are huge. I did some stats on this for RHEL when I worked on RHEL QA, using "sloccount". I believe the largest by SLOC was openoffice, closely followed by the kernel (in the millions of SLOC), then a big dropoff to the 100k SLOC packages, then a long tail.

Amount of code touched
What is the build-time difference between old and new versions of the src.rpm? This isn't the whole story (a one-line bug can still kill you), but it's part of the story. A rebase might contain a fix for bugs you care about, but might also touch 50 other subsystems.

Amount of testing elsewhere
One advantage of a rebase is that you are sharing source code changes with other people, and so if there is a problem, someone else might have already run into it. This isn't a panacea: yes, there are plenty of ways in which we can have Fedora-specific bugs, but it is one difference between a tarball rebase versus cherry-picking patches.

(random thought: could Bodhi have integration with other distributions update systems and warn us about analogous updates that are breaking other people? or is Fedora always the first into the minefield, finding the bugs for other distributions?)

Noarch vs architecture independent
The former are typically much simpler than the latter. The latter has specific risks (e.g. word-size assumptions). To what extent can we mitigate these risks with automated testing?

Programming Language
Each programming language exhibits its own sets of emergent behavior. For example (and this is grossly oversimplifying):

C code tends to exhibit buffer-overflow bugs, poor unit testing, poor error-handling
C++ code can be more prone to compiler/static linker/dynamic linkage bugs than C code

etc. I don't want to populate this list too much as this kind of thing is prone to unhelpful programming language flamewars.

Problems inherent to packaging
Each software delivery system exhibits its own set of flaws, and our RPM/yum is no exception. To what extent does, say, rpmlint cover the types of thing that go wrong, and to what extent can we extend it to help us?

Build system
Although the Fedora packaging guidelines manage to impose some sanity on this, there are many ways in which packages get configured and built.

Some examples:

the GNU autotools: configure.in, Makefile.am, leading to a "configure" used during the build to generate a Makefile. This can be prone to "silently" dropping functionality when the buildroot changes. It's sometimes possible to detect such breakage by looking at the "Requires" metadata of the built packages (can we automate this in Bodhi?)
hand-written one-of-a-kind Makefile written by upstream. Given that each is unique, each will have unique problems
python setup.py, using distutils/setuptools.
cmake

etc

Security fixes
Security fixes probably should be treated differently from non-security fixes: many people expect that the former should happen as a matter or course, that if someone has distributed software, they should also promptly distribute security fixes. This seems to be regarded as some kind of natural entitlement within software in a way that other kinds of update aren't, and so our update process probably should reflect this special quality ascribed to security flaws (I suspect I'm getting grumpy and middle-aged in my attitudes here, sorry)

Critical versus Speciality packages
Is this a package that needs to work for an essential Fedora functionality to work, or is it more of a "leaf" within the dependency graph. For example, if this package breaks, could it prevent a user from running yum, or running a graphical browser to search for information on the bug?

I like our "critical path" approach: some packages are definitely more critical than others. The exact details might need tuning, of course.

Paid versus Volunteer
I'm in the very fortunate position that I'm being paid to work on Fedora, and thus I'm professionally responsible for doing some tasks that aren't fun. Others volunteer their time and effort on Fedora, and I think it's important that their time should be fun, or, at least satisfying for some of the higher levels of Maslow's hierarchy of needs. (I happen to enjoy most of what I do in Fedora, and I do spend evenings and weekends on it too).

I hope this is a constructive addition to the debate. What other variability is meaningful in the context of "candidate updates"? I probably missed some.

Update on 2to3c

2010-02-21T17:35:23Z

My 2to3c tool now has a website: https://fedorahosted.org/2to3c/

I haven't yet sanely packaged it (I'm working on that) but I have done some work since my last blog post:
- It's somewhat more robust at handling errors from spatch
- Beginnings of a selftest suite (ultimately I want to be able to take C code, run the tool, compile it against multiple runtimes, then execute it in each one, verifying that the module still works)
- Some (possibly crazy) hacks to try to better handling preprocessor macros when reworking module initialization.

John Palmieri has used the tool to help with porting the DBus python bindings to Python 3. He's had to do a lot of manual work, but apparently 2to3c did save him some time

This is more of a shovel than a silver bullet, perhaps.

Help with this would be most welcome. I'm at PyCon, BTW, and will be around for a few of the sprint days, if anyone's interested in working on this.

Debugging the innards of Python... with Python

2010-02-13T06:39:22Z

I've been trying a new approach for debugging the internals of Python.

One of the strengths of the C implementation of Python ("CPython") is how easy it is to wrap libraries written in C/C++ so that they're callable from Python code.

The drawback of this is that you're running "native" machine code, and that code can crash the python process, or worse, corrupt the internals of the python process so that the crash seems to be coming from somewhere else.

Time to break out the debugger...

Unfortunately, practically everything inside CPython is a pointer to a PyObject structure, with some additional type-specific data lurking after it in memory. A typical debugger by default just shows you the addresses of these structures, or shows you the two values they contain ("ob_refcnt" and "ob_type", which aren't necessarily enlightening).

I typically use the "gdb" debugger. Python has long had a file named "gdbinit" which contains various hooks to make it easier to debug CPython within gdb (In Fedora and RHEL that file can be found in our "python-devel" packages).

If you copy this file to ~/.gdbinit and restart gdb you can then use "pyo" to print a more human-readable representation of a PyObject*, and use "pyframe" and other commands to figure out where the currently selected thread of the process is in Python code.

This approach has some drawbacks:

it isn't very robust - it effectively injects calls to the repr() function into the process being debugged (the so-called "inferior process" in debugger parlance). If the data in the inferior process is corrupt, attempting to print it can lead to another segmentation fault ("SIGSEGV") within that process inside the implementation of "repr". For example, if the allocator of the inferior process has become corrupt, then Python won't be able to create the string representation of the data without a crash (be it in Python's memory arena code or the underlying heap), and you've got another SIGSEGV. If one of the objects inside a list has been splatted with 0xDEADBEEF garbage, attempting to print the repr() object will crash, and you won't even know that you had a list.
you have to go into gdb manually and run these commands by hand. Automated crash reports can't get at the "real" data, most significantly: what lines of Python code were being executed when this crash happened?
as well as being a manual task, it's hard to do this correctly; any mistakes when doing this will typically cause a SIGSEGV in the inferior process
the script is written in gdb's own language and is thus hard to work with and extend
complicated intermeshing of C and Python (e.g. invocations of callbacks that then trigger other callbacks each time crossing a language boundary) can be hard to figure out

Is there a better way?

(...drum roll...)

As of gdb 7.0, it's now possible to write custom "pretty-printers" for data in gdb, using Python as the extension language. One of the reasons for this was to improve visualization of C++ data in the debugger, so that if you try to print a C++ std::string, rather than get this craziness:
(gdb) print str $1 = {static npos = 4294967295, _M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x804a014 "hello world"}}

you now get:
(gdb) print str $1 = hello world

The above is thanks to python hooks running inside the gdb process, using a "gdb" module to walk the graph of gdb.Value and gdb.Type objects representing the state of the process being debugged.

What this is building up to is that I've written a similar set of Python hooks for CPython itself.

The code can be seen in this git repository. I've been going through the various types within CPython's internals, writing Python code to run inside the debugger. For each type I've written a Python class that runs inside the debugger process, and knows how to convert an object in the process being debugged into an object within the debugger, and can then print it.

To demonstrate, let's try breaking Python:

The real use case for this for me is when a library decides to do error handling using "assert", and takes down the whole process, or has a crasher bug. Rather than pick on a 3rd-party library, let's inject our own segfault. We can do this by abusing the "ctypes" module, by picking a random location in the process' address space and trying to read it as a string. We can do this as a one-liner, but let's set up some data so we can see what it looks like in the crash report:

>>> class Foo:
...     def bar(self):
...         from ctypes import string_at
...         string_at(0xDEADBEEF) # this code will (probably) cause Python to segfault
... 
>>> f = Foo()
>>>
>>> # Let's assign some data of various kinds to the instance:
>>> f.someattr = 42
>>> f.someotherattr = {'one':1, 'two':2L, 'three':[(), (None,), (None, None)]}
>>>
>>> # Now let's trigger the segfault
>>> f.bar()

At this point we've generated a segmentation fault inside Python.

Let's see the old behavior of a backtrace, using the "bt" command:
Program received signal SIGSEGV, Segmentation fault. __strlen_sse2 () at ../sysdeps/i386/i686/multiarch/strlen.S:87 87 pcmpeqb (%esi), %xmm0 Current language: auto The current source language is "auto; currently asm". (gdb) bt #0 __strlen_sse2 () at ../sysdeps/i386/i686/multiarch/strlen.S:87 #1 0x07113d30 in PyString_FromString (str=0xdeadbeef <Address 0xdeadbeef out of bounds>) at Objects/stringobject.c:116 #2 0x00167e18 in string_at (ptr=0xdeadbeef <Address 0xdeadbeef out of bounds>, size=-1) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/_ctypes.c:5348 #3 0x0018247f in ffi_call_SYSV () at src/x86/sysv.S:61 #4 0x001822b0 in ffi_call (cif=<value optimized out>, fn=<value optimized out>, rvalue=<value optimized out>, avalue=<value optimized out>) at src/x86/ffi.c:213 #5 0x00171315 in _call_function_pointer (pProc=0x167de0 <string_at>, argtuple=0xb7f3d02c, flags=4357, argtypes=0xb7f45a4c, restype=0x80f3dc4, checker=0x0) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/callproc.c:815 #6 _CallProc (pProc=0x167de0 <string_at>, argtuple=0xb7f3d02c, flags=4357, argtypes=0xb7f45a4c, restype=0x80f3dc4, checker=0x0) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/callproc.c:1162 #7 0x0016a6f2 in CFuncPtr_call (self=0xb7f9d5dc, inargs=0xb7f3d02c, kwds=0x0) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/_ctypes.c:3857 #8 0x070c478c in PyObject_Call (func=0xb7f9d5dc, arg=0xb7f3d02c, kw=0x0) at Objects/abstract.c:2492 #9 0x0716069c in do_call (f=0x80f37bc, throwflag=0) at Python/ceval.c:3917 #10 call_function (f=0x80f37bc, throwflag=0) at Python/ceval.c:3729 #11 PyEval_EvalFrameEx (f=0x80f37bc, throwflag=0) at Python/ceval.c:2389 #12 0x07162642 in PyEval_EvalCodeEx (co=0xb7f3bda0, globals=0xb7f3768c, locals=0x0, args=0x80ec788, argcount=1, kws=0x80ec78c, kwcount=0, defs=0xb7f45d78, defcount=1, closure=0x0) at Python/ceval.c:2968 #13 0x07160983 in fast_function (f=0x80ec644, throwflag=0) at Python/ceval.c:3802 #14 call_function (f=0x80ec644, throwflag=0) at Python/ceval.c:3727 #15 PyEval_EvalFrameEx (f=0x80ec644, throwflag=0) at Python/ceval.c:2389 #16 0x07161b79 in fast_function (f=0x80eb1cc, throwflag=0) at Python/ceval.c:3792 #17 call_function (f=0x80eb1cc, throwflag=0) at Python/ceval.c:3727 #18 PyEval_EvalFrameEx (f=0x80eb1cc, throwflag=0) at Python/ceval.c:2389 #19 0x07162642 in PyEval_EvalCodeEx (co=0xb7f2e578, globals=0xb7fc70b4, locals=0xb7fc70b4, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2968 #20 0x071627a3 in PyEval_EvalCode (co=0xb7f2e578, globals=0xb7fc70b4, locals=0xb7fc70b4) at Python/ceval.c:522 #21 0x0717d94b in run_mod (mod=<value optimized out>, filename=<value optimized out>, globals=0xb7fc70b4, locals=0xb7fc70b4, flags=0xbffff2fc, arena=0x80e8628) at Python/pythonrun.c:1335 #22 0x0717f4a6 in PyRun_InteractiveOneFlags (fp=0x5b5420, filename=0x71c3e7d "<stdin>", flags=0xbffff2fc) at Python/pythonrun.c:840 #23 0x0717f6ab in PyRun_InteractiveLoopFlags (fp=0x5b5420, filename=0x71c3e7d "<stdin>", flags=<value optimized out>) at Python/pythonrun.c:760 #24 0x0717f7eb in PyRun_AnyFileExFlags (fp=0x5b5420, filename=<value optimized out>, closeit=0, flags=0xbffff2fc) at Python/pythonrun.c:729 #25 0x0718c212 in Py_Main (argc=1, argv=0xbffff3f4) at Modules/main.c:599

Looking closely at the above, we can see that there's a problem inside ctypes, but it's hard to see what triggered the problem within the Python code.

Below is a screendump of what it looks like with my new debugging hooks (warning: some very long lines; hopefully this won't get too badly mangled)

I've highlighted some of it: we can see

the location we're at (file/line/function) within the Python code at each point in the call stack
the dictionary of globals
the arguments passed to functions

and so on - there's almost too much information. This should come in handy for nasty issues when the stack involves C code calling into Python and back again (e.g. when callbacks get invoked); should also be useful for debugging all threads at once in a multithreaded app.

There are probably many things that could be improved about this code, but I'm already finding it very useful, and I hope other people will too.

I've integrated this code into Fedora's Python 2 and Python 3 builds for the upcoming Fedora 13 release so that the hooks should get automatically loaded by the debugger. It should all Just Work - the crash-handling tool will fetch the debugging hooks and the bug report that's generated should contain all of this rich information on what's going on inside python - without the user needing to be a debugging guru.

(gdb) bt
#0  __strlen_sse2 () at ../sysdeps/i386/i686/multiarch/strlen.S:87#1  0x07113d30 in PyString_FromString (str=0xdeadbeef <Address 0xdeadbeef out of bounds>) at Objects/stringobject.c:116
#2  0x00167e18 in string_at (ptr=0xdeadbeef <Address 0xdeadbeef out of bounds>, size=-1) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/_ctypes.c:5348
#3  0x0018247f in ffi_call_SYSV () at src/x86/sysv.S:61
#4  0x001822b0 in ffi_call (cif=<value optimized out>, fn=<value optimized out>, rvalue=<value optimized out>, avalue=<value optimized out>) at src/x86/ffi.c:213
#5  0x00171315 in _call_function_pointer (pProc=0x167de0 <string_at>, argtuple=(3735928559L, -1), flags=4357, argtypes=(<builtin_function_or_method at remote 0xb7f45d2c>, <builtin_function_or_method at remote 0xb7f45d4c>), restype=
    <_ctypes.SimpleType at remote 0x80f3dc4>, checker=<unknown at remote 0x0>) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/callproc.c:815
#6  _CallProc (pProc=0x167de0 <string_at>, argtuple=(3735928559L, -1), flags=4357, argtypes=(<builtin_function_or_method at remote 0xb7f45d2c>, <builtin_function_or_method at remote 0xb7f45d4c>), restype=
    <_ctypes.SimpleType at remote 0x80f3dc4>, checker=<unknown at remote 0x0>) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/callproc.c:1162
#7  0x0016a6f2 in CFuncPtr_call (self=0xb7f9d5dc, inargs=(3735928559L, -1), kwds=<unknown at remote 0x0>) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/_ctypes.c:3857
#8  0x070c478c in PyObject_Call (func=<CFunctionType at remote 0xb7f9d5dc>, arg=(3735928559L, -1), kw=<unknown at remote 0x0>) at Objects/abstract.c:2492
#9  0x0716069c in do_call (f=File /usr/lib/python2.6/ctypes/__init__.py, line 492, in string_at (ptr=3735928559L, size=-1), throwflag=0) at Python/ceval.c:3917
#10 call_function (f=File /usr/lib/python2.6/ctypes/__init__.py, line 492, in string_at (ptr=3735928559L, size=-1), throwflag=0) at Python/ceval.c:3729
#11 PyEval_EvalFrameEx (f=File /usr/lib/python2.6/ctypes/__init__.py, line 492, in string_at (ptr=3735928559L, size=-1), throwflag=0) at Python/ceval.c:2389
#12 0x07162642 in PyEval_EvalCodeEx (co=0xb7f3bda0, globals=
    {'Union': <_ctypes.UnionType at remote 0x17c120>, 'c_wchar': <_ctypes.SimpleType at remote 0x80fb32c>, 'c_bool': <_ctypes.SimpleType at remote 0x80fab54>, 'c_double': <_ctypes.SimpleType at remote 0x80f7a0c>, 'CFUNCTYPE': <function at remote 0xb7f3264c>, '__path__': ['/usr/lib/python2.6/ctypes'], 'byref': <builtin_function_or_method at remote 0xb7f40eec>, 'pointer': <builtin_function_or_method at remote 0xb7f40d6c>, 'alignment': <builtin_function_or_method at remote 0xb7f40eac>, '_memmove_addr': 4962832, 'c_longlong': <_ctypes.SimpleType at remote 0x80f8544>, 'c_short': <_ctypes.SimpleType at remote 0x80f407c>, 'get_errno': <builtin_function_or_method at remote 0xb7f39f4c>, '__file__': '/usr/lib/python2.6/ctypes/__init__.pyc', '_calcsize': <builtin_function_or_method at remote 0xb7f96bec>, 'c_ulong': <_ctypes.SimpleType at remote 0x80f5974>, 'c_int': <_ctypes.SimpleType at remote 0x80f5124>, 'c_int32': <_ctypes.SimpleType at remote 0x80f5124>, 'memmove': <CFunctionType at remote 0xb7f9d4a4>, '_sys': <module at remote 0xb7fa308c>, '_cast': <CFunctionType at remote 0xb7f9d574>, 'addressof': <builtin_function_or_method at remote 0xb7f40f0c>, 'ArgumentError': <type at remote 0x80f2fdc>, 'c_buffer': <function at remote 0xb7f32614>, 'c_longdouble': <_ctypes.SimpleType at remote 0x80f821c>, 'cdll': <LibraryLoader at remote 0xb7f459ac>, 'memset': <CFunctionType at remote 0xb7f9d50c>, 'string_at': <function at remote 0xb7f32a04>, 'sizeof': <builtin_function_or_method at remote 0xb7f40ecc>, '_FUNCFLAG_PYTHONAPI': 4, 'create_string_buffer': <function at remote 0xb7f325dc>, 'set_errno': <builtin_function_or_method at remote 0xb7f40d2c>, '_pointer_type_cache': {<_ctypes.SimpleType at remote 0x80f9ec4>: <_ctypes.PointerType at remote 0x80fb864>, <_ctypes.SimpleType at remote 0x80fb32c>: <_ctypes.PointerType at remote 0x80fb504>, <NoneType at remote 0x72061e0>: <_ctypes.SimpleType at remote 0x80fa6a4>}, '_Pointer': <_ctypes.PointerType at remote 0x17bd00>, 'create_unicode_buffer': <function at remote 0xb7f326bc>, 'c_long': <_ctypes.SimpleType at remote 0x80f5124>, 'c_char_p': <_ctypes.SimpleType at remote 0x80fa37c>, '__builtins__': {'bytearray': <type at remote 0x71fb540>, 'IndexError': <type at remote 0x71ff0e0>, 'all': <builtin_function_or_method at remote 0xb7fafccc>, 'help': <_Helper at remote 0xb7f814ec>, 'vars': <builtin_function_or_method at remote 0xb7fb280c>, 'SyntaxError': <type at remote 0x71fed60>, 'unicode': <type at remote 0x720e2c0>, 'sorted': <builtin_function_or_method at remote 0xb7fb274c>, 'isinstance': <builtin_function_or_method at remote 0xb7fb22cc>, 'copyright': <_Printer at remote 0xb7f81d6c>, 'NameError': <type at remote 0x71feac0>, 'BytesWarning': <type at remote 0x72006c0>, 'dict': <type at remote 0x7205960>, 'input': <builtin_function_or_method at remote 0xb7fb224c>, 'oct': <builtin_function_or_method at remote 0xb7fb246c>, 'bin': <builtin_function_or_method at remote 0xb7fafd8c>, 'SystemExit': <type at remote 0x71fe2e0>, 'StandardError': <type at remote 0x71fdf60>, 'format': <builtin_function_or_method at remote 0xb7fb20ac>, 'repr': <builtin_function_or_method at remote 0xb7fb268c>, 'UnicodeDecodeError': <type at remote 0x71ff540>, 'False': <bool at remote 0x71f9624>, 'RuntimeWarning': <type at remote 0x7200340>, 'bytes': <type at remote 0x7209c80>, 'iter': <builtin_function_or_method at remote 0xb7fb230c>, 'reload': <builtin_function_or_method at remote 0xb7fb264c>, 'Warning': <type at remote 0x71ffee0>, 'round': <builtin_function_or_method at remote 0xb7fb26cc>, 'dir': <builtin_function_or_method at remote 0xb7faff4c>, 'cmp': <builtin_function_or_method at remote 0xb7fafe4c>, 'set': <type at remote 0x7207000>, 'list': <type at remote 0x7204420>, 'reduce': <builtin_function_or_method at remote 0xb7fb260c>, 'intern': <builtin_function_or_method at remote 0xb7fb228c>, 'issubclass': <builtin_function_or_method at remote 0xb7fb22ec>, 'apply': <builtin_function_or_method at remote 0xb7fafd4c>, 'EOFError': <type at remote 0x71fe820>, 'locals': <builtin_function_or_method at remote 0xb7fb238c>, 'BufferError': <type at remote 0x71ffe00>, 'slice': <type at remote 0x7207900>, 'FloatingPointError': <type at remote 0x71ff8c0>, 'sum': <builtin_function_or_method at remote 0xb7fb278c>, 'buffer': <type at remote 0x71f9840>, 'getattr': <builtin_function_or_method at remote 0xb7fb20cc>, 'abs': <builtin_function_or_method at remote 0xb7fafc8c>, 'exit': <Quitter at remote 0xb7fd1d2c>, 'print': <builtin_function_or_method at remote 0xb7fb256c>, 'IndentationError': <type at remote 0x71fee40>, 'True': <bool at remote 0x71f9630>, 'FutureWarning': <type at remote 0x7200420>, 'ImportWarning': <type at remote 0x7200500>, 'None': <NoneType at remote 0x72061e0>, 'hash': <builtin_function_or_method at remote 0xb7fb218c>, 'len': <builtin_function_or_method at remote 0xb7fb234c>, 'credits': <_Printer at remote 0xb7f8156c>, 'frozenset': <type at remote 0x72070e0>, '__name__': '__builtin__', 'ord': <builtin_function_or_method at remote 0xb7fb24ec>, 'super': <type at remote 0x720ade0>, 'TypeError': <type at remote 0x71fe040>, 'license': <_Printer at remote 0xb7f8170c>, 'KeyboardInterrupt': <type at remote 0x71fe3c0>, 'UserWarning': <type at remote 0x71fffc0>, 'filter': <builtin_function_or_method at remote 0xb7fb206c>, 'range': <builtin_function_or_method at remote 0xb7fb25ac>, 'staticmethod': <type at remote 0x72035e0>, 'SystemError': <type at remote 0x71ffb60>, 'BaseException': <type at remote 0x7200a60>, 'pow': <builtin_function_or_method at remote 0xb7fb252c>, 'RuntimeError': <type at remote 0x71fe900>, 'float': <type at remote 0x72028a0>, 'GeneratorExit': <type at remote 0x71fe200>, 'StopIteration': <type at remote 0x71fe120>, 'globals': <builtin_function_or_method at remote 0xb7fb210c>, 'divmod': <builtin_function_or_method at remote 0xb7faff8c>, 'enumerate': <type at remote 0x71fdaa0>, 'Ellipsis': <ellipsis at remote 0x7207840>, 'LookupError': <type at remote 0x71ff000>, 'open': <builtin_function_or_method at remote 0xb7fb24ac>, 'quit': <Quitter at remote 0xb7fd120c>, 'basestring': <type at remote 0x7209ba0>, 'UnicodeError': <type at remote 0x71ff380>, 'zip': <builtin_function_or_method at remote 0xb7fb284c>, 'hex': <builtin_function_or_method at remote 0xb7fb21cc>, 'long': <type at remote 0x7204f60>, 'next': <builtin_function_or_method at remote 0xb7fb244c>, 'int': <type at remote 0x7203a40>, 'chr': <builtin_function_or_method at remote 0xb7fafe0c>, '__import__': <builtin_function_or_method at remote 0xb7fafc6c>, 'type': <type at remote 0x720ac20>, '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", 'Exception': <type at remote 0x71fde80>, 'tuple': <type at remote 0x720a6a0>, 'UnicodeTranslateError': <type at remote 0x71ff620>, 'reversed': <type at remote 0x71fdb80>, 'UnicodeEncodeError': <type at remote 0x71ff460>, 'IOError': <type at remote 0x71fe660>, 'hasattr': <builtin_function_or_method at remote 0xb7fb214c>, 'delattr': <builtin_function_or_method at remote 0xb7faff0c>, 'setattr': <builtin_function_or_method at remote 0xb7fb270c>, 'raw_input': <builtin_function_or_method at remote 0xb7fb25ec>, 'PendingDeprecationWarning': <type at remote 0x7200180>, 'compile': <builtin_function_or_method at remote 0xb7fafecc>, 'ArithmeticError': <type at remote 0x71ff7e0>, 'str': <type at remote 0x7209c80>, 'property': <type at remote 0x71fcec0>, 'MemoryError': <type at remote 0x71ffd20>, 'ImportError': <type at remote 0x71fe4a0>, 'xrange': <type at remote 0x7206640>, 'KeyError': <type at remote 0x71ff1c0>, 'coerce': <builtin_function_or_method at remote 0xb7fafe8c>, 'SyntaxWarning': <type at remote 0x7200260>, 'file': <type at remote 0x7201d80>, 'EnvironmentError': <type at remote 0x71fe580>, 'unichr': <builtin_function_or_method at remote 0xb7fb27cc>, 'id': <builtin_function_or_method at remote 0xb7fb220c>, 'OSError': <type at remote 0x71fe740>, 'DeprecationWarning': <type at remote 0x72000a0>, 'min': <builtin_function_or_method at remote 0xb7fb242c>, 'UnicodeWarning': <type at remote 0x72005e0>, 'execfile': <builtin_function_or_method at remote 0xb7fb202c>, '__package__': <NoneType at remote 0x72061e0>, 'complex': <type at remote 0x71fc840>, 'bool': <type at remote 0x71f9560>, 'ValueError': <type at remote 0x71ff2a0>, 'NotImplemented': <NotImplementedType at remote 0x72061e8>, 'map': <builtin_function_or_method at remote 0xb7fb23cc>, 'any': <builtin_function_or_method at remote 0xb7fafd0c>, 'max': <builtin_function_or_method at remote 0xb7fb240c>, 'object': <type at remote 0x720ad00>, 'TabError': <type at remote 0x71fef20>, 'callable': <builtin_function_or_method at remote 0xb7fafdcc>, 'ZeroDivisionError': <type at remote 0x71ffa80>, 'eval': <builtin_function_or_method at remote 0xb7faffcc>, '__debug__': <bool at remote 0x71f9630>, 'ReferenceError': <type at remote 0x71ffc40>, 'AssertionError': <type at remote 0x71ff700>, 'classmethod': <type at remote 0x7203500>, 'UnboundLocalError': <type at remote 0x71feba0>, 'NotImplementedError': <type at remote 0x71fe9e0>, 'AttributeError': <type at remote 0x71fec80>, 'OverflowError': <type at remote 0x71ff9a0>}, '_FUNCFLAG_USE_ERRNO': 8, '_memset_addr': 4962944, '_dlopen': <builtin_function_or_method at remote 0xb7f40e0c>, '__name__': 'ctypes', 'RTLD_LOCAL': 0, 'c_int16': <_ctypes.SimpleType at remote 0x80f407c>, '_SimpleCData': <_ctypes.SimpleType at remote 0x17bde0>, 'wstring_at': <function at remote 0xb7f32a3c>, 'c_void_p': <_ctypes.SimpleType at remote 0x80fa6a4>, 'set_conversion_mode': <builtin_function_or_method at remote 0xb7f40dec>, 'PyDLL': <type at remote 0x80fc3dc>, 'DEFAULT_MODE': 0, 'LittleEndianStructure': <_ctypes.StructType at remote 0x17c040>, 'c_uint64': <_ctypes.SimpleType at remote 0x80f8d54>, 'c_ulonglong': <_ctypes.SimpleType at remote 0x80f8d54>, '_FUNCFLAG_USE_LASTERROR': 16, '_cast_addr': 1490912, 'ARRAY': <function at remote 0xb7f3279c>, 'c_ushort': <_ctypes.SimpleType at remote 0x80f48ac>, '__doc__': 'create and manipulate C data types in Python', '_check_size': <function at remote 0xb7f32684>, 'CDLL': <type at remote 0x80fbeac>, '_wstring_at': <CFunctionType at remote 0xb7f9d644>, 'c_ubyte': <_ctypes.SimpleType at remote 0x80f9564>, 'RTLD_GLOBAL': 256, 'c_char': <_ctypes.SimpleType at remote 0x80f9ec4>, 'c_uint32': <_ctypes.SimpleType at remote 0x80f5974>, 'c_float': <_ctypes.SimpleType at remote 0x80f71fc>, 'SetPointerType': <function at remote 0xb7f32764>, 'resize': <builtin_function_or_method at remote 0xb7f40dcc>, '_c_functype_cache': {(<_ctypes.SimpleType at remote 0x80f5124>, (), 1): <_ctypes.CFuncPtrType at remote 0x8100b14>, (<_ctypes.SimpleType at remote 0x80fa6a4>, (<_ctypes.SimpleType at remote 0x80fa6a4>, <_ctypes.SimpleType at remote 0x80f5124>, <_ctypes.SimpleType at remote 0x80f5974>), 1): <_ctypes.CFuncPtrType at remote 0x80fd84c>}, '---Type <return> to continue, or q <return> to quit---
_os': <module at remote 0xb7fa314c>, '_wstring_at_addr': 1494192, 'cast': <function at remote 0xb7f329cc>, 'c_int8': <_ctypes.SimpleType at remote 0x80f9a14>, 'c_byte': <_ctypes.SimpleType at remote 0x80f9a14>, 'c_int64': <_ctypes.SimpleType at remote 0x80f8544>, 'c_voidp': <_ctypes.SimpleType at remote 0x80fa6a4>, '_string_at_addr': 1474016, '_FUNCFLAG_CDECL': 1, 'pythonapi': <PyDLL at remote 0xb7f45a0c>, 'PYFUNCTYPE': <function at remote 0xb7f327d4>, '_CFuncPtr': <_ctypes.CFuncPtrType at remote 0x17bb40>, '_endian': <module at remote 0xb7fa3944>, '__package__': 'ctypes', 'c_uint16': <_ctypes.SimpleType at remote 0x80f48ac>, 'BigEndianStructure': <_swapped_meta at remote 0x81006ec>, 'pydll': <LibraryLoader at remote 0xb7f459ec>, '__version__': '1.1.0', 'Structure': <_ctypes.StructType at remote 0x17c040>, 'c_uint': <_ctypes.SimpleType at remote 0x80f5974>, 'py_object': <_ctypes.SimpleType at remote 0x80f3dc4>, 'c_wchar_p': <_ctypes.SimpleType at remote 0x80fae7c>, '_string_at': <CFunctionType at remote 0xb7f9d5dc>, 'c_size_t': <_ctypes.SimpleType at remote 0x80f5974>, 'c_uint8': <_ctypes.SimpleType at remote 0x80f9564>, 'LibraryLoader': <type at remote 0x80fc704>, 'Array': <_ctypes.ArrayType at remote 0x17bc20>, 'POINTER': <builtin_function_or_method at remote 0xb7f40d4c>}, locals=<unknown at remote 0x0>, args=0x80ec788, argcount=1, kws=0x80ec78c, kwcount=0, defs=0xb7f45d78, defcount=1, closure=
    <unknown at remote 0x0>) at Python/ceval.c:2968
#13 0x07160983 in fast_function (f=
    File <stdin>, line 2, in bar (self=<Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, string_at=<function at remote 0xb7f32a04>), throwflag=0) at Python/ceval.c:3802
#14 call_function (f=
    File <stdin>, line 2, in bar (self=<Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, string_at=<function at remote 0xb7f32a04>), throwflag=0) at Python/ceval.c:3727
#15 PyEval_EvalFrameEx (f=
    File <stdin>, line 2, in bar (self=<Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, string_at=<function at remote 0xb7f32a04>), throwflag=0) at Python/ceval.c:2389
#16 0x07161b79 in fast_function (f=File <stdin>, line 1, in <module> (), throwflag=0) at Python/ceval.c:3792
#17 call_function (f=File <stdin>, line 1, in <module> (), throwflag=0) at Python/ceval.c:3727
#18 PyEval_EvalFrameEx (f=File <stdin>, line 1, in <module> (), throwflag=0) at Python/ceval.c:2389
#19 0x07162642 in PyEval_EvalCodeEx (co=0xb7f2e578, globals=
    {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}, locals=
    {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, 
    closure=<unknown at remote 0x0>) at Python/ceval.c:2968
#20 0x071627a3 in PyEval_EvalCode (co=0xb7f2e578, globals=
    {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}, locals=
    {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}) at Python/ceval.c:522
#21 0x0717d94b in run_mod (mod=<value optimized out>, filename=<value optimized out>, globals=
    {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}, locals=    {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}, flags=0xbffff2fc, arena=0x80e8628) at Python/pythonrun.c:1335
#22 0x0717f4a6 in PyRun_InteractiveOneFlags (fp=0x5b5420, filename=0x71c3e7d "<stdin>", flags=0xbffff2fc) at Python/pythonrun.c:840
#23 0x0717f6ab in PyRun_InteractiveLoopFlags (fp=0x5b5420, filename=0x71c3e7d "<stdin>", flags=<value optimized out>) at Python/pythonrun.c:760
#24 0x0717f7eb in PyRun_AnyFileExFlags (fp=0x5b5420, filename=<value optimized out>, closeit=0, flags=0xbffff2fc) at Python/pythonrun.c:729
#25 0x0718c212 in Py_Main (argc=1, argv=0xbffff3f4) at Modules/main.c:599
#26 0x080485c7 in main (argc=1, argv=0xbffff3f4) at Modules/python.c:23

Python memory usage: is it worth sharing constant data?

2009-12-19T01:19:52Z

I had some interesting conversations at FUDcon Toronto [1] about python's memory footprint, and about ways of reducing the whole-system memory pressure of a collection of Python processes, by sharing more pages of memory between them.

We discussed sharing memory between python processes.

One approach to doing this would be to write some kind of support for starting python as a "zygote process", where we keep a started-up python process around, forking from it, so that the freshly forked process (in theory) uses only memory for the pages that become different (copy-on-write). As I understand it, this is how Chromium implements its tabs.

Bernie Innocenti apparently tried this for OLPC's activities (each a PyGTK process) but only got 10-20% sharing. The problem is that every python object embeds a reference count, and constantly increments/decrements those ob_refcnt fields, leading to unsharing of the pages.

One way of improving this might be to invent a magic refcount value (e.g. 0) that means "don't refcount this", so that pages can be shared (stuff that's been created at vm startup will probably be long-lived). (do we need the ability to "seal" an object, to optimize for the common case where nothing's been monkeypatched?)

(Or we could completely scrap refcounting and go to a full gc model, but I'm looking for easy wins here)

A similar approach that doesn't require us to have a zygote process is to use KSM (Kernel SamePage Merging). This is a feature added in recent versions of the Linux kernel, where a program can mark regions of its address space. The kernel can try to hash these pages, and pages that are bit-for-bit identical will be shared in RAM across the various processes.

KSM was developed by Qumranet hackers (now at Red Hat) for use in KVM. Perhaps Python could use it?

Unfortunately I don't think this approach will work either: all of the various PyObject* pointers cross-referencing the data in memory will be non-equal, and that will kill the sharing; the pages are unlikely to be bit-for-bit identical.

One idea for achieving equality of pages was to mimic how we build emacs: as I understand it, we build emacs as a process, then take a core-dump of it. On starting up emacs we bring the coredump back to life. (I may be horribly mischaracterizing this - it seems very fragile to me).

An approach that sounded promising was to try to consolidate the representation of the immutable blobs of data in the loaded python modules: the docstrings, the constant strings and unicode representations; the bytecode blobs (actually these are PyStringObjects).

The idea was a new variant of .pyc/.pyo. A .pyc file is a hierarchical representation of a parsed python module, containing everything needed at runtime (e.g. optimized bytecodes blobs rather than source strings, with links to the constants needed in the code) serialized to disk using the "marshal" module. It's analogous to pickle, except that the marshal format only caters to strict hierarchies of objects, whereas pickle supports cross-references, and this the marshal code can be simpler (and hopefully more efficient).

So in our proposed variant of .pyc, we would split the data into two streams:
- control data for building the hierarchy of objects, to be thrown away after the module is loaded
- "large" data to be mmap-ed, to persist in the process' address space after the module is loaded, with the kernel sharing all instances of this data in RAM between all python processes.

This would require hooks in PyStringObject (need PyBytesObject to do it for py3k) e.g. a new ob_sstate: SSTATE_INTERNED_MMAPPED, which places the bytes in a pointer elsewhere in the address space.

Some approaches to doing this:
- use the .pyc format as is. Unfortunately I don't think this works: currently they're written to disk as (size, bytes) without a nul terminator, whereas PyStringObject assumes that ob_sval is nul-terminated.
- come up with a simple variant of .pyc that splits the marshalled data into two streams (as above), storing offsets into the second stream within the first whenever writing out e.g. a PyStringObject
- use the ELF format directly: ELF is a binary format supporting multiple chunks/streams of data, with room for expansion, and a library and command-line tools for manipulating them. We could invent some new types of section. However I suspect that tools for dealing with ELF files are only readily-available on Linux (it happens to be the native format for executables and shared libraries) (we came up with the name ".pye" to refer to these ELF-based bytecode files)

Another idea was linkage: to try to link together all of the .pyc files per RPM into one big file, linking together the large sections as far as possible, or perhaps a big sqlite db mapping dotted path to .pyc files for standard modules. The idea here was to try to reduce the number of syscalls needed to locate the .py files.

As it turned out, this seems to have been a classic case of optimizing without measuring.

Looking at the "stat" case, starting up the python interpreter 100 times:

[david@brick ~]$ time (for i in $(seq 1 100) ; do python -c "pass" ; done)

real	0m3.129s
user	0m2.328s
sys	0m0.652s

...so the bulk of the time taken is in user-space, rather than waiting on the kernel. (We tried this on the OLPC "boot animation" workload, and I believe the real slowdown is an accidental syncronous call that should have been asyncronous, that's stalling on waiting for a socket to close).

On my return from the conference I spent some time trying to capture real measurements to justify a possible pyc rewrite.

To look at the memory usage of all of those shared docstrings, I wrote some systemtap scripts.

You can see the latest version here: http://dmalcolm.fedorapeople.org/systemtap/marshal.stp

I trieds various approaches to instrumentation.

The one I've settled on is to instrument returns from the r_object() call in Python/marshal.c, and to record PyObject* instances returned from that function that are of ob_type "str" (i.e. are PyStringObject instances) and have ob_refcnt == 1 (i.e. they are shared with anything, and haven't been interned).

Assuming I've instrumented things correctly, a simple startup of the interpreter under systemtap:
$ stap -v marshal.stp -c"python -c'pass'"
has this usage of unique strings (the "value" is the length of the string); note that this includes docstrings, constant strings, and bytecode blobs:
(snipped, full output here: http://dmalcolm.fedorapeople.org/systemtap/output-of-marshal.stp-on-python-pass.txt )

Total cumulative size of r_object() calls returning strings with refcnt==1:  192K
value |-------------------------------------------------- count
    0 |                                                     0
    1 |@@                                                  54
    2 |@@@@@@@@@@@@@@                                     291
    4 |@@@@@@@@@@@                                        238
    8 |@@@@@@@@@@@@@@                                     281
   16 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@    959
   32 |@@@@@@@@@@@@@@@@@@@@@                              432
   64 |@@@@@@@@@                                          196
  128 |@@@@@@                                             138
  256 |@@@@                                                97
  512 |@@                                                  45
 1024 |                                                    11
 2048 |                                                     5
 4096 |                                                     1
 8192 |                                                     1
16384 |                                                     0
32768 |                                                     0

so (assuming my method is correct) we'd save 192K of mmap-ed data per python process.

For the OLPC case, each "activity" on the laptop is a python process that typically imports the GTK and DBus modules.

This shows a larger saving: 431K per python process:

$ stap -v marshal.stp -c"python -c'import gtk; import dbus'"

(output snipped; full output here: http://dmalcolm.fedorapeople.org/systemtap/output-of-marshal.stp-on-python-import-gtk-and-dbus.txt )

Total cumulative size of r_object() calls returning strings with refcnt==1:  431K
value |-------------------------------------------------- count
    0 |                                                      0
    1 |@                                                    65
    2 |@@@@@@@@@@@@@                                       534
    4 |@@@@@@@@@@@@@@                                      565
    8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                   1302
   16 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@   1958
   32 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                      1173
   64 |@@@@@@@@@@@                                         478
  128 |@@@@@@@@                                            336
  256 |@@@@@                                               216
  512 |@@                                                   87
 1024 |                                                     22
 2048 |                                                     10
 4096 |                                                      2
 8192 |                                                      1
16384 |                                                      0
32768 |                                                      0

Similarly, I suspect that there may be savings if you have numerous python web apps on one box (mod_wsgi daemon mode?), or via KSM savings as above if dealing with multiple guest VMs running python on one host.

Worth pursuing?

I also started looking at "yum"'s memory usage; see http://dmalcolm.fedorapeople.org/systemtap/output-of-marshal.stp-on-yum-whatprovides-python.txt . I wrote a systemtap script to try to instrument the various levels of memory allocation inside the python runtime; see http://dmalcolm.fedorapeople.org/systemtap/python-mem.stp ; unfortunately this script doesn't work yet, owing to a systemtap bug. Hopefully when that's fixed we can get some real insight into this.

[1] with Bernie Innocenti, Colin Walters, Luke Macken, Adam Jackson and others; please forgive me if I've forgotten you.

2to3c: an implementation of Python's 2to3 for C code

2009-11-21T00:54:51Z

I'm hoping that we'll package python 3 versions of as many modules as possible in Fedora 13, so the easier it is to port them, the better.

To that end, I've written a tool to help people port their C python extensions from Python 2 to Python 3.

It uses the Coccinelle tool to apply a series of "semantic patches" to .c files. I also had to code one of the refactorings in python with regular expressions (due to the need to manipulate preprocessor macros containing commas).

Sample session, running on a tarball of dbus-python:

[david@brick 2to3]$ ./2to3c --help
Usage: 2to3c [options] filenames...

Options:
  -h, --help   show this help message and exit
  -w, --write  Write back modified files
[david@brick 2to3]$ ./2to3c ../../python3/packaging/modules/by-hand/dbus-python/devel/dbus-python-0.83.0/_dbus_bindings/*.c > dbus-python.patch 

[david@brick 2to3]$ diffstat dbus-python.patch
 abstract.c       |   28 ++----
 bus.c            |    4 
 bytes.c          |   16 +--
 conn.c           |    7 -
 containers.c     |   21 ++--
 float.c          |    6 -
 generic.c        |    4 
 int.c            |   31 ++-----
 libdbusconn.c    |    5 -
 mainloop.c       |    3 
 message-append.c |    4 
 message.c        |   17 +--
 module.c         |  243 ++++++++++++++++++++++++++++++++++++++++++++-----------
 pending-call.c   |    3 
 server.c         |    7 -
 signature.c      |    6 -
 string.c         |    9 --
 17 files changed, 267 insertions(+), 147 deletions(-)

[david@brick 2to3]$ head -n 30 dbus-python.patch
--- ../../python3/packaging/modules/by-hand/dbus-python/devel/dbus-python-0.83.0/_dbus_bindings/abstract.c.orig 
+++ ../../python3/packaging/modules/by-hand/dbus-python/devel/dbus-python-0.83.0/_dbus_bindings/abstract.c 
@@ -54,7 +54,7 @@
 
     if (!vl_obj)
         return 0;
-    return PyInt_AsLong(vl_obj);
+    return PyLong_AsLong(vl_obj);
 }
 
 dbus_bool_t
@@ -76,7 +76,7 @@
         }
     }
     else {
-        PyObject *vl_obj = PyInt_FromLong(variant_level);
+        PyObject *vl_obj = PyLong_FromLong(variant_level);
         if (!vl_obj) {
             Py_DECREF(key);
             return FALSE;
@@ -127,7 +127,7 @@
     Py_DECREF(key);
 
     if (!value)
-        return PyInt_FromLong(0);
+        return PyLong_FromLong(0);
     Py_INCREF(value);
     return value;
 }

You can see the full patch it generated here: http://dmalcolm.fedorapeople.org/dbus-python.patch

It hasn't done all of the work, there are some places involving the preprocessor where it didn't quite generate correct code, and there are some remaining issues - for example, a human is going to have to decide whether the strings are bytes or unicode.

However, I think this ought to save a lot of time: it takes care of a lot of the tedious parts of such patches.

The public git repo can be seen here:
http://fedorapeople.org/gitweb?p=dmalcolm/public_git/2to3c.git;a=tree

You should be able to download it by cloning it thus:

git clone git://fedorapeople.org/home/fedora/dmalcolm/public_git/2to3c.git

Patches most welcome! (send them to dmalcolm@redhat.com) I intend to license this under LGPLv2.1, but am happy to relicense as the upstream Python community see fit.

Static analysis of CPython's .c code

2009-11-16T23:09:54Z

I've been hearing good things about Coccinelle for a while now, a tool for working with C code.

For example, it's been used for automating tedious (and error-prone) work on the Linux kernel.

I decided it was time to take it for a test-drive on CPython code.

I occasionally run into problems with the PyArg_ParseTuple API. It's a convenient API - it makes it very easy to marshal the objects passed as parameters to Python function calls into their C equivalents using a mini-language. The downside of this approach is that the compiler can't check such code for type safety, and so it's an area where bugs can lurk.

So I've written a tool which can detect such problems. You can download it from my Fedora people page using this command:
git clone git://fedorapeople.org/home/fedora/dmalcolm/public_git/check-cpython.git

or simply read the code online here:
http://fedorapeople.org/gitweb?p=dmalcolm/public_git/check-cpython.git;a=tree

To run it you'll need Coccinelle (which includes the "spatch" tool). On Fedora you can install it using this command:
yum install coccinelle

You should then be able to run it thus:
spatch -sp_file pyarg-parsetuple.cocci buggy.c

init_defs_builtins: /usr/share/coccinelle/standard.h HANDLING: buggy.c buggy.c:13:socket_htons:Mismatching type of argument 1 in ""i:htons"": expected "int *" but got "unsigned long *"

thus correctly finding the bug (an old one, fixed in http://svn.python.org/view?view=rev&revision=34931 )

Early days yet, but this seems promising. Does anyone know of any other non-proprietary tools that can do this kind of thing?

(I've posted more info to python-dev list here)

librpm and python 3(000)

2009-10-15T21:29:38Z

I've had a go at hacking up librpm's python bindings so that they work with both python 2 and python 3.

Here's my progress so far, trying out some sample rpm python code with py3k. Some things appear to be working, but I really need a unit test suite to do this properly, I think.


[david@brick rpm]$ PYTHONPATH=/home/david/coding/python3/rpm-python-bindings/install-prefix/lib/python3.1/site-packages python3
Python 3.1.1 (r311:74480, Oct  1 2009, 12:20:21) 
[GCC 4.4.1 20090725 (Red Hat 4.4.1-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpm
/home/david/coding/python3/rpm-python-bindings/install-prefix/lib/python3.1/site-packages/rpm/__init__.py:9: DeprecationWarning: Type rpm.hdr defines tp_reserved (formerly tp_compare) but not tp_richcompare. Comparisons may not behave as intended.
  from rpm._rpm import *
# (clearly I need to fix this)
>>> rpm.addMacro("_dbpath", "/var/lib/rpm")
>>> ts = rpm.TransactionSet()
>>> mi = ts.dbMatch()
>>> for h in mi:
...     print("%s-%s-%s" % (h['name'], h['version'], h['release']))
b'im-chooser'-b'1.2.6'-b'3.fc11'

etc. (note how print has become a function, and how the data is coming back as "bytes", not as strings)

For the gory details, I sent my patches here:
http://lists.rpm.org/pipermail/rpm-maint/2009-October/002528.html

Does anyone have a good test suite for the python rpm API?

Whence came "codebase"?

2009-06-24T20:27:59Z

The word "codebase" has been bugging me lately. I've noticed myself using it, and it's starting to grate on me.

It appears to mean a collection of computer code, from an implementation perspective. Users of the word might refer to the "foo-1.0 codebase" as opposed to the "foo-2.0 codebase, where we totally rewrote everything".

Assuming that this is what the word means, it's a useful one. However, where did this word come from, and when? Does it have a more precise meaning, or has it spread across the internet with the rough (but useful) meaning above?

The only definitions I can find are within Wikipedia or Wictionary (neither of which I'm a fan of). I don't see it in the Jargon File either (but ditto, frankly). It also seems to have a (different) specific meaning for Java applets, and is the name of a proprietary database product.

Does anyone have an etymology for this word?

squeal 0.4

2009-06-05T01:04:45Z

I've put together a tarball release of my SQL/command-line collision, formerly "show", now "squeal"

Tarball here: https://fedorahosted.org/released/squeal/squeal-0.4.tar.gz
SRPM for Fedora/RHEL 5 here: http://people.redhat.com/dmalcolm/python/squeal-0.4-3.src.rpm

Here's what happened other than the name change:

New Data Sources ¶

Text files and streams ¶

Squeal can now deal with arbitrary text files, and on stdin (using "-").

It uses the first line of the input to determine the number of columns, giving you an input source of string columns named "col0", "col1", ..., up to N-1.

By default it splits columns on whitespace, but you can use -F a.la. "awk" to specify a field separator, e.g.:

   cat some-file.txt | squeal -F: --format=html -

to generate an HTML table from a colon-separated file.

You can also specify a regular expression containing groups, which will become the columns, e.g.:

   squeal col0, count from -r "(\<[^\>]+\>) (.*)" irc-log.txt group by col0 order by count desc

to figure out who's the chattiest in an IRC log.

Of course, if you need more complicated parsing, it's probably worth writing a dedicated data-source backend.

Archives ¶

You can now issue SQL-like queries upon the contents of various kinds of archive files: .zip, .tar, .tar.bz2, .tar.gz2, and .rpm. For example, here's a query on the payload of an RPM file:

    $ squeal "total(size)" from ~/rpmbuild/RPMS/noarch/squeal-0.4-1.fc10.noarch.rpm
    total(size)|
    -----------+
       171407.0|

tcpdump/Wireshark files ¶

I wrote a proof-of-concept backend for querying tcpdump files, e.g.:

$ squeal "count(*)", "total(length)", src_mac, dst_mac from test.pcap group by src_mac, dst_mac

to analyse the quantity of network traffic between pairs of machines.

Internally it's merely invoking tcpdump turn the file back into textual form, then carve up with regexps, so it's not at all robust yet.

/var/log/maillog* (sendmail and spamd) ¶

I wrote an experimental parser for /var/log/maillog

Query Syntax Improvements ¶

Squeal can split its own arguments, to minimize the amount of escaping that you have to do within the shell. You can now pass in mixed queries like:

      $ squeal "count(*), total(size) host from" /var/log/httpd/*error_log* \
      "order by total(size) desc"

where some of the arguments are split by the shell, and some by squeal's parser.

You can now type "count" rather than "count(*)", provided "count" isn't a column name (it's a pain to type, and to have to escape this from the shell). So the above query can be simplified further to:

      $ squeal "count, total(size) host from" /var/log/httpd/*error_log* \
      "order by total(size) desc"

Output ¶

I added two new formatting options:

"text" : outputs as lines of space-separated fields
"table" : outputs an ascii-art table

Bugfixes ¶

The httpd log backend now supports parsing logs containing usernames
The syslog backend can now deal with single-digit dates within a month(!)
Deal with absolute and relative paths when path-matching input filenames; only check for Augeas below /etc
Various internal cleanups
Started a unit test suite.
Support Python < 2.5 by using earlier versions of sqlite; runs on RHEL 5
Work around an issue seen sometimes with the RPM backend
Detect exceptions in execution of the internal sqlite queries, and log them

Enjoy!

"show" is now "squeal"

2009-05-25T03:07:48Z

My command-line/SQL hybrid needed a new name, since "show" was regarded as too generic.

Thanks everyone who suggested a name, and for the feedback. "squeal" was my favourite, so "show" is now named squeal.

I'm working on a release; it's gained a few new features since I last blogged.

(Thanks to the Fedora Infrastructure team for doing the rename)

How not to render 3D graphics: 40 ways to get a blank black screen

2009-04-12T21:52:27Z

I've learned most of these the hard way:

The monitor is in power-saving mode
Your video card has two outputs, and the monitor is plugged into the wrong one.
The screensaver kicked in while you were talking to your boss, but your program has grabbed input focus.
You've forgotten to flip the display buffers. You're drawing everything OK, but not displaying it.
There's a bug in your memory management. You're rendering to a different area of video RAM than the one the monitor is reading from.
Your scene is too complex; either the CPU or CPU crashed before the first frame finished rendering.
The "near" clip plane is further away than the "far" clip plane
You're rejecting the wrong side of each line as you walk your BSP tree; your universe is hiding just beyond the corner of your eye.
You forgot to set up the bounding-boxes for your objects, and all of them are being rejected as too small
You have a bug in your matrix library. The entire universe has collapsed to (0,0,0).
You have a bug in your quaternion library. All of your rotated objects are collapsing to points.
You have a bug in your back-face culling. Every face is being culled.
Your camera is too narrow angle; the universe has collapsed to a single point directly ahead of you
Your camera's so wide-angle... (insert mother joke)
You've confused the origins of screen space and of the frame buffer, and all of the scene is being rendered off the side of the framebuffer (and culled per-fragment)
You're using index-colour, and you forgot to set up a palette: all colours are black.
Your scale is wildly wrong: the scene is vastly larger than you expect. You are viewing a single texel on a single triangle in the scene.
Your scale is wildly wrong: the scene is vastly smaller than you expect. The entire world has collapsed to a tiny point in the centre of the screen.
The camera is outside the scene and pointing the wrong way. Did you try looking behind you? Above you? etc
You forgot to add any lights to the scene. Darkness remains over the surface of the deep.
You forgot to set the textures. All of the scene is being rendered with a blank texture.
You forgot to set any UV coordinates. The entire scene is being rendered with the colour at (0,0) in their textures.
You're doing integer multiplication of fractions and everything is coming out as zero.
You've forgotten to offset the objects in the scene in world space and relative to the camera. You are viewing all of the visible objects in the scene from inside, and back-face culling ensures that nothing is visible. (Insert bad pun about "Lost In Translation")
You've forgotten to pop matrices from the transformation stack, and you overflowed the stack after a few frames. (You should check error codes at least once per frame)
Your collision algorithms aren't good enough. The object representing the camera has fallen through the floor and is hurtling at great speed downwards into nothingness whilst the world disappears high above.
Your geometry shader has a bug, and all geometry in the scene is appearing behind the camera
Your fragment shader failed to compile, and everything is coming out as black.
You forgot to clear the z-buffer, and all of your fragments are being rejected.
Your scene is foggier than a lazy TV remake of Sherlock Holmes.
You forgot to set up alpha values, and the entire world is fully-transparent.
You're standing in front of a black wall.
The game logic has decided that we're fading out the screen, in preparation for the next level.
You set up all of your data correctly, but a stray pointer is trashing one/all of the above.
The in-game menus have appeared, overlaid in front of your scene. Unfortunately, due to poor state management, they're far too big, and a single corner of one letter "A" is obscuring the entire screen with black.
Your radiosity renderer doesn't have enough photons.
Your volume shader is too opaque.
Your bidirectional reflectance distribution functions aren't reflective enough
Time is running at a different rate than it should be, and your simulation code has either destroyed all of the objects in the scene, or none have been created yet.
There's a FIXME in your code that really needs fixing...

IM IN UR UZER-BIN BRAKIN UR TAB COMPLEESHUN

2009-04-01T21:44:51Z

I'm hoping to get my SQL/command-line mashup into Fedora, but, alas "show" is too generic to go into /usr/bin.

Some names have been suggested:
- "squeal", a mispronunciation of "sql" (thanks Nalin and Jeremy).
- shelect

Anyone got any other ideas?

My current favourite name is squeal.

(The current project location is here and the package review request is here)

Be excellent to each other

2009-03-30T21:21:44Z

I wholeheartedly agree with Seth's comment here.

More command-line SQL

2009-03-25T02:32:39Z

I've done a bit more hacking on my command-line/SQL mashup, currently called "show".

It can now handle /var/log/messages, /var/log/secure (and the rotated logs), so you can issue a command like this:

  $ show /var/log/secure* where message like \"%authentication failure%\"

and browse the results

For example, here's a query with aggregation:
$ show "count(*)", source from /var/log/messages group by source order by "count(*)" desc limit 5

count(*)|source        |
--------+--------------+
1635    |kernel        |
1398    |NetworkManager|
98	|ntpd          |
70	|avahi-daemon  |
63	|dhclient      |

Going beyond log files, I used the rather wonderful Augeas library to get parsers for many of the files in /etc, and wrote a backend to leverage this, so you can write things like:

  $ show /etc/passwd where shell !=\'/sbin/nologin\'

and

  $ show /etc/yum.repos.d/*.repo where gpgcheck != \'"1"\'

(it's a little dumb about string vs numeric types, and shell escaping requires lots of quotes here)

I extended the ncurses table-browsing UI so that you can scroll horizontally as well as vertically, which helps when the columns are wide.

The Fedora infrastructure team set up a hosted project for me, so you can see the source here:
https://fedorahosted.org/show/browser (thanks!)

An up-to-date SRPM can be grabbed from here:
http://people.redhat.com/dmalcolm/show-0.3-1.fc10.src.rpm

and you can grab the source via git here:
$ git clone git://git.fedorahosted.org/show.git

Thanks to everyone for the great feedback on my previous post.

I suspect some kind of integration with Func for running queries over groups of machines would be a good next step for this tool (oh, and fixing up the Trac instance)

Is /usr/bin/show too generic?

SQL for the command line: "show"

2009-03-23T00:37:00Z

I found myself analyzing some apache log files the other day, and found myself wanting a SQL interface to the logs.

Now, it's possible to send the logs directly into a database, but that wasn't how the machine was configured.

This got me thinking. We have many different log formats, and many different sources of data. All of our tools seem to have different interfaces.

For example, why should I write regular expressions and shell pipelines to get at my logs?
Why do I have to learn a custom syntax ("rpm -qa --queryformat='various things'") for looking at the software I have installed? Why does e.g. the audit subsystem have its own query format?

Why can't I just use SQL, and write SELECT statements to drill down into all of this data?

So I started writing a SELECT statement for the command line.

I didn't want to use SELECT as caps are a recipe for RSI, and annoyingly, "select" is a bash builtin.

So I've picked "show" as the command; it doesn't seem to be taken by anything on my system. (it's a SQL command, but hopefully that's not going to be too confusing)

The idea is that it looks at the FROM part of the query, and looks up a data source based on this. For example, if you write


$ show host, "count(*)", "total(size)" from /var/log/httpd/*access_log* group by host;

it figures out that you're looking at apache logs, looks up an appropriate backend, and makes a table "on the fly", so that it can run the query and give you the results:


     host|count(*)|total(size)|
---------+--------+-----------+
127.0.0.1|      10|    27679.0|

(You have to either use quotes or escaping to deal with parentheses and * characters from the shell)

You can use filters using "WHERE":


$ show distinct request from /var/log/httpd/*access_log* where status = 404
request                  |
-------------------------+
GET /favicon.ico HTTP/1.1|

If you specify more than one filename it adds a "filename" column:


# show filename, "count(*)", "total(size)" from /var/log/httpd/*access_log* group by filename order by "total(size)" desc
                       filename|count(*)|total(size)|
-------------------------------+--------+-----------+
/var/log/httpd/ssl_access_log.4|    1921| 12824849.0|
    /var/log/httpd/access_log.3|     222|  6207367.0|
/var/log/httpd/ssl_access_log.3|     741|  2210799.0|
    /var/log/httpd/access_log.4|     268|   626711.0|
/var/log/httpd/ssl_access_log.1|       8|    13351.0|
/var/log/httpd/ssl_access_log.2|       5|     7305.0|
    /var/log/httpd/access_log.2|       4|     6995.0|
    /var/log/httpd/access_log.1|       2|      288.0|

Naturally, this isn't just for apache logs.

Here I'm querying the yum logs. The backend code deals with the changes that happened in the logging format, without me having to worry about this in my query:


[root@brick select]# show from /var/log/yum.log* where 'name like "kernel%"' limit 5
           time|    event|           name|  arch|epoch|  version|     release|        filename|
---------------+---------+---------------+------+-----+---------+------------+----------------+
Feb 14 20:00:03|Installed|kernel-firmware|noarch| None|2.6.27.12|170.2.5.fc10|/var/log/yum.log|
Feb 14 20:00:28|  Updated| kernel-headers|  i386| None|2.6.27.12|170.2.5.fc10|/var/log/yum.log|
Feb 14 20:15:11|Installed|   kernel-devel|  i686| None|2.6.27.12|170.2.5.fc10|/var/log/yum.log|
Feb 14 21:05:53|Installed|         kernel|  i686| None|2.6.27.12|170.2.5.fc10|/var/log/yum.log|
Feb 14 21:12:41|Installed|     kernel-PAE|  i686| None|2.6.27.12|170.2.5.fc10|/var/log/yum.log|

I also wrote "rpm" and "proc" backends:


[david@brick ~]$ show name, "count(*)" from rpm group by name having "count(*)>1"
                     name|count(*)|
-------------------------+--------+
               gpg-pubkey|       4|
jakarta-commons-validator|       2|
 java-1.6.0-openjdk-devel|       2|
                   kernel|       3|
             kernel-devel|       2|
               kernel-xen|       3|
                  libgnat|       2|
                  openssl|       2|

Looking in RPM database by vendor:


[david@brick select]$ show vendor, "count(*)" from rpm group by vendor
            vendor|count(*)|
------------------+--------+
              None|      12|
    Fedora Project|    2042|
              Koji|       2|

It got tiresome typing "*", and "from" all the time, so you can omit these:


# show rpm where release not like \'%fc%\' order by name limit 4
    name|epoch|version|release|  arch|        vendor|
--------+-----+-------+-------+------+--------------+
 MAKEDEV| None|   3.24|      1|  i386|Fedora Project|
   PyXML| None|  0.8.4|     10|  i386|Fedora Project|
  autofs|    1|  5.0.3|     36|  i386|Fedora Project|
automake| None| 1.10.1|      2|noarch|Fedora Project|

There's a --format option, which can currently be used to emit tables in HTML format. Other formats could be written e.g. xml, json, yaml, odf spreadsheets, etc.

For bonus points, I wrote an ncurses UI for browsing tabular results. The command detects if stdout is connected to a tty, and if so, goes into the UI, otherwise it sends text, so you can use it for shell pipelines etc.
Here's a screenshot from

show rpm order by name

I'm already finding this useful.

I'm hoping to host this as a Fedora project. For now, you can grab the code from my Red Hat page here:
http://people.redhat.com/dmalcolm/show-0.2-1.fc10.src.rpm

There's plenty of scope for writing new table backends for other data sources/file formats, improving the UI, writing new output formats etc Any ideas?

Things to do when you're waiting for a compile

2008-04-16T21:15:44Z

I implemented XInclude support for docbook-lint just now, or rather, a first pass at it. Most documents I've been pointing it at turned out to be built from a short "core" document that has a collection of XInclude elements pointing at the rest of the content, so this was a must-have feature.

Seems to work, but needs testing. Hopefully I'll have a chance to try hooking it up with Publican soon (dear lazyweb: it's not clear how to get at the source directly on that page), and maybe make the toolchain a little better.

Trying to improve the signal:noise ratio

2008-04-13T23:40:45Z

I've been avoiding this for a while, but thanks to Jeremy's prodding, I've started this blog. I've got a few coding projects in-flight at the moment, and it's about time I let the world know about them. So far I've set up a home page for docbook-lint, but there are some others on my hard drive that ought to see the light of day. More to follow... I hope.