One Liner At a Time

Just some one-liners. 'Cause, you know, to post something, and to each one its own.

even_map = dict(('key_%s' % i, i) for i in range(1, 9) if i % 2 == 0)

even_map = Hash[(1..9).find_all { |i| (i).even? }.collect {
    |i| ["key_#{i}", i] }]

(def even-map (into {} (for [i (range 1 9) :when (= (mod i 2) 0)]
                         [(keyword (str "key_" i)) i])))

Labelled Reduction as A Good Thing

One small thing (of the many) in Python 2.6 I like (and have waited for since it appeared as a recipe), is collections.namedtuple. It is very useful in itself, but the fact that the stdlib has been adapted to use it throughout is quite nice. Consider the following code:

from urlparse import urlparse
print urlparse(
        "http://localhost:8080/doc/something;en?rev=1")

If run with Python 2.5, you get this tuple:

('http', 'localhost:8080', '/doc/something',
 'en', 'rev=1', '')

, whereas in 2.6 it is a namedtuple:

ParseResult(
        scheme='http', netloc='localhost:8080',
        path='/doc/something',
        params='en', query='rev=1', fragment='')

The last one is unpackable just like a regular tuple, but you can access the parts as attributes as well. This little "data struct" is quite handy, since I don't like to access tuples by index, but quite often pass them around and only access some piece of them at a time.

(With these you don't need to create full-fledged classes for every kind of instrumental data. (Sometimes coupling data and functionality in a single paradigm may be a coarse hammer, treating nails and iron chips alike..) Nor resort to the use of dictionaries where you really want a "restricted value lens", if you will.. But this is another rant altogether.)

Of course, there's lots more to enjoy in 2.6 (the enhanced property for decorator use, ABC:s, json, 2to3 etc).

On a related note, do check out Swaroop C H:s excellent and free books on Python (2.x + 3.0(!)): A Byte of Python. And if you're into Vim (you should be, IMHO) his new A Byte of Vim.

Out To Transmogrify Resource Descriptions

Finally, I've bundled another release of Oort. And OortPub! I've modularized the package into two Eggs, separating the core O/R things from the (experimental) WSGI web stuff.

At the cheeseshop you'll now find them both, at

• Oort (0.4), and
• OortPub (0.1)

respectively. I did this not the least since the core Oort package is very useful on its own, in conjunction with other tools altogether. I've used it for a lot of stuff lately, none of which involved OortPub. Check out e.g. the QueryContext for some new features..

I hope this will become useful to others as well.

The website got a little overhaul as well, to reflect this change, and to include some more documentation (not so much more yet, but the basic stuff is in place). Oh, by the way, it's generated using Oort objects (among other things like Genshi, ReStructuredText, google-code-prettify and of course RDFLib).

OMG! Snakes on a Phone!

So, I got this new cell phone from work. A Nokia 6120 Classic, which has just the size and features I wanted (and more, I'm sure). I picked the white/silver model, to go with my kitchen supplies (in honour of the fact that with Series 60, you can basically install the Kitchen Sink if you so choose to).

Going further, I of course wanted to play Monkey Island and such on the amazing ScummVM. Well, admire the running of; actually playing through them might be a little straining.. Though great screen clarity makes it quite possible to discern the orange and pink on the red herring you steal from the Loom™ seagull outside the Scumm Bar kitchen, it's quite thumb-numbing to actually snatch it in time..

And then, surely, I couldn't wait to get to the real goodies: Python. Specifically, The Python for Series 60 distro by Nokia (open source). I knew it was quite capable, but seriously; it basically lets you p0wn your phone.

(Half a decade ago I had fun running Python on my old Psion Revo; but that didn't go much further than some file tinkering and exploration of new-style classes (metaprogramming and whatnot) during commuting. But this is different. The same. But different.)

You go ahead by following the instructions, basically downloading the PythonForS60_1_4_0_3rdEd.sis and PythonScriptShell_1_4_0_3rdEd.sis from sourceforge, wiring or beaming them to the phone and running them to install (py, then shell). I also got the PythonForS60_1_4_0_doc.pdf, and marvelled at its beauty. I mean, apart from greater parts of the Python Standard Library, a mere glance at the modules provided for S60 stuff gives you an impression of the power at hand:

graphics, e32db, messaging, inbox, location, sysinfo, camera, audio, telephone, calendar, contacts, keycapture, topwindow, gles, glcanvas

Now, I haven't gotten very far yet; mostly tinkered with urllib to download some stuff from the web (which just worked (after a nice confirmation dialogue from the phone to actually access the net)), rendering the obligatory mandelbrot and then peeking at the real deal by looking at the SMS inbox contents using the interactive console.

Or actually, using

$ screen /dev/tty.bt_console

from my OS X Terminal (iTerm really). Magically set up by simply following this excellent guide.

Perhaps screen scraping wasn't really what I thought of when wanting Python on the phone, but you can't deny it's cool to be able to (perhaps awaiting flatrate subscription to not drain my wallet in the process though).

Speaking of screens and webs again; damn that WebKit-based browser in the phone really worked! It actually ran the JavaScript-based SyntaxHighlighter, thus rendering the Oort Tutorial quite faithfully. I did not expect that (nor my rekindled desire for highly variable width web sites..).

[And speaking of Oort; I've added some somewhat nifty stuff (if I may say so) lately; accessible in the svn repo at google code for now. A release should be coming up. Pre- or post my planned split into core and web/publication stuff I haven't decided yet. There really is a working RDF-to-object mapper underneath the touted WSGI stuff (which is just a minimal layer).]

Now I should go on to do something productive (like shaving a yak).

Cutting Edges

For all of you out there writing droves of RDF, churning out endless amounts of Notation 3, and doing it all in Vim.. (we must be in the millions I'm sure), be sure to check out my little RDF Namespace-complete — giving you RDF Vocabulary/Model Omni-Completion for Vim 7+. Requires Vim compiled with Python and RDFLib.

On a related note, I've been using RDFLib a lot lately (around the clock as it seems). It's an honor to contribute in an ever so small way to it, and it's great to see it progressing so well (it's been mature for years if that needs to be said). The SPARQL-support is working, and along with Chimezie Ogbuji's FuXi there is now an even stronger Python RDF platform to work with.

Oort In Space, In Time, At 0.3.2

Oort is now at 0.3.2. See the release history for details. Also, check out the tutorial — now functional at last.

Apart from fixes, the API is slightly cleaner and RdfQueries can now be updated, directly or by loading a dictionary. This means that JSON to RDF is a no-brainer when using Oort to do the semantic enhancements. (This is very new though, use with care.)

Knowledge, The Bits and Pieces

In a recent post by Lee Feigenbaum, he talks about Using RDF on the Web. Naturally I find this very interesting. In my work on the Oort toolkit, I use an approach of "removing dimensions": namespaces, I18N (optionally), RDF-specific distinctions (collections vs. multiple properties) and other forms of graph traversing. This is done with declarative programming very similar to ORM-tools in dynamic languages (mainly class declarations with attributes describing desired data selection). The resulting objects become simple value trees — with the added bonus of automatic JSON-serialization.

Ideally, this approach will be deterministically reversible. I have not yet implemented it, but the idea is that the declared classes (RdfQueries in Oort — let's call them "facets" here) could take JSON as input and reproduce triples. Using checksums of the JSON would make over-the-web editing possible.

Since the task of "updating" a subgraph is somewhat difficult at best, I think a basic "wipe and replace" approach may be simplest. There are many dangers here of course (removing a relation must not remove knowledge about its object — unless perhaps if that object is a bnode..).

Albeit all of this is Python to the core now, nothing in the design — declarative as it is — prevents the approach from being more general. Indeed, such facets, were they themselves serializable, could be used as structured content retrieval over-the-web too. Ok, maybe I'm reinventing SPARQL now.. Or should I use SPARQL for their remote execution? It seems reasonable (I mean that's exactly how ORMs do SQL).

Now, I seem to end up in the RPC/RESTful camp with this. A solution to that could be: use the facets on the client, having them use SPARQL for retrieval. Then you have clients working against any kind of SPARQL endpoint, mashup-style. Still, if facets are completely reversible, they may be powerful, aware tools, and perhaps an alternative to SPARQL in intercommunication? That's a pipe dream for me right now though.

The SPARQL-way is of course only for reading data. Fine, the RDF model may be too expressive (fine-grained) to be practical for direct over-the-web editing in specific situations anyway. A confined approach such as this JSON+checksums+"careful with nested knowledge" may be better for this.

I think of JSON-RPC here as I view e.g. microformats with GRDDL — it's leverage, not a final solution. RDF models are the ideal, we may just need reversible simplifications to gain mass usability. I touched upon JSON for integrated apps, but stuff like Atom for "talking to strangers" in response to a previous post by Lee. His post which I refer to above hints at better stuff than just Atom if we want RDF all the way also in more specific apps.

So, what I'm saying is really: I also desire the best of two worlds. The RDF model is sound and extremely valuable, but after all, simple domain-specific object manipulation is what makes the web go round. A solution may be some form of O/R mapping for RDF. The difference is not wanting to forget the data layer (there is no monster in the closet as there is with raw SQL..), just streamline some of the work with it.

Let's hope 2007 will be a good year for the knowledge revolution.

Cobwebs and Rainbows, Round 3

Web 3.0. Like 2.0, but meaningful and integrated (as opposed to unqualified punyformats and mess-ups). And sensible (whatever that means.. "think Scandinavian Ajax" — to cite my boss at V.).

I'll be damned if that revolution won't finally take us from the information age to the knowledge age. Of course, this would mean yet another technological shift — one I've been longing for for a long time. Let's see (disregarding "Enterprise Web X" here; sorry Java and whatNot- uhm.. dotNet):

Web 1.0

Perl + random flatfiles and some raw SQL (websafe black, white and blue)

Web 1.5

PHP + SQL (off-black on white, green and blue hues, some orange)

Web 2.0

Rails + generated SQL (pink on white, huge fonts, lots of attitude — think adolescence)

Web 3.0

Python + RDF (color is not an issue)

Or I'll die trying. Truly. Die. Trying. (Well at least I may end up drinking as much in my thirties as I did in my twenties. Cheers.)

Then for 4.0, I suppose fully OWL-enabled reasoners and modernized but still brutal OCaml could be fun.

[I'm not utterly serious here, but a bit. Please don't take too much offense.]

Fvyyl Indeed

Oh. My. God. I had no idea.

From now on, I shall only do pynff-based OO. Or BB, as it were.

#!/usr/bin/env python
#-*- coding: rot13 -*-

qrs whfg_n_fvyyl_rknzcyr():
   cevag "Bu, ubj fvyyl guvf vf.".qrpbqr('rot13')

vs __anzr__ == '__main__':
   riny('just_a_silly_example')()

Yes, it runs fine. Try it. (Note: in Vim you can use g? on selected text. Just so you know.)

Oort To Got Itself Updated

Version 0.3.1 of Oort is out, with some fixes, cleanup and new stuff. See the release history for details. (Also, note that Oort now resides at http://oort.to/).

I've been using RDF a lot lately (in work with legal information for the swedish government), which gave me opportunity for more real-life testing of this toolkit. I hope you find it of value too.

The base technologies of Oort — RDFLib, Genshi and Paste — have gotten updates lately too. Be sure to check them out.

One Oort to RDF, o-dot-three

Another one hits the Dust.. feed. Uhm. Ok, seriously.. I just put up Oort 0.3, with some new stuff. Mainly a rudimentary template for Paste. And the switch from Kid to Genshi for templating.

I've also begun writing some kind of tutorial.. Like many others, I use dp.SyntaxHighlighter for the code samples (the Oort pages themselves are written in reStructuredText and assembled with the other stuff through Python and Genshi).

And the Oort pages are looking a tad bit better, I hope. ;)

Nightly Rewrites

After literally an all-nighter yesterday Oort 0.2 is now up for grabs. A very sparse example at that page will serve as a stand-in for a proper introduction for a while. After the cleanup mentioned earlier, I can finally toy with some operator overloading sugar (>> for now). There are some tests included now, and then there is the addition of RdfDisplay and JsonDisplay to remove the need for templates to carry out serialization of query results to those formats.

Within the Outer Space

While closures are very interesting and useful, they may be less of a wise tool when implementing an extendable toolkit. The first version of Oort used them extensively in the oort.rdfview package. My code was quite illegible and obviously a case of failing to be "clever". A more "properly dressed" class-based approach was (as I honestly felt even when I first stepped off the beaten path) evidently much more easy to maintain. Still, state change is less evident in the flat representation of a class, which doesn't give a structural hint of the intended temporal aspect. Using stuff like IllegalStateException doesn't alleviate this much at all (it's a "formally polite design" though). The impression of "loading" a state and bundling a nice nest of code to unfold at each state change is a quite powerful aspect of closures, I guess.

So I thought, let's do that in a somewhat declarative manner.. And thus I (just now) sketched up a way to declare a state change as an hierarchical closure nest which must be unfolded in the proper sequence:

class stateflow(object):
    def __init__(self, start_func):
        self.start_func = start_func
        self.steps = []
        self.reset()
    def start(self, *args, **kw):
        self._started = True
        return self.start_func(*args, **kw)
    def next(self, step_func):
        self.steps.append(step_func)
    def __getattr__(self, name):
        return self.__dict__.get(name) or self._get_next(name)
    def _get_next(self, name):
        if not self._started:
            raise BadStateError("Not started.")
        step_func = self.steps[self._at_step]
        if step_func.__name__ != name:
            raise BadStateError("Must call %s before %s."
                    % (step_func.__name__, name))
        self._at_step += 1
        return step_func
    def reset(self):
        self._started = False
        self._at_step = 0

class BadStateError(LookupError):
    pass

#== Let's test it ==

@stateflow
def query(database, subject):
    @query.next
    def with_query_data(predicate):
        @query.next
        def run_query(context):
            return database[subject][predicate] % context

database = {'s': {'p': "value in %s"}}

query.start(database, 's')
query.with_query_data('p')
assert query.run_query("context") == "value in context"

query.reset()
query.start(database, 's')
try:
    query.run_query("context")
except BadStateError, mess:
    assert str(mess) == "Must call with_query_data before run_query."
else:
    assert False

query.reset()
try:
    query.run_query("context")
except BadStateError, mess:
    assert str(mess) == "Not started."
else:
    assert False

Pretty interesting, I guess. Won't be putting that into my toolkit just yet though (if ever). So, dear closures, it was a nice experience, but I just have to let some of you go. Till next time, keep it simple!

(Oort 0.2 is on the way, by the by. I oort to create an introductory tutorial for it.. And a logo. Yes yes, of course; without a logo, what kind of a toolkit would it be?)

Whisky Pasting Space Clouds

My spirits somewhat rekindled by this nice presentation about WSGI, Paste and Pylons, I finally undertook the actually easy task of switching from CherryPy to a pure Paste-utilizing WSGI. Why? What? Last things first: Oort, as previously mentioned, is a little web toolkit for building RDF-driven web apps. It has a very specific scope, and thus the why-answer is: because it's better suited as a small WSGI-thing composable with whatever else you want to build legac* ehrm, regular web apps with. So go ahead and publish your RDF Graphs (I'm sure you have an abundance of them) with Oort, right among your other object branches in CherryPy, or mod_python, or among the URLs of a Pylons app, or whatever.

WSGI is a really neat thing, being so simple (to use) as it is. Same goes for RDF, in my opinion, but that's another rant. (A quite reoccurring one at that, if you happen to know me..)

Oh, where is Oort, you may ask? I finally put it up for display at the CheeseShop tonight. Like so many other things, it's alpha software and all. It is used for real work though. Next thing could be a little how-to (and why not a nice logo..).

Logix

I get a fuzzy feeling (with a dash of vertigo) when I look at Logix. For something managing to utterly violate many of the finer points of restriction that make Python programming resistant to cases of obfuscated cleverness, Logix is one of the coolest things I've seen lately (I guess I'm a year late to the party though). I've been doing some Ruby inquiries recently, to capture the finer things and evaluating my potential Python bias (while interesting, I see no reason to abandon my current favorite; still: YAR (Yet Another Rant)). I've also taken some time to look closer at Haskell, with some fun effects on my sleep cycle.

Although I might sometimes ponder if the Python syntax could be tweaked some to great effect, I often realise that I'm picking daisies instead of harvesting for the winter. Python is brutally straight-forward and effective as it is. Still, Logix is what I obviously overlooked when I saw Pluvo earlier this year and felt the lure of language experiments. Logix may be in a state of flux, but I really wish that it would prosper as somewhat of a hyper-intelligent, mind-boggling but brutally powerful layer on top of Python. If nothing else, it would give me and everyone else believing that this or that could be a better syntax for the job an easy way to test it. And possibly discover that it was a better syntax for a job, but perhaps not the right job. Still, Logix can definitely make declarative programming and design of DSLs a breeze. Or perhaps a tornado.

For a concise example, see this excellent article.

The Tenfold Error in Language Choice

Sometimes I just know I must have missed some convienience method..

static String makeEncodedValue(String value) {
   try {
       MessageDigest digester =
               MessageDigest.getInstance("MD5");
       byte[] digest = digester.digest(
               value.getBytes());

       StringBuffer buf = new StringBuffer();
       for (int i=0; i < digest.length; i++) {
           int nValue = ((int) digest[i]) & 0xFF;
           buf.append(Integer.toHexString(nValue));
       }
       return buf.toString();
   } catch (NoSuchAlgorithmException e) {
       // ... must.. do.. error.. recovery..
   }
   return value;
}

Oh..

import md5

def makeEncodedValue(value):
    return md5.md5(value).hexdigest()

Time for coffee.

All About the Package

I'm currently in the process of cleaning up and packaging Oort - a Python-based thingy used for making web apps driven by RDF graphs (there will be more where that comes from, rest assured).

Following is my little list of good starters when doing Python package development.

Most obvious: take a good look at setuptools (especially the concepts and usage of eggs, easy_install and pypi).

For testing, Nose is my one-stop-shop for now. Avoids the articificality of the xUnit-API; uses automatic discovery (with filtering); finds doctests if desired; uses coverage.py if available.

It's probably always wise to follow the practises of the most modern and well-renowned packages. I think Paste is a good example of this.