journal.stuffwithstuff.com

40 Songs

2018-05-19T00:00:00-07:00

I turned 40 today. Well, actually, as I’m writing this, I’m not yet 40. And, unless you happen to be reading this on the very day I post it, it’s no longer my birthday. Chronology in text is weird.

I’m not gonna lie. This number is hitting me hard. Intellectually, I get that the fact that it’s a round number is merely an artifact of our base ten numeral system, which is in turn an arbitrary quirk of the evolutionary history that led to us having ten fingers. I get it.

But what it really feels like is the midpoint. As a male in the US, 40 isn’t that far from the halfway point of average life expectancy. I have more memories accrued than new experiences to anticipate. In the great hallway of life, there are more doors behind me than ahead, which makes the regret of those unopened ones all the more acute.

I was talking about all this with my sister-in-law a few months ago. Her milestone is coming soon too. She plans to celebrate by writing a list of her top forty favorite songs. I’ll be damned if I don’t love a good list, so here’s mine.

It’s in chronological order of my personal relationship to the song. It appears when it appeared in my life. The luxury of this being my list is that I get to scramble time as it suits me.

40 is a pretty long list of songs. Heck, some days it feels like a long list of years. So we’re going to need some speed, and what better way to get moving then…

La Grange - ZZ Top

As you’ll find out, I live for a good build-up, and there is none better than this. A couple of sticks on the edge of the snare and a palm-muted riff and holy crap what just happened to my pulse. ZZ Top is possibly the tightest band that ever lived. Every punchy ghost hit on the snare is so locked into the guitar and bass that it must have been played by a single giant six-armed demon.

If your car is on fumes, wheezing out its last few yards, threatening to leave you stranded in the desert, crank this song up and I promise you you’ll get another mile or two out of the tank.

Sweet Dreams (Are Made of This) - Eurythmics

This is the first song that I distinctly remember hearing for the first time. Picture tiny me standing in front of a TV, staring mouth agape at Annie Lennox’s bright red hair and that weird-ass video, wondering what the hell was going on and why I liked it so much.

I sometimes wonder if imprinting on this song at such a young age twisted my brain. Lord knows I love a synth bassline and a four on the floor kick to an unhealthy degree. That thing I have for short hair on women must have come from somewhere.

Maneater - Daryl Hall & John Oates

I’m just old enough to remember roller rinks. My older brother and I would go, and I have vivid memories of how cool it felt to be in that dark, cavernous room, surrounded by pulsing sound.

“Maneater” is one of the songs I remember from there. (“Mickey” is another.) It ticks almost every checkbox for what I still love in a song: tons of reverb, high string pads, machine-precision drums, rhythmic bassline locked to a minor chord progression. I know I should hear 80s cheese when I listen to it today, but it still sounds just as spacious and moody today as it did then.

What I also remember is strapping on my skates every time and never once having the courage to venture off the carpet onto the rink. It took me thirty years and a friend’s skating party to face that fear and finally get out there.

Send Me an Angel - Real Life

Alright, I gotta speed this up if we’re going to get through all forty in a reasonable amount of time. There is a special place in the dark–of course it’s dark–corner of my soul for sad synthpop. I don’t know what it is, but slap some maudlin lyrics on top of a dance beat and I’m all over it. Fast + sad is my jam.

The whole genre pushes my buttons, but “Send Me an Angel” stands out because:

David Sterry really goes for it lyrically. It’s hard to top “if a girl walks up and carves her name in my heart, I’ll turn and run away” for abject pathos.
That little synth choir melody is six notes of absolute perfection.

Decades later, I was at a show for a friend of a friend’s band. The singer said they were going to do a weird cover and if anyone knew the song, to shout out the name of the band. He played the first three notes of that melody and it was like the heavens opened above, presumably so angels could watch me yelling “Real Life!” over and over at the top of my lungs like an idiot.

Enjoy the Silence - Depeche Mode

Speaking of synthpop songs with aching melodies. I’ve listened to Depeche Mode so much over the years that they are all over my memories. But the best, the one I’ve shared with literally no one until now…

My best friend in elementary school had an older sister. My friend and were two total nerds (still are), but his sister was so cool–serious and artistic. Exhibit A: giant Depeche Mode posters on her wall. Exhibit B: a complete and utter disinterest in interacting with us for even a second. At that age, that was about all it took for a crush to blossom.

Walk Like an Egyptian - Bangles

At some point in my childhood, I informed my parents that I liked music and desired the ability to listen to it in my bedroom. My Dad, being the music snob he is, solved that problem by buying young me an honest-to-God record player. No cassette tapes in this household, young man!

One of the first records I got was “Different Light” by the Bangles. I spent a lot of time listening to the bassline in this song, gazing at the photos on the cover, and having thoughts about Susanna Hoffs I wasn’t quite old enough to process yet.

Istanbul (Not Constantinople) - They Might Be Giants

Look, I don’t know what happened. I’ll blame it on incredible middle school awkwardness, but I went through a phase where I, I shit you not, listened to nothing but TMBG. For like two years. I still know all of the lyrics to every single song on “Flood”, “Lincoln”, “Apollo 18”, and “John Henry”.

They aren’t my thing very much these days, but the lyrics are still up there in the wetware should a particularly odd karaoke night have need of them.

Run Through the Jungle - CCR

I mentioned intros, right? This is another of the greats. I think most people like the happy-dumb CCR hits, but for me I want the ones that remind me that the US was goin’ through some shit when those songs came out. This tops that list.

Nights in White Satin - Moody Blues

One of the things I truly lament about my generation and the generations to follow is our acute self-awareness. Ironic detachment of the 90s followed by ironic attachment left us basically unable to produce art without constantly obsessing about what the art says about the artist.

Can you imagine anyone today sitting down to record a rock album with an orchestra, and then slapping a poem on the end? And doing it with complete, heartfelt, unironic sincerity? It’s an ability that seems to be completely lost.

Thankfully, this song embedded itself in my subconscious before that cultural shift happened. I love every single bit of this song, completely, totally.

Stayin’ Alive - The Bee-Gees

Speaking of things I love un-ironically. For a nerdy white dude, I have spent a surprisingly large amount of time shaking my ass on various dancefloors across the United States. Not with any particular skill mind you, but with great enthusiasm.

I like basically any kind of dance music, and disco is certainly on that list. The genre is a distillation of everything that makes a song danceworthy, with everything unnecessary filtered out. Picking one disco song is hard, and picking one played by white guys is a borderline travesty, but I’d be lying if I said I didn’t know every single cymbal crash in this song and had my arm thrust, finger extended, for each.

Groove is in the Heart - Deee-Lite

Bootsy’s bassline. That beat. If this don’t get your booty movin’, your booty must be dead.

Free Fallin’ - Tom Petty

Thomas Earl Petty is another one like Depeche Mode where it’s hard to know where to slot him in. His music is a constant companion in my life. Narrowing his presence on this list down to a single song was hard enough. “You Wreck Me”, “Learning to Fly”, “American Girl”, “Breakdown” are all strong contenders.

I picked this one because it spans what I think of as the two sides of Petty – the straightforward timeless American rock, and the bittersweet character studies.

Stand By Me - Ben E. King

A high school friend got a cheap acoustic guitar and I eventually got my hands on it. Tinkering around, I figured out the bassline to “Stand By Me”, playing that guitar like a bass, upside down. (I’m left-handed and it was strung right-handed.) This was the first bassline I ever learned.

Did you know King never intended to record this? He wrote it for the Drifters and only reluctantly recorded it himself when he had some extra time in the studio. Listen to that vocal performance. Can you imagine being so good that you can just toss something like that out there?

Children - Robert Miles

Apparently, there was a whole house and techno scene going on in the world. Maybe if I’d been a kid in Chicago or Detroit, I would have noticed. But none of that made its way to southern Louisiana until this instrumental, piano-driven electronica song improbably came down from space and landed on commercial radio.

The first time I heard this song was a revelation. I didn’t even know music like this existed. I loved everything about it, the rigid tempo, heartbeat kick, offset bass, heart-stirring melody. I couldn’t get enough.

Dark + Long (Dark Train) - Underworld

Right around this time, I met a guy at work who wore these weird pants with huge legs. What was that about? He told my brother and I he was a “raver” and there was this music called “techno” and you could hear it at these things called “raves”.

In one sitting, he gave us a rundown of artists that kept my ears full for several years. The band that I forged the strongest bond with is Underworld. I rushed out and got the “Pearl’s Girl” EP, then later “Dubnobasswithmyheadman” and “Second Toughest in the Infants”. Eventually, I had almost everything they ever recorded.

I met some of my closest college friends by bonding over Underworld. We once drove 900 miles from Baton Rouge to Chicago, non-stop, because that was the closest place to us that Underworld was playing during the “Beaucoup Fish” tour. Still one of the best shows I ever saw.

Picking a single Underworld song is hard. Picking this particular song is frustrating because for most, it’s associated with Trainspotting. But that’s not what I think when I hear this. To me, it’s driving home from raves at the State Palace Theatre in New Orleans as the sun comes up in our rearview mirror.

Halcyon + On + On - Orbital

A lot of techno at the 90s was, well, kind of dumb. Maybe a good beat to dance to, but it didn’t leave you thinking or feeling much of anything. Aphex Twin took care of the “thinking” part. Orbital took care of feeling. They showed that music made with computers could have as much heart as anything else.

I’m not the kind of person to fret about what song I want played at my funeral, but if I had to pick, this might be it.

Roads - Portishead

Oh, the tremolo on the Rhodes, mirroring Beth Gibbons’ vibrato. Those ghost hits on the snare. The strings. That bassline. Every ounce of this is flawless.

Like most people at the time, I got into trip-hop. Some of it doesn’t hold up, but some of it, like this, I seem to respond to more and more the older I get.

Gorecki - Lamb

Around this time, I was working at an Internet start-up during the first dotcom bubble. Amazon had just started selling CDs and they had this new “Customers who bought this also bought” thing. Today, we are inundated with machine learning that knows every single connection between ever human artifact every produced. It’s hard to remember what the world was before recommendation engines.

But there was a time where if you liked some song, you might be totally unable to find other stuff like it, even if that other stuff existed. If you were lucky, the guy at the music store knew stuff. That was literally it.

So when I typed in the one trip-hop band I knew into the search box at www.amazon.com, and then saw a list of other bands I might also like, and other bands linked to from them, and so on, it was like the gates to Paradise had opened up.

I ordered a stack of CDs, the largest music purchase of my life. They showed up a week later. I took them home, put Lamb’s debut album in, and put on my headphones. The first time this song came on, I was moved nearly to tears.

Granted, I was going through some girlfriend stuff at the time, so tears weren’t as far away as usual, but it’s still the most profoundly emotional listening experience of my life.

La Femme d'Argent - Air

One day, the roommate of the girl that caused the aforementioned troubles put in a CD in her car stereo. She said, “You like electronic stuff, you might like this.” I was totally flummoxed by what came on. It sounded like it had been recorded in the 70s. Was it even “electronic” music? She insisted it had come out recently. I’d never heard anything like it. It was, and still is, magnificent. An album that stands outside of time and genre.

Much much later, it occurred to me that I had dated the wrong roommate.

Kathy’s Song - Apoptygma Berzerk

I love this song first because it reminds me of discovering a whole new genre of music (synthpop and EBM), a circle of friends (“the Baton Rouge goth community” as my friend used to say) and a new bar (The Spanish Moon) all at the same time.

Going to that bar led to throwing parties with the same people and music, which led to me DJing at our house, which eventually led to me DJing at the Spanish Moon, which was a huge step for me to get past some of my crippling shyness.

I love this song second because my sister-in-law’s name is “Cathy”, and I played this at her wedding to my brother on the day that I got to officially call her family.

Remember (Mood II Swing Mix) - BT

There are a number of BT songs that could be on this list, but I picked this remix because it’s the track I’ve listened to the most over the years. “Flaming June” and “Poseidon” are up there too.

But, also, because every time I hear that little fuzz effect come in at :15, I can still perfectly picture the owner of The Spanish Moon sprinting across the dancefloor towards the DJ booth, panic in his eyes, because he thought I’d blown his sound system.

Autumn Tactics - Chicane

That couple year span when “progressive house” was the thing brought a lot of great artists to my attention. Of all of them, I probably sunk more time into Chicane’s first two albums than anything else. This isn’t really a house track – I don’t know how to categorize it, honestly – and maybe that’s why it’s held up so well for me. I’ve been listening to it for close to twenty years, and I still haven’t tired of it.

Poor Leno - Röyksopp

We’re getting closer to the modern era, witnessed by the fact that I stumbled onto this song from a video on the Internet. This song led to me buying turntables. I was going to clubs listening to house all the time, but most of the DJs were playing stripped down tribal or tech house that sounded soulless and empty to me. I didn’t necessarily need a full set of lyrics, but I at least wanted a chord progression.

I realized that if I ever wanted to hear this track or others like it on a dancefloor, I was going to have to make it happen myself. I filled up my online shopping cart at Turntable Lab. A few weeks later some very large, very heavy boxes showed up at work, and I was off learning how to beatmatch.

Tracey In My Room - EBTG vs Soul Vision

This song is a two-fer. Tracy Thorn from Everything but the Girl is one of my favorite vocalists. When Ben Watt took EBTG in a decidedly electronic direction with “Walking Wounded” and then “Temperamental”, I was right there with him.

But around this time was also when I started to get fully into house music, the more heartfelt the better. This record, a mash-up of a house song and the vocals from “Wrong” merges those two better than it has any right to.

Unspoken - Four Tet

In my twenties, my friends and I used to hit Park Ave. CDs every Tuesday when new music came out. This album was at a listening station on an end cap. When the first song came on, it was as if the lights in the building gradually dimmed, leaving nothing but sound.

Back of my Hand - Gemma Hayes

We went to this record store so often that the employees would hook us up with samplers and other merch. This song was on one. It’s not my usual genre and I can’t say I’ve listened to much else by her. But the production on this song pushes it out of generic singer-songwriter and into something really interesting to me.

I love the contrast between the folksy guitar – twelve-string? double-tracked? – and that that tinny mechanical drum loop. When the fully-EQed beat and organ drops, my heart goes a-flutter.

I had this song as my alarm clock for several years. It was a gentle way to wake up and the end result was the lyrics embedding themselves in my subconscious. Years later, my kids started asking me to sing them bedtime songs. This was one of the few whose words I remembered and whose melody fit within my not-so-wide vocal range.

Listening brings back peaceful mornings sipping coffee in Orlando, and gentle nights in Washington stroking my daughters’ hair as they fall asleep. It’s hard to imagine a better pair of bookends.

Such Great Heights - Postal Service

The drag about being into electronic music in Orlando back then was, well, the other people into it. Instead of the fun rave scene of New Orleans where kids were down to Earth and just wanted a good time, Orlando was a club scene – overdressed bros and over-made-up woman desperately trying to impress each other while trying not to look desperate.

Indie dance music and Orlando’s Independent Bar (called “Barbarella” at the time) saved me from that. The indie night there remains the absolute best dancefloor, and the most fun crowd I’ve ever experienced. Like clockwork, every Friday night ended with a packed, sweaty, grinning mass of euphoric people.

There are a number of songs I could pick to be the anthem for that time in my life, but this one also happens to mark a great point in a relationship (before the relationship went no-so-great).

Move Your Feet - Junior Senior

One of the few things I love as much as dance music is pixel art, so this song’s video pushes all of the buttons on the 747 control panel of my heart.

September - Earth, Wind, and Fire

I expected more soul music to make this list, but somehow it didn’t. It would have been a real shame to omit this gem. Thursdays were soul night at I-Bar and I can’t hear this song without picturing my friend Amy cutting up the dancefloor.

This song gets twice my love because I also used to spin the fantastic Phats & Small house remix.

Under the Milky Way - The Church

You know those songs that reside in a primordial part of your brain? I can’t recall ever not knowing this song. I’m wired towards the low end of the frequency spectrum. I play bass, and tend to focus on the bass and rhythm side of songs. That extends to vocals. I can’t do histrionic screechy singers. Nothing sets my soul at ease quite like a soft baritone.

This is another song I used to sing to my kids as they fell asleep.

Maybe Tomorrow - Stereophonics

The beginning of the video for this song is a dirty lie. This song wasn’t crafted by imperfect human hands. It was plucked, flawless, from some sonic vein deep within the Earth.

Stop - Black Rebel Motorcycle Club

Hanging out at Independent Bar opened one of the most fulfilling chapters of my life. It was there that I met my friend Mikey, which led to us starting a band. Playing music with others was a transcendent experiences – to hear four people produce one single harmonious sound. To give birth to something better than we could have made on our own. Being on stage is a bonus. Just to play is the thing.

My bandmates introduced me to a lot of great rock, including BRMC. I used to listen to this track on the way to shows and by the time I got to the gig, that bassline had me feeling like I could walk through a brick wall.

Baby in Two - Pernice Brothers

My drummer Shannon introduced me to Pernice Brothers. Our band days are over, we live about three thousand miles apart, but he will always be my drummer and I will always be his bassist. There’s a special kind of love that forms only between two halves of a rhythm section and this album is the soundtrack to mine.

Cupid - Sam Cooke

God, did they know how to record back then. Take one of the world’s greatest voices. Add beautiful, authentic reverb, and just enough tape saturation, and you get a sound so rich I can practically taste it.

But the technical merits are an aside. The real reason this is here is because it marks the time when I met my wife. This song is inseparable from her, from us.

Blue Ridge Mountains - Fleet Foxes

By this time, I was burned out from eight years in the game industry. Tired of the heat and sameness Florida–every month indistinguishable from the previous, the days an unending blur of sunlight.

My wife and I discovered the Pacific Northwest on a work trip. We played this album non-stop during our honeymoon on the Olympic Peninsula, and by the end we were ready to uproot and move. Back in Florida, during the months it took for me to find work in Seattle, this record was a constant reminder of the promise awaiting us out west.

I Will Follow You Into the Dark - Death Cab for Cutie

There is a Seattle city ordinance that you must like Ben Gibbard.

Shooting Stars - Bag Raiders

My love of disco and house naturally turned into a love of nu-disco. I don’t know what’s going on in Australia, but there’s a whole pile of bands out of Sydney and Melbourne that sound like they took everything I love about the 70s and 80s, mashed it all together, and somehow made it sound fresh again.

One of the joys of being a Dad is playing music with my kids. I get to choose the songs that will form their subconscious musical memory. They love this album, perhaps largely because I used to pick them up and dance with them in the kitchen every time we played it. I hope when they are much older and hear this, they still think of me.

Elevate - St. Lucia

Like I said, the 80s is new again. I won’t lie. Listening to music that apes a genre that I still remember the first time it came around makes me feel pretty old. That weird cringe I get when I see fashion come full circle and the young folk start wearing stuff I still have jammed in the back of my closet.

I try to ignore that twinge because, honestly, St. Lucia is fantastic. This isn’t some ironic winking aping of the past (looking at you The Darkness and Steel Panther). St. Lucia is a gushing love letter to everything great about beachy 80s pop. It’s mai tais, sand between your toes, coconut sunscreen. Endless summer and eternal youth.

Fade Into You - Mazzy Star

We’re nearing the end of the list, nearly through forty years of living and listening. It’s been a long drive down a winding road, and the sun will be up soon.

I’m on Fire - Bruce Springsteen

Forty feels like all my younger selves are lost, receded far into the past. Yet, at the same time, they still burn vibrantly alive inside, refusing to be forgotten.

Long Names Are Long

2016-06-16T00:00:00-07:00

One smart thing Google does is rigorous code reviews. Every change, before you can land it, gets reviewed in at least two way. First, someone on the team does a normal review to make sure the code does what it’s supposed to.

But, then, there’s a second layer of review called readability. It makes sure the code is, well, readable: Is it easy to understand and maintain? Does it follow the style and idioms of the language? Is it well-documented?

Dart usage inside Google is cranking up, so I’ve been doing a long ton of these kind of code reviews. As a language designer, it’s fascinating. I get a first-hand view into how people use Dart, which is really useful for evolving it. I have a clearer picture of which mistakes are common and which features are heavily used. I feel like an ethnographer journaling the lives of natives.

But, anyway, that’s not what this is about. Heck, it’s not even about Dart. What I want to talk about is something I see in a lot of code that drives me up the wall: identifiers that are too damn long.

Yes, names can be too short. Back when C only required external identifiers to be unique up to the first six characters; auto-complete hadn’t been invented; and every keypress had to be made uphill, in the snow, both ways; it was a problem. I’m glad we now live in a futuristic utopia where keyboard farts like p, idxcrpm, and x3 are rare.

But the pendulum has swung too far in the other direction. We shouldn’t be Hemingway, but we don’t need to be Tennessee Williams either. Very long names also hurt the clarity of the code where they are used. Giant identifiers dwarf the operations you’re performing on them, are hard to visually scan, and force extra line breaks which interrupt the flow of the code.

Long class names discourage users from declaring variables of that type, leading to massive, gnarly nested expressions instead of hoisting things out to locals. Long method names obscure their equally important argument lists. Long variables are annoying to use repeatedly, leading to sprawling method chains or cascades.

I’ve seen identifiers over 60 characters long. You could fit a haiku or a koan in there (and likely enlighten the reader more than the actual chosen name did). Fear not, I am here to help.

Choosing a Good Name

A name has two goals:

It needs to be clear: you need to know what the name refers to.
It needs to be precise: you need to know what it does not refer to.

After a name has accomplished those goals, any additional characters are dead weight. Here’s some guidelines I use when I names things in my code:

1. Omit words that are obvious given a variable’s or parameter’s type

If your language has a static type system, users usually know the type of a variable. Methods tend to be short, so even when looking at local variable whose type was inferred, or in a code review or some place where static analysis isn’t available, it rarely takes more than scanning a few lines to tell what type a variable has.

Given that, it’s redundant to put the type in the variable’s name. We have rightfully abandoned Hungarian notation. Let it go.

// Bad:
String nameString;
DockableModelessWindow dockableModelessWindow;

// Better:
String name;
DockableModelessWindow window;

In particular, for collections, it’s almost always better to just use a plural noun describing the contents instead of a singular noun describing the collection. If the reader cares more about what’s in the collection, the name should reflect that.

// Bad:
List<DateTime> holidayDateList;
Map<Employee, Role> employeeRoleHashMap;

// Better:
List<DateTime> holidays;
Map<Employee, Role> employeeRoles;

This also applies to method names. The method name doesn’t need to describe its parameters or their types—the parameter list does that for you.

// Bad:
mergeTableCells(List<TableCell> cells)
sortEventsUsingComparator(List<Event> events,
    Comparator<Event> comparator)

// Better:
merge(List<TableCell> cells)
sort(List<Event> events, Comparator<Event> comparator)

This tends to make callsites read better:

mergeTableCells(tableCells);
sortEventsUsingComparator(events, comparator);

Is it just me, or is there an echo echo in here here?

2. Omit words that don’t disambiguate the name

Some people tend to cram everything they know about something into its name. Remember, the name is an identifier: it points you to where it’s defined. It’s not an exhaustive catalog of everything the reader could want to know about the object. The definition does that. The name just gets them there.

When I see an identifier like recentlyUpdatedAnnualSalesBid, I ask:

Are there updated annual sales bids that aren’t recent?
Are there recent annual sales bids that were not updated?
Are there recently updated sales bids that aren’t annual?
Are there recently updated annual bids not related to sales?
Are there recently updated annual sales things that are not bids?

A “no” for any of these usually points to an extraneous word.

// Bad:
finalBattleMostDangerousBossMonster;
weaklingFirstEncounterMonster;

// Better:
boss;
firstMonster;

Of course, you can go too far. Shortening that first example to bid might be a little too vague. But, when in doubt, leave it out. You can always add qualifiers later if the name proves to cause a collision or be imprecise but it’s unlikely you’ll come back later to trim the fat.

3. Omit words that are known from the surrounding context

I can use “I” in this paragraph because you can see this post is by Bob Nystrom. My dumb face is right up there. I don’t need to keep saying “Bob Nystrom” everywhere here (despite Bob Nystrom’s temptation to aggrandize Bob Nystrom by doing so). Code works the same way. A method or field occurs in the context of a class. A variable occurs in the context of a method. Take that context for granted and don’t repeat it.

// Bad:
class AnnualHolidaySale {
  int _annualSaleRebate;
  void promoteHolidaySale() { ... }
}

// Better:
class AnnualHolidaySale {
  int _rebate;
  void promote() { ... }
}

In practice, this means that the more deeply nested a name is, the more surrounding context it has. That in turn means it usually has a shorter name. The effect is that identifiers with shorter scopes have shorter names.

4. Omit words that don’t mean much of anything

I used to see this a lot in the game industry. Some people succumb to the temptation to inflate their identifiers by adding Serious Business sounding words. I guess it makes their code feel more important and, by extension, makes them feel more important.

In many cases, the words carry no meaningful information. They’re just fluff or jargon. Usual suspects include: data, state, amount, value, manager, engine, object, entity, and instance.

A good name paints a picture in the mind of the reader. Calling something a “manager” doesn’t convey any image to the reader about what the thing does. Does it do performance evaluations? Give raises?

Ask yourself “Would this identifier mean the same thing if I removed the word?” If so, the word doesn’t carry its weight: vote if off the island.

Applying the Guidelines… to Waffles

To give you a feel for how these rules work in practice, here’s an example that breaks all of these rules. This contrived example is heart-breakingly close to real code I’ve seen in reviews:

class DeliciousBelgianWaffleObject {
  void garnishDeliciousBelgianWaffleWithStrawberryList(
      List<Strawberry> strawberryList) { ... }
}

We know from the type that it takes a list of strawberries (#1), so let’s cut that out:

class DeliciousBelgianWaffleObject {
    void garnishDeliciousBelgianWaffle(
        List<Strawberry> strawberries) { ... }
}

Unless our program has foul-tasting Belgian waffles, or waffles of other nationalities, we can drop those adjectives (#2):

class WaffleObject {
  void garnishWaffle(List<Strawberry> strawberries) { ... }
}

The method is inside a WaffleObject, so we know what it’s going to garnish (#3):

class WaffleObject {
  void garnish(List<Strawberry> strawberries) { ... }
}

Obviously it’s an object. Everything is an object. That’s kind of what “object-oriented” means (#4):

class Waffle {
  void garnish(List<Strawberry> strawberries) { ... }
}

There, much better.

I think these are pretty simple guidelines. You may think it’s pointless to worry about this stuff, but I believe that naming things is one of the most fundamental tasks we do when programming. Names are the structure we impose on the formless sea of bits that is computing.

The Hardest Program I've Ever Written

2015-09-08T00:00:00-07:00

The hardest program I’ve ever written, once you strip out the whitespace, is 3,835 lines long. That handful of code took me almost a year to write. Granted, that doesn’t take into account the code that didn’t make it. The commit history shows that I deleted 20,704 lines of code over that time. Every surviving line has about three fallen comrades.

If it took that much thrashing to get it right, you’d expect it to do something pretty deep right? Maybe a low-level hardware interface or some wicked graphics demo with tons of math and pumping early-90s-style techno? A likely-to-turn-evil machine learning AI Skynet thing?

Nope. It reads in a string and writes out a string. The only difference between the input and output strings is that it modifies some of the whitespace characters. I’m talking, of course, about an automated code formatter.

Introducing dartfmt

I work on the Dart programming language. Part of my job is helping make more Dart code, readable, idiomatic, and consistent, which is why I ended up writing our style guide. That was a good first step, but any style guide written in English is either so brief that it’s ambiguous, or so long that no one reads it.

Go’s “gofmt” tool showed a better solution: automatically format everything. Code is easier to read and contribute to because it’s already in the style you’re used to. Even if the output of the formatter isn’t great, it ends those interminable soul-crushing arguments on code reviews about formatting.

Of course, I still have to sell users on running the formatter in the first place. For that, having great output really does matter. Also, I’m pretty picky with the formatting in my own code, and I didn’t want to tell users to use a tool that I didn’t use myself.

Getting that kind of quality means applying pretty sophisticated formatting rules. That in turn makes performance difficult. I knew balancing quality and speed would be hard, but I didn’t realize just how deep the rabbit hole went.

I have finally emerged back into the sun, and I’m pleased with what I brought back. I like the output, and the performance is solid. On my laptop, it can blow through over two million lines of code in about 45 seconds, using a single core.

Why is formatting hard?

At this point, you’re probably thinking, “Wait. What’s so hard about formatting?” After you’ve parsed, can’t you just walk the AST and pretty-print it with some whitespace?

If every statement fit within the column limit of the page, yup. It’s a piece of cake. (I think that’s what gofmt does.) But our formatter also keeps your code within the line length limit. That means adding line breaks (or “splits” as the formatter calls them), and determining the best place to add those is famously hard.

Check out this guy:

experimentalBootstrap = document.querySelectorAll('link').any((link) =>
    link.attributes['rel'] == 'import' &&
        link.attributes['href'] == POLYMER_EXPERIMENTAL_HTML);

There are thirteen places where a line break is possible here according to our style rules. That’s 8,192 different combinations if we brute force them all 1. The search space we have to cover is exponentially large, and even ranking different solutions is a subtle problem. Is it better to split before the .any()? Why or why not?

What is up with the skulls?

I had two goals with this article: to explain how dartfmt works, and to show a realistic picture of how a real programmer solves a difficult problem with all of the messiness that entails. Alas, the first is more than long enough to try your patience, so I shunted all of the dead ends and failed attempts to footnotes. Click the skulls to laugh at my misfortune.

In Dart, we made things harder on ourselves. We have anonymous functions, lots of higher-order functions, and—until we added async and await—used futures for concurrency. That means lots of callbacks and lots of long method chains. Some Dart users really dig a functional style and appear to be playing a game where whoever crams the most work before a single semicolon wins.

Here’s real code from an amateur player:

_bindAssignablePropsOn.forEach((String eventName) => node
    .addEventListener(eventName, (_) => zone.run(() => bindAssignableProps
        .forEach((propAndExp) => propAndExp[1].assign(
            scope.context, jsNode[propAndExp[0]])))));

Yeah, that’s four nested functions. 1,048,576 ways to split that one. Here’s one of the best that I’ve found. This is what a pro player brings to the game:

return doughnutFryer
    .start()
    .then((_) => _frostingGlazer.start())
    .then((_) => Future.wait([
          _conveyorBelts.start(),
          sprinkleSprinkler.start(),
          sauceDripper.start()
        ]))
    .catchError(cannotGetConveyorBeltRunning)
    .then((_) => tellEveryoneDonutsAreJustAboutDone())
    .then((_) => Future.wait([
          croissantFactory.start(),
          _giantBakingOvens.start(),
          butterbutterer.start()
        ])
            .catchError(_handleBakingFailures)
            .timeout(scriptLoadingTimeout, onTimeout: _handleBakingFailures)
            .catchError(cannotGetConveyorBeltRunning))
    .catchError(cannotGetConveyorBeltRunning)
    .then((_) {
  _logger.info("Let's eat!");
});

(The funny names are because this was sanitized from internal code.) That’s a single statement, all 565 characters of it. There are about 549 billion ways we could line break it.

Ultimately, this is what the formatter does. It applies some fairly sophisticated ranking rules to find the best set of line breaks from an exponential solution space. Note that “best” is a property of the entire statement being formatted. A line break changes the indentation of the remainder of the statement, which in turn affects which other line breaks are needed. Sorry, Knuth. No dynamic programming this time 2.

I think the formatter does a good job, but how it does it is a mystery to users. People get spooked when robots surprise them, so I thought I would trace the inner workings of its metal mind. And maybe try to justify to myself why it took me a year to write a program whose behavior in many ways is indistinguishable from cat.

How the formatter sees your code

As you’d expect from a program that works on source code, the formatter is structured much like a compiler. It has a front end that parses your code and converts that to an intermediate representation 3. It does some optimization and clean up on that 4, and then the IR goes to a back end 5 that produces the final output. The main objects here are chunks, rules, and spans.

Chunks

A chunk is an atomic unit of formatting. It’s a contiguous region of characters that we know will not contain any line breaks. Given this code:

format /* comment */ this;

We break it into these chunks: format /* comment */ this;.

Chunks are similar to a token in a conventional compiler, but they tend to be, well, chunkier. Often, the text for several tokens ends up in the same chunk, like this and ; here. If a line break can never occur between two tokens, they end up in the same chunk 6.

Chunks are mostly linear. For example, given an expression like:

some(nested, function(ca + ll))

We chunk it to the flat list: some( nested, function( ca + ll)).

We could treat an entire source file like a single flat sequence of chunks, but it would take forever and a day to line break the whole thing 7. With things like long chains of asynchronous code, a single “statement” may be hundreds of lines of code containing several nested functions or collections that each contain their own piles of code.

We can’t treat those nested functions or collection literals entirely independently because the surrounding expression affects how they are indented. That in turn affects how long their lines are. Indent a function body two more spaces and now its statements have two fewer spaces before they hit the end of the line.

Instead, we treat nested block bodies as a separate little list of chunks to be formatted mostly on their own but subordinate to where they appear. The chunk that begins one of these literals, like the { preceding a function or map, contains a list of child block chunks for the contained block. In other words, chunks do form a tree, but one that only reflects block nesting, not expressions.

The end of a chunk marks the point where a split may occur in the final output, and the chunk has some data describing it 8. It keeps track of whether a blank line should be added between the chunks (like between two class definitions), how much the next line should be indented, and the expression nesting depth at that point in the code.

The most important bit of data about the split is the rule that controls it 9.

Rules

Each potential split in the program is owned by a rule. A single rule may own the splits of several chunks. For example, a series of binary operators of the same kind like a + b + c + d uses a single rule for the splits after each + operator.

A rule controls which of its splits break and which don’t. It determines this based on the state that the rule is in, which it calls its value. You can think of a rule like a dial and the value is what you’ve turned it to. Given a value, the rule will tell you which of its chunks get split.

The simplest rule is a “hard split” rule. It says that its chunk always splits, so it only has one value: 0. This is useful for things like line comments where you always need to split after it, even in the middle of an expression 10.

Then there is a “simple” split rule. It allows two values: 0 means none of its chunks split and 1 means they all do. Since most splits are independent of the others, this gets used for most of the splits in the program.

Beyond that, there are a handful of special-case rules. These are used in places where we want to more precisely control the configuration of a set of splits. For example, the positional argument list in a function list is controlled by a single rule. A function call like:

function(first, second, third)

Will have splits after function(, first,, second,, and third). They are all owned by a single rule that only allows the following configurations:

// 0: Don't split at all.
function(first, second, third)

// 1: Split before the first.
function(
    first, second, third)

// 2: Split before only the last argument.
function(first, second,
    third)

// 3: Split before only the middle argument.
function(first,
    second, third)

// 4: Split before all of them.
function(
    first,
    second,
    third)

Having a single rule for this instead of individual rules for each argument lets us prohibit things like:

function(
    first, second,
    third)

Constraints

Grouping a range of splits under a single rule helps us prevent split configurations we want to avoid like this, but it’s not enough. There are more complex constraints we want to enforce like: “if a split occurs inside a list element, the list should split too”. That avoids output like this:

[first, second +
    third, fourth]

Here, the list and the + expression have their own rules, but those rules need to interact. If the + takes value 1, the list rule needs to as well. To support this, rules can constrain each other. Any rule can limit the values another rule is allowed to take based on its own value. Typically, this is used to make a rule inside a nested expression force the rules surrounding itself to split when it does 11.

Finally, each rule has a cost. This is a numeric penalty that applies when any of that rule’s chunks are split. This helps us determine which sets of splits are better or worse than others 12.

Rule costs are only part of how overall fitness is calculated. Most of the cost calculation comes from spans.

Spans

A span marks a series of contiguous chunks that we want to avoid splitting. I picture it like a rubber band stretching around them. If a split happens in any of those chunks, the span is broken. When that happens, the solution is penalized based on the cost of the span.

Spans can nest arbitrarily deeply. In an expression like:

function(first(a, b), second(c, d))

There will be spans around a, b and c, d to try to keep those argument lists from splitting, but also another span around first(a, b), second(c, d) to keep the outer argument list from splitting.

If a split occurs between a, and b, the a, b span splits, but so does the first(a, b), second(c, d) one. However, if a split occurs after first(a, b), then the a, b span is still fine. In this way, spans teach the formatter to prefer splitting at a higher level of nesting when possible since it breaks fewer nested spans.

Parsing source to chunks

Converting your raw source code to this representation is fairly straightforward. The formatter uses the wonderful analyzer package to parse your code to an AST. This gives us a tree structure that represents every single byte of your program. Unlike many ASTs, it even includes comments.

Once we have that, the formatter does a top-down traversal of the tree. As it walks, it writes out chunks, rules, and spans for the various grammar productions. This is where the formatting “style” is determined.

There’s no rocket science here, but there are a lot of hairy corner cases. Comments can appear in weird places. We have to handle weird things like:

function(argument, // comment
    argument)

Here, we normally would have a split after the first argument owned by an argument list rule. But the line comment adheres to the , and has a hard split after it, so we need to make sure the argument list rule handles that.

Whitespace is only implicitly tracked by the AST so we have to reconstitute it in the few places where your original whitespace affects the output. Having a detailed test suite really helps here.

Once we’ve visited the entire tree, the AST has been converted to a tree of chunks and a bunch of spans wrapped around pieces of it.

Formatting chunks

We’ve got ourselves a big tree of chunks owned by a slew of rules. Earlier, I said a rule is like a knob. Now we get to dial them in.

Doing this naïvely is infeasible. Even a small source file contains hundreds of individual rules and the set of possible solutions is exponential in the number of rules.

The first thing we do is divide the chunk list into regions we know can’t interfere with each other. These are roughly “lines” of code. So with:

first(line);
second(line);

We know that how we split the first statement has no effect on the second one. So we run through the list of chunks and break them into shorter lists whenever we hit a hard split that isn’t nested inside an expression.

Each of these shorter chunk lists is fed to the line splitter. Its job is to pick the best set of values for all the rules used by the chunks in the line. In most cases, this is trivial: if the whole line fits on the page, every rule gets set to zero—no splits—and we’re done.

When a line doesn’t fit, the splitter has to figure out which combination of rule values produces the best result. That is:

The one with the fewest characters that go over the column limit.
The one with the lowest cost, based on which rules and spans were split.

Calculating the cost for a set of rule values is pretty easy, but there are still way too many permutations to brute force it. If we can’t brute force it, how do we do it?

How line splitting works

I’m a college dropout so my knowledge of algorithms was fairly, um, rudimentary. So before I interviewed at Google, I spent two days in a hotel room cramming as many of them—mostly graph traversal—in my head as I could. At the time, I thought graphs would never come up in the interviews…

Then I had multiple interview questions that reduced down to doing the right kind of traversal over a graph. At the time, I thought this stuff would never be relevant to my actual job…

Then I spent the past few years at Google discovering that damn near every program I have to write can be reduced down to some kind of graph search. I wrote a package manager where dependencies are a transitive closure and version constraint solving is graph based. My hobby roguelike uses graphs for pathfinding. Graphs out the wazoo. I can do BFS in my sleep now.

Naturally, after several other failed approaches, I found that line splitting can be handled like a graph search problem 13. Each node in the graph represents a solution—a set of values for each rule. Solutions can be partial: some rules may be left with their values unbound.

From a given partial solution (including the initial “no rules bound” one), there are edge to new partial solutions. Each binds one additional rule to a value. By starting from an empty solution and walking this graph, we eventually reach complete solutions where all of the rules have been bound to values.

Graph search is great if you know where your destination is and you’re trying to find the best path. But we don’t actually know that. We don’t know what the best complete solution is. (If we did, we’ve be done already!)

Given this, no textbook graph search algorithm is sufficient. We need to apply some domain knowledge—we need to take advantage of rules and conditions implicit in the specific problem we’re solving.

After a dozen dead ends, I found three (sort of four) that are enough to get it finding the right solution quickly:

Bailing early

We are trying to minimize two soft constraints at the same time:

We want to minimize the number of characters that overflow the line length limit. We can’t make this a hard constraint that there is no overflow because it’s possible for a long identifier or string literal to overflow in every solution. In that case, we still need to find the one that’s closest to fitting.
We want to find the lowest cost—the fewest split rules and broken spans.

The first constraint dominates the second: we’ll prefer a solution with any cost if it fits one more character in. In practice, there is almost always a solution that does fit, so it usually comes down to picking the lowest cost solution 14.

We don’t know a priori what the cost of the winning solution will be, but we do know one useful piece of information: forcing a rule to split always increases the cost.

If we treat any unbound rule as being implicitly unsplit 15, that means the starting solution with everything unbound always has the lowest cost (zero). We can then explore outward from there in order of increasing cost by adding one rule at a time.

This is a basic best-first search: we keep a running queue of all of the partial solutions we’ve haven’t explored yet, sorted from lowest cost to highest. Each iteration, we pop a solution off.

If the solution completely fits in the page width, then we know we’ve won the overflow constraint. Since we’re exploring in order of increasing cost, we also know it’s the lowest cost. So, ta-da!, we found the winner and can stop exploring. Otherwise, if it has any unbound rules, we enqueue new solutions, each of which binds one of those to a value.

We basically explore the entire solution space in order of increasing cost. As soon as we find a solution that fits in the page, we stop.

Avoiding dead ends

The above sounds pretty promising, but it turns out that there can be an imperial ton of “low-cost but overflowing” solutions. When you’re trying to format a really long line, there are plenty of ways it can not fit, and this algorithm will try basically all of them. After all, they’re low cost since they don’t have many splits.

We need to avoid wasting time tweaking rules that aren’t part of the problem. For example, say we’re looking at a partial solution like this:

// Blog-friendly 40-char line limit:    |
function(
    firstCall(a, b, c, d, e, f, g, h),
    secondCall("very long argument string here"));

There are a bunch of ways we can split the arguments to firstCall(), but we don’t need to. Its line already fits. The only line we need to worry about is the secondCall() one.

So, when we are expanding a partial solution, we only bind rules that have chunks on overflowing lines. If all of a rule’s chunks are on lines that already fit, we don’t mess with it. In fact, we don’t even worry about rules on any overflowing line but the first. Since tweaking the first line will affect the others, there’s no reason to worry about them yet 16.

This dramatically cuts down “branchiness” of the graph. Even though a partial solution may have dozens of unbound rules, usually only a couple are on long lines and only those get explored.

Pruning redundant branches

This gets us pretty far, but the splitter can still go off the deep end in some cases. The problem is that within large statements, you still run into cases where how you format part of the statement is mostly independent of later parts.

Take something like:

// Blog-friendly 40-char line limit:    |
new Compiler(
    assertions: options.has('checked-mode'),
    annotations: options.has('annotations'),
    primitives: options.has('primitives'),
    minify: options.has('minify'),
    preserve: options.has('preserve'),
    liveAnalysis: check(options.has('live'), options.has('analysis')),
    multi: options.has('multi'),
    sourceMap: options.has('source-map'));

Each of those named arguments can be split in a few different ways. And, since those are less nested—which means fewer split spans—than that nasty liveAnalysis: line, it will try every combination of all of them before it finally gets down to the business of splitting that check() call.

The best way to split the liveAnalysis: line is the best way to split it regardless of how we split assertions: or annotations:. In other words, there are big branches of the solution space that initially differ in irrelevant ways, but eventually reconvene to roughly the same solution. We traverse every single one of them.

What we need is a way to prune entire branches of the solution space. Given two partial solutions A and B, if we could say not just “A is better than B” but “every solution we can get to from A will be better than every solution we can get to from B” then we can discard B and the entire branch of solutions stemming from it.

It took some work, but I finally figured out that you can do this in many cases. Given two partial solutions, if one has a lower cost than the other and:

They have the same set of unbound rules (but their bound rules have different values, obviously).
None of their bound rules are on the same line as an unbound rule.
None of their bound rules place constraints on an unbound rule.

If all of those are true, then the one with a lower cost will always lead to solutions that also have a lower cost. Its entire branch wins. We can discard the other solution and everything that it leads to. Once I got this working, the formatter could line split damn near anything in record time 17.

An escape hatch

Alas, that “damn near” is significant. There are still a few cases where the formatter takes a long time. I’ve only ever seen this on machine generated code. Stuff like:

class ResolutionCopier {
  @override
  bool visitClassDeclaration(ClassDeclaration node) {
    ClassDeclaration toNode = this._toNode as ClassDeclaration;
    return javaBooleanAnd(
        javaBooleanAnd(
            javaBooleanAnd(
                javaBooleanAnd(javaBooleanAnd(javaBooleanAnd(
                        javaBooleanAnd(javaBooleanAnd(
                            javaBooleanAnd(javaBooleanAnd(javaBooleanAnd(
                                    _isEqualNodes(node.documentationComment,
                                        toNode.documentationComment),
                                    _isEqualNodeLists(
                                        node.metadata, toNode.metadata)),
                                _isEqualTokens(node.abstractKeyword,
                                    toNode.abstractKeyword)), _isEqualTokens(
                                node.classKeyword, toNode.classKeyword)),
                            _isEqualNodes(
                                node.name, toNode.name)), _isEqualNodes(
                            node.typeParameters, toNode.typeParameters)),
                        _isEqualNodes(
                            node.extendsClause, toNode.extendsClause)),
                    _isEqualNodes(
                        node.withClause, toNode.withClause)), _isEqualNodes(
                    node.implementsClause, toNode.implementsClause)),
                _isEqualTokens(node.leftBracket, toNode.leftBracket)),
            _isEqualNodeLists(
                node.members,
                toNode.members)),
        _isEqualTokens(
            node.rightBracket,
            toNode.rightBracket));
  }
}

Yeah, welcome to my waking nightmare. Unsurprisingly, code like this bogs down the formatter. I want dartfmt to be usable in things like presubmit scripts where it will have a ton of weird code thrown at it and it must complete in a reliable amount of time.

So there is one final escape hatch. If the line splitter tries, like, 5,000 solutions and still hasn’t found a winner yet, it just picks the best it found so far and bails.

In practice, I only see it hit this case on generated code. Thank God.

Finally, output

Once the line splitter has picked values for all of the rules, the rest is easy. The formatter walks the tree of chunks, printing their text. When a rule forces a chunk to split, it outputs a newline (or two), updates the indentation appropriately and keeps trucking.

The end result is a string of (I hope!) beautifully formatted Dart code. So much work just to add or remove a few spaces!

Footnotes

1 Yes, I really did brute force all of the combinations at first. It let me focus on getting the output correct before I worried about performance. Speed was fine for most statements. The other few wouldn’t finish until after the heat death of the universe.

2 For most of the time, the formatter did use dynamic programming and memoization. I felt like a wizard when I first figured out how to do it. It worked fairly well, but was a nightmare to debug.

It was highly recursive, and ensuring that the keys to the memoization table were precise enough to not cause bugs but not so precise that the cache lookups always fail was a very delicate balancing act. Over time, the amount of data needed to uniquely identify the state of a subproblem grew, including things like the entire expression nesting stack at a point in the line, and the memoization table performed worse and worse.

3 The IR evolved constantly. Spans and rules were later additions. Even the way chunks tracked indentation changed frequently. Indentation used to be stored in levels, where each level was two spaces. Then directly in spaces. Expression nesting went through a number of representations.

In all of this, the IR’s job is to balance being easy for the front-end to produce while being efficient for the back end to consume. The back end really drives this. The IR is structured to be the right data structure for the algorithm the back end wants to use.

4 Comments were the one of the biggest challenges. The formatter initially assumed there would be no newlines in some places. Who would expect a newline, say, between the keywords in abstract class? Alas, there’s nothing preventing a user from doing:

abstract // Oh, crap. A line comment.
class Foo {}

So I had to do a ton of work to make it resilient in the face of comments and newlines appearing in all sorts of weird places. There’s no single clean solution for this, just lots of edge cases and special handling.

5 The back end is where all of the performance challenges come from, and it went through two almost complete rewrites before it ended up where it is today.

6 I started from a simpler formatter written by a teammate that treated text, whitespace, and splits all as separate chunks. I unified those so that each chunk included non-whitespace text, line split information, and whitespace information if it didn’t split. That simplified a lot.

7 When I added support for better indentation of nested functions, that broke the code that split source into separately splittable regions. For a while, a single top-level statement would be split as a single unit, even if it contained nested functions with hundreds of lines of code. It was… not fast.

8 Ideally, the split information in a chunk would describe the split before the chunk’s text. This would avoid the pointless split information on the last chunk, and also solve annoying special-case handling of the indentation before the very first chunk.

I’ve tried to correct this mistake a number of times, but it causes a near-infinite number of off-by-one bugs and I just haven’t had the time to push it all the way through and fix everything.

9 Rules are a relatively recent addition. Originally each chunk’s split was handled independently. You could specify some relations between them like “if this chunk splits then this other one has to as well”, but you could not express things like “only one of these three chunks may split”

Eventually, I realized the latter is what I really needed to get argument lists formatting well, so I conceived of rules as a separate concept and rewrote the front and line splitter to work using those.

10 At first, I thought hard splits weren’t needed. Any place a mandatory newline appears (like between two statements) is a place where you could just break the list of chunks in two and line split each half independently. From the line splitter’s perspective, there would be no hard splits.

Which would work… except for line comments:

some(expression,
   // with a line comment
   rightInTheMiddleOfIt);

This has to be split as a single unit to get the expression nesting and indentation correct, but it also contains a mandatory newline after the line comment.

11 There used to be a separate class for a “multisplit” to directly handle forcing outer expressions to split when inner ones did. Once rules came along, they also needed to express constraints between them, and eventually those constraints were expressive enough to be able to handle the multisplit behavior directly and multisplits were removed.

12 I spent a lot of time tuning costs for different grammar productions to control how tightly bound different expressions were. The goal was to allow splits at the places where the reader thought code was “loosest”, so stuff like higher precedence expressions would have higher costs.

Tuning these costs was a nightmare. It was like a hanging mobile where tweaking one cost would unbalance all of the others. On more than one occasion, I found myself considering making them floating point instead of integers, a sure sign of madness.

It turns out spans are what you really want in order to express looseness. Nested infix operators then fall out naturally because you have more spans around the deeper nested operands. The parse tree gives it to you for free.

These days, almost every chunk and span has a cost of 1, and it’s the quantity of nested spans and contained chunks that determine where it splits.

13 I had known that clang-format worked this way for a long time, but I could never wrap my head around how to apply it to dartfmt’s richer chunk/rule/span system.

I took a lot of walks along the bike trail next to work trying to think through a way to get graph search working when the two numbers being optimized (overflow characters and cost) are in direct opposition, and we don’t even know what the goal state looks like. It took a long time before it clicked. Even then, it didn’t work at all until I figured out the right heuristics to use to optimize it.

14 For a long time, overflow and cost were treated as a single fitness function. Every overflow character just added a very high value to the cost to make the splitter strongly want to avoid them.

Splitting overflow out as a separate metric turned out to be key to getting the graph search to work because it let us order the solutions by cost independently of overflow characters.

15 I went back and forth on how an unbound rule should implicitly behave. Treating it as implicitly split gives you solutions with fewer overflow characters sooner. Treating it as unsplit gives you lower costs.

16 Oh, God. I tried a million different ways to reduce the branchiness before I hit on only looking at rules in the first long line. I’m still amazed that it works.

I could also talk about how controlling branchiness lets us avoid reaching the same state from multiple different paths. After all, it’s a graph, but everything I’ve described talks about it like it’s a tree. By carefully controlling how we extend partial solutions, we ensure we only take a single path to any given complete solution.

Before I got that working, I had to keep a “visited” set to make sure we didn’t explore the same regions twice, but just maintaining that set was a big performance sink.

17 Discarding overlapping branches is the last macro-optimization I did and its behavior is very subtle. Correctly detecting when two partial solutions overlap took a lot of iteration. Every time I thought I had it, one random weird test would fail where it accidentally collapsed two branches that would eventually diverge.

That bullet list was paid for in blood, sweat, and tears. I honestly don’t think I could have figured them out at all until late in the project when I had a comprehensive test suite.

What the Hero Sees: Field-of-View for Roguelikes

2015-09-07T00:00:00-07:00

This is a record for me. I’ve been procrastinating this post for eight years. I moved four times, got married, had two kids, and ported my roguelike to a language that didn’t exist when I first wrote the code this post is about. You can thank Simon Andersson for prodding me to finally write it down.

Every good game, or genre of games, has a pyramid of rewards. From simple pleasures that mete out a droplet of endorphins when you click the next button up to the deep, abiding feeling of accomplishment you get from slaying the final boss on nightmare mode.

The roguelike genre standardizes many of these, and one of my favorites is the joy of exploring the dungeon. You start out on a black screen, only one tiny room visible. As you walk around, the map incrementally fills in—a perfect graphical representation of your own knowledge and mastery increasing.

Click and drag the hero to explore. Click walls and floors to alter the dungeon.

There wouldn’t be much to explore if your hero could see through walls. The second they entered the dungeon, the entire map would be filled in, all of the crypt’s hidden secrets laid bare to warrior and player alike. To prevent that, we need to simulate something that seems trivial: walls blocking the hero’s view.

In the roguelike scene, this is referred to as field of view, and there are a number of ways to do it. Many of the posts linked there talk about “light” and “shadow” as well, but they calculate the same thing as visilibity. In both cases, we’re trying to find the set of tiles that can be reached by rays emanating from some point source. I’ll use both terms interchangeably.

Brute force line-of-sight?

The simplest solution is to repurpose your line-of-sight code. You already need code to determine if there is an open line from one point to another on the map. You use that to tell if things like arrows and fireballs reach their target or bounce harmlessly off the dungeon wall.

This is invariably done using Bresenham’s line algorithm. It’s one of the true classics of graphics programming—an elegant, simple algorithm from 1962 that’s still useful today. (This kind of living connection to CS’s past is one of the things I love about hacking on a roguelike. How often do you have a good reason to recode a procedure originally devised for a 1950s-era drum plotter?)

As you’d imagine for an algorithm designed to run on a machine that took punch cards, it’s very efficient… for tracing a line between two points. But field of view is different: we need to scan the entire dungeon—or at least the part that fits on the player’s screen—and calculate the visibility of every tile.

You actually can run Bresenham a few thousand times whenever the hero moves on a modern machine, but doing that feels, well, like cheating somehow. Can we come up with something more efficient?

The answer is, of course, “yes”. (It would be a short post if it wasn’t.) And, in fact, a lot of others have already done so. But, one lazy sunny Saturday morning in 2006, I didn’t see any I liked and wanted to come up with one that made more sense to me.

Pieces of eight

If you have the mind of a programmer, the first thing you do when presented with a problem is to break it into multiple (hopefully) smaller problems. Our goal is to calculate the entire field of view surrounding the hero, but we can slice that 360° problem into 45° pie pieces. One looks like this:

Click to paint a wedge.

This wedge is called an octant, and it’s common in 2D algorithms. We can paint every tile in that triangle like so:

for (var row = 1; row < maxDistance; row++) {
  for (var col = 0; col <= row; col++) {
    var x = hero.x + col;
    var y = hero.y - row;

    paint(x, y);
  }
}

If we take that wedge and apply some transformations, we can cover the entire field around the hero. The above code is most of the way there. If you squint, you can see there are two coordinate systems. The row and col variables are in the octant’s coordinate space. Meanwhile, x and y are in real tile space—what you see on screen.

The first two lines inside the loops map octant space to tile space. Using just + and - and row and col, there are eight ways to calculate x and y. Each represents a reflection or 90° rotation of the original octant. If we enumerate them all, we get:

Vec transformOctant(int row, int col, int octant) {
  switch (octant) {
    case 0: return new Vec( col, -row);
    case 1: return new Vec( row, -col);
    case 2: return new Vec( row,  col);
    case 3: return new Vec( col,  row);
    case 4: return new Vec(-col,  row);
    case 5: return new Vec(-row,  col);
    case 6: return new Vec(-row, -col);
    case 7: return new Vec(-col, -row);
  }
}

Each case represents a different octant, starting at the top and going clockwise around the circle. Painting them all covers the whole view (with a bit of innocuous overlap):

Click to paint the whole view.

From here on out, we only have deal with a single triangle’s worth of row and col, and we can cover the entire field of view just by running the same code eight times, once for each octant.

A Line of Shadows

Another way to approach a problem is to negate it, and that’s what this algorithm does. Instead of calculating which tiles are visible, it figures out which are hidden, which put it in a family of algorithms that do “shadow casting”. Before I explain it, try it out yourself:

Drag the slider up to advance the shadow line. Click anywhere else to change the dungeon.

We start at the hero and work upwards one row at a time. As we sweep through the octant, we incrementally update a data structure called the shadow line. It’s the white line you see next to the slider. It tracks which parts of the row are in the shade of opaque tiles on previous rows and which aren’t.

The line is a series of segments, each representing one obscured region of the line. We can define this like so:

class ShadowLine {
  final List<Shadow> _shadows = [];
}

class Shadow {
  num start;
  num end;

  Shadow(this.start, this.end);
}

The interesting question is, “What is the range of start and end?” The naïve answer is to use tile coordinates. If we’re five rows in and the shadow line is five tiles long, the segment coordinates would range from 0 to 5.

The problem is that since light expands outwards from a point, the shadows stretch out as they get farther away. We don’t want to have to recalculate the segment positions each time we advance a row and the rays spread out.

Instead, we store their slopes. Regardless of what row we’re on, they always range from 0 (the short edge of the octant) to 1 (the diagonal edge). They are distance-independent. This is what the black line in the demo above shows. As you click to add and remove wall, you can see new shadows appear, but they don’t move or grow as you sweep the row up and down.

Projecting a tile

The tricky part is calculating those slopes given some tile in the octant. There are a couple of corner cases to consider. Literally. A tile is a square, and the shadow it projects goes from one corner of the square to another.

Given our canonical octant, we know the tile will be above and to the right of the hero. That means the projected shadow’s extent will always be from the top-left corner of a tile to the bottom-right corner. The other two corners lie in the middle of the shadow. (This isn’t strictly true if the tile is straight up from the hero, but we can safely ignore that.)

What we need, then, is to calculate the slopes of those two corners of a tile. The math is a kind of fussy, but it’s:

/// Creates a [Shadow] that corresponds to the projected
/// silhouette of the tile at [row], [col].
Shadow projectTile(int row, int col) {
  var topLeft = col / (row + 2);
  var bottomRight = (col + 1) / (row + 1);
  return new Shadow(topLeft, bottomRight);
}

This function has two uses. The obvious one is that we call this for each opaque tile and add its result to the shadow line. But the projection comes into play before that too.

You can think of the result of this function as the shadow that the tile casts past itself, but it also describes the projection from the hero to this tile. In other words, it describes which angles need to be unblocked for this tile to be visible.

When we scan a row, we call projectTile() on every tile—transparent or opaque— and compare it to the existing shadow line. If the tile’s projection is covered by the shadow line, we know it can’t be seen. If it isn’t, it can.

An interesting edge case is tiles whose projection is partially covered by the shadow line. Different games take different approaches here. Mine is considered permissive: if you can see any part of a tile, it’s visible. A tile’s projection has to be totally covered by the shadow to be hidden. If you want something less permissive, this algorithm is easy to tweak.

Let’s code! First, we’ll add a method to see if one shadow totally covers another:

class Shadow {
  /// Returns `true` if [other] is completely covered by this shadow.
  bool contains(Shadow other) {
    return start <= other.start && end >= other.end;
  }

  ...
}

Then we use that to see if any shadow in the line covers the tile:

class ShadowLine {
  bool isInShadow(Shadow projection) {
    for (var shadow in _shadows) {
      if (shadow.contains(projection)) return true;
    }

    return false;
  }

  ...
}

Using that, we can determine the visibility of every tile in a row. Given a ShadowLine in line, and a set of tiles in tiles, it’s:

for (var col = 0; col <= row; col++) {
  var projection = _projectTile(row, col);
  var pos = start + transformOctant(row, col, octant);
  tiles[pos].isVisible = !line.isInShadow(projection);
}

Inky pools of shadows

We can calculate which tiles are obscured by the shadow line, but that isn’t very useful since our shadow line is always empty right now. Let’s fix that.

As we trace the row, each time we hit an opaque tile, we add it to the shadow line. If the shadow line was a simple list of these little shadow segments, the list would get longer and longer. In a dense dungeon, the hero may be near hundreds of solid tiles. Having to walk through an increasingly long list of shadow segments to see if a tile is obscured would get slower and slower.

Fortunately, I have a simple fix that makes the algorithm get faster as tiles occlude more of the view. This is the part where I think my algorithm is pretty cool, and it’s why I’m excited to share it with you.

Often, when a new segment is added to the shadow line, it overlaps other shadows. When that happens, we merge it with the existing shadows. The end result is that the shadow line will have a single Shadow object for each contiguous range of obscured area.

This does mean adding a new shadow to the line is more complex. There are a handful of cases:

The shadow is contained within an existing one. That means the new shadow doesn’t cover any new territory, so we can discard it.
The shadow doesn’t overlap any other ones. In this case, we insert it in sorted order between the segments that come before and after it.
The shadow overlaps another shadow on its starting edge. We take the previous shadow and grow it to encompass the new shadow’s endpoint and discard the new one.
The shadow overlaps another shadow on its ending edge. Do the same thing, but in reverse: grow the following shadow to cover the new one.
The shadow overlaps shadows on both ends. This is the fun one. We take the previous shadow and extend it to cover the next shadow’s endpoint. Then we discard both the new shadow and that next one.

The first case doesn’t change the list of shadows at all. In the second case, the list of shadows gets longer. In the next two, adding a new shadow doesn’t grow the list, it just shifts an endpoint. The last case is the fun one: there, the list gets shorter.

(Pop quiz! Why don’t we have to worry about cases where a shadow overlaps more than two existing ones?)

Here’s the entire method to add a shadow to the line:

class ShadowLine {
  void add(Shadow shadow) {
    // Figure out where to slot the new shadow in the list.
    var index = 0;
    for (; index < _shadows.length; index++) {
      // Stop when we hit the insertion point.
      if (_shadows[index].start >= shadow.start) break;
    }

    // The new shadow is going here. See if it overlaps the
    // previous or next.
    var overlappingPrevious;
    if (index > 0 && _shadows[index - 1].end > shadow.start) {
      overlappingPrevious = _shadows[index - 1];
    }

    var overlappingNext;
    if (index < _shadows.length &&
        _shadows[index].start < shadow.end) {
      overlappingNext = _shadows[index];
    }

    // Insert and unify with overlapping shadows.
    if (overlappingNext != null) {
      if (overlappingPrevious != null) {
        // Overlaps both, so unify one and delete the other.
        overlappingPrevious.end = overlappingNext.end;
        _shadows.removeAt(index);
      } else {
        // Overlaps the next one, so unify it with that.
        overlappingNext.start = shadow.start;
      }
    } else {
      if (overlappingPrevious != null) {
        // Overlaps the previous one, so unify it with that.
        overlappingPrevious.end = shadow.end;
      } else {
        // Does not overlap anything, so insert.
        _shadows.insert(index, shadow);
      }
    }
  }

  ...
}

OK, so that’s kind of hairy, but it’s not deep magic, just a bunch of different cases to handle. Clever readers are probably wondering why we don’t do a binary search to find the insertion point. The list is sorted after all. If you want to be super smart, go for it. In practice, I don’t think it makes much of a difference. The maximum size of the list is small enough that a linear search may actually be faster.

(Pop quiz two! What is the maximum size of the list? Show your work.)

Very clever readers may have noticed we don’t check for the first case, a completely contained shadow here. That’s because we’ve already done that check. Earlier, when we detect if this tile is visible, that also tells us if it’s shadow is contained. If it is, we don’t bother calling add().

There’s another simple optimization we can do. If we get to the point where the shadow line is a single segment from 0 to 1—in other words, the whole line is in shadow—then we can skip all of the projection calculation, updating, etc. Every tile will be hidden after that. Here’s how we detect that:

class ShadowLine {
  bool get isFullShadow {
    return _shadows.length == 1 &&
        _shadows[0].start == 0 &&
        _shadows[0].end == 1;
  }

  ...
}

Putting it all together

OK, so we have code to:

Walk over every octant.
Walk every tile in an octant.
Update the tile’s visibility.
Update the shadow line if the tile is opaque.

Let’s stitch the last few pieces together along with a dash of bounds checking. Building on top of what we have above, here’s the top-level code to update the visibility of the whole dungeon:

void refreshVisibility(Vec hero) {
  for (var octant = 0; octant < 8; octant++) {
    refreshOctant(hero, octant);
  }
}

void refreshOctant(Vec hero, int octant) {
  var line = new ShadowLine();
  var fullShadow = false;

  for (var row = 1;; row++) {
    // Stop once we go out of bounds.
    var pos = hero + transformOctant(row, 0, octant);
    if (!tiles.bounds.contains(pos)) break;

    for (var col = 0; col <= row; col++) {
      var pos = hero + transformOctant(row, col, octant);

      // If we've traversed out of bounds, bail on this row.
      if (!tiles.bounds.contains(pos)) break;

      if (fullShadow) {
        tiles[pos].isVisible = false;
      } else {
        var projection = projectTile(row, col);

        // Set the visibility of this tile.
        var visible = !line.isInShadow(projection);
        tiles[pos].isVisible = visible;

        // Add any opaque tiles to the shadow map.
        if (visible && tiles[pos].isWall) {
          line.add(projection);
          fullShadow = line.isFullShadow;
        }
      }
    }
  }
}

And there we have it. It runs very fast in wide open areas since there will be few shadow segments and the list is short. Likewise, it runs fast in closed areas since the shadow list will also be short—it will contain a small number of long segments. It performs the worst in “spotty” areas with lots of small trees or pillars, but even there, the performance is pretty solid.

I’d love to say I implemented a bunch of other algorithms and this one came out the winner, but honestly I was too lazy to that. I will say that this has never showed up as a hot spot when I’ve profiled the game. That’s good enough for me, and I hope this will be helpful for you too.

If you want to see all of the code for these demos, it’s here. Or, in the context of my game here.

What Color is Your Function?

2015-02-01T00:00:00-08:00

I don’t know about you, but nothing gets me going in the morning quite like a good old fashioned programming language rant. It stirs the blood to see someone skewer one of those “blub” languages the plebians use, muddling through their day with it between furtive visits to StackOverflow.

(Meanwhile, you and I, only use the most enlightened of languages. Chisel-sharp tools designed for the manicured hands of expert craftspersons such as ourselves.)

Of course, as the author of said screed, I run a risk. The language I mock could be one you like! Without realizing it, I could have let the rabble into my blog, pitchforks and torches at the ready, and my fool-hardy pamphlet could draw their ire!

To protect myself from the heat of those flames, and to avoid offending your possibly delicate sensibilities, instead, I’ll rant about a language I just made up. A strawman whose sole purpose is to be set aflame.

I know, this seems pointless right? Trust me, by the end, we’ll see whose face (or faces!) have been painted on his straw noggin.

A new language

Learning an entire new (crappy) language just for a blog post is a tall order, so let’s say it’s mostly similar to one you and I already know. We’ll say it has syntax sorta like JS. Curly braces and semicolons. if, while, etc. The lingua franca of the programming grotto.

I’m picking JS not because that’s what this post is about. It’s just that it’s the language you, statistical representation of the average reader, are most likely to be able grok. Voilà:

function thisIsAFunction() {
  return "It's awesome";
}

Because our strawman is a modern (shitty) language, we also have first-class functions. So you can make something like like:

// Return a list containing all of the elements in collection
// that match predicate.
function filter(collection, predicate) {
  var result = [];
  for (var i = 0; i < collection.length; i++) {
    if (predicate(collection[i])) result.push(collection[i]);
  }
  return result;
}

This is one of those higher-order functions, and, like the name implies, they are classy as all get out and super useful. You’re probably used to them for mucking around with collections, but once you internalize the concept, you start using them damn near everywhere.

Maybe in your testing framework:

describe("An apple", function() {
  it("ain't no orange", function() {
    expect("Apple").not.toBe("Orange");
  });
});

Or when you need to parse some data:

tokens.match(Token.LEFT_BRACKET, function(token) {
  // Parse a list literal...
  tokens.consume(Token.RIGHT_BRACKET);
});

So you go to town and write all sorts of awesome reusable libraries and applications passing around functions, calling functions, returning functions. Functapalooza.

What color is your function?

Except wait. Here’s where our language gets screwy. It has this one peculiar feature:

1. Every function has a color.

Each function—anonymous callback or regular named one—is either red or blue. Since my blog’s code highlighter can’t handle actual color, we’ll say the syntax is like:

blue•function doSomethingAzure() {
  // This is a blue function...
}

red•function doSomethingCarnelian() {
  // This is a red function...
}

There are no colorless functions in the language. Want to make a function? Gotta pick a color. Them’s the rules. And, actually, there are a couple more rules you have to follow too:

2. The way you call a function depends on its color.

Imagine a “blue call” syntax and a “red call” syntax. Something like:

doSomethingAzure(...)•blue;
doSomethingCarnelian()•red;

When calling a function, you need to use the call that corresponds to its color. If you get it wrong—call a red function with •blue after the parentheses or vice versa—it does something bad. Dredge up some long-forgotten nightmare from your childhood like a clown with snakes for arms hiding under your bed. That jumps out of your monitor and sucks out your vitreous humour.

Annoying rule, right? Oh, and one more:

3. You can only call a red function from within another red function.

You can call a blue function from with a red one. This is kosher:

red•function doSomethingCarnelian() {
  doSomethingAzure()•blue;
}

But you can’t go the other way. If you try to do this:

blue•function doSomethingAzure() {
  doSomethingCarnelian()•red;
}

Well, you’re gonna get a visit from old Spidermouth the Night Clown.

This makes writing higher-order functions like our filter() example trickier. We have to pick a color for it and that affects the colors of the functions we’re allowed to pass to it. The obvious solution is to make filter() red. That way, it can take either red or blue functions and call them. But then we run into the next itchy spot in the hairshirt that is this language:

4. Red functions are more painful to call.

For now, I won’t precisely define “painful”, but just imagine that the programmer has to jump through some kind of annoying hoops every time they call a red function. Maybe it’s really verbose, or maybe you can’t do it inside certain kinds of statements. Maybe you can only call them on line numbers that are prime.

What matters is that, if you decide to make a function red, everyone using your API will want to spit in your coffee and/or deposit some even less savory fluids in it.

The obvious solution then is to never use red functions. Just make everything blue and you’re back to the sane world where all functions have the same color, which is equivalent to them all having no color, which is equivalent to our language not being entirely stupid.

Alas, the sadistic language designers—and we all know all programming language designers are sadists, don’t we?—jabbed one final thorn in our side:

5. Some core library functions are red.

There are some functions built in to the platform, functions that we need to use, that we are unable to write ourselves, that only come in red. At this point, a reasonable person might think the language hates us.

It’s functional programming’s fault!

You might be thinking that the problem here is we’re trying to use higher-order functions. If we just stop flouncing around in all of that functional frippery and write normal blue collar first-order functions like God intended, we’d spare ourselves all the heartache.

If we only call blue functions, make our function blue. Otherwise, make it red. As long as we never make functions that accept functions, we don’t have to worry about trying to be “polymorphic over function color” (polychromatic?) or any nonsense like that.

But, alas, higher order functions are just one example. This problem is pervasive any time we want to break our program down into separate functions that get reused.

For example, let’s say we have a nice little blob of code that, I don’t know, implements Dijkstra’s algorithm over a graph representing how much your social network are crushing on each other. (I spent way too long trying to decide what such a result would even represent. Transitive undesirability?)

Later, you end up needing to use this same blob of code somewhere else. You do the natural thing and hoist it out into a separate function. You call it from the old place and your new code that uses it. But what color should it be? Obviously, you’ll make it blue if you can, but what if it uses one of those nasty red-only core library functions?

What if the new place you want to call it is blue? You’ll have to turn it red. Then you’ll have to turn the function that calls it red. Ugh. No matter what, you’ll have to think about color constantly. It will be the sand in your swimsuit on the beach vacation of development.

A colorful allegory

Of course, I’m not really talking about color here, am I? It’s an allegory, a literary trick. The Sneetches isn’t about stars on bellies, it’s about race. By now, you may have an inkling of what color actually represents. If not, here’s the big reveal:

Red functions are asynchronous ones.

If you’re programming in JavaScript on Node.js, everytime you define a function that “returns” a value by invoking a callback, you just made a red function. Look back at that list of rules and see how my metaphor stacks up:

Synchronous functions return values, async ones do not and instead invoke callbacks.
Synchronous functions give their result as a return value, async functions give it by invoking a callback you pass to it.
You can’t call an async function from a synchronous one because you won’t be able to determine the result until the async one completes later.
Async functions don’t compose in expressions because of the callbacks, have different error-handling, and can’t be used with try/catch or inside a lot of other control flow statements.
Node’s whole shtick is that the core libs are all asynchronous. (Though they did dial that back and start adding ___Sync() versions of a lot of things.)

When people talk about “callback hell” they’re talking about how annoying it is to have red functions in their language. When they create 4089 libraries for doing asynchronous programming, they’re trying to cope at the library level with a problem that the language foisted onto them.

I promise the future is better

People in the Node community have realized that callbacks are a pain for a long time, and have looked around for solutions. One technique that gets a bunch of people excited is promises, which you may also know by their rapper name “futures”.

These are sort of a jacked up wrapper around a callback and an error handler. If you think of passing a callback and errorback to a function as a concept, a promise is basically a reification of that idea. It’s a first-class object that represents an asynchronous operation.

I just jammed a bunch of fancy PL language in that paragraph so it probably sounds like a sweet deal, but it’s basically snake oil. Promises do make async code a little easier to write. They compose a bit better, so rule #4 isn’t quite so onerous.

But, honestly, it’s like the difference between being punched in the gut versus punched in the privates. Less painful, yes, but I don’t think anyone should really get thrilled about the value proposition.

You still can’t use them with exception handling or other control flow statements. You still can’t call a function that returns a future from synchronous code. (Well, you can, but if you do, the person who later maintains your code will invent a time machine, travel back in time to the moment that you did this and stab you in the face with a #2 pencil.)

You’ve still divided your entire world into asynchronous and synchronous halves and all of the misery that entails. So, even if your language features promises or futures, its face looks an awful lot like the one on my strawman.

(Yes, that means even Dart, the language I work on. That’s why I’m so excited some of the team are experimenting with other concurrency models.)

I’m awaiting a solution

C# programmers are probably feeling pretty smug right now (a condition they’ve increasingly fallen prey to as Hejlsberg and company have piled sweet feature after sweet feature into the language). In C#, you can use the await keyword to invoke an asynchronous function.

This lets you make asynchronous calls just as easily as you can synchronous ones, with the tiny addition of a cute little keyword. You can nest await calls in expressions, use them in exception handling code, stuff them inside control flow. Go nuts. Make it rain await calls like a they’re dollars in the advance you got for your new rap album.

Async-await is nice, which is why we’re adding it to Dart. It makes it a lot easier to write asynchronous code. You know a “but” is coming. It is. But… you still have divided the world in two. Those async functions are easier to write, but they’re still async functions.

You’ve still got two colors. Async-await solves annoying rule #4: they make red functions not much worse to call than blue ones. But all of the other rules are still there:

Synchronous functions return values, async ones return Task<T> (or Future<T> in Dart) wrappers around the value.
Sync functions are just called, async ones need an await.
If you call an async function you’ve got this wrapper object when you actually want the T. You can’t unwrap it unless you make your function async and await it. (But see below.)
Aside from a liberal garnish of await, we did at least fix this.
C#‘s core library is actually older than async so I guess they never had this problem.

It is better. I will take async-await over bare callbacks or futures any day of the week. But we’re lying to ourselves if we think all of our troubles are gone. As soon as you start trying to write higher-order functions, or reuse code, you’re right back to realizing color is still there, bleeding all over your codebase.

What language isn’t colored?

So JS, Dart, C#, and Python have this problem. CoffeeScript and most other languages that compile to JS do too (which is why Dart inherited it). I think even ClojureScript has this issue even though they’ve tried really hard to push against it with their core.async stuff.

Wanna know one that doesn’t? Java. I know right? How often do you get to say, “Yeah, Java is the one that really does this right.”? But there you go. In their defense, they are actively trying to correct this oversight by moving to futures and async IO. It’s like a race to the bottom.

C# also actually can avoid this problem too. They opted in to having color. Before they added async-await and all of the Task<T> stuff, you just used regular sync API calls. Three more languages that don’t have this problem: Go, Lua, and Ruby.

Any guess what they have in common?

Threads. Or, more precisely: multiple independent callstacks that can be switched between. It isn’t strictly necessary for them to be operating system threads. Goroutines in Go, coroutines in Lua, and fibers in Ruby are perfectly adequate.

(That’s why C# has that little caveat. You can avoid the pain of async in C# by using threads.)

Remembrance of operations past

The fundamental problem is “How do you pick up where you left off when an operation completes”? You’ve built up some big callstack and then you call some IO operation. For performance, that operation uses the operating system’s underlying asynchronous API. You cannot wait for it to complete because it won’t. You have to return all the way back to your language’s event loop and give the OS some time to spin before it will be done.

Once it is, you need to resume what you were doing. The usual way a language “remembers where it is” is the callstack. That tracks all of the functions that are currently being invoked and where the instruction pointer is in each one.

But to do async IO, you have to unwind discard the entire C callstack. Kind of a Catch-22. You can do super fast IO, you just can’t do anything with the result! Every language that has async IO in its bowels—or in the case of JS, the browser’s event loop—copes with this in some way.

Node with its ever-marching-to-the-right callbacks stuffs all of those callframes in closures. When you do:

function makeSundae(callback) {
  scoopIceCream(function (iceCream) {
    warmUpCaramel(function (caramel) {
      callback(pourOnIceCream(iceCream, caramel));
    });
  });
}

Each of those function expressions closes over all of its surrounding context. That moves parameters like iceCream and caramel off the callstack and onto the heap. When the outer function returns and the callstack is trashed, it’s cool. That data is still floating around the heap.

The problem is you have to manually reify every damn one of these steps. There’s actually a name for this transformation: continuation-passing style. It was invented by language hackers in the 70s as an intermediate representation to use in the guts of their compilers. It’s a really bizarro way to represent code that happens to make some compiler optimizations easier to do.

No one ever for a second thought that a programmer would write actual code like that. And then Node came along and all of the sudden here we are pretending to be compiler back-ends. Where did we go wrong?

Note that promises and futures don’t actually buy you anything, either. If you’ve used them, you know you’re still hand-creating giant piles of function literals. You’re just passing them to .then() instead of to the asynchronous function itself.

Awaiting a generated solution

Async-await does help. If you peel back your compiler’s skull and see what it’s doing when it hits an await call you’d see it actually doing the CPS-transform. That’s why you need to use await in C#: it’s a clue to the compiler to say, “break the function in half here”. Everything after the await gets hoisted into a new function that it synthesizes on your behalf.

This is why async-await didn’t need any runtime support in the .NET framework. The compiler compiles it away to a series of chained closures that it can already handle. (Interestingly, closures themselves also don’t need runtime support. They get compiled to anonymous classes. In C#, closures really are a poor man’s objects.)

You might be wondering when I’m going to bring up generators. Does your language have a yield keyword? Then it can do something very similar.

(In fact, I believe generators and async-await are isomorphic. I’ve got a bit of code floating around in some dark corner of my hard disc that implements a generator-style game loop using only async-await.)

Where was I? Oh, right. So with callbacks, promises, async-await, and generators, you ultimately end up taking your asynchronous function and smearing it out into a bunch of closures that live over in the heap.

Your function passes the outermost one into the runtime. When the event loop or IO operation is done, it invokes that function and you pick up where you left off. But that means everything above you also has to return. You still have to unwind the whole stack.

This is where the “red functions can only be called by red functions” rule comes from. You have to closurify the entire callstack all the way back to main() or the event handler.

Reified callstacks

But if you have threads (green- or OS-level), you don’t need to do that. You can just suspend the entire thread and hop straight back to the OS or event loop without having to return from all of those functions.

Go is the language that does this most beautifully in my opinion. As soon as you do any IO operation, it just parks that goroutine and resumes any other ones that aren’t blocked on IO.

If you look at the IO operations in the standard library, they seem synchronous. In other words, they just do work and then return a result when they are done. But it’s not that they’re synchronous in the sense that it would mean in JavaScript. Other Go code can run while one of these operations is pending. It’s that Go has eliminated the distinction between synchronous and asynchronous code.

Concurrency in Go is a facet of how you choose to model your program, and not a color seared into each function in the standard library. This means all of the pain of the five rules I mentioned above is completely and totally eliminated.

So, the next time you start telling me about some new hot language and how awesome its concurrency story is because it has asynchronous APIs, now you’ll know why I start grinding my teeth. Because it means you’re right back to red functions and blue ones.

Rooms and Mazes: A Procedural Dungeon Generator

2014-12-21T00:00:00-08:00

Several months ago I promised a follow-up to my previous blog post about turn-based game loops in my roguelike. Then I got completely sidetracked by self-publishing my book, Game Programming Patterns, and forgot all about it. I totally left you hanging.

Well, I finally got some time to think about my roguelike again and today, I’m here to… keep you hanging. Alas, you are at the mercy of my wandering attention span! Instead of game loops, today we’re going to talk about possibly the most fun and challenging part of making a roguelike: generating dungeons!

Go ahead and click the little box below to see what we end up with:

Click it again to restart it.

Pretty neat, huh? If you want to skip the prose, the code is here.

One of my earliest memories of computing is a maze generator running on my family’s Apple IIe. It filled the screen with a grid of green squares, then incrementally cut holes in the walls. Eventually, every square of the grid was connected and the screen was filled with a complete, perfect maze.

My little home computer could create something that had deep structure—every square of the maze could be reached from any other—and yet it seemed to be chaotic—it carved at random and every maze was different. This was enough to blow my ten-year-old mind. It still kind of does today.

What’s in a dungeon?

Procedural generation—having the game build stuff randomly instead of using hand-authored content—is amazing when it works well. You get a ton of replayability because the game is different every time. As the person implementing the game, you also get the critical feature of not knowing what you’re going to get even though you wrote the code. The game can surprise you too.

People get into procedural generation because it seems easier. Hand-authoring content is obviously a lot of work. If you want your game to have a hundred levels, you have to make a hundred things. But make one little random level generator and you can have a hundred levels, a thousand, or a million, for free!

Alas, it doesn’t quite work out that way. You see, defining the procedure is a hell of a lot harder than just sitting down and banging out some content. You have to take some very nebulous, artistic chunk of your brain, figure out precisely what it’s doing, and translate that to code. You’re coding a simulation of yourself.

It must balance a number of technical and aesthetic constraints. For mine, I focused on:

It needs to be fairly efficient. The generator only runs when the player enters a new level, so it doesn’t have to be as super fast, but I still don’t want a several second pause giving the player time to question whether they should be playing a game or doing something more productive with their life.
The dungeon needs to be connected. Like the mazes on my old green-screen Apple, that means from any point in the dungeon, there is a way—possibly circuitous—to any other point.

This is vital because if player has to complete a quest like “find the magic chalice” or “kill the cockatrice”, it’s pretty cruel if the dungeon drops that in some walled-off room the player can’t get to. It also avoids wasting time generating and populating areas the player can never see.
Moreso, I want dungeons to not be perfect. “Perfect” in the context of mazes and graphs (which are synonymous) means there is only one path between any two points. If you flatten out all of the windy passages, you’ll discover your twisty maze is really just a tree all crumpled up. Passageways branch but never merge. Im-perfect mazes have loops and cycles—multiple paths from A to B.

This is a gameplay constraint, not a technical one. You could make a roguelike with perfect dungeons, and many simple roguelikes do that because generators for those are easier to design and implement.

But I find them less fun to play. When you hit a dead end (which is often), you have to do a lot of backtracking to get to a new area to explore. You can’t circle around to avoid certain enemies, or sneak out a back passage. Neither can the bad guys, for that matter.

Fundamentally, games are about making decisions from a set of alternatives. At a literal level, perfect dungeons only give you one path to choose from.
I want open rooms. I could make dungeons just be nothing but mazes of narrow passages, but then you could never get surrounded by a horde of monsters. It would feel claustrophic and kill a bunch of interesting combat tactics.

Wide open areas are critical for area effect spells, and big dramatic battles. They also provide space for interesting decorations and themed areas. Vaults, pits, traps, treasure rooms, etc. Rooms are the high points of the hero’s journey.
I want passageways. At the same time, I don’t want the dungeon to just be rooms. There are some games that create levels this way where doors directly join room to room. It works OK, but I find it a bit monotonous. I like the player feeling confined part of the time, and having narrow corridors that the player can draw monsters into is a key tactic in the game.
All of this needs to be tunable. Many roguelikes have one huge multi-floor dungeon where depths vary in difficulty but not much else. My game is different. It has a number of different areas. Each has its own look and feel. Some may be small and cramped, others spacious and orderly.

I solve this partially by having multiple distinct dungeon generation algorithms. Outdoor areas use an entirely different process. (I should probably write about that too sometime. Look, another unfulfilled promise!) But coding a new dungeon generator from scratch for every area is a huge time sink. Instead, I want the generator to have a bunch of knobs and levers I can tweak so I can make a number of areas that share the same code but have their own feel.

A room with a view

I’ve been working on this game pretty much forever (it’s gone through four different implementation languages!) and I’ve tried a number of different dungeon generators. My main source of inspiration is a game called Angband. The only thing I’ve sunk more of my life into than working on my game is playing that one.

Angband is fantastically old. When it forked off of Moria, Nancy Kerrigan had just taken a round of melee damage from a club-wielding troll. On machines of that time, it was much harder to make a fast dungeon generator, and Angband’s is pretty simple:

Sprinkle a bunch of randomly located, non-overlapping rooms.
Draw random corridors to connect them.

To ensure rooms don’t overlap, I just discard a room if it collides with any previously placed one. To avoid a possible infinite loop, instead of trying until a certain number of rooms are successfully placed, I do a fixed number of attempts to place rooms. Failure becomes more common as the dungeon gets fuller—after all, you can only fit so many rooms in a given area—but tuning this gives you some control over room density, like so:

Attempts: 200

A dark and twisty passageway

Most of the dungeon generators I’ve written start with this. The hard part, by far, is making good passageways to connect them. That’s really what this post is about—a neat way to solve that problem.

Angband’s solution is brute force but surprisingly effective. It picks a pair of rooms—completely ignoring how far apart they are—and starts a passageway that wanders randomly from one (hopefully) to the other. It’s got a few clever checks to keep things from overlapping too much but passageways can and do cut through other rooms, cross other passages or dead end.

I tried implementing that a number of times but (likely failures on my part) never got to something I really liked. The corridors I ended up with always looked too straight, or overlapped other stuff in unattractive ways.

Then, a few months ago, I stumbled onto a description of a dungeon generator by u/FastAsUcan on the /r/roguelikedev subreddit. His generator, Karcero, is based on Jamis Buck’s dungeon generator. If you’ve ever done any procedural dungeon generation, you know—or should know—who Buck is. He’s got a ton of great articles on random mazes.

Years ago, I remember seeing an actual dungeon generator he wrote for use with pen-and-paper Dungeons & Dragons. Unlike most of his maze stuff, this had actual rooms, and the results looked great.

But, at the time, I didn’t know how it worked. How do you go from mazes to open winding corridors and rooms? I tucked this open question away in the corner of my mind and immediately forgot about it.

The post by FastAsUcan provides the answer. It works like so:

Make a perfect maze. There are a number of different algorithms for this, but they’re all fairly straightforward.
Make the maze sparse. Find dead end passages and fill them back in with solid rock.
Pick some of the remaining dead ends and cut holes in them to adjacent walls. This makes the maze imperfect. (Remember, this is a good thing!)
Create rooms and find good locations to place them. “Good” here means not overlapping the maze but near it so you can add a door and connect it.

The magic step, and the piece I was missing, is sparseness. A normal maze fills every single square of the world, leaving no areas where you can fit a room. The trick that Jamis and FastAsUcan do here is to carve the whole maze and then uncarve the dead ends.

Doing that is actually pretty easy. A dead end is just a tile that has walls on three sides. When you find one of those, you fill that tile back in. That may in turn make the tile it connects to a dead end. Keep doing this until you run out of dead ends and you’ll end up with lots of solid area where rooms can be placed.

Of course, if you do that starting with a perfect maze and run to completion, you’ll erase the whole maze! A perfect maze has no loops so everything is a dead end if you follow passages long enough. Jamis’ solution is to not erase all of the dead ends, just some. It stops after a while. Something like this:

Corridors to leave: 1000

Once you do that, you can start placing rooms. The process Jamis uses for this is interesting. He picks a room size and then tries to place it on every single location in the dungeon. Any location that overlaps a room or passageway is discarded. The remaining positions are “ranked” where rooms that are near passageways are better. It then picks the best position and places the room there, and puts some doors between the room and the passage.

Rinse, lather, repeat and you’ve got yourself a dungeon.

Rooms then mazes

I went ahead and coded this up exactly as described. It went OK, but I found that the process of placing rooms was pretty slow. It works well for dungeons of the small size you do for a tabletop role-playing game, but not so much at the scale of a computer roguelike.

So, I did some tinkering and came up with a slight variation. My contribution is pretty minor, but I thought it would be worth writing down. (Honestly, I just think it’s fun to watch animated dungeon generators, and the prose is pure fluff.)

Where Buck and Karcero start with the maze and then add the rooms, mine does things in the opposite order. First, it places a bunch of random rooms. Then, it iterates over every tile in the dungeon. When it finds a solid one where an open area could be, it starts running a maze generator at that point.

Maze generators work by incrementally carving passages while avoiding cutting into an already open area. That’s how you ensure the maze only has one solution. If you let it carve into existing passages, you’d get loops.

This is conveniently exactly what you need to let the maze grow and fill the odd shaped areas that surround the rooms. In other words, a maze generator is a randomized flood fill algorithm. Run this on every solid region between the rooms and we’re left with the entire dungeon packed full of disconnected rooms and mazes.

Each color here represents a different region of connected tiles.

Looking for a connection

All that remains is to stitch those back together into a single continuous dungeon. Fortunately, that’s pretty easy to do. The room generator chooses odd sizes and positions for rooms so they are aligned with the mazes. Those in turn fill in all of the unused area, so we’re assured that each unconnected region is only a single tile away from its neighbors.

After filling in the rooms and mazes, we find all of those possible connectors. These are tiles that are:

Solid rock.
Adjacent to two regions of different colors.

Here they are highlighted:

We use these to tie the regions together. Normally we think of the entire dungeon as a graph with each tile a vertex, but we’re going to go up a level of abstraction. Now, we treat each region of tiles as a single vertex and each connector as an edge between them.

If we use all of the connectors, our dungeon would be way too densely connected. Instead, we carve through just the connectors we need to get each region connected to the whole once. In fancy terms, we’re finding a spanning tree.

The process is pretty straightforward:

Pick a random room to be the main region.
Pick a random connector that touches the main region and open it up. In the demo, it does that by placing a door, but you can do an open passageway, locked door, or magical wardrobe. Be creative.

Note that this process is agnostic about rooms and mazes. It just deals in “regions”. That means it can connect rooms directly to other rooms sometimes. You can avoid that if you want, but I find the resulting dungeons more fun to play.
The connected region is now part of the main one. Unify it. In the demo, I use a little flood fill algorithm to color in the newly merged region because it looks pretty. In a real implementation, you don’t need to mess with tiles. Just make a little data structure that tracks “region X is now merged”.
Remove any extraneous connectors. There are likely other existing connectors that connect the two regions that just merged. Since they no longer connect two separate regions and we want a spanning tree, discard them.
If there are still connectors left, go to #2. Any remaining connectors imply that there is still at least one disconnected region. Keep looping until all of the unconnected regions are merged into the main one.

Earlier, I said that I don’t want a perfect dungeon because they make for crappy gameplay. But, since this creates a spanning tree, that’s exactly what we’ve got. We only allow a single connector between any two regions so our dungeon is a tree and there’s only a single path between any two points.

Fixing that is pretty simple. In step 3, when we cull the unneeded connectors, we give them a slight chance of being opened up. Something like:

if (rng.oneIn(50)) _carve(pos, CELL_DOOR);

This occasionally carves an extra opening between regions. That gives us the imperfect loops we want to make the dungeon more fun to play in. Note that this is also easily tunable. If we make the chance more likely, we get more densely connected dungeons.

Uncarving

If we stop here, we’ll get dungeons that are packed chock full of maze corridors, most of which are dead ends. That has a certain sadistic appeal, but isn’t exactly what I’m going for. The last remaining step is the “sparseness” pass described earlier.

Now that we’ve got all of our rooms connected to each other, we can remove all of the dead ends in the maze. When we do that, the mazes are reduced to just the winding set of passageways needed to connect the rooms to each other. Every corridor is guaranteed to go somewhere interesting.

What we ended up with

In summary:

Place a bunch of random non-overlapping rooms.
Fill in the remaining solid regions with mazes.
Connect each of the mazes and rooms to their neighbors, with a chance to add some extra connections.
Remove all of the dead ends.

I’m pretty happy with it so far. It’s not perfect, though. It tends to produce annoyingly windy passages between rooms. You can tune that by tweaking your maze generation algorithm, but making the passageways less windy tends to make them wander to the edge of the dungeon, which has its own strange look.

The fact that rooms and mazes are aligned to odd boundaries makes things simpler and helps fill it in nicely, but it does give the dungeon a bit of an artificially aligned look. But, overall, it’s an improvement over what I had before, and the dungeons it makes seem to be pretty fun to play.

If you want to see for yourself, you can play the game right in your browser. The code for these demos is here, but it’s pretty gnarly. Making them animated adds a lot of complexity. Instead, here is the much cleaner implementation my game uses.

As a bonus for making it this far, here’s a super dense giant dungeon. I find it hypnotic:

How My Book Launch Went

2014-11-20T00:00:00-08:00

Greetings, superfans! When we last tuned in, I was just about to “launch” my self-published book. I put that in quotes because it’s an awfully serious-sounding word for what was really just doing some clicking on my laptop. If my last post was the climax of my book writing adventure, consider this the denouement. You’ve been with me this long, it’s the least I can do.

I do hold back one detail, though: while I talk about sales here, I won’t be putting cash numbers on it. If you try hard enough, you can calculate them yourself, but I feel weird sharing financial details. Strangely, it’s not because I’m shy about strangers knowing it as much as I am friends and family. So much room for awkwardness there.

I am a Marketer and I do Marketer-y Things

As I mentioned before, I decided to self-publish about halfway through writing the manuscript. That meant I had to wear all of the hats that a publisher has dedicated noggins for: editor, designer, proofreader, and… marketer. This last one is the least fun for me, but I know it’s important.

I read a bit online and what a lot of people said was that email lists are gold. I like that they are pull-based instead of push-based. Instead of jamming myself and my book down strangers’ throats, they sign up of their own volition. I only get in touch with the people who actually want to hear from me.

So I created a MailChimp account—chosen over other mailing list providers mainly because of their cute logo—and slapped a little “sign up!” form on every page of the site. Whenever I put a new chapter online, I got a traffic bump and some fraction of those people were kind or foolhardy enough to grant me their email address.

I did this pretty early in the writing process so that I could give it time to grow before the book was complete. By putting the whole book online and releasing it chapterly (like a Dickens serial for the Internet age) I had amassed a surprisingly large list by the time I was ready to ~~try to squeeze money out of them~~ tell them my book was complete. Around 8,000 people, which still seems like a crazy large number to me.

I also, despite the content of my tweets, managed to acquire a fair number of twitter friends. (I don’t like thinking of them “followers”. Celebutants and TEDx thought/cult leaders have “followers”. I just have Internet pals.)

A New Front Page

Now that the book is a product as much as it is a web site, I redid the front page to highlight both of those. It still, of course, links to the web version. And, of course, the entire contents of the book are still readable on the web. But I also added links to the various sites where you could get the print version, Kindle, and EPUB.

I wanted to emphasize the “bookyness” of it, the physicality, so I got out my camera and my macro lens and took a bunch of honest-to-God photos of it:

Wait… if the camera is in the photo, how did I take this shot?

There’s something weird and meta about taking a picture of myself taking photos of a book whose cover I designed using a photo I took of an illustration I drew. I should take a picture of myself writing this blog post just to go full Inception.

Next, I spent way too long coming up with a design and text for the front page. Seriously, it took me as long to make what is effectively a glorified list of hyperlinks as it did to write, illustrate, and edit a full chapter of the book. I think it’s worth it, though. The front page is the reader’s first impression of the book. If I make you feel that page is beautiful and high quality, it’s a short hop to having faith that the inside of the book is too.

I ended up with this:

Click to zoom!

This Part of the Blog Post is About Writing a Blog Post

I feel a little squirmy writing this. Like I said, I’m uncomfortable with self-promotion. I hoped, wanted, to get a decent amount of… attention? traffic? validation of my worth as a human? when I announced the for-sale versions of the book. Readers had been asking for them for a while, so I knew there would be some excitement just by saying, “Here they are.”

But that didn’t feel really newsworthy to me. There wasn’t much to discuss, just “here they are. moneys plz.” People like a story, and I like telling them, especially when the protagonist is yours truly. I’ve been really lucky to have a lot of encouragement on this project from readers, and I thought one way to say thanks would be to give a window into what the production process was like. It’s a side of book-making that I don’t see very much online.

Also, it let me blather about fonts for six thousand words. Seriously, the blog post I wrote about turning the manuscript into a physical book is longer than almost every chapter in the book. Clearly, I need an editor.

This also took a lot of time—about a week—but I really enjoyed writing it all down. I did my best to make it an entertaining read. While I hoped it would drive traffic to the book, I did honestly want it to be a worthwhile read in and of itself. I believe that, when played well, life is a non-zero-sum game.

I <3 Filling Out Web Forms

I was ready to upload the new site, and the new blog post. All that was left was to actually make the book available. This involved uploading various files—inside PDF for the print version, cover, EPUB file, MOBI file—to various sites whose forms cover the full gamut of usability. Clicking the final “OK” on CreateSpace, Kindle Direct Publishing, and Smashwords was both nerve-wracking and anti-climactic.

Then the waiting game. Most of these sites do various amounts of automated and manual validation that take time to process. I noticed a typo that forced me to re-upload to CreateSpace. Kindle found a couple of spelling errors (surprising given how many rounds of editing I’d done by now). Smashwords asked me to have an explicit cover page.

Once those were all happy, the listings appeared. If you knew to look for them, you could have bought the book.

CreateSpace and Smashwords also distribute to other sales channels (things like Lightning Source and book stores for the former; iBooks, Nook, Kobo, etc. for the latter). I wanted to wait for those sales channels to populate before I pulled the switch but… well, I got impatient.

As soon as I saw the Amazon page for my book, I just couldn’t wait any more. I pushed the new version of the site and the blog post. I wrote an email and blasted the mailing list. I tweeted. I mentioned it on G+.

What I didn’t do this time was post it to reddit. It felt a little too self-serving for how reddit rolls. I did post a link to the blog on Hacker News since that crowd is more accepting of self-promotion.

I had tabs open for the Smashwords sales dashboard, CreateSpace royalties, Kindle Direct Publishing sales report, twitter, and Hacker News. I refreshed those suckers like my life depended on it.

Command, We Are Ready for Launch

This moment, as dumb as it sounds, was actually pretty stressful. A surprisingly large number of self-published books just sink to the depths leaving nary a ripple on the surface. You hear horror stories of writers whose novels sell zero copies. Nonfiction is a lot easier, but I still had no idea if people really liked the book enough to drop real cash on it.

Refresh refresh refresh refresh. A nervous sweat trickled down my neck.

A few minutes later, Smashwords reported two sales. Holy shit! Someone bought the book! I was a professional writer!

Then people started retweeting me. Someone posted a blurb about it on the gamedev subreddit and someone else did on the programming subreddit.

People started saying super nice things! About me! It was awesome! Also, I was drinking in celebration at the same time so that multiplied all the good feelings.

How I felt.

I feverishly replied to as many comments, tweets, and emails as I could. (All the while still refreshing the other tabs.) Eventually, I maxed out my serotonin levels and had to crash. Like some candy raver in an E puddle in the back of the chill-out room, I passed out, totally spent.

The Next Day

I woke up the next morning feeling like a kid on Christmas. A kid who wasn’t entirely certain he’d been good enough for Santa to bring him presents. Morning coffee in hand, I opened up my laptop and checked all the dashboards.

While I slept, I’d sold enough copies to pay off all of the expenses of the book. That isn’t actually that much: all I’d paid for was ISBN numbers, a couple of fonts, freelance copy-editing, and a business license. But, still, I was in the black now.

I replied to as many people as I could and then headed into the office to ~~get some work done~~ spend the rest of the day refreshing my browser. Aggregate sales stats started showing up in the various dashboards:

Kindle sales on the first day.

Smashwords (EPUB) sales on the first day.

CreateSpace doesn’t draw a pretty picture for you, but the raw numbers were doing the same thing. I told my coworkers and they got all excited for me.

The rest of the day was basically a blur of responding to people and telling my wife and team how high the numbers were going. At some point, when I refreshed the Amazon page, I noticed a little banner had appeared next to my book:

No way.

It turns out Amazon tracks book sales very quickly. For the brief period of time where I had saturated social media, enough sales were coming in to actually make it the best-selling game programming book for a little while.

Sorry “Learning Python”.

Wait, did I say “game programming”? I meant #1 out of all programming books:

I did a victory lap at this point.

At it’s peak, my book was the #7 best-selling book in the entire computer category:

Up there with Minecraft!

Around this time, I started referring to myself in the third person as “best-selling author Robert Nystrom”. Of course, most of the other books in those lists had been there for weeks or months, but I had my moment in the sun and I was committed to making the most of it.

Return to Normalcy

The peak, naturally, only lasted about a day. I was most curious to see what the right side of the graph would look like. Would it fall to zero? A slow trickle? A steady passive income?

When I sent out the email for the book, one thing I asked was for people to write honest, detailed reviews. I figured a book page with a good set of reviews would give it the air of legimitacy for future customers who stumbled onto the book on their own. That in turn would help the book have a longer sales tail.

Because people are awesome and I love every one of them, people did indeed do this. It took a while, but reviews started trickling in.

Other pleasant surprises started happening too. The book finally made its way through the other channels. I put it up on Google Play, and Smashwords got it onto iBooks, Nook, and some other places. I finally figured out a place to sell the PDF. (If you go to the book’s site now, these are all listed there.)

In a mistake that I don’t understand but can’t bear to correct, Apple even decided to spice up my name a bit:

Robert Nyström, now 15% more metal.

Then, a few days later, the most magical part of this whole thing started happening (though, honestly, the umlaut is a close second). That’s about how long it took for print copies to get shipped to people.

Soon, friends and family—some who I haven’t seen in years—started sending me pictures of their copies of the book. Kind strangers in countries I couldn’t find on a map tweeted pictures of themselves opening the shipping box with excitement to see my dumb face staring out at them.

It’s hard to articulate how profoundly gratifying that is. Through almost the entire six years I’ve been working on this book, it’s been a purely digital thing. While I spend an astonishing amount of my life on the Internet, it still never quite feels “real” to me.

Is rearranging some polarization on an SSD really making something? Is a few words glowing on an LCD actually communicating with a person, or just some pixelated avatar?

But seeing photos of real live people holding an actual physical thing filled with words and illustrations I put together, and seeing them happy to have it in their hands finally made it click for me that I made something real that actually affected people.

How We’re Looking Now

Things have mostly settled down since then, though I do still have lots of email to catch up on (sorry!). Now that the book is in pretty much every sales channel, eBook walled garden, and international site for Amazon, I’m finally getting to the point where I can think of the book as done and put it into the background of my life.

So far, buyers have been trickling in at a decent rate. Here’s what my Kindle sales look like as of writing this post:

EPUB is similar, and I think print is too.

The book has been out about two weeks, and I’ve sold:

796 print copies. 555 in the US, 145 in Great Britain, 95 in Europe, and 1 directly from createspace.com
274 Kindle copies. I’ll spare you the per country break-down, but it is really cool to see that I’ve sold it in India, Japan, Brazil, and other places.
70 copies from smashwords. This is people who wany to buy the EPUB file directly.
7 copies from iBooks. The book only appeared a few days ago, so I expect this to catch up with the other channels over time.

That’s a grand total of 1,147 copies sold in the first two weeks. I won’t be retiring off of this, but I’m really really happy with that number, and unbelievably grateful to everyone who bought a copy.

While I’m planning to take a good long break from serious projects first, this level of success has certainly gotten me pumped about writing another book. If I do, I’m 100% confident now in my approach of putting the entire book online for free. I’m certain I wouldn’t have sold anywhere near this many copies if the book’s web site hadn’t built an audience who were ready and eager to get their hands on a copy.

It turns out giving something away is a great way to get something back.

Zero to 353 Pages: Bringing My Web Book to Print and eBook

2014-11-03T00:00:00-08:00

Look what I made!

It’s a funny feeling to spend nearly six years making something and then finally hold it in your hand. It’s simultaneously, “This is it!” and “Is this it?” How can it be so small when it feels like such a big piece of my life?

Most of that time, the book existed entirely on the web. It was less “book” and more “book-length manuscript that you can read with your browser”. The web site is still the book’s real home in some ways. If you want to know more about game programming and software architecture, take a look. You can read the whole thing online, for free, because I love you.

I already wrote a post about the writing part of the process. That was the real mountain to climb. But, once I reached the summit and decided (1) I wanted to also have print and eBook editions and (2) I wanted to do it all myself, I learned that one does not simply walk out of Mordor either.

This gratuitously, vaingloriously long post is about climbing back down the mountain—all of the stuff required to turn a web site into a book. Remember on Mr. Rogers Neighborhood where they would take a trip to a factory to see how cute little piglets are turned into hotdogs or something charming like that? This is like that, but marginally less incarnadine.

Where it Starts

On April 22nd, I finished the third draft of the last chapter of the book. The next day, I uploaded it and modestly told a few friends. And by that, I mean I milked it for all it was worth on all the social networks.

As soon as I shook out all of the bugs readers reported, I went straight into working on the print and eBook versions. Like I mentioned in the last post, I work on the book every single day. Not breaking the chain has somehow been just enough of a mindtrick to get me to overcome my usual inability to finish anything bigger than breakfast.

By this point, I was actually superstitious about breaking it. I really wanted to hold the physical book in my hands, so until that was done, I was afraid to take even single day’s break.

I did break the chain on one day. It was my birthday in Kauai. I spent the morning playing on the beach with my kids, the afternoon snorkeling with sea turtles, the evening grilling burgers for my friends, and the night boozing it up. The book kinda slipped my mind.

The first step was doing yet another editing pass over the manuscript. At this point, I’d done three drafts of each chapter, but ink on paper is a lot harder to fix than a web site, so I wanted to give it one more round of scrutiny.

This was also, strangely, the first time I’d read the book front to back. The chapters are relatively unconnected, sort of like recipes in a cookbook, and I wrote them out of order. Everytime I finished a chapter, the next one I started was the one that seemed like the most fun right then.

I found a bunch of places where I said the same thing twice near each other in the book, but years apart in my life when I wrote them. I fixed those, and tons of other style and tone issues. Then I found a freelance copy editor and got her to do a pass over it.

Despite having done four drafts at this point, she still found dozens of mistakes that I managed to miss. If you ever write book, I highly recommend this step. You’ll be astonished at how much you miss that a good editor will find.

eBooks, Ugh

I’m going to mix up the chronology here and talk about the eBook version first. In reality, I interleaved working on this and the print version and going back and forth over chapters with my copy editor, Lauren. The eBook stuff is less exciting than the print design, so let’s just get it out of the way.

I was actually in pretty good shape going in to this. I wrote the book in markdown, and had a little Python script that takes those files, some CSS, and an HTML template, and burps out the web site.

If you’ve never done an eBook before (you lucky devil, you), they come in two predominant flavors: EPUB and MOBI. Most of the world uses EPUB, but Kindle demands MOBI in some sort of gigantic distributed prank on the entire writing community.

An EPUB file is basically a zip file containing a web site. Seriously. Take your static site and run this on it:

cd my_awesome_site
zip -X0 my_awesome_site.epub mimetype
zip -Xur9D my_awesome_site.epub *

Now you’re 90% of the way to having an eBook. It’s the other 10% that makes you want to claw your eyes out. See, eBooks are a lot like web sites… circa 2004. Instead of this fancy HTML5 stuff which is clearly, like, living in the future with all of its new tags that don’t need to be closed, EPUB requires XHTML, the evil mutant offspring of HTML and XML. The turducken of the markup world.

Worse, eBook readers handle CSS about as well as Netscape Navigator and IE4 did. If you lived through the horror show that was web design in the ‘90s where you had three browsers open all day, you know what I’m talking about. Except that now each of those “browsers” is a separate physical device that you have to jump through hoops to get your “site” on.

Getting an updated .mobi file into the Kindle app on my tablet involved some combination of Dropbox, an Android file manager, going to the Amazon website to delete the previous version, sending an email to a mysterious dead drop address granted me by Amazon, waiting for the Kindle app to crash a few times, and, on occasion, blood sacrifice.

The things I do for love.

Oh, and did I mention the XML? SO MUCH XML. The EPUB format was clearly written by uptight pencil-pushers whose OCD was permanently triggered by the chaos of the web. You need a content.opf XML manifest that lists every single file in your eBook. You also need a container.xml manifest that points to that .opf file. You need a manifest for your manifest.

The .opf file needs a <spine> tag containing the ordered list of stuff in the book. And a <guide> containing… an ordered list of the stuff in the book. Also, there’s a separate—mandatory, of course!—toc.nxc file containing, you guessed it, an ordered list of the stuff in the book. Not only that, each item in the TOC is not just in order but explicitly manually numbered:

<navPoint id="copyright" playOrder="1">
  <navLabel><text>Copyright</text></navLabel>
  <content src="copyright.html" />
</navPoint>

<navPoint id="acknowledgements" playOrder="2">
  <navLabel><text>Acknowledgements</text></navLabel>
  <content src="acknowledgements.html" />
</navPoint>

<navPoint id="dedication" playOrder="3">
  <navLabel><text>Dedication</text></navLabel>
  <content src="dedication.html" />
</navPoint>

See those little playOrder attributes? It’s like double-entry bookkeeping, minus the fun and excitement of double-entry bookkeeping.

But, despite my ranting here, it’s not really hard. It’s sort of like counting to a thousand by hitting + and 1 over and over on your calculator. (I had a fun childhood.) Once you’ve got a happy little EPUB file, you throw it at KindleGen, a rancorous old app from Amazon, and it begrudgingly converts it to a MOBI for you. Behold:

It took me almost six years to make these two files.

If you ignore the fact that it took about a hundred iterations to please the EPUB validator Gods, this was straightforward. Now, let’s talk about the fun stuff!

My Dirty Little Secret

Warning: Seriously ardent exposition on design lies ahead. The rest of this post will be like watching two of your best friends who after years of awkward sexual tension finally hook up and now aren’t so much showering you with public displays of affection as they are public displays of not-always-entirely-dry humping. I really really like fonts.

When I first started the book, I wrote a little promise to myself in my pink locked diary that I never show anyone. (Except now you, I guess, Dearest Reader.) That promise was, “If I can finish writing this whole damn manuscript, I will let myself typeset it.”

I know that probably sounds strange but if you haven’t figured out I’m a bit off center by now, you must be skimming this post for the pictures.

Here’s one of a tiny pumpkin pie I baked!

Most of my life, I had one cranial hemisphere in the artist bucket and one in the logical bin. I grew up drawing and painting, but usually on graph paper.

A quick trip down my résumé makes stops at computer animator, web designer, UI designer, part-time UI programmer, full-time UI programmer, and tools programmer, before finally landing on regular full-on programmer. My left brain metastasized and took over pretty much the whole light show upstairs.

I love programming, and I’m happy to spend almost all day doing stuff that’s not that visual, but, man, sometimes I really miss design. Thus the vow: if I could complete this giant ball of logical thought, I would reward myself by fully giving into my artistic side.

You Mean I Get To Pick a Page Size?

Of the design stuff I have done in the past, almost all of it has been for the web. There’s a lot to like about web design, but it’s sort of like body painting. No matter how steady you are with your airbrush, there’s an inherit squishiness and unreliability to the underlying medium.

When you can’t trust the user not to futz with the size of their browser window or even control what fonts they have access to, it’s hard to design something that looks exactly the way you want. But, print, now that’s a different story.

The first thing I did was decide how big the pages would be. Let me repeat that for you web folks: I, the designer, got to choose the page size. Because Amazon is the giant gorilla of the publishing industry and CreateSpace is its cuddly orangutan buddy, I decided to go with that for handling the print-on-demand production of the book. They have a few different trim sizes you can pick from.

I grabbed a bunch of programming books I had nearby, took my ruler out, and measured those suckers. I wanted my book to be like a real programming book, so I measured their dimensions, margins, line height, font size, the works. I took some averages, settled on 7.5" × 9.25" and that was that.

Fonts, Fonts, Fonts

Now that I knew how big my canvas was, it was time to start painting it. And, for a book, that means fonts. Sweet, delicious, heavenly fonts.

As a web designer, I was used to—paraphrasing Henry Ford here—having any serif I wanted, as long as it was Georgia. (And I remember the days before Georgia, a ghastly fever dream thankfully cured by heroic doctor Matthew Carter.)

Google Web Fonts turned things up a notch, but once you filter out all of the fonts on there that don’t have italics, or have shitty metrics, or incomplete character sets, or only one weight, what you’re left with is… well, Georgia starts looking pretty good again.

But, now. Now! I could just go out and buy a fucking font and then use it. Any font I wanted! It was intoxicating. Incredible. Overwhelming. Paralyzing.

I spent nights in a hallucinogenic daze going through dozens of serif (for body copy), sans serif (asides and headers), and monospace (code, naturally) fonts. For all I know, it could have been weeks. Time had lost all meaning. When my fugue dissipated, I had a beard, a Grateful Dead tattoo, and three fonts.

Ironically, the first two, Source Sans Pro and Source Code Pro, are Google Web Fonts, and are what the site uses:

Not pictured: beard and tattoo.

I don’t know if this is astute branding on my part or just a failure of imagination, but do I think Adobe really nailed it with Source Code Pro—even when in print—and it pairs just as perfectly with Source Sans as theirs names would lead you to believe.

That just left a body font, the most important font in the book. Like I do, I over-constrained the hell out of the problem:

I wanted a relatively small x-height. I don’t dig the current trend of really big x-heights, especially not in print. They look like a schoolgirl’s loopy handwriting to me, minus hearts over the “i"s. If I’m going to print this bad boy at 2400 DPI, it can handle some nice small bowls.
But not too small. The prose is heavily seasoned with inline code, so the body and code fonts need to have similar vertical metrics or it will look like Danny DeVito and Arnold Schwarzenegger in Twins.
I love Old Style typefaces. Garamond, Jenson, and Bembo are my homeboys. Low contrast in line thickness, angled stress and a bit of humanist irregularity, aww yiss. I can’t stand the wimpy horizontal strokes of Didone fonts. I seriously get irrationally angered by them.
But a font that’s too calligraphic would clash with the other fonts. A page should look like a cohesive whole. That won’t happen if the body text looks enscribed by a medieval monk’s quill pen while the code and headers are machined out by a robot.

If you distill that down, what I really wanted was a font with the metrics of an Old Style, but the letterforms of a cleaner modern typeface. Also, it needed a heavier weight and a spritely italic. I swear, italics trigger some kind of delectable synaesthetic response in me like the electric tingle of a battery on my bathing suit area.

After days of poring over typography blogs and font sites, lo and behold, I found a match. My Manic Pixie Dreamgirl in OpenType Format. Sina Nova.

Sorry for the linguistic liberties, Vlad. I’m sure you understand.

God, just look at it there. And wait until you see the ligatures. It’s enough to make a girl swoon.

I had assembled my army, now it was time to draw battlelines.

So Many Grids

Some people fantasize about building roller coasters in their backyards or being smothered in pugs. I’ve always dreamt of designing using an honest-to-God grid system.

I’ve tried to approximate them for the web but it always ends up leaving me disappointed and slightly ashamed like trying to use role-play to reinvigorate a relationship you both know is dead. Not this time, though. I had a brand new copy of InDesign, and I wasn’t afraid to use it!

For those who don’t know, a grid system organizes the content on a page. You come up with the dimensions of an invisible grid that overlays the page and everything slots within those. When done well, the text hangs together in placid harmony like a zen garden.

Coming up with a good grid, especially for this book, was quite hard. Again, I had a bunch of constraints:

Of course, it has to take into account the page size, and the margins. Wider margins on the inside so you don’t get to close to the spine. We’re an actual book now!
If you’ve read any of the book, you know it’s full of asides that provide commentary and additional info. The longest one is a few paragraphs and they often need to appear right next to certain passages of text.
Of course, being a programming book, we’ve got lots of code samples. Those need to be fairly wide so you don’t have to wrap the lines very much.

The code is particularly tricky. I like a pretty large line height (think single- versus double-spaced) for prose, but code looks wrong if you spread it out that much. Most grids use a single line height for the vertical rhythm of the page, but that wouldn’t work for both prose and code (or for the asides for that matter).

After days of tinkering—wonderful, relaxing, theraputic, tinkering—I had something I liked. To get there, I picked a random chapter and mocked it up. You can’t design a layout without something to lay out, and I believe the design should take into account the actual content, so no lorem ipsum for me. I ended up with this:

Mmm.

Three 2” columns with a ¼“ gutter, two for the main text and one for the asides. I used the same spacing vertically for large elements like headers on chapter title pages.

For the main vertical rhythm, I came up with a fractional line height. The core baseline was 4.5 pt. Prose fell on every third line, and code and aside every second. That let the latter snuggle in a little tighter while still having some semblance of a vertical cadence across the entire page:

9 pt for code and asides, 13.5 pt for prose.

I set up a running footer, and designed the section and chapter headers. Not gonna lie: I was pretty pleased with myself. There may have been some celebratory drinking.

Oh no, XML Again?

Once I had my grid and fonts, all that was left to do was typeset the whole book. That meant bringing in all of the text. I couldn’t just copy and paste the web pages into InDesign. I’d already carefully authored (in markdown) emphasis, bold, lists, headers, subheaders, etc. I did not want to start from plaintext and do that all over again.

Fortunately, InDesign has a feature called "XML import”. You can take an arbitrary XML document, import it to InDesign, and automatically map XML tags to paragraphic and character styles you’ve created in InDesign. Unfortunately, this feature seems to have been implemented by a narcoleptic intern who sidestepped any code review process.

It’s one of the buggiest pieces of nominally commercial-grade software I’ve ever used. I, on more than one occasion, managed to get it to completely corrupt an InDesign file beyond repair. (Fortunately, help was just a git reset away.)

I cracked open my little Python script and hacked it up to convert the markdown to HTML and then, through an unholy series of regexes, mash that into something approximating XML. Stuff like:

def clean_up_code_xml(code):
  # Ditch most code formatting tags.
  code = re.sub(r'<span class="(k|kt|mi|n|nb|nc|nf)">([^<]+)</span>',
                r"\2", code)

  # Turn comments into something InDesign can map to a style.
  code = re.sub(r'<span class="(c1|cn)">([^<]+)</span>',
                r"<comment>\2</comment>", code)

And:

def clean_up_xhtml(html):
  # Ditch newlines in the middle of blocks of text. Out of sheer malice,
  # even though they are meaningless in actual XML, InDesign treats them
  # as significant.
  html = re.sub(r"\n(?<!<)", " ", html)

  # Also collapse redundant whitespace.
  html = re.sub(r" +", " ", html)
  html = html.replace("> <", "><")

  # Re-add newlines after closing paragraph-level tags.
  html = html.replace("</p>", "</p>\n")
  html = html.replace("</h2>", "</h2>\n")
  html = html.replace("</h3>", "</h3>\n")
  html = html.replace("</li>", "</li>\n")
  html = html.replace("</ol>", "</ol>\n")
  html = html.replace("</ul>", "</ul>\n")
  html = html.replace("</pre>", "</pre>\n")
  html = html.replace("</aside>", "</aside>\n")
  html = html.replace("</blockquote>", "</blockquote>\n")

  return html

Forgive me Father, for I have sinned. In my defense, I wasn’t writing a program to convert any HTML to XML. As long as it worked on the 90k words of my book, it was Correct™ as far as I’m concerned. This gave me a folder full of more-or-less XML files, one for each chapter. All that was left was to typeset them.

I’m a Designer!

Starting at chapter one, the introduction, I imported its XML into InDesign. Then the work started. Each aside had to be pulled out of the main body text, restyled, and put into the sidebar. I only wanted leading indentation on paragraphs that followed previous ones, so every “first” paragraph that followed a header or code sample needed to be styled. I could never get lists to work in the XML, so I restyled all of those manually.

Then the real work started. I redid every illustration to look sharp in a black-and-white, high resolution medium. I brought all sixty-something of them into Photoshop, removed the graph lines, upscaled the hell out of them, did some shenanigans to smooth the edges (but not too smooth since that’s part of their charm) and then saved them as 2400 DPI TIFFs:

Original web version on the left, upscaled and monochrome print version the right.

Then the real, hard work started. You see, all of that is just grunt work. You could probably get it done on Mechanical Turk if you wanted to, or just let Knuth solve it with some clever dynamic programming. Where the art comes into play is dealing with this one little problem: where do you put the page breaks?

First off, you want to minimize widows and orphans—things like when the last line of a paragraph is at the top of the next page. InDesign can automate this, and even handle more complex rules like:

Keep headers with at least a line or two of the paragraph that follows them.
Don’t let a block of code be split across pages at all.
If a code block follows a paragraph of body copy, keep at least the last line of the body text with the code.

This stuff is why people get all excited about InDesign. It rocks.

But, once I set up all of those rules, I discovered I’d basically turned InDesign into HAL 9000. Except, instead of refusing to open the pod bay doors, what it was forced to do was break a lot of things really early.

You see, once I set up all of these rules, the result was that big chunks of text—headers, body text, and code—were all inseparably glued to each other. Since I told InDesign not to split in the middle, the only thing it could do was move the whole kit and caboodle to the next page, leaving a huge blank area at the bottom of the previous page, like:

Most of the left (verso) page is wasted space.

There’s no magic bullet to fix this. I just took it a page at a time and tried to organize and distribute the whitespace as nicely as I could. Usually, I could fill in a gap in one page by pushing a bit of stuff from earlier pages down. Other times, I’d tweak the code to shave off a line or two, just enough to get it to fit. On occasion, I’d break code samples into separate pieces so it could span pages.

See for yourself:

Now imagine this playing back 24 times slower.

It was tedious, challenging work. I started on June 19th. Working every single day, I reached the end of the last chapter exactly two months later. There was more drinking.

Baby Got Back (Matter)

So, the print edition was done, right? Nope! Real books aren’t just a pile of chapters. They’ve got a copyright page, title page, table of contents, and all sorts of stuff like that. The bits of those that come before the meat of the book are called front matter and the rest are, obviously, back matter.

I spent a few more days designing a table of contents. (I just had to make it pretty. InDesign handled actually filling it in.) I wrote a copyright page, and acknowledgements, even a dedication:

Showing this page to my wife was one of the highlights of this whole adventure.

Then I moved onto the back matter. I guess a non-fiction book should have an index, right? Where’s the “automatically index my whole book button?” What’s that? There isn’t one? Well… shit.

InDesign can create an index for you, bless its little heart, but it doesn’t actually know English. You have to sprinkle the index references throughout your book. What it does is figure out what pages those references are on and builds the list of index entries at the end.

I spent two weeks going over the entire book again, adding index entries for anything I could imagine someone wanting to look up. I don’t promise that it’s a great index, but it is an index, God dammit.

While I was at it, I cross-referenced the whole book. See, the web version has these magical things called “hyperlinks”. When one chapter mentions a concept on another one, you can “click” them with your “mouse” and the computer takes you straight to the referenced chapter! It’s like living in the future but with less Stallone and Wesley Snipes homo-erotically punching each other.

Paper, alas, does not support web standards. Instead you just put “(see page 123)” and the reader, poor plebian, has to manually turn to that page theirself. You can create cross references in InDesign and it will automatically track the referenced section and keep the page number up to date. I found every place where there was a link in the web version and manually created a cross-reference.

Getting My Grubby Paws On It

Phew! Book production is a lot of work! But this was finally nearing the end! I printed the whole book, all three-hundred-plus pages of it on my laser printer. It was heavy! I made a big heavy thing all by myself. Full of words!

I went to the store and bought a red pen. I proofread every single page. I read my book, on paper. It, finally, after all these years, started to feel like a real thing.

This took a while to print.

I went back and fixed everything that got the red pen treatment, mainly just minor formatting bugs, then I exported a PDF. (Brief interlude where I zoom into 1000% and drool over Sina Nova again.) I uploaded it to CreateSpace. Oh boy oh boy oh boy.

Oh, crap. I’m not done yet. I forgot the cover!

Cover Me, I’m Going In!

If you go by self-publishing blogs, designing a cover is the hardest, most important, most agonized over, most terribly done part of book writing. For every blog post telling you how important it is to not screw it up, there are a hundred self-published books whose cover is some amalgamation of still-watermarked screen-res stock photography, fonts that come pre-installed on Windows 95, and a design aesthetic clearly based on the belief that if one drop shadow is good, ten must be better.

I didn’t want to be that guy.

To check myself, I spent some time looking at other game programming book covers. I even made a little page of thumbnails so I could put my cover mockups in there to see how they compared:

Where’s my book hiding in there?

There’s a delicate art to crafting a design that gets the readers attention, but not in that “what the hell was he thinking?” way.

The cover I’d had in the back of my head for a long time was something hand-drawn. The book is full of little flow-charty illustrations, and I like how their informality contrasts with the technical content of the book.

After trying a bunch of different mocks for other ideas, I spent an afternoon drawing one big illustration that combined a bunch my favorites from the book. I took a ton of photos of it at different angles with a macro lens and then tried to find a composition for the text that I liked. You can see the evolution here:

In chronological order.

Hopefully, you don’t hate the final one too much.

My Esteemed Publisher

Oh, I left out one piece of the puzzle! Look on the bottom right corner of those mocks. See that? Here, look closer:

A little logo.

To publish a book, you need an ISBN number. CreateSpace can give you a free one, but then you can’t use that anywhere else, and it associates the “publisher” of that number with CreateSpace, which felt weird to me. I’d also need separate ISBN numbers for the two eBook versions.

ISBNs work a bit like domain names. Each country has an appointed registrar—the company who is allowed to distribute ISBNs to publishers in one country. In the US, that’s Bowker.

It’s a pretty smart racket. They basically just hand out IDs, a process that could be automated with five lines of Perl code and a copy of Apache running on an Arduino. In return for that, they get to charge you $125 for a single ISBN number. Or you can get 10 for $275. Since everyone publishes both print and eBook versions, you basically always get the 10-pack. They know what they’re doing.

It’s certainly the most money I’ve ever paid for 130 digits, but whatever. I sent them my credit card deets and started filling out the form. Whereupon I reached a mandatory field for “publisher”. Apparently, “Yours Truly”, “I Just Did It Myself”, “Fake Vanity Press”, and “Can’t I Just Skip This?” are not valid values for that field.

There was only one thing to do: It’s business time. I got myself a business license. I am a real deal publisher. Look, it says so right here:

I guess this blog means I’ve now posted this conspicuously?

Now, when I was looking at other game programming books, I saw a bunch of others that were clearly self-published. The dead giveaways were “publishers” that were either (a) the author’s name or an anagram of such, (b) obviously the name of a pet, or © one of the “we’ll give you an ISBN for free” companies.

I don’t think there’s anything wrong with self-publishing, obviously, but it does carry a stigma to some readers. I don’t want them to see a jokey “publisher” and think the book is some amateur hour affair. While my writing style is light-hearted, I actually do take the content really seriously.

Older, established publishers tend to agglomerate over time yielding titles like “Harcourt, Brace & Howe”, “Harcourt Brace Jovanovich”, “Reed Elsevier”, and “Houghton Mifflin Riverdeep”. So I picked two stuffy-sounding names and stuck them together. Then I debated how to join them—“+” (too mathy), “|” (too '90s), “-” (hyphen, en-dash, or em-?), and “/” (too much like a boxing match)—before finally settling on just a plain space.

I even drew a logotype, just for kicks.

Although, for the record, I did still name it after my pets. I just took their names—Ginny and Benny—and classed them up. As the CEO and CFO of the Genever Benning empire, they are skilled executives, though a bit prone to farting in board meetings.

Upload it, Already!

OK, so we’ve got typeset chapters, front matter, back matter, a cover, ISBN numbers, a camera-ready PDF. All systems are go go go! I uploaded everything to CreateSpace, waited for their “manual” review process to complete and ordered a couple of proof copies.

The next few days were like waiting for Santa and then… in his traditional brown UPS suit, Santa arrived! The cover looked great! The back cover was even better than I’d hoped! (Sorry, you’ll have to get your hands on a copy to see it.) The binding looked solid! It looked like a professional quality book! I was super pumped!

Imagine you muster up the courage to crawl out of your nerd hole and ask the captain of the cheerleading team to go to prom with you. Wonder of wonders, she says “yes”! That’s how I felt.

With my wife looking over my shoulder, I cracked it open.

Then imagine you look that cheerleader in the eye and the realization crawls down your spine that her “yes” was laden with sarcasm you missed the first time around. She would never in a million years go to prom with you.

Somehow, despite my meticulous measuring and scrupulous adherence to CreateSpace’s guidelines, my layout was bad. The text was too small. The top margin too short. Worst of all, the inner margin was too narrow, making it hard to read text near the spine.

Merde.

OK, Let’s Do it All Again

As you surely realize by now, changing any of the metrics of the book is a huge undertaking. Sure, you can just edit the master and all of the pages update. But that in turn affects how the text wraps, which then totally undoes all of my careful fitting of blocks of stuff onto different pages. That grueling two-month period where I laid out each page? Out the window now.

I went back to the drawing board. I cracked open the master. I started re-measuring things. Part of the problem was that (unsurprisingly) I’d over-constrained myself. In addition to needing decent margins, a good-sized sidebar, and the right line height, I also wanted measurements that were relatively round numbers. A column width of 1.35728261" is no fun to work with.

In the process of rounding some of those measurements to the nearest nice round number, I’d strayed away from actual good metrics. After bumping up the text size a bit, I spent days trying to come up with a column width, gutter size, and line height that would fit within the page margins and be easy to read.

Eventually, I found a way out: decimal inches. Most of my print work has used… shall we say… imperial measurements? Things like 16pt or 3/16". In other words, usually some power of two fraction of an inch. But that’s not the only option. You can go French revolution and actually do things like 1.3". InDesign won’t bat an eye at it.

After a bunch of monkeying around, I found a new grid. Instead of a vertical grid where prose is every three grid lines and code is every two, I bumped the fraction to ¾. This opened up the code and asides a bit relative to the text. I brought down the top margin and gave myself more than enough breathing room near the spine.

All that was left to do was update all of the pages. By this point, I was angry and fired up. I was so close to thinking the book was done and I just wanted it to be over. I burned through those pages, working on them practically every waking moment. This time, I got the whole three-hundred-something pages done in a week:

Fortunately, years playing Pokemon have given me fantastic grinding skills.

I uploaded a new PDF, crossed my fingers and waited for the new proof to arrive. When it did… God what a sigh of relief. It looked fine. Totally readable. Hallelujah.

That readability was great because it made it much easier for me to notice all the dumb mistakes I’d made in my hurried re-layout. Somehow, I’d managed to break all of the cross references, and sprinkle typos through much of the code. I did another proof-reading pass on the actual proof:

Every note is a mistake.

I fixed those, and uploaded it again. And… that’s it. I know there are still mistakes lurking in there, and thinking about them kills me. But, at some point, the value of getting the damn thing in people’s hands outweighs the value of trying to keep making it better.

Kicking it Out the Door

The print edition was done, and I made a slew of final changes to the eBook versions—mainly getting the cover in and working. Finally, only three things were left to do:

Redo the front page of the site to mention the new formats.
Upload everything to the various market places and put them on sale.
Write this blog post.

If you’re reading this, it looks like I got those done too! You can see for youself:

iBooks should be coming soon but Apple is busy manually reviewing erotica submissions so it may be a few weeks and I was too impatient to wait for that.

This whole production ended up taking six months. It was a ton of work, but I don’t regret doing it. For better or worse, I can now hold this book and know that it’s mine. From cover to cover, every word, picture, and bit of ink was up to me. I had a ton of help from my copy editor and from every kind reader who sent a bug report or pull request, and the book is immensely better thanks to their input. But, ultimately, the decisions were all mine.

What I’m feeling now is a curious mixture of relief, gratitude, and trepidation. Relief that it’s done and I actually pulled off completing a large project. Immense gratitude to everyone who encouraged me to keep going. I know I wouldn’t have finished without that.

But, finally, trepidation. People—you—have been really supportive of the book, which is truly the best feeling in the world. But there’s a big difference between saying you like the book and spending cold hard cash on it. I’ve never written this for the money, but the number of copies it sells will, in some ways, legitimize it in my mind. And that’s entirely outside of my control now.

I feel like I’m walking on stage, alone, squinting through the footlights to see if there’s anyone in the audience.

A Turn-Based Game Loop

2014-07-15T00:00:00-07:00

Now that my roguelike written in Dart is open source, I wanted to talk about a piece of it that I put a lot of time into. Well, I actually poured way too much of my life into lots of parts of this game, and maybe I’ll write about those too, but for now let’s start where most games start: with the game loop.

Here’s the game I’m talking about. You can play it here. Don’t get freaked out by all of the text below! There’s interactive demos just a few paragraphs down!

I’m a bit obsessed with software architecture in games. Since this roguelike is my hobby project, I’ve fully indulged myself. You know that guy who always wanted to be a railroad engineer but ended up an accountant or something, and then spends years in his basement endlessly tinkering on a model railroad? I’m like that guy, but for software architecture. And roguelikes.

That may be the nerdiest thing I’ve ever written, which, knowing me, is really saying something.

I’ve refactored and rewritten the game loop more times than I can count (there’s probably some “iteration” metajoke hiding in this sentence), and I won’t drag you through the whole history of it. Instead, we’ll go through its current form, building it up a piece at a time.

I have a couple of high-level goals:

The game engine should be strictly separated from the user interface. I’ve gone back and forth between a pixel art UI and a more old school ACSII-based one, and it’s important for me that the engine supports both. (The current Dart version is all ASCII-based right now, ironically rendered using the canvas API.) That means the engine can’t be coupled to any details of how the game is displayed to the user. Like a business app, I want a real model/view separation.
Monsters and player-controlled characters should be treated uniformly. Most of the engine just deals with “actors”, the superclass of both Monster and the player-controlled, gender-nonspecific Hero. I want to minimize special treatment that the player’s avatar receives and treating it like just another entity in the game does that implicitly.

Slicing up the game loop

Hauberk—like most roguelikes—is a turn-based game. Each actor in the game makes a move one at a time. When it’s the player’s turn, all of the monsters halt, awkwardly motionless like the world’s strangest game of freeze tag, until the player makes their move.

At the core of the game engine is the game loop. Its job is to iterate over the actors in the level and tell each to take its turn. In a simple game, it would look something like:

void gameLoop() {
  while (stillPlaying) {
    for (var actor in actors) {
      actor.update();
    }
  }
}

However, since the engine is decoupled from the user interface, it is driven externally. The engine has a main Game class. The UI owns an instance of that and tells it to process. The engine only processes one “step” of gameplay before returning control back to the UI.

(I’m being intentionally vague about “step” here. If I get a chance, I’ll write a follow-up post about how the engine’s game loop and the browser event loop interact. It’s pretty tricky.)

What this means is that Game needs to track the last actor who took a turn so it can pick up where it left off. Something like:

class Game {
  final actors = <Actor>[];
  int _currentActor = 0;

  void process() {
    actors[_currentActor].update();
    _currentActor = (_currentActor + 1) % actors.length;
  }
}

Astute readers like yourself are probably thinking this sounds like a good case for a generator. Indeed, in the previous C# incarnation of my game, I did exactly that. When Dart gets generators, I’ll probably use them. In the meantime, it’s just a bit more verbose.

Actions for actors

We’ve got a little resumable game loop now. When each actor’s turn comes up, its update() is called and the actor does whatever it does. A monster might pick a direction to walk. Then, the consequences of that have to be handled: it may walk into another actor in which case that triggers combat. It needs to handle walking into a wall or a door.

All told, even for a simple bit of behavior like “take a step”, there’s a decent amount of logic, but you’ll note that all of that applies equally well to monsters and heroes. Brave warriors can stumble into walls too, and the consequences are the same. It would be good to share code for this.

The obvious answer is to push the walking code up into the shared Actor base class. But if we do that for everything—walking, melee combat, ranged attacks, inventory management, magic, etc.—we’ll end up with an Actor class that contains damn near the whole game. Super gross.

Instead, we’ll make a classic game architecture decision. We’ll separate deciding what behavior to perform from executing the behavior. In other words, we’ll use the Command pattern. In Hauberk, these are called actions.

The game loop asks each actor to give it an action, then it tells the action to execute itself, like:

void process() {
  var action = actors[_currentActor].getAction();
  action.perform();
  _currentActor = (_currentActor + 1) % actors.length;
}

There are different action classes for each atomic thing an actor in the world can do. There is a WalkAction, OpenDoorAction, EatAction, etc.

This pulls all of that behavior out of the base Actor class. Even better, it separates all of the actions from each other. If you’re adding or changing a thing that actors can do, you can just poke at one isolated little action class. It feels nice and decoupled and it’s easy to add new actions to the game. (As of today, there are nineteen different actions, and I expect to add a bunch more.)

It also, of course, helps us treat monsters and heroes uniformly. Since the Action classes all work on instances of Actor, they can all be used by monsters and heroes alike. (There are some exceptions since heroes have some capabilities monsters don’t have. Right now, monsters don’t have inventory so all of the inventory management actions don’t apply to them.)

Acting at speed

We’ve got a basic loop working now, but our game is a bit too turn-based. Every monster and the hero all proceed in lockstep. You move one step, they all move one step, kind of like the ancient Robots game. You can never outrun or be outrun. See for yourself:

Use arrow keys or iopkl;,./ to move. Press t to teleport.

To fix that, we want actors to move at different speeds.

Of course, this is “speed” in the turn-based sense, not literally moving quicker in realtime. What it means is that a “faster” actor gets to take turns more frequently than other actors. If you’re twice as fast as a green slime, then you’ll get to take two steps for every one it gets.

This mechanic is de rigueur in roguelikes, and the literature is rife with ways to implement it. The system I’m using is almost exactly what Angband uses, because it’s awesome.

It works like this: Every actor has an energy level. Each time the game loop reaches an actor, it grants it a bit of energy. When the actor’s energy reaches a certain threshold, it has enough to take a turn and perform an action. Otherwise, the game loop just moves on to the next actor. It may take several cranks through the game loop before an actor accumulates enough juice to actually take a turn. (And, in fact, for all but the fastest actors, it does.)

When the actor performs an action, that burns energy, and they’re back in the “waiting to get enough energy to go” state. Right now, all actions consume the same amount of energy, which means every action takes the same amount of “time” to perform. I could vary this so that, for example, archers could shoot arrows more frequently than they could swing a sword.

The way speed comes into play is simple: faster actors get more energy each turn. That means they’ll cross the threshold in fewer revolutions of the game loop, so they’ll get to move more often. It’s as simple as that.

The neat thing about it is that by accumulating energy over several turns, you can have actors that move at arbitrary fractions of each other’s speed. You could have an actor that gets five moves for every seven moves another actor gets if you wanted. (Of course, what you’d see during game play is that every now and then the second actor would get a double turn. It’s just that the “every now and then” averages out to 7/5 over time.)

Enough verbiage, let’s see it in action:

Same controls before but the game loop is slowed down so you can watch it parcel out energy.

The > points to the actor whose turn it is. It’s usually stuck waiting on you. After you make a move, you can watch the game loop race around, doling out bits of energy. When an actor’s bar reaches the right edge, it takes a move and the bar resets.

The cool thing about this system is that applies to all actors. Some game engines update the hero separately from monsters in the main game loop, but that makes speed much trickier to handle. By treating the hero as just another actor, the hero can be both slower and faster than other monsters automatically.

The one special thing about heroes

Treating the hero as just another actor is mostly swell, but there is one thing about heroes that makes them unique—they’re controlled by the player. The game loop I showed runs fine as long as actors generate their own actions. That’s true in the case of AI-driven monsters, but the hero can’t see through your glassy computer screen and discern your intentions through cyber-telepathy. It needs user input.

We already have two constraints that make this harder:

Because we want to separate the engine from the user interface, it can’t directly call into input-handling code.
Because the game runs in a browser, it can’t block waiting for user input. The browser don’t play ‘dat. You have to return to the event loop and let it tell you when an event comes in.

When the game loop asks a hero for its action, the hero can’t just stop the game and wait for the player to push a button. Instead, we’ll let the user interface inject input into the game. The input handling code can create an action for the hero ex-nihilo and jam it in the engine’s piehole, like:

void handleInput(Keyboard keyboard) {
  switch (keyboard.lastPressed) {
    case KeyCode.G:
      game.hero.setNextAction(new PickUpAction())
      break;

    case KeyCode.I:         walk(Direction.NW); break;
    case KeyCode.O:         walk(Direction.N); break;
    case KeyCode.P:         walk(Direction.NE); break;
    case KeyCode.K:         walk(Direction.W); break;
    case KeyCode.L:         walk(Direction.NONE); break;
    case KeyCode.SEMICOLON: walk(Direction.E); break;
    case KeyCode.COMMA:     walk(Direction.SW); break;
    case KeyCode.PERIOD:    walk(Direction.S); break;
    case KeyCode.SLASH:     walk(Direction.SE); break;
  }
}

void walk(Direction dir) {
  game.hero.setNextAction(new WalkAction(dir));
}

That call to setNextAction() stuffs the given action into a field in the hero. When the game loop asks the hero what it wants to do, it barfs that back up:

class Hero extends Actor {
  Action _nextAction;

  void setNextAction(Action action) {
    _nextAction = action;
  }

  Action getAction() {
    var action = _nextAction;
    // Only perform it once.
    _nextAction = null;
    return action;
  }

  // Other heroic stuff...
}

(In the actual game, there’s actually a level of indirection here to handle multi-step behaviors like running, but we’ll ignore that here.)

This keeps the engine from reaching out to the user interface and lets the UI pass input to the engine at its leisure. The only problem is what happens when the game engine is told to process the hero’s turn and the UI hasn’t given it an input yet.

To handle that, the loop just checks for the actor failing to cough up an action. When that happens, it bails and returns control back to the user interface:

void process() {
  var action = actors[_currentActor].getAction();

  // Don't advance past the actor if it didn't take a turn.
  if (action == null) return;

  action.perform();
  _currentActor = (_currentActor + 1) % actors.length;
}

If the interface tells the game engine to process but hasn’t given instructions to the hero, the engine does nothing and bounces control back to the UI. Note how setNextAction() can be called at any point in time. This works seamlessly with the speed system without the UI having to be aware of it. It just throws hero actions at the engine and tells it to process. The engine takes care to ensure the simulation only ratchets forward at the right time.

In fact, now that I think about it, if you had multiple player-controlled heroes driven by different inputs, it would automatically handle that too. They could even be coming over a network and the engine won’t care. Groovy.

The way this interacts with the browser’s own event loop is actually a good bit more complex than this when you take into account visual effects, but I think I’ll have to save that for another post. For now, let’s keep our attention on the pristine confines of the engine.

Fat fingers

We’re pretty far along with our game loop now. We’ve got it doing the stuff it needs to do, so we can start looking at making the game more pleasantly usable. Usability means fallibility. People make mistakes, and usability is about accommodating that.

For example, let’s say the player tries to make the hero walk into a wall. Right now, that creates a walk action. When action is processed it prevents the hero from walking through the wall, but it still burns that turn. If he’s trying to run away from a foul beastie, that slip up could cost the hero his life. Some games are OK with that, but I don’t want to be that punishing. Roguelikes are unforgiving enough as it is.

The demos so far work this way now. Go back and try running into a wall. See how your mortal enemies approach in the midst of your ineptitude? That’s what we want to fix. When the player tries to do an action that isn’t possible, we want to make sure we don’t waste a turn on it.

One way to handle that would be to validate the turn in the user interface. In the input handling, we check the tile that the hero wants to walk into and make sure it’s a floor tile. If it isn’t, the user interface shows an error and doesn’t send an action to the game. From the engine’s perspective, it only receives beautiful, correct user actions.

But doing that validation is actually pretty complex. Maybe the hero has an insubstantiation spell and can walk through walls right now. Maybe the tile isn’t floor but is something the hero can tunnel through, but only if they have a shovel in their inventory.

What I’m describing are game mechanics, and game mechanics belong in the engine. In particular, most of them belong in actions. We’ll put the solution to this problem in there too. When an action is processed, we’ll let it return a value indicating success. If it fails, the game loop considers it to have never happened. It’s as simple as:

void process() {
  var action = actors[_currentActor].getAction();
  if (action == null) return;

  var success = action.perform();

  // Don't advance if the action failed.
  if (!success) return;

  _currentActor = (_currentActor + 1) % actors.length;
}

This makes the engine more robust: you can throw arbitrary actions at it and it will handle them gracefully. It also keeps all of the code for a single mechanic—including validation—in one place: in the relevant action. Yay for encapsulation!

Now, try braining yourself against the dungeon’s stone boundary:

Do what I mean, not what I said

Success/failure handles cases where the action the player picked is totally bogus, but sometimes the game can infer out what they were trying to do. For example, if you try to make the hero walk into a closed door instead of using the dedicated “open door” command, odds are pretty good you want to open the damned door. Likewise, if you try to walk into a monster, that’s a good time to consider swinging a sword.

I know this sounds obvious, but you’d be surprised how many roguelikes don’t do this. Improving usability is one of my main goals for my game, so I care about this stuff. I’ve got a pretty simple solution too.

When an action is validating itself, it can fail outright like we saw, but it can also respond with an alternate action. It lets the action say, “no, you really mean this”.

Since the perform() method on Action can return success, failure, or another action, we’ll make a little class to wrap that up:

class ActionResult {
  static final SUCCESS = const ActionResult(true);
  static final FAILURE = const ActionResult(false);

  /// An alternate [Action] that should be performed instead of
  /// the one that failed.
  final Action alternative;

  /// `true` if the [Action] was successful and energy should
  /// be consumed.
  final bool succeeded;

  const ActionResult(this.succeeded)
  : alternative = null;

  const ActionResult.alternate(this.alternative)
  : succeeded = true;
}

When an action is executing, it returns ActionResult.SUCCESS to say everything went fine, ActionResult.FAILURE to say nothing happened, or it can return an ActionResult with .alternate pointing to a new action to perform instead.

The game loop processes that:

void process() {
  var action = actors[_currentActor].getAction();
  if (action == null) return;

  while (true) {
    var result = action.perform();
    if (!result.succeeded) return;
    if (result.alternate == null) break;
    action = result.alternate;
  }

  _currentActor = (_currentActor + 1) % actors.length;
}

We do this in a loop because an alternate may itself return an alternate, so we keep trying until we bottom out on an action that succeeds or fails. This turns out to be a handy feature for a number of things in the full game:

When you use an item, the “use item” action looks up the specific thing the item does (shoot a fireball, teleport, etc.) and returns that as an alternate. When you “use” an equippable item, it returns the “equip” action as an alternate.
If an actor “walks” but in no direction, it becomes a “rest” action which regains a bit of health.
If an actor walks into a door, it returns the “open door” action as an alternate.
If an actor tries to walk into another, it returns the “attack” melee action as an alternate.

The last three are particularly handy because they’re equally applicable to monsters. The monster AI code doesn’t have to check for doors or opponents. Instead it just tries to make the monster walk where it wants to go and the action system handles opening doors to get there and attacking the hero when it reaches them. When the monster can’t figure out where to go, it rests automatically.

Believe me, anything you can do to simplify your AI code is a good idea. Enough jibber-jabber, let’s see it in action:

Notice the little room has a door now. If you try to walk into it, the game loop will go through the walk’s alternate action to do the open. You can close it by pressing “C” when standing next to it.

The wizard is smart enough to open doors too (sorry, troll and slug, no Mensa membership for you) and no changes to the pathfinding AI were needed to enable it. He just tries to walk through the door and miraculously succeeds.

The end… or is it?

Crap, I’ve already burned three thousand words of your attention and there’s still cool stuff to talk about! I’m gonna have to split this into a two parter.

What we have now is a pretty solid (in my opinion, naturally) game loop. Defining behavior in actions lets us treat the player-controlled hero and monsters uniformly. We’ve got a flexible speed system, and it’s all nicely separated from the user interface.

If you’re building a relatively simple turn-based game, this is probably enough.

But, while ASCII-art-based roguelikes aren’t known for their visual spectacle, I want something that with a bit more pizazz than what I described here can dish up. I want the player to see an arrow arc across the room before plunking into the meaty face of a troll. I want fireballs to flare outwards in a ring of death. And when that flame touches incendiary items laying on the ground, those should in turn trigger a cascading conflagration.

The game loop I have in the full game can do that. When I can find the time to carve out a few more standalone demos and hack up some prose, I’ll show you how. In the meantime, the code for the game is here, and the code for the demos in this post are here.

If you want to try out the game, you can play it here.

Dropping Loot

2014-07-05T00:00:00-07:00

I got started hacking on my own roguelike after spending several years avidly playing Angband. Like most projects, I had a few itches I wanted to scratch and it kind of took on a life of its own. One thing that annoyed me about Angband was how monsters dropped loot when they died.

In Angband, any monster that drops stuff can drop pretty much anything. Monsters have a level, and if they drop loot, it just randomly picks any item near the monsters level. This bugged me for two reasons:

It felt flavorless and unrealistic. For example, the description of a novice mage is:
```
The Novice mage  (Red 'p')
=== Num:38  Lev:2  Rar:1  Spd:+0  Hp:6d4  Ac:6  Exp:6
He is leaving behind a trail of dropped spell components.
```
But when you hack one to pieces, he’s as likely to drop a sword as he is anything magical. This makes monsters feel too similar each other: each one is basically a spin on the one giant Wheel Of Loot.
It made it impossible for the player to seek certain items. Let’s say you’ve got a good kit of armor except you really could use some high quality boots. How do you fill in that gap?

In Angband, the answer is “kill a whole ton of stuff and wade through piles of loot you don’t care about”. Especially near the end game where you’re looking for just a few specific pieces of gear, you spend a lot of time just killing dragons and hunting through mountains of loot. 95% of it gets left on the ground.

I wanted to solve this by making different monsters drop different stuff. Kill a warrior, and you should get weapons and armor. Kill a wizard, get wands and scrolls.

Monster-specific loot

My first pass at this was to give each monster its own set of drops. I had already made monsters data-driven. I came up with my own little text format so monsters were defined in text something like:

dwarf miner
    color     = DarkGray
    depth     = 22
    health    = 44
    attacks
        hammers = 15t5
    description
        Covered from head to toe is dust and dirt, you can barely
        make out the form of this weary dwarf.

For each monster (or at least, each one that drops loot), I added a “drops” section, like:

drops
    hammer (50%)

And, behold, kill a dwarven miner and there’s a fifty-fifty chance he’ll drop a hammer. But dwarves like gems too, right? So I added that in too. Only the problem is that there’s a bunch of different kinds of gems, so I ended up with something like:

drops
    hammer (50%)
    amethyst (10%)
    sapphire (7%)
    emerald (4%)
    ruby (3%)
    diamond (2%)
    blue diamond (1%)

It uses the percentages to pick one of the drops. Notice how the better gems are rarer? That’s fine for a miner, which is a pretty weak dwarf, but a dwarf chieftan should tend to drop better gems:

drops
    amethyst (3%)
    sapphire (5%)
    emerald (8%)
    ruby (10%)
    diamond (8%)
    blue diamond (5%)

Now consider that there’s a few different types of hammers too… Things got pretty nasty pretty quickly. It was a huge chore to maintain these giant tuned drop tables for each of potentially hundreds of monsters. Ugh.

The first thing I realized was that there was a lot of overlap between monsters. Dwarven miners and dwarven warriors drop the same gems with the same frequency. So I made “macros” in my drop language. I could define:

(gem)
    hammer (50%)
    amethyst (10%)
    sapphire (7%)
    emerald (4%)
    ruby (3%)
    diamond (2%)
    blue diamond (1%)

Then in dwarven miner and warrior, just add:

drops
    (gem)

The (gem) would then expand to that little weighted table of gems. That worked fine when the probabilities of each gem were the same, but it meant I couldn’t reuse these tables across dwarves of different difficulty. Stronger dwarves drop the same gems but with different probabilities.

So I came up with a different kind of drop that selects from a bunch of child drops based on level. This let me define:

(gem)
    one near level
        amethyst (23)
        sapphire (38)
        emerald (52)
        ruby (62)
        diamond (87)
        blue diamond (95)

The “one near level” part would take the monster’s level into account, plus a bit of randomness, to pick child drop from within it. Now I could reuse (gem) across a range of different difficulties. Of course, I still had to define lots of tables for all of these different “kinds” of drops. There was a table for gems, stones, hammers, daggers, etc. Piles of data.

After all of that, I found it wasn’t fun to play. The problem was that even with these tables, killing a monster was… boring. Every time you killed a drwarf, you knew what you were going to get. You might get a better gem (yay), but nothing surprising.

Simpler Sequences

A few years ago, I ported the game to Dart. When I did, I tried to simplify as many things as I could. I didn’t want to rewrite this whole giant DSL macro language thing. Instead, I noticed that every item seemed to be a member of some sequence. Emeralds are always in a sequence of gems, a hammer is in a sequence with warhammers and mattocks, etc.

Instead of building some complex general-purpose loot DSL and then implementing this logic using that, I just built sequences directly into the loot code. When defining a monster (now we’re in Dart code), you’d list the drops:

breed("goblin peon", lightBrown, 16, [
  attack("stab[s]", 6)
], drop: [
  chanceOf(10, "Spear")
]);

The “Spear” means it tends to drop spears. But it also implicitly means it can drop other items in the same sequence that spears are in. This lets you express both a sequence and how deep into the sequence a monster should tend to drop. To have a stronger monster drop items from the deep end of the sequence, just name an item from that end.

This let me express pretty much what I could before, but it was a hell of a lot simpler. The only limitation is that an item could only be a part of one sequence. In practice, that works OK.

Unfortunately, though, it didn’t solve the boringness problem. I learned the hard way that a huge part of the fun of roguelikes is the “lottery effect”. Every time you kill a monster there should be a small chance of getting something really amazing. That “maybe this one will be the big one” anticipation is one of the key emotions in the game.

With sequences, you could get something good, but never something surprising. Eventually, I hit upon a solution and, so far, I really like it.

Item Groups

Instead of sequences of items, I now have a hierarchy of them. So, for example, a stilleto is:

equipment/weapon/dagger/Stilleto

I’m not generally a fan of hierarchies but I found it surprisingly easy to lump almost every item into one. Items also have a “level”, which describes their relative value. Higher level stuff is better.

When specifying a drop for a monster, I can specify an item name to drop that specific item. But I can also specify the name of part of the path and a level, like:

breed("goblin peon", lightBrown, 16, [
  attack("stab[s]", 6)
], drop: [
  chanceOf(10, "spear:3"),
  chanceOf(5, "healing:2"),
], meander: 2, flags: "few open-doors");

The "spear:3" means “drop something from the ‘spear’ group around level 3”. You don’t have to specify the entire path since path components are unique. So “spear” is equivalent to “equipment/weapon/spear”.

So far, this is pretty much equivalent to the sequences from before. The neat part is what happens when the drop is generated. When picking an item, there’s a small chance that it will walk up the group chain. So “spear:3” will usually drop a spear, but there’s a chance it will drop any weapon (the parent group). There’s an even smaller chance it will drop any piece of equipment (the grandparent).

Any monster has a chance of dropping almost any item, so you have that pleasant anticipation. At the same time, the probabilities are weighted so that each monster still has a unique “feel” to their drops, and you can seek out monsters that are more likely to drop what you want.

When specifying a drop, you can also directly specify one of the parent groups. For example:

breed("goblin warrior", gray, 32, [
  attack("stab[s]", 14)
], drop: [
  chanceOf(20, "equipment:6")
], meander: 1, flags: "protective open-doors");

Here, the "equipment:6" means a goblin may drop any kind of equipment. This makes it really easy to specify monsters that define wide sets of drops. That’s good for high level boss monsters that can serve up an assortment of loot.

At the same time, defining the items is pretty simple. You basically just need to categorize and assign a level for each item. It looks like this:

group(r"\", "equipment/weapon/spear");
weapon("Pointed Stick", 5, brown, "stab[s]", 7);
weapon("Spear", 25, gray, "stab[s]", 12);
weapon("Angon", 35, lightGray, "stab[s]", 16);
weapon("Lance", 45, white, "stab[s]", 24);
weapon("Partisan", 55, darkGray, "stab[s]", 36);

(The numbers after the name are their levels.) Once you do that, all of the rest of the drop behavior falls out naturally. If I add a new item to an existing group, every monster will then start dropping it, with the right probabilities.