By Sam Neubardt, Software Engineer

We recently released Fabric mission control, a new dashboard on fabric.io that shows the pulse of all your apps. In order to build mission control, we needed to query for metrics across multiple apps, which would’ve been very difficult to integrate into our existing REST endpoints. Therefore, we decided to create a new service to handle GraphQL queries (Fin Hopkins went over GraphQL basics and introduced our usage of it in a previous post). To do this we used GraphQL.js (a library that serves as the reference implementation of the GraphQL specification) as our core query resolver. We were incredibly happy with GraphQL.js as it’s flexible and easy to work with, however, little is prescribed in terms of production operation.

Before putting this new tool into production, we needed to solve two common problems that almost all developers face: how can we protect our customer’s data and be confident that abuse will not impact the availability of the site?

For some extra context, we store data across a number of backend systems. Rather than embed access rules in each of them, we decided to consolidate authorization logic in our GraphQL server. Having a single set of access rules for resources reduces the responsibilities of the services that own those resources. The flexibility of GraphQL.js allowed us to integrate these rules with our schema to build a single representation of the available data and the conditions that allow access to it.

Schema basics: GraphQL.js 101

Before we dive into our solution, here’s a refresher on GraphQL.js. GraphQL requires a schema that describes the entities in your system and their relationships with each other. You can define a new type with GraphQL.js using the GraphQLObjectType constructor:

Authorizing access to the GraphQL schema

Now that you’re all caught up on GraphQL.js let’s take a look at how we added authorization to our schema.

A schema’s resolvers are the key abstraction through which its behavior is defined. So we modified the resolve functions in our schema to check authorization. Given this type in our schema:

And since resolvers are just functions we can modify their behavior using function composition, which allows us to abstract this pattern:

We even factored the authorization checker out so it could be reused:

The authorization primitives we wrote are highly reusable which makes it easy to check access across the schema. Using makeAuthorizedResolver ensures that we’re checking authorization consistently across the code base. We’ve found that the best way to prevent errors is to build systems that make it easy to do the right thing. But we wanted to take things a step further; to not only make it easy to check authorization on every field but to require it.

Mandating authorization in GraphQL schemas

Requiring explicit authorization on every field in the schema gives us a strong guarantee that we’re controlling access to our customer’s data. To do this we took advantage of the flexibility of JavaScript to build a new abstraction into our resolve functions. Since functions are objects we can assign new fields to them at runtime — in this case information to mark them as a special “authorization” function:

If a field in a type we define is missing a resolve function tagged as one that checks authorization, then tests will fail. This moves authorization to the forefront of an engineer’s mind as they make changes to the schema.

We even generalized this into a facility for checking arbitrary properties on our resolvers by “tagging” additional fields onto the resolver function and verifying their presence in tests. This is a shared code base (meaning engineers may not have all the necessary context when making changes), and people make mistakes. Verifying our schema in tests helps us ensure that the appropriate guard rails are in place to prevent human error from taking down the site, and has proved to be an effective strategy as we’ve increased the number of developers working on our GraphQL server.

Preventing abuse without sacrificing agility

After implementing authorization in our schema we moved on to figure out how to protect our GraphQL server from abuse. Since it handles queries for arbitrary data served by multiple backend services a bad query could impact other systems that our customers rely on 24/7. Even if no one makes intentionally malicious requests, there’s still the chance that we’d accidentally ship a bad query to production (could you imagine taking down the site due to a preventable mistake?). However, we didn’t want to put up fences that would get in the way of product development. We identified and implemented a couple of key techniques to protect the availability of our system while maintaining the properties that allowed us to work fast with GraphQL.

First, we restricted the total concurrency of a single request. Since our GraphQL server is dispatching requests to other services asynchronously a single query can result in many concurrent downstream requests. We’re primarily concerned with limiting the impact to other services, and preventing requests from getting made felt more straightforward than parsing a query to attempt to infer its impact statically. Large queries have little effect on our GraphQL service itself as it’s a service composition layer with little logic of its own. We can easily scale it out if needed since it’s totally stateless (our backend services require more attention to increase the capacity of their data stores as well).

To enforce this restriction, requests to other services must first acquire a permit on a per-query semaphore before executing. The total concurrency per query is limited by the number of available permits.

We also added a limit to the total execution time of a request. Long running queries are stopped after this timeout elapses. This, along with the concurrency limit, places an upper bound on the concurrent work allowed by a single inbound request — in effect limiting the “breadth” of a query.

We can use these limits to understand the worst case performance of our system. Understanding and limiting the work that could be done by the service allows us to be good clients. These restrictions are all instrumented so we can keep track of their impact and understand if we need to adjust them. We’ve found that these techniques have provided a good degree of safety against overwhelming the server, without getting in the way as we write queries. The instrumentation we added makes it clear during development if a query gets restricted, giving the developer the chance to improve it before going to production.

Success!

Overall we’ve been really pleased working with GraphQL. It was great to quickly integrate GraphQL with other services that allowed us to rapidly iterate on customer features, rather than non-user facing infrastructure. Head over to fabric.io to see it in action!

By Fin Hopkins, Software Engineer

A few weeks ago, we shipped mission control, a new dashboard for Fabric that shows you the most important information across all of your apps on the same page. To build it, we turned to two fairly new technologies, GraphQL and Relay, because of the flexibility they bring to fetching data. Although investigating and productionizing these new technologies added an upfront cost to the project, we found that they paid off over the course of development and we’re excited by their potential as we develop more for Fabric.

What does it take to responsibly dive headfirst into a new technology and framework? Here’s how we went about it:

Bringing together the data

Just from the description of mission control (“all your apps, all their important information”), we knew that we were going to faceplant into the limitations of our existing API architecture. Before mission control, pages on fabric.io were limited to showing one kit’s worth of information for one app at a time, and our RESTful APIs and backend data providers reflected that design. If we were going to show you the most important data for any of your kits, for all of your apps, in under 500 network requests from the browser, we’d need to build something new.

That “new” could have been mission control–specific endpoints in our Ruby on Rails API layer, but, as we’ll see, that would have been forcing an architecture we already weren’t satisfied with to do something it wasn’t suited for. Instead, we took this as an opportunity to vet GraphQL, a technology that, on its surface, was tailor-made to our problem. If this experiment went well, we could see ourselves adopting GraphQL for more features, bringing its benefits to all Fabric teams.

Rendering a page with the information diversity and density we were going for requires making multiple API calls to a variety of backend systems, such as those for Answers, Crashlytics, and Beta. A RESTful public endpoint that combined those sources would actually spend most of its time waiting for those API responses, something our existing Ruby on Rails setup is decidedly poor at. Making the API calls in serial would take far too long, but parallel requests in Rails would require bringing in a concurrency library like EventMachine, and even then we’d still be tying up one of our limited Unicorn processes for the duration of the request.

GraphQL for a better server

GraphQL, developed at Facebook, is a query language for, well, graphs of data. You start with a strongly-typed schema of your data, which is defined principly as objects and their fields. A user Account might have strings for name and email address, but it might also have a “connection” to a list of Apps. Those Apps will have their own fields, like name, icon, or current number of active users. An App might also have a connection back to a list of member Accounts. GraphQL lets you start at a “root” in your schema — say, the current account — and query for the subset of fields you’d like returned. In our case, one query might be for the account’s starred Apps, their names, icons, and the timestamp / value pairs of their daily and monthly active user data.

It’s important to note that the queries come from the client. For mission control, that’s a webapp written in JavaScript. The server is a general engine that can fulfill any query that matches the schema. That flexibility will be very important later on.

By using the reference GraphQL implementation, graphql-js, we would have access to Node’s native concurrency, giving us the “wait fast” behavior we want. Backend API requests could happen in parallel, and running out of Unicorn workers during those waits wouldn’t be a concern. This has since been proven in production: loading the mission control details for an app can approach two dozen internal API requests but still return in only a few hundred milliseconds to the browser.

Relay for a better process

Adopting GraphQL would also give us the opportunity to adopt Relay, another Facebook library that integrates GraphQL and React (we love React). Relay essentially annotates React components with a GraphQL query that describes the data they need for rendering. Defining the query next to the rendering code allows a component to be developed and maintained in isolation, rather than in concert with server-side API changes. That means it’s faster and easier to make changes during development.

We knew we would be rapidly iterating mission control in response to usability studies and beta tester feedback, so anything that reduced what we needed to change to bring in new data would pay off and speed those cycles up. Having new data available with just a tweak to the GraphQL query and a hot reload in the browser changed how we were able to work. We iterated on mission control in mob sessions with our product manager and designer, and Relay let us instantly respond to their feedback and see changes working live.

Adopting graphql-js

Though we’ll have more to say later about specific productionization and hardening steps we took, putting graphql-js on top of our JSON-returning backend services was straightforward. Though any unification of previously-disparate data sources will require some smoothing over of inconsistencies, on the whole we were able to craft a solid schema out of the most important Crashlytics, Answers, and Beta data.

Our best advice to new graphql-js users is to build your schema out slowly, so you can see how your data model will get consumed by a frontend client, and also to lean heavily on DataLoader to dedupe, cache, and adapt multiple one-off requests to use bulk endpoints. We also have become fans of marking any field we can as “not-null,” for reasons we’ll explain in a future post. (Spoiler: you can automatically generate valid Relay data for testing and development.)

Speed bumps in production

Development was going swimmingly against development backends and fixture data, but once we got mission control loaded up in production we saw that it actually took quite a long time to see any apps on the page. Despite all of the parallel fetches and concurrent processing in our GraphQL server, we’d see the loading spinner for sometimes as long as 10 seconds! It was a terrible experience for what was due to become the new fabric.io home page.

In that first iteration, our by-the-book use of Relay and GraphQL led us to an extreme that was ill-suited to our backend’s characteristics. Relay works by gathering up the data requests for all the (transitive) sub-components of a particular root component and combining them into a single query to the backend. That meant that we were making a single request for all of the mission control data at once: metrics, issue counts, velocity alerts, and so on, our first page of 25 apps. To fulfill such a query, our GraphQL server was making scores of requests to our backend services, and, even though they were mostly in parallel, it couldn’t return anything to the browser until they had all completed.

In an ideal world, these requests would all be consistently performant, but in our situation the backend services would occasionally hit a snag that would add a few hundred milliseconds or even a couple of seconds to their response times, delaying the rendering of anything in the browser. As we added to mission control and the GraphQL query grew, the number of internal requests — and the chance of this happening — grew along with it.

An even worse case would happen when one of the dozens of internal requests timed out or had an otherwise transient error. This would add an error to the response — which Relay interprets as a failure — requiring a second attempt from the client while the loader still spun. Of course, there was no guarantee that this next attempt wouldn’t have its own transient errors, leading to a sort of starvation by flakiness.

Though Relay’s mega-query was an efficient use of client-side network connections, it was trading that against the time-to-first-render of the page. Luckily, while Relay and GraphQL were leading us into making these giant, flaky queries, they also provided the tools we needed to fix them.

Smaller batches of data

Our solution was easy enough: split up the queries. We knew we could request the basic information about your apps in a small number of bulk requests to backends that had very predictable performance. With that data, we could start rendering rows for each app and make the page functional for navigation while we brought in the metrics and highlights information. Separating each app’s detailed data into its own request would isolate a slow response to just a single app, not the whole page.

To accomplish this in Relay, we made our AppRow component take “app” and “appDetails” props, each with its own fragment. The former fragment requested the bare-bones data and was referenced by our page’s top-level RootContainer. We then wrapped each AppRow in its own RootContainer to load the “appDetails” fragment (while passing through the “app” prop).

It’s important to note that this refactor of the queries was entirely done client-side. Because of GraphQL’s flexibility, we were free to make the connection / responsiveness tradeoff that was appropriate to the experience we wanted. If we had resigned to make this change with a legacy-style, fixed API, we would have had to split out the code on the server and then maintain two endpoints instead of one. Furthermore, a different frontend client (on mobile, perhaps) wouldn’t have been able to choose its own performance tradeoff without writing yet another endpoint to the same data.

GraphQL is not a magic wand

This was an important lesson for us and one that anyone adopting GraphQL will have to pay attention to: combining data from multiple sources is a big advantage of using GraphQL, but the more distinct backends or internal requests that go into servicing a single query, the more you will have to consider the case where one of them fails.

This experience also caused us to rethink how we wanted to handle retries and backoff. We were hoping to keep all retry logic on the client for operational simplicity, but, if just one internal request had a transient failure, a client-side retry would not only waste all the work that did succeed from the first request, it would be at risk for its own transient failure. Since most of our backend failures and timeouts were transient, rather than consistently slow, we added a retrying wrapper around request-promise in our GraphQL server to re-do those requests. This simple change eliminated 99% of the errors when resolving queries, and the expense of a second or two of extra latency in cases where a retry occurred after a timeout.

Going fast

While we can’t compare our GraphQL / Relay implementation to mission control built with RESTful endpoints on Ruby on Rails, we’re satisfied with both the pace and the experience of developing mission control. We’re confident that “go slow to go fast” paid off. Even if the “go slow” made things take longer than just following our old API pattern, we can now be in “go fast” mode from here on out, as can everyone else doing client development at Fabric. GraphQL has let us build a platform for ourselves that is delightful to build upon.

We took Relay to production with eyes open about its pre-release status. We expected a few rough edges, and we found some: pulling in connections a page at a time was awkward; after the page was open for a while polling for updates, running a mutation (starring an app, for example) could generate a query with 30K of duplicate fields. We were able to write workarounds in both cases (a mixin for paging in connections, and an alternate query tracker that continuously collapses), and we’ve found the maintainers at Facebook are very responsive and helpful when we run into things.

For a project like mission control, GraphQL and Relay were a near-perfect solution, and the cost of building it any other way justified the investment. Now that we have these pieces in place, we see ourselves using them again and again.

Stay tuned for our next post, where Sam Neubardt is going to share more about how we extended graphql-js to make a productionized GraphQL server.