When a page is visited in “subsequent” mode, the window.onload() event is not fired at all because there is no new HTML document being downloaded. We cannot track the performance of these page views if we rely on the window.onload() event.

RUM for single-page applications

We need to automatically detect the start of a “subsequent” visit and detect when the page has been rendered to measure the PLT for both “app launch” and “subsequent” navigations. When considering our options, we looked at the different ways to do so:

• Using Resource Timing API to detect when an AJAX call has been made to identify the start of a “subsequent” visit.

• Using MutationObserver to detect changes in DOM and detecting end of network activity using Resource Timing API.

These approaches were either unreliable or inaccurate for our use case. For example, we make AJAX requests to pre-fetch some data without the user initiating a “subsequent” visit. Also, there might be network activity happening that does not impact user experience. We want PLT to closely match when the user has seen content on the screen, instead of network activity.

We figured out that the most reliable way we could detect these events is by letting the application tell us when they’ve happened by using a simple API. Below is an example of how the application would use the API to mark these events.

The main problem with this approach is that each page would have to write the instrumentation code, which is very tedious. Thankfully, some SPA frameworks, like Ember, have rich lifecycle hooks that we can use to add this instrumentation automatically.

Below is an example of how we automate the instrumentation for most cases in our Ember addon.

We know the start of “app launch” navigation based on navigationStart from Navigation Timing API. We listen to the router’s willTransition event to detect the start of a “subsequent” navigation. We also listen to the router’s didTransition event and add work in the afterRender queue to notify that the page has been rendered for both “app launch” and “subsequent” navigations. We now have the start and end times of page renderings for both “app launch” and “subsequent” navigations and can therefore calculate PLT based off of these times.

With this approach, we have automated the measurement of PLT for many single-page applications within LinkedIn that are built on top of Ember, AngularJS, and Marionette frameworks.

Granular metrics for finding optimization opportunities

If we want to debug any performance issues, we need to collect more granular metrics to breakdown the PLT and understand where the bottlenecks are. Traditional RUM libraries would rely on Navigation Timing API and Resource Timing API to provide these granular metrics about the HTML and resource download times. But single-page applications spend a significant amount of time on the browser doing JavaScript execution after the main HTML/JS/CSS are downloaded. In order to identify the key milestones or phases during this JavaScript execution, we added granular metrics using the User Timing API. Once we added these granular metrics, we were able to construct the waterfall below, which helps us visualize bottlenecks easily.

Waterfall for a typical “app launch” page in a single-page application

 

When we launched our new LinkedIn web application built on Ember, we analyzed the RUM data to find optimization opportunities. Below are a couple of optimizations that we implemented based on RUM data.

Occlusion culling (lazy rendering)

When we looked at the granular metrics charts, we found that about 30% of the page load time was being spent in the “render” phase. The render phase is where we build the DOM for the components on the page after the data is available. We also noticed that the browser’s main thread is not yielded for a paint until the DOM is created for all the components on the page.

If we yield the browser main thread earlier, after the DOM is created for only components that are in the viewport, we can have a much faster user experience, as we do not need to render components below the viewport before the browser does a paint. We refer to this optimization technique, where we avoid/defer rendering of content outside the viewport, as “occlusion culling.”

At a high level, the major performance issue here is that all the components on the page have the same priority and all of them are rendered/painted at the same time. So we developed a solution where we can give different priorities to components based on when they need to be rendered. This is illustrated in the image below:

The first category is the blue-colored components, which are marked as “non occludable.” They have the highest priority and are always rendered on the page irrespective of the viewport height. We keep all the components that fit in the typical viewport in this category.

The second category is the green-colored components, which are marked as “occludable” but have lesser priority and might be rendered incrementally in the next animation frame, depending on these two conditions:

• If the viewport height is longer than a typical viewport, these components will be rendered;

• If the component is configured to pre-render for smooth scrolling experience, even if it is outside the viewport.

The third category is the red-colored components, which are marked as “occludable” and are not rendered (culled) initially because they are outside the viewport. They will be rendered as we scroll and they enter into the viewport.

Using this approach, we incrementally render/paint components and can have a faster First Meaningful Paint of the content in the viewport. Additionally, since we avoid doing the work to render some components that are outside the viewport, we can also have a faster Time To Interactive. When we implemented this optimization for some pages, we noticed that the time spent in render phase improved by 50% at both the 50th and 90th percentile.

Lazy data fetching

After we improved the render phase, we noticed that 20% of the page load time was being spent in the “Transition” phase, according to the granular metrics charts. The transition phase is where we wait for the data to arrive, normalize the data, and then push it to Ember data store. This work is heavily dependent on the amount of data that needs to be processed. If we lazily fetch data that is not needed for First Meaningful Paint, we can reduce the amount of data that needs to be processed and have a faster First Meaningful Paint. So we split the data fetching process into two calls at a high level: one to fetch data needed for First Meaningful Paint and the other to fetch the remaining data needed for the page. We applied this optimization for some pages and noticed that the transition phase improved by up to 40%.

Another advantage of the optimizations described above is that they apply to both “app launch” and “subsequent” navigations because these optimizations are in the time spent on the client browser after the application has been downloaded and booted.

Lessons learned

These are just some of the optimizations which were implemented and validated using RUM data as we ramped our new LinkedIn web application. Some best practices and lessons learned as we went through this journey are:

• Defer work not needed for First Meaningful Paint. This principle yielded us good results in many cases, including the optimizations described above. There were many other optimizations, like lazily loading application code (JS/CSS) not needed for First Meaningful Paint, that also fall under this category.

• Analyze RUM data thoroughly in addition to performance data obtained from synthetic and development environments. There are a wide variety of devices, networks, and users in the real world, and it is not easy to simulate all of them in a synthetic environment. In many cases, the optimizations in the real world were significantly higher than what we noticed in a synthetic environment.

• Always do an A/B experiment to accurately measure the gains from a given optimization. Since we are running many optimization experiments in parallel, we cannot attribute the gains accurately to a specific experiment without putting the optimization behind an A/B experiment. This also helps us in informing how much gain the same experiment would give in other pages and applications. It is also a good idea to continue to do these experiments periodically to see how they are performing, as bottlenecks shift over time.

• To iterate quickly on new ideas, we should validate and refine them in a synthetic performance testing environment. We have a synthetic performance testing framework where we can do many runs of a test in an isolated environment to validate and refine the idea before pushing out to production. This framework has been very useful for iterating quickly on new ideas.

Acknowledgements

We would like to thank David He, Ruixuan Hou, and Krati Ahuja for helping in the design and implementation of our RUM solution for single-page applications. We’d also like to thank Chris Thoburn, Chad Hietala, and Jeba Singh Emmanuel who helped in implementing the optimizations described in this blog. And finally, thanks to Ritesh Maheshwari for his input and feedback while writing this blog post.

What are some of the coolest projects that you and your team have been working on?

Right now, we’re working on building a control plane for Project Falco. It’s exciting to think about how to build a control plane that interacts with LinkedIn’s business specifically, rather than simply buying vendor-driven architectures and trying to shape them to fit our needs. At some point, it would be great to be able to give part of the control plane we’re building to the community so that it could have a positive impact on the way companies build their networks.

A second project I’m working on is improving our edge security. Edge security is always fun because of the community aspect, and the blend of security and routing.

 

What other projects are you involved in outside of working on LinkedIn’s network?

I’ve been involved with the Internet Engineering Task Force (IETF) for about 20 years now. My involvement in the IETF currently includes serving on the Routing Area Directorate, where I review drafts and act as a general “helping hand” for the Routing Area Directors wherever it is needed. I have, in the past, served as a co-chair for working groups in the area of routing protocols security, and participated in the writing and editing of a number of internet standards. Right now I’m also co-chairing two working groups, one for a routing protocol called BABEL, and another for I2RS, the Interface to the Routing Systems. I also serve as a liaison between LinkedIn and the Internet Society (ISOC), which is the governing body for IETF and and the Internet Research Task Force (IRTF), and on the technical advisory board at SDxE, a technical show focused on the software-defined enterprise. I am also active, where possible, in a number of user groups, such as NANOG and LACNOG.

In addition, I’m currently working on a book with Pearson about networking basics. Not another book on network basics, right? The idea here is completely unique in the introductory networking space, and deeply technical. This will be my eleventh book on technology published by Pearson, and will be targeted at the collegiate market.

 

Compared to other places you've worked, how do you like working at LinkedIn?

I’ve worked on a lot of enterprise networks, even at a large scale, and at a number of large vendors. What’s interesting to me about working at LinkedIn is the focus on hyperscale and moving towards engineering our own network solutions rather than only considering vendor options. We’re not just going out and buying a vendor solution; we’re thinking about what we need specifically and then building something that fits our environment.

What are your favorite things to do when you’re not at the office?

I’m currently pursuing a doctorate in apologetics and culture at Southeastern Baptist Theological Seminary. This degree falls under the philosophy department, so I’m reading a lot of works on ontology, epistemology, and the mind/body problem. My specific area of work is around technology and the human person, particularly in the areas of privacy and various forms of social engineering. I’m right at the intersection of philosophy, culture, technology, and Christian belief. I also teach classes for local homeschoolers, in addition to writing the technical books mentioned above. It can be tough to balance everything, but it helps that I don’t watch television, nor am I a heavy user of social media. I don’t really consider LinkedIn to be social media so much as professional media, so LinkedIn is the one exception.

At a high level, here’s what happens:

1. A user or a scheduled job requests a service be profiled on a specific host.

2. This request is passed to a REST-based API server. That server deploys the profiler if necessary (if another service on the host has used the profiler already, we don’t need to re-deploy the profiler but can instead reuse that same profiler) and then signals the profiler to attach to the specified service.

3. The profiler sends its data through a scalable pipeline. After post-processing, it can be publicly viewed on a web-based GUI.

Profiling requests
Profiling requests can come from both users and approved services (e.g., automated testing).

In addition to profiling a service on demand, users can also schedule to profile during regular events, such as traffic shifts. For flexibility, the framework supports these requests coming in from anywhere—as long as they’re authenticated.

REST-based API server
Our REST-based API server serves the scheduled and unscheduled start/stop profiling requests. The server checks whether the profiler is already deployed, and if not, deploys it. Then it tells the profiler to attach to the specified service via a Kafka message.

JVM profiler
The current profiler we have is a sampling profiler based upon JMX. It can connect and profile any JVM-based applications on the same host with no disruption.

The general workflow of the profiler is to collect stack traces and elapsed CPU time for each Java thread via MXBeans at regular intervals, and then to post-process the data in a separate thread. The data is aggregated and periodically sent to Kafka.

Samza
Samza is the scalable streaming platform used within LinkedIn. Even if multiple profilers send data simultaneously to the Kafka topic, Samza can catch up with the produced messages and push them to our remote data store. Samza also provides the benefit of no data loss. We use Samza to pull the profiling data from Kafka and push it to a MySQL database.

GUI/web application
The profiling data is visualized through a web application built with ember.js, flask and flamegraph technology. Flamegraph originates from Brendan Gregg in Netflix; it’s a visualization technique for CPU stack traces. The flamegraph is rendered as interactive SVG, which allows the user to zoom in and zoom out of stack traces easily. The way to read a flamegraph is to read each cell in a given layer as a method call, and the cells in the layer above it as its children method calls. Thus, the highest cells are the deepest method calls.

Performance debugging portal

For each profiling request, the user gets a unique page with:

• Information about the profiled service, plus optional comments;
• Different display modes, such as sample counts or CPU time;
• Widgets to help the user debug performance issues, including top hot leaf methods, leaf-first view, highlight, filter, known issues, thread status, and others;
• A flamegraph for the sampled stack traces.

Highlight or filter

The stack traces can be overwhelming to the user. We need to make them manageable, and allow the user to focus on specific sections. To do this, we introduced highlighting and filtering functions. The user can highlight or filter any searchable string (for example, a method name, package name, line number, or a regex combination of those). The flamegraph will re-render and only list the stack traces containing the string in a method name and highlight the specific matches.

Leaf-first mode

The above diagram shows the leaf-first mode, which reverses the stack traces (callee listed at bottom, caller at top). This helps developers spot the code paths where the majority of time is spent. For example, this service spends a significant portion of time in the leftmost stack traces; this may indicate a bottleneck.

Top hot leaf methods

The GUI lists the top hot leaf methods for the profiling event shown above.

If the user is interested in a certain leaf method, clicking the method name will automatically filter out the stack traces pertinent to that method, as shown above.

Thread states visualization

We provide a visualization of thread states at each sampling point. Users can easily find if there’s any thread contention issue and see which method is blocking.  

Comparison

Users can also compare profiles. In the figure above, the red color shows an increase of CPU samples compared to the baseline, while the blue color shows a decrease.

Automatic integration with performance test frameworks
The API server provides endpoints to allow trusted services to start/stop profiling events automatically. Some services have test frameworks to catch performance regression issues; we’ve integrated with one such internal system already. The internal framework makes profiling requests to ODP during its performance test runs, providing profiling data that helps find performance regressions.

Performance improvements

In the last few months, many performance improvements and fixes have been made across the LinkedIn stack through usage of ODP. Detecting the bad apples, especially in commonly-used library code, has been a huge win through the profiler, leading to reduced latency and/or CPU usage in many services. Although the issues encountered are varied, some common patterns have emerged:

Exception handling:
JVM exception handling can be slow (orders of magnitude slower on occasion)—but in other cases, the effect is negligible.

Reflection:
Using Reflection in the JVM can be slow. This slowness can show up in surprising places; even getting a class name can have an effect.

Logging:
Logging is a very common event in services, and it’s expected to be cheap. However, old logging frameworks, short-lived logger objects, and function evaluations during logging have all been found to sometimes have performance effects.

Summary

In this post, we’ve given an overview of ODP, our infrastructure for on-demand service profiling. We’ve described its architecture, its features, and some of the wins that LinkedIn has already experienced through the use of this framework. We hope that you see the benefit of such a framework, and can apply something similar for your own systems.

Acknowledgments

The development and use of ODP at LinkedIn has been a significant cross-team effort. We wish to thank Brandon Duncan, Josh Hartman, Haiying Wang, Kumar Pasumarthy and Jason Johnson (and their respective teams)... and of course, all the users of the framework.

LinkedIn engineers require tooling that scales really well, and we never stop improving it. Even at a smaller scale, providing great tools for engineers is key to winning business and retaining top talent. This post is about working with code that lives in many separate code repositories, while still being productive and efficient in the process!

Repository and SCM agnostic development at LinkedIn

At LinkedIn, we want to create a development environment where the underlying SCM (source control management) system becomes an implementation detail. We built an abstraction layer on top of the source code repositories called “multiproduct.” Think of a multiproduct layer as a code base that is a unit of building, testing, and releasing some software, such as a web app, a service, or a set of software libraries. Multiproducts at LinkedIn can be checked out, and a new feature can be implemented and submitted for review, without interacting directly with the SCM system. A software project, wrapped with multiproduct abstraction, can have the source code stashed in multiple separate code repositories. At the moment, this is a standard scenario for certain kinds of projects at LinkedIn, like services that keep configs in a separate repository or open source wrapper projects that keep public code in GitHub. Multiproduct abstraction does not completely override interactions with SCM system. On a daily basis, engineers rebase their branches, work with revision history, and interact with SCM system in other ways. More about the multiproduct concept will be covered in future articles. Jens Pillgram-Larsen’s post “Find the Seams” is a great introduction.

LinkedIn's multiproduct abstraction presents a boundary between code bases. The challenge emerges when a code change needs to span this boundary. For example, when a common library developed in a separate multiproduct changes its API, clients that use it (have a binary dependency on the library) need to be updated. So, there must be a change in the library code (multiproduct A) and in the client code (multiproduct B). This generates more overhead: one change in the library requires the new version to be published, and then another change is needed in the client code so that it uses the new version of the library. The result is more independent changes, manual ordering work, more CI builds, and binary version management.

There are, however, benefits of the boundary between multiproducts. Working with software components integrated at a binary level via dependency management enables teams to move faster by making it possible to ship ambitious, incompatible changes without the need to update all consumers at once. In multiproduct, thoughtful design of the public API is a necessary practice, and this positively affects the overall architecture. Each team needs to continually care about dependency management to avoid having a complicated dependency graph, unnecessary dependencies, and dependency cycles. It is more work at first, but it pushes the teams harder to keep the architecture clean and healthy.

LinkedIn engineers collaborating on a project at one of our regular developer bootcamps

Open source partnerships for a better build experience

At LinkedIn, we want it all: a clean architecture of software components integrated at the binary level and the convenience of cross-multiproduct refactoring. In order to deliver this capability, we sponsored a new feature in the Gradle build system called Composite Builds. In addition, the LinkedIn Development Tools team updated our custom IntelliJ IDEA plugin to take advantage of Gradle Composite Builds and deliver a great developer experience.

Gradle is a robust and extensible open source build platform. At LinkedIn, Gradle serves as the core of the build automation framework for all software stacks we support: JVM, Android, iOS, C++, JavaScript/Web, Python, and more. We recently open sourced support for Python with Gradle—try it out and enjoy Gradle’s amazing dependency management in the Python world! Driving automation of major technology stacks at LinkedIn using the same build tool gives engineers a consistent developer experience. It also reduces the maintenance cost of the build infrastructure by reusing our custom LinkedIn Gradle plugins.

At LinkedIn, we leverage our open source partnership with Gradle Inc. to sponsor new features in Gradle. The partnership gives LinkedIn an opportunity to further enhance our internal build automation experience while adding value to the entire Gradle community. The new features become part of the standard Gradle distribution, free and open sourced for the community to use.

The Gradle Composite Builds feature “connects” separate Gradle builds into one single, cohesive build. This means that we can treat separate multiproducts  as a single multiproduct, built with a single Gradle invocation. A critical feature of Composite Builds is the ability to integrate with IntelliJ IDEA—a very popular IDE. We can import separate multiproducts into a single IntelliJ IDEA window. This allows cross-repository IDE experience: very convenient code navigation and cross-multiproduct refactoring.

To maximize the developer experience at LinkedIn, we have added support for cross-multiproduct development into the LinkedIn IntelliJ IDEA plugin. An IDE is the place where engineers spend most of their time, so adding features into the IDE has huge opportunities to boost engineering productivity at a large organization. We recognize this at LinkedIn, so custom IDE integration via plugins is a key component of our engineering tooling roadmap. If you want to see a demo of how cross-multiproduct refactoring is implemented at LinkedIn, check out the first seven minutes of this video from the Gradle meetup LinkedIn hosted in November 2016.

We believe Gradle Composite Builds is an example of a disruptive change LinkedIn can make by partnering with open source companies to improve developer productivity. The added benefit here is not only for those of us at LinkedIn, but also for all organizations that use Gradle. Never before has it been so easy to setup the IDE to work with code from separate code bases!

Bluepill runs tests in parallel using multiple simulators. The main features supported are:

• Running tests in parallel by using multiple simulators.
• Automatically packing tests into groups with similar running time.
• Running tests in headless mode to reduce memory consumption.
• Generating a junit report after each test run.
• Reporting test running stats, including test running speed and environment robustness.
• Retrying when the Simulator hangs or crashes.

Here’s a demo of Bluepill in action:

Using Bluepill

It is quick and easy to start using Bluepill! In a simplified scenario, you just need to run the following command, and Bluepill will kick off four simulators to run your tests in parallel. By the end of the test run, it will generate a report in ./output.

./bluepill -a ./Sample.app -s ./SampleAppTestScheme.xcscheme -o ./output/

Alternatively, you can have a configuration file like the one below:

And run ./bluepill -c config.json

A full list supported options are listed here.

Open sourcing Bluepill

We’re more than happy to announce that Bluepill has been open sourced under the BSD 2-Clause license and that the code is available on Github. Contributions and suggestions are welcome!

Acknowledgements

Bluepill was created by Ashit Gandhi, Jarek Rudzinski, Keqiu Hu, and Oscar Bonilla. It was inspired by parallel iOS test and Facebook’s xctool. The Bluepill icon was created by Maria Iu.

This is the final post of the series “Every Day Is Monday in Operations.” Throughout this series we’ve discussed our challenges, shared our war stories, and walked through the learnings we’ve gained as Operations leaders. You can read the introduction and find links to the rest of the series here.

If Operations fails, so does your company—it is, as you might say, mission-critical. The delicate trust a company holds with its customers can be shattered by a single sustained outage. Just look at one outage from 2012 which cost millions of dollars per minute while a bug was in production. When a major incident occurs, no customer cares about who was at fault, the extenuating circumstances, or how you’re going to do better next time. They leave, wondering why they trusted your company in the first place. In Operations, failure is not an option.  

As we’ve retold our war stories over the course of this series, we can’t help but be reminded of this heated exchange from the movie “Apollo 13.”  It’s a variant on a conversation we found ourselves having during many of our own major incidents: it doesn’t matter if it looks impossible; we must find a way to succeed.

David and I believe that the incredible combination of expectations, difficulty, and risk associated with each outage warrants the guidance we’ve put forward in this series. We’ve learned these lessons the hard way over the years and hope that by sharing them, readers can gain the benefits without the pain.

This, then, is the greatest gift we can give back to our colleagues: our experiences, the lessons learned, and the axioms we’ve developed from our 45 combined years of experience as Operations leaders.

Implementing a culture of reliability

When looking to apply these lessons yourself, there is some good news: you can start anywhere. For the most part, each of the axioms stands on its own, without relying on the others for merit. That said, I highly recommend starting with what gets measured gets fixed. Once you understand the problems, you can begin fixing them.

Whether you are the senior member of your team or new to the industry, these axioms will serve you well. Your tenure experience, or years of service do not matter. By applying these 10 lessons, teams can create a culture of reliability backed by engineers who can think on their feet, are empowered to make the difficult calls, and who can scale their group farther than you can imagine.

Conclusion

Understanding the 10 axioms we’ve presented is simple; implementing them can be a bit more challenging. This isn’t rocket science, but every day is Monday in Operations, and that means you have to overcome constant changes. You are only as good as your lieutenants, and to make site up your highest priority, you’ll need their help. Also, don’t assume that everyone is on the same page when you start to implement these lessons—you need to communicate, communicate, communicate about the goals and processes you want to use to achieve them.

Operations is a team sport, and if you are at a loss for where to start, remember: what gets measured gets fixed. If you can only measure two things, remember that MTTD and MTTR are key. To improve how you handle the site incidents you do discover after measuring, you must attack the problem, not the person. When it’s time to attack the problem and get to the source, look to the code, because the code doesn’t lie.

Finally: do your best and never give up.

To ask us questions about this post or the entire “Every Day is Monday” series, please join the conversation here. We’ll be checking in on the comments throughout business hours, so let us know what you think!

I made friends with a robot with a rainbow afro, because that’s what you do in Japan

 

What are some of the coolest projects that you and your team have been working on?

We have some really exciting changes to the desktop experience and to how we find and get content in front of our members that you’ll be able to see very soon. Those changes include new ways to see extremely tailored content about niche topics. We’ll be the source for professional news, even in deeply technical areas.

 

What other projects are you involved in outside of your work on the editorial team?

In addition to writing content and to scouring the web for good stuff, I also help edit the Engineering Blog you’re reading now. Since I was previously an engineer and am now an editor, it was a pretty natural fit. I also work to help maintain the R&D culture in the New York office, making sure that engineers and product people feel connected to headquarters and that we have similarly-awesome activities to bring us together.

 

What are your favorite kinds of engineering stories to read on the blog?

I like stories that focus on solving a problem and explaining how it was done, with examples. I love reading incident postmortems or reading about infrastructure scaling and growth. One of my recent favorites is the “Every Day is Monday in Operations” series, mainly because I felt like I was sharing in the story and learning about the challenges for the first time.

 

What is the most challenging part of your job?

Figuring out what people want to read now, and—more importantly—will want to read in the future. I try to produce at least one long-form piece a week, and sometimes thinking about what will be the hot topic or what will get engineers talking takes a lot more time than I’d expect!

 

Compared to other places you've worked, how do you like working at LinkedIn?

I like working at LinkedIn because many of the leaders embrace our value of taking intelligent risks—my move to New York, my promotion to management, and my most recent career change to editorial were all intelligent risks that ultimately made LinkedIn more successful. At other places where I’ve worked, I feel like these things would have been more difficult or impossible to have happen, but here it seemed like we all agreed it was a logical move and it was done.

 

What are your favorite things to do when you’re not at the office?

Traveling and travel hacking. I optimize points, miles, signup offers, and all that kind of stuff to take trips. I recently took a month off to take an Alaskan cruise and a trip to Japan, and I didn’t spend nearly as much money as you’d think to do those things. I also have written modules for my home automation system so that I get Telegram messages when things happen at home and so that I can control my house via the bot.

 

What’s something about you not found on your LinkedIn profile?

In keeping with the traveling mention above, I’ve flown enough miles to have flown to the moon (at perigee).

What does this look like? A tree!

At LinkedIn, we’ve already built call trees between different services, which are found inside our data centers. If we apply this concept on other systems like CDN, third party ads, browsers, etc., we get a bigger call tree.

Simplifying client-side waterfall

We use Resource Timing data to build the client-side call tree model. However, the raw waterfall contains many page-level navigation timing metrics, like redirect duration, first byte time, page download time, etc., as well as more than a hundred resource timing entries for all the downloads associated with the page—HTTP calls with varying URLs and resource types. This makes it hard to determine the cause of slowness. Bottlenecks need to be actionable. For example, if profile pictures are generally slow to download, it means our media CDN needs to be investigated, rather than everyone’s profile pictures.

To solve this problem, we came up with bottleneck types that we assigned to each resource/metric in the waterfall in order to get actionable bottlenecks.

Note that we saw a lot of gaps in waterfalls where no network activities happened. After debugging locally, we found that there is a lot of heavy native code execution (js/parsing/rendering) during these gaps. To measure this more precisely, we started using user timing API to measure the key rendering paths so that we get more insight.

Call trees analysis

Once we have the “combined tree,” the next challenge is how to analyze these trees to find the performance bottlenecks of each page. Basically, there are two kinds of calls that hurt performance:

1. Slow calls

2. Sequential/blocking calls

Developers are very sensitive to the latency of individual requests, but sometimes neglect the importance of parallelization of calls. Let’s use two hypothetical pageviews as example. The first one parallelizes the CSS and JS calls with the HTML request, but the second one does not. The latency for all calls is the same, but the page load time has 1,100ms difference! The bottleneck here is that the HTML request is blocking the CSS and JS requests.

BOSS penalizes both slow calls and blocking calls. In the example above, our bottleneck analysis will give the HTML 38.9% for parallelled page view and 72.2% for unparallelled page view. Even if the duration of each call is the same, the blocking HTML call gets more penalties in our analysis and is marked as the bottleneck of the page. The algorithm we use is based on existing service contribution tools, and looks into the call tree between services. Given that client-side data also fits into our generic call tree model well, and we now have a “combined call tree,” we can apply heuristics from service contribution to the client side.

To understand the algorithm that calculates the bottleneck contribution, let’s use a simplified version of page view call tree. The timeline is divided into different segments based on the start and end duration of each call. For each time segment, if there are multiple calls happening in parallel, we will contribute this segment evenly to every call. In the example below, the 90ms segment is divided into 3 pieces and each service gets assigned a 30ms contribution. For the 100ms segment of Call A, there are no other calls in parallel and it gets blamed for the whole time segment, which will result in a very high contribution for Call A. Calls with high contributions will be the top bottlenecks. As a result, Call A gets 195ms, which is 60.9% of the total page load time of 320ms, due to the first 100ms blocking segment plus its long duration.  Call B gets 20.3% and Call C gets 18.8%, since they are well-paralleled with each other.

Analyzing performance bottlenecks at scale

Processing the call tree data is non-trivial. We are getting millions of page view records and each record creates a call tree with hundreds of nodes. Here are the requirements of our data processing system and the solutions we picked:

1. Scalable to massive amount of data: we picked a Kafka + Hadoop solution to handle the data, which has proven to be very successful at LinkedIn.
2. Ability to slice and dice data into different dimensions and aggregated metrics: what is the bottleneck for the slowest 10% of members? What is the bottleneck for members using a 4G Network? What is the bottleneck for members who only visit our website once a week? We picked Apache Pig as the language of choice to perform these tasks in Hadoop.
3. Fast iterations on call tree analysis algorithms: the logic of processing a call tree is complicated and needs to be tuned in multiple iterations. It is impractical to use Pig to handle the logic and test it in Hadoop every time. To solve this, we use UDFs (user defined functions) written in Java/Python to handle the call tree analysis logic and write unit tests for fast iterations.

With the scalable data analysis system available, the next challenge is building an algorithm to analyze the call tree.

UI for bottleneck analysis: putting it all together

On the UI side, we built the following components to assist the analysis:

1. Bar chart and pie charts to highlight top bottlenecks;
2. Stacked trending chart to show how the bottleneck changes over time;
3. A table for easy sorting and look up.

On the same UI, users can also view the latency distribution of page views in a scatter plot and click each point to get the full waterfall of the page.

With this powerful tool, users can easily find the bottlenecks with a few clicks.

Real-world example

Here is a bottleneck analysis we ran for our LinkedIn desktop home page in March 2016. Each type of bottleneck has a contribution ratio, which means “by removing this bottleneck, how much site speed improvement we can have.” Here is the table that lists top bottlenecks and the solutions to fix each.

After finding the bottlenecks, the next step was locating the particular requests/code causing the bottlenecks. We examined a couple of waterfalls and found that a long gap happens frequently after downloading the ads. That indicates that there are heavy JavaScript executions after downloading the ads and the next requests are waiting for the JavaScript to finish execution.

The fix was easy: we just made the ads load in an unblocking way so that images and other things can be loaded at the same. After running A/B testing, it turned out that our home page became 21% faster in page load time. At the same time, we’ve seen a boost in engagement metrics; after deferring ads, users are more engaged with the site.

And in our bottleneck analysis data, the contribution of gaps and ads calls dropped significantly.

 

Conclusion

We have built a bottleneck analysis tool, BOSS, that can process data at scale and produce actionable optimization items. We already have several success stories so far but plan to make additional improvements, including:

1. More focus on server-side analysis. So far, BOSS works well for client-side analysis. We want to do the same on the server side. For example, bottleneck analysis for different API endpoints, service calls across different data centers, etc.

2. Auto suggestion for performance optimization. Currently, the tool is used in a passive mode: analysis is only done when a user visits our tool and wants to do some optimization. Instead, we want to automatically run analysis for every page and send optimization suggestions directly to page owners. In this way, we can bubble up performance issues easily and make improvements early.

Acknowledgements

Thanks to David He and Ritesh Maheshwari for the invaluable input and feedback. Thanks to Toon Sripatanaskul for his awesome service contribution algorithm. Thanks to Swapnil Ghike and Joseph Zemek for their pioneering efforts for this project. And thanks to Oliver Tse for being our avid user and for his valuable feedback. And finally, thanks to Steven Pham and Dylan Harris for their help in developing the BOSS UI.

• Async I/O: To evaluate the performance for asynchronous I/O, we use the asynchronous I/O client of Rest.li in an AsyncStreamTask. This client uses efficient non-blocking HTTP connections and does not require threads waiting for I/O completion. We set task max concurrency to be 1 first, and then increased it to be 3. The results are similar to the previous experiment with thread pool size 10 and then 30, and the resource utilization of CPU and memory is much less, since there are no additional threads waiting for I/O.

Part II: RocksDB

Setup
We use a Samza job that reads/writes to RocksDB. The job keeps count of how many times each input message key has been seen. For each message, it extracts the key, looks it up from the local RocksDB store, and updates the count. The input stream has 8 partitions and is preloaded with 1G messages, with message size around 100 bytes. Each key is a unique number ranging from from 1 to 1G, so each process will have cache miss and access RocksDB. The job uses one container and runs for 30 minutes.

Baseline
The baseline of the performance test processes about 55K messages/second in the legacy single-thread execution model.

Experiment results
Since the task is synchronous, we run the job in the built-in thread pool. The test results are the following:

When thread pool size is 0, all tasks run on the event loop thread, and we see a small overhead of synchronization. When we increase the thread count, the multithreading model performs better and better. In the last experiment, we set the max concurrency per task to 3, and we see both processing rate and CPU usage increase more. During all the tests, the memory heap usage is mostly the same.

Part III: CPU-bound

Setup
Here, we conduct an experiment that compares the single-node peak performance with the previously published results. We used the same test job, which has randomly-generated keys of key space (1M) and let the cache size also be 1M objects to avoid the cache miss after the cache warms up. The job consumes from a topic with 48 partitions. The lookup and updates will happen mostly in cache, which is periodically flushed to RocksDB. The job has 24 containers and runs in a single worker-node Yarn cluster.

Baseline
As mentioned in the published results, Samza can process around 1.1M messages per second on a single node.

Experiment results
When running all tasks in the asynchronous event loop thread (thread pool size is 0), performance is on par with the baseline. After increasing the thread pool size to be 1, the performance degrades to about ⅓. This is caused by the overhead of the thread scheduling (which is about 5 microseconds on average in this host), much longer than the process time. This study shows that for CPU-bound jobs, single-thread execution is still optimal, and the asynchronous event processing model performs on par with the legacy synchronous processing model in this case.

Summary

In practice, we see both performance and efficiency gains in running Samza jobs with the new asynchronous processing API. We expect future improvements in resource utilization in our current Samza jobs.

Acknowledgements

Thanks to the Samza engineering teams at LinkedIn for the invaluable help with the design and implementation of this feature. Thanks to all Samza customers for their feedback on the use cases of remote access in practice.

In this example, the task makes an asynchronous request to some remote service (http://example.com/resource) using Jersey client, and triggers the task callback from the Jersey callback invocation thread. The callback will notify the Samza engine that the message has been processed completely by the task, and the next message will be dispatched for the task to process (assuming task max concurrency is 1).

Event loop

In Samza, the event loop in a container is responsible for running multiple user tasks for consuming/producing messages, windowing, and checkpointing. (For more information on windows and checkpoints in Samza, see this article). To support callback-based asynchronous processing, we implemented a brand new event loop. The following flowchart illustrates how the event loop works for one task:

The event loop works as follows:

1. When a message event (1) comes from the system consumers, Samza will check Cond 1. Internally, Samza keeps a counter of the outstanding callbacks for each task, and invokes 1a task.processAsync() if the counter value is less than the max concurrency allowed. When the user task finishes the asynchronous process, it will trigger callback in the user thread. Then in 1b, the callback will notify the event loop and update the counter.

2. When a window timer event (2) comes, Samza will check Cond2 to make sure there are no outstanding task actions. Then Samza will invoke 2a task.window(). When the user window function completes, the event loop will be notified as in 2b and continue to process future events.

3. When a checkpoint event (3) comes, Samza will do the same check as for a window event (2). Then Samza will invoke 3a checkpoint for the task. When the checkpoint completes in 3b, the event loop will be notified and continue to process future events.

4. Loop 1 to 4 runs until all tasks have outstanding actions (processAsync, window, or checkpoint). Then the loop will block itself until the next task becomes available to handle new events.

Guaranteed semantics

In order to provide useful asynchronous processing in Samza core, we designed the following semantic models and guarantees. With them, the user job is relieved from error-prone synchronization so that it can truly focus on the processing itself.

Parallelism
With the above processing model, Samza provides multiple levels of parallelism. If the callback is invoked synchronously, all tasks will be run in the same event loop thread. In this scenario, the model provides the traditional Samza container-level parallelism. If the callback is invoked asynchronously, Samza supports task-level parallelism. Tasks can process in parallel without blocking each other. This is because the invocation of processAsync in one task doesn’t require waiting for the callback/window/commit from other tasks. Furthermore, Samza supports within-task-level parallelism. This is achieved by allowing multiple processAsync invocations for one task without waiting for the callbacks to complete. The max concurrency, configured by task.max.concurrency, is enforced when scheduling the next processAsync inside the event loop.

It is also straightforward to run the synchronous tasks in parallel with multithreading. Samza ships with a built-in thread pool which makes multithreading work out of the box. Without any code change, the user can simply configure the property job.container.thread.pool.size and then the tasks are executed in parallel. However, asynchronous I/O is the prefered way to access remote data due to superior performance and better scalability for more parallelism.

Ordering
Let’s discuss the ordering guarantees under different scenarios of parallelism. For parallelism on the container or task level, messages are guaranteed to be processed in order. For parallelism within a task, Samza guarantees processAsync will be invoked in order for a task. The processing or completion, however, can go out of order. With this guarantee, users can implement sub-task-level data pipelining with customized ordering and parallelism. For example, users can use a keyed single thread executor pool to have in-order processing per key while processing messages with different keys in parallel.

Checkpointing
Samza guarantees it will only checkpoint the messages that are completely processed. Samza uses a low watermark model for checkpointing. It maintains a queue of the completed callbacks, sorted by the callback sequence number. When checkpointing happens, the max offset of the contiguous callback sequence (the low watermark) will be committed. The head of the queue will advance to the next callback for the future checkpointing.

Memory visibility and happen-before semantics
The following semantics are guaranteed in the above processing model:

• Event processing within Samza is thread-safe. You can safely access your job’s state in the local RocksDB key-value store, write messages, and checkpoint offsets in the task threads. Any other state or code shared between tasks, e.g., global variables or static fields, is not thread-safe if it can be accessed by multiple threads. Samza guarantees the mutual exclusiveness of process, window, and commit so there will be no concurrent modifications among these operations and any state change from one operation will be fully visible to the others.

• Each processAsync is guaranteed to happen before the next invocation of processAsync of the same task. If task max concurrency is 1, the completion of the processing is guaranteed to happen before the next invocation of processAsync of the same task. If task max concurrency is greater than 1, there is no such happens-before constraint.

• window is called when no invocations to processAsync are pending and no new processAsync invocations can be scheduled until it completes. Samza guarantees that all previous processAsync invocations happen before an invocation of window. An invocation of window is guaranteed to happen before any subsequent processAsync invocations. The Samza engine is responsible for ensuring that window is invoked in a timely manner.

• checkpoint is guaranteed to only cover events that are fully processed. It happens only when there are no pending processAsync or window invocations. All preceding invocations happen before checkpointing, and checkpointing happens before all subsequent invocations.

Summary

Samza has made the first attempt to bridge the gap between stream processing and asynchronous I/O for remote data access, which is commonly found in modern application/service asynchronous application/service platforms. Samza provides support for different granularities of parallelism, from the container-level to concurrency within a task. Samza also supports both in-order and out-of-order process. Finally, Samza provides practical semantic guarantees to reduce the user job complexity.

Acknowledgements

Thanks to the Samza engineering teams at LinkedIn for the invaluable help with the design and implementation of this feature. Thanks to all Samza customers for their feedback on the use cases of remote access in practice.