Rollbar

Ruby Exception Handling - Primer for Dealing with Errors in Ruby

Ruby is a popular open source programming language that is highly flexible, and used for everything from basic "hello world" apps to sophisticated, dynamic programs. Whether you've been programming in Ruby for years or you're a complete beginner, tracking down errors in your Ruby app is simple and easy. Let's go through some basic Ruby error handling, and discover how easy it can be to integrate Rollbar into your Ruby app to automatically log and report your exceptions.

Raise and Rescue Exceptions

Ruby's default exception handling is to terminate the program in the event of an exception. That's not really useful when you're trying to build a complex web application for multiple users. Luckily there's a way around this - declaring exception handlers. Exception handlers are blocks of code that are called if an exception occurs in the execution of another block of code in your program. For the most basic Ruby exception handling, you need to know how to Raise and Rescue an exception.

When you Raise an exception, you stop the normal flow of the program, and execute the code that deals with handling an error. This code can either deal with the error in some way, or terminate the program. If you provide a Rescue clause in your error handler, you can choose how to deal with the exception; without it, the program will simply terminate.

Ruby Exception Hierarchy

Ruby has a predefined Exception class, with quite a few subclasses that can be used for basic error handling. The hierarchy of these subclasses is as follows (with commentary):

Fatal: used internally by Ruby when it must exit due to a fatal error. Errors of this class can't be rescued.
NoMemoryError: raised when memory allocation fails.
ScriptError: used when a script is unable to be executed. This can be due to errors in any of the subclasses of ScriptError.
- LoadError: raised if a required file doesn't load.
- NotImplementedError: raised when a feature isn't implemented on the current platform (i.e., trying to call a method that isn't supported by the operating system).
- SyntaxError: occurs when trying to execute code with invalid syntax.
SecurityError: raised when a potentially unsafe operation is attempted.
SignalException: used when a signal is received on a process.
- Interrupt: raised if the interrupt signal is received (i.e., a user pressed Ctrl-C).
StandardError: used for the most common types of exceptions. Usually the subclasses will be raised rather than StandardError itself. However, when a rescue clause without a specific StandardError subclass is executed, StandardError will be raised.
- ArgumentError: occurs when arguments are incorrect (i.e., the wrong number of arguments).
- FiberError: raised if an invalid operation is being attempted on a Fiber (i.e., trying to call a dead Fiber).
- IndexError: used when the index is invalid.
- KeyError: occurs if a specified key is not found.
- StopIteration: raised to stop the iteration.
- IOError: raised during an input/output operation failure.
- EOFError: occurs when reaching the end of the file by some IO operations.
- LocalJumpError: raised if Ruby cannot yield as requested in the code.
- NameError: raised if a name is undefined or invalid.
- NoMethodError: used when a method is called on a receiver that doesn't have the method defined, and the receiver doesn't respond with method_missing.
- RangeError: raised if a numerical value is out of range.
- FloatDomainError: occurs when attempting to convert certain float values to unsupported classes.
- RegexpError: raised if a given regular expression is invalid.
- RuntimeError: used when given an invalid operation. This is a generic error class.
- SystemCallError: raised for low-level, platform-dependent errors. These exceptions look like Errno:xxx.
- ThreadError: occurs if an invalid operation is being attempted on a Thread.
- TypeError: raised when an object is not of the expected type.
- ZeroDivisionError: used if attempting to divide an integer by 0.
SystemExit: raised if the exit method is called to terminate the script.
SystemStackError: occurs during a stack overflow (i.e., in the case of an accidental infinite loop).

Basic Ruby Error Handling

Okay, so now we know what Ruby's exception hierarchy looks like, and why we need to Raise and Rescue our application's exceptions. But how do you actually do this? If you're a beginner or just new to Ruby, this might all sound fantastic in theory, but you need to see some code to actually understand what you're supposed to do.

Here's an example program that handles an exception:

def handle_exception
    puts 'Lalala, our app is running without errors.'
    raise 'Oh no, a error has occurred.'
  rescue
    puts 'Hooray the error has been rescued instead of terminating the program.'
end

handle_exception

Level Up: Ruby + Rollbar = ❤️

So far, so good. We've got our Ruby error handling down, and we're feeling like old pros. :-) But let's say you want to keep track of the errors that are occurring in your application, maybe even get some alerts when a certain number of exceptions are raised. What do you do? You could write some custom code and try to do this all yourself...or you could simply integrate your application with Rollbar and set your error handling, logging, and alerting on auto-pilot.

Getting Started with Rollbar

To get started, you'll need to sign up for a Rollbar account and name and create the new Ruby 'project' you will be monitoring with Rollbar. You get two weeks free when you sign up, so you can see if Rollbar is right for you.

Once you've signed up for an account you'll receive a server-side access token which you'll need to save for later.

Installing the Gem

Installing the Gem is super easy! First, add this line to your application's Gemfile:

gem 'rollbar'

And then execute:

$ bundle install

If you don't use bundler, execute this instead:

$ gem install rollbar

Unless you are using JRuby, we suggest also installing Oj for JSON serialization. Add this line to your Gemfile:

gem 'oj', '~> 2.12.14'

and then run bundle install again.

Basic Ruby Integration

Rollbar isn't dependent on Rack or Rails for most of its functionality, but if you're using Rack or Rails, there is more info on using both of these Ruby frameworks below. In a plain Ruby project, assuming you've installed the Rollbar gem, you just need to do these three easy steps:

Require Rollbar
Configure Rollbar
Send Rollbar data

require 'rollbar'

Rollbar.configure do |config|
  config.access_token = "POST_SERVER_ITEM_ACCESS_TOKEN"
  # Any other configuration settings you might wish to have
end

Rollbar.debug("Running Script")

begin
  run_script ARGV
rescue Exception => e # Never rescue Exception *unless* you reraise in rescue body
  Rollbar.error(e)
  raise e
end

Rollbar.info("Script ran successfully")

See the configuration reference guide on GitHub for further information on more advanced settings and granular configuration settings.

Using Rails

Run the following command from your Rails root:

$ rails generate rollbar POST_SERVER_ITEM_ACCESS_TOKEN

Be sure to replace POST_SERVER_ITEM_ACCESS_TOKEN with your project's post_server_item access token, which you can find in the Rollbar.com interface.

That will create the file config/initializers/rollbar.rb, which initializes Rollbar and holds your access token and other configuration values.

If you want to store your access token outside of your repo, run the same command without arguments and create an environment variable ROLLBAR_ACCESS_TOKEN that holds your server-side access token:

$ rails generate rollbar
$ export ROLLBAR_ACCESS_TOKEN=POST_SERVER_ITEM_ACCESS_TOKEN

For Heroku users:

If you're on Heroku, you can store the access token in your Heroku config:

$ heroku config:add ROLLBAR_ACCESS_TOKEN=POST_SERVER_ITEM_ACCESS_TOKEN

Using Sinatra

Initialize Rollbar with your access token somewhere during startup:

Rollbar.configure do |config|
  config.access_token = 'POST_SERVER_ITEM_ACCESS_TOKEN'
  config.disable_rack_monkey_patch = true
  # other configuration settings
  # ...
end

Then mount the middleware in your app, like:

require 'rollbar/middleware/sinatra'

class MyApp < Sinatra::Base
  use Rollbar::Middleware::Sinatra
  # other middleware/etc
  # ...
end

Using Other Rack Frameworks

Initialize Rollbar with your access token somewhere during startup:

Rollbar.configure do |config|
  config.access_token = 'POST_SERVER_ITEM_ACCESS_TOKEN'
  # other configuration settings
  # ...
end

Be sure to replace POST_SERVER_ITEM_ACCESS_TOKEN with your project's post_server_item access token, which you can find in the Rollbar.com interface.

The gem monkey patches Rack::Builder so Rollbar reports will be sent automatically without any other action. If you prefer to disable the monkey patch, apply this change to your config::

Rollbar.configure do |config|
  config.disable_rack_monkey_patch = true
  # other configuration settings
  # ...
end

If you disabled the Rack::Builder monkey patch or it doesn't work for the Rack framework you are using, then add our Rack middleware to your app:

require 'rollbar/middleware/rack'

use Rollbar::Middleware::Rack

Test Your Installation

If you're not using Rails, you may first need to add the following to your Rakefile:

require 'rollbar/rake_tasks'

You may also need to add an :environment task to your Rakefile if you haven't already defined one. At minimum, this task should call Rollbar.configure() and set your access token.

task :environment do
  Rollbar.configure do |config |
    config.access_token = '...'
  end
end

To confirm that it worked, run:

$ rake rollbar:test

This will raise an exception within a test request; if it works, you'll see a stacktrace in the console, and the exception will appear in the Rollbar dashboard.

Integration with Rollbar.js

The gem has a cool feature - you can configure the gem to enable Rollbar.js on your site to report your JavaScript errors using Rollbar.js.

Rollbar.configure do |config|
  # common gem configuration
  # ...
  config.js_enabled = true
  config.js_options = {
    accessToken: "POST_CLIENT_ITEM_ACCESS_TOKEN",
    captureUncaught: true,
    payload: {
      environment: "production"
    }
  }
end

The Hash passed to #js_options= should have the same available options that you can find in Rollbar.js, using symbols or strings for the keys.

For further information, you'll want to check out the gem docs for advanced ways to use the gem.

Diagnosing and debugging Ruby errors with Rollbar

Now that you have Rollbar integrated into your Ruby application, any errors that occur will be automatically captured and viewable from your dashboard and errors page within Rollbar (screenshot below). You'll be able to easily see what errors are occurring, how often they occur and the full context and analytics into your Ruby applications errors. Rollbar provides a detailed stack trace of every ruby exception allowing you to see exactly what's happening in your application when an error occurs. You can also view the user's browser and OS where the error occurred.

Errors are grouped by cause and frequency so that you can prioritize which ones to work on first. You'll be able to view trends over time. If a certain error spikes in frequency, you'll know about it right away. Below are more features worth noting:

Errors automatically get grouped by root cause.
Tons of context data like stack traces, request params, URL and environment.
Integrate your favorite tools like Slack, GitHub, JIRA, Pivotal Tracker, Trello, PagerDuty.
Tons of context data like stack traces, request params, URL and environment.
View trends by type of error, browser, code deployment, OS, location, users, host.
Mark an item as Resolved, Mute or Assign errors to others on your team to fix.
Keep track of code deployments and correlate them them to new error occurrences.

With Rollbar, you can rest assured that you'll know about any bugs that might make their way into your Ruby projects. If you run into any issues, we're always available to help at support@rollbar.com. Happy hacking!

Better workflows for managing errors

Happy Holidays Rollbar users! 🎁 We're gifting you a handful of updates to a key feature in Rollbar, the Live Error Feed. We recently rolled out some improved fucntionality to the Live Error Feed, most notably the ability to quickly view and assign ownership of errors with your team. Instantly know who's working on what and ensure that urgent production issues don't get overlooked.

Here's an overview of how these updates will help you and your team members better manage errors in Rollbar:

Assign individual errors directly within the feed

Multi-select and assign errors

Filter to view all assigned or unassigned errors

Filter by owner

Quickly Resolve and Mute errors

Easily change severity of errors

Improved look and feel

By streamlining the live error feed, we hope that you and your team will save more time in your day-to-day debugging efforts. We appreciate your feedback. Email support@rollbar.com and let us know if there's anything you think we could do better.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you defeat annoying errors in your applications. :-)

Introducing Error Merging ⎌

We're excited to introduce merging (and un-merging) of errors! Merging errors lets you combine multiple errors into one 'group' for easier management and more accurate metrics. All past and future occurrences of any merged errors will automatically be combined and grouped. Today's merged errors are tomorrow's error groupings. :-)

Merge errors

When you encounter a duplicated error, you'll want to create a new 'group'. Select one or more errors from the same environment in the error Items feed. Slide the toggle in the box above to 'Merge', set appropriate Level, Status, Owner, and Source values, enter a name for the new item, and click 'Merge'. Done, error merge success!

Un-merge errors

Merged the wrong error by mistake? No problem! You can easily un-merge errors in Rollbar.

Search for merged errors

By default, errors which have been merged into a group will no longer appear. To view these errors, you can search for is:member. To only show grouped errors, you can search for is:group.

We hope that you and your team will save more time in your day-to-day debugging efforts with Error Merging. Check out the error merging documentation for more details. We appreciate your feedback. Email support@rollbar.com and let us know your thoughts.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you defeat annoying errors in your applications. :-)

Monitoring Errors in Android Apps

Wed, 09 May 2018 00:00:00 GMT

When developing mobile apps it’s important to monitor errors so that you can understand your user’s experience. You need deeper insight than just a crash report because errors could cause a degraded user experience or a drop in key behavioral metrics. Your team needs to know quickly when there are production problems either with the app itself or with your backend services so you can fix the issue before more customers are impacted.

Rollbar’s Android SDK lets you track and analyze errors that happen in your Android native applications, and even trace problems to backend services and third party APIs. It provides you with a live error feed from your application, including complete stack traces and contextual data to debug errors quickly. We also track the environment the error is coming from (prod or staging), the server that generated the error, and even the user’s session data. You can then quickly assign ownership of errors to your team and track when they are fixed. Learn more about Rollbar’s product features for Android.

Below, you can see that we've created an example app that triggers an exception when the user clicks on a button. The error message is tracked in Rollbar, including a stack trace where you can see the line of code that caused the error. Rollbar captures errors that occur anywhere in the app. You can follow along with our example using the source code on GitHub.

Adding Rollbar in your code

Visit https://rollbar.com and sign up for an account if you haven’t done so yet. Next, create your project and select Other from the list of notifiers. Select the client side access token that is generated for you. You’ll need this to configure Rollbar in the steps below.
Open the app.gradle of your project, and add the following dependencies.

compile('com.rollbar:rollbar-java:1.1.0')
compile('com.rollbar:rollbar-android:1.1.0@aar')

Add the project access token from step 1 in your app manifest file under the application section. You can find the manifest file in your project directory.

<meta-data android:name="com.rollbar.android.ACCESS_TOKEN" android:value="ACCESS_TOKEN" />

The app does not provide you the meta-data section by default, so you need explicitly to add the Rollbar access token. The manifest file looks like this after adding the meta-data section.

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
   package="rollbar.com.rollbarexample">

   <application
       android:allowBackup="true"
       android:icon="@mipmap/ic_launcher"
       android:label="@string/app_name"
       android:roundIcon="@mipmap/ic_launcher_round"
       android:supportsRtl="true"
       android:theme="@style/AppTheme">
       <activity android:name=".MainActivity">
           <intent-filter>
               <action android:name="android.intent.action.MAIN" />
               <category android:name="android.intent.category.LAUNCHER" />
           </intent-filter>
       </activity>

       <meta-data android:name="com.rollbar.android.ACCESS_TOKEN" android:value="YOUR_ACCESS TOKEN" />
   </application>

</manifest>

Open the code for your launcher activity, and then initialize Rollbar in the onCreate method.

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    Rollbar.init(this);   
}

Test Rollbar with an example Android app

To test that it’s working, let’s create an activity that will generate an error message. In the example below, you’ll be able to generate an error by clicking the "Generate an error" button.

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
   xmlns:app="http://schemas.android.com/apk/res-auto"
   xmlns:tools="http://schemas.android.com/tools"
   android:layout_width="match_parent"
   android:layout_height="match_parent"
   tools:context="rollbar.com.rollbarexample.MainActivity">
    <Button
        android:layout_margin="30dp"
        android:layout_centerInParent="true"
        android:text="Generate an error"
        android:id="@+id/clickMe"
        android:layout_width="match_parent"
        android:layout_height="50dp" />
</RelativeLayout>

public class MainActivity extends AppCompatActivity {
   private Button button;
   private Rollbar rollbar;
   @Override
   protected void onCreate(Bundle savedInstanceState) {
       super.onCreate(savedInstanceState);
       setContentView(R.layout.activity_main);

       button=(Button)findViewById(R.id.clickMe);
       Rollbar.init(this);
       button.setOnClickListener(new View.OnClickListener() {
           @Override
           public void onClick(View view) {
             throwError();
           }
       });
   }

   private void generateError(){
       String test=null;
       test.toString(); // This will throw null pointer exception
   }
}

When you click the "Generate an error" button, it will trigger the GenerateError method. In this method, we have added a bug which is throwing a null pointer exception.

Viewing errors in Rollbar

Open Rollbar to see what these errors look like in your account’s item page. The error we just generated should be called java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String java.lang.String.toString()'.

Get more details by clicking on the item. You can now see a traceback showing you the exact source code file, method, and line number that generated the error.

Rollbar also provides a lot of other contextual information that will help you troubleshoot and debug problems faster. Along the row of sub-tabs above you can get extra drill-down information on each individual occurrence and how many times it occurred, the people affected in client-side apps and how many people were impacted, and much more. You can use this to keep an eye on the error rate and drill down to troubleshoot high-impact errors.

It’s pretty easy to get error tracking across your entire app thanks to Rollbar’s Android SDK. It only takes a few minutes to set up, and you will have way more context to track and debug problems faster in the future. You’ll know about errors right away so your users can experience your app the way it was meant to be.

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your web and mobile application errors.

Meet the Rollbar Team - Jesse Gibbs (Head of Product)

Thu, 26 Apr 2018 00:00:00 GMT

We're happy to introduce Jesse Gibbs, our Head of Product, who has been with Rollbar two years this May . In addition to leading the product team, Jesse is the day-to-day product owner for new feature development in the Rollbar web app.

After growing up in Tahoe, Jesse studied Computer Science at UC Davis. Originally majoring in Physics, he took an Intro to CS course because it was a requirement and soon realized he preferred programming to physics. In addition to his BS, Jesse has an MBA from UC Berkeley, which makes him a great fit to lead Rollbar's product development.

After graduation, Jesse worked as a developer on issue-tracking and test automation software and ended up as the Product Manager because no one else was doing it. At first, he was informally making decisions about what to make and how to do it. As the team grew and roles were formalized, he was asked to be the PM. He found he really enjoyed doing product management; he Likes the fact that you get a full sense of what it takes to build a product and sell it. As he said, "You’re talking to customers, engineers, sales people, marketers, and customer support. I like getting the big picture. It uses a lot of different parts of your brain, and scratches the problem solving itch. If you enjoy talking with people then you get a lot of that too. It brings a lot of new challenges."

Jesse enjoys working in companies where developers are the end users, because he can relate to them and likes solving their problems. He also loves that developers will take pieces of a product and build off of it. After a few years of working on products that weren't developer-focused, Jesse was excited to come to Rollbar since he would get to move back into working on products for developers. In addition to being a developer-focused company, Jesse liked the company culture, which he describes as pragmatic, laid back, and friendly. For the future, Jesse is really excited to both improve the product for our core use cases and to expand it to help both our current as well as new types of users to use Rollbar for critical day-to-day work.

Outside of work, Jesse enjoys cycling and spending time with his wife and their dog Charlie(@charliethechiyorkiepm on Instagram).

We are excited that Jesse decided to join us in our mission of making software development easy!

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your web and mobile application errors.

Top 10 errors from 1000+ Ruby on Rails projects (and how to avoid them)

Wed, 18 Apr 2018 00:00:00 GMT

To give back to our community of developers, we looked at our database of thousands of projects and found the top 10 errors in Ruby on Rails projects. We’re going to show you what causes them and how to prevent them from happening. If you avoid these "gotchas," it'll make you a better developer.

Because data is king, we collected, analyzed, and ranked the top 10 Ruby errors from Ruby on Rails applications. Rollbar collects all the errors for each project and summarizes how many times each one occurred. We do this by grouping errors according to fingerprinting. Basically, we group two errors if the second one is just a repeat of the first. This gives users a nice overview instead of an overwhelmingly big dump like you’d see in a log file.

We focused on the errors most likely to affect you and your users. To do this, we ranked errors by the number of projects experiencing them across different companies. We intentionally looked at the number of projects so that high-volume customers wouldn't overwhelm the data set with errors that are not relevant to most readers.

Here are the top 10 Rails errors:

You’ve probably noticed some familiar faces in there already. Let’s dig in and take a look at the errors in a bit more detail to see what might cause them in your production application.

We'll provide example solutions based on Rails 5, but if you’re still using Rails 4 they should point you in the right direction.

1. ActionController::RoutingError

We start with a classic of any web application, the Rails version of the 404 error. An ActionController::RoutingError means that a user has requested a URL that doesn’t exist within your application. Rails will log this and it will look like an error, but for the most part it is not the fault of your application.

It may be caused by incorrect links pointing at or from within your application. It may also be a malicious user or bot testing your application for common weaknesses. If that’s the case, you might find something like this in your logs:

ActionController::RoutingError (No route matches [GET] "/wp-admin"):

There is one common reason you might get an ActionController::RoutingError that is caused by your application and not by errant users: if you deploy your application to Heroku, or any platform that doesn’t allow you to serve static files, then you might find that your CSS and JavaScript doesn’t load. If this is the case, the errors will look like this:

ActionController::RoutingError (No route matches [GET] "/assets/application-eff78fd93759795a7be3aa21209b0bd2.css"):

To fix this and allow Rails to serve static assets you need to add a line to your application’s config/environments/production.rb file:

Rails.application.configure do
  # other config
  config.public_file_server.enabled = true
end

If you aren’t interested in logging 404 errors caused by ActionController::RoutingError then you can avoid them by setting a catch all route and serving the 404 yourself. This method is suggested by the lograge project. To do so, add the following at the bottom of your config/routes.rb file:

Rails.application.routes.draw do
  # all your other routes
  match '*unmatched', to: 'application#route_not_found', via: :all
end

Then add the route_not_found method to your ApplicationController:

class ApplicationController < ActionController::Base
  protect_from_forgery with: :exception
  def route_not_found
    render file: Rails.public_path.join('404.html'), status: :not_found, layout: false
  end
end

Before implementing this, you should consider whether knowing about 404 errors is important to you. You should also keep in mind that any route or engine that is mounted after the application loads won’t be reachable as they will be caught by the catch all route.

2. NoMethodError: undefined method '[]' for nil:NilClass

This means that you are using square bracket notation to read a property from an object, but the object is missing, or nil, and thus it does not support this method. Since we are working with square brackets, it’s likely that we’re digging through hashes or arrays to access properties and something along the way was missing. This could happen when you’re parsing and extracting data from a JSON API or a CSV file, or just getting data from nested parameters in a controller action.

Consider a user submitting address details through a form. You might expect your parameters to look like this:

{ user: { address: { street: '123 Fake Street', town: 'Faketon', postcode: '12345' } } }

You might then access the street by calling params[:user][:address][:street]. If no address was passed then params[:user][:address] would be nil and calling for [:street] would raise a NoMethodError.

You could perform a nil check on each parameter and return early using the && operator, like so:

street = params[:user] && params[:user][:address] && params[:user][:address][:street]

While that will do the job, thankfully there is now a better way to access nested elements in hashes, arrays and event objects like ActionController::Parameters. Since Ruby 2.3, hashes, arrays and ActionController::Parameters have the dig method. dig allows you to provide a path to the object you want to retrieve. If at any stage nil is returned, then dig returns nil without throwing a NoMethodError. To get the street from the parameters above you can call:

street = params.dig(:user, :address, :street)

You won't get any errors from this, though you do need to be aware that street may still be nil.

As an aside, if you are also digging through nested objects using dot notation, you can do this safely in Ruby 2.3 too, using the safe navigation operator. So, rather than calling

street = user.address.street

and getting a NoMethodError: undefined method street for nil:NilClass, you can now call.

street = user&.address&.street

The above will now act the same as using dig. If the address is nil then street will be nil and you will need to handle the nil when you later refer to the street. If all the objects are present, street will be assigned correctly.

While this suppresses errors from being shown to the user, if it still impacts user experience, you might want to create an internal error to track either in your logs or in an error tracking system like Rollbar so you have visibility to fix the problem.

If you are not using Ruby 2.3 or above you can achieve the same as above using the ruby_dig gem and ActiveSupport's try to achieve similar results.

3. ActionController::InvalidAuthenticityToken

Number 3 on our list requires careful consideration as it is related to our application's security. ActionController::InvalidAuthenticityToken will be raised when a POST, PUT, PATCH, or DELETE request is missing or has an incorrect CSRF (Cross Site Request Forgery) token.

CSRF is a potential vulnerability in web applications in which a malicious site makes a request to your application on behalf of an unaware user. If the user is logged in their session cookies will be sent along with the request and the attacker can execute commands as the user.

Rails mitigates CSRF attacks by including a secure token in all forms that is known and verified by the site, but can't be known by a third party. This is performed by the familiar ApplicationController line

class ApplicationController < ActionController::Base
  protect_from_forgery with: :exception
end

So, if your production application is raising ActionController::InvalidAuthenticityToken errors it could mean that an attacker is targeting the users of your site, but the Rails security measures are keeping you safe.

There are other reasons you may be unintentionally receiving this error though.

Ajax

For example, if you are making Ajax requests from your front end, you need to ensure you are including the CSRF token within the request. If you are using jQuery and the built in Rails unobtrusive scripting adapter then this is already handled for you. If you want to handle Ajax another way, say using the Fetch API, you'll need to ensure you include the CSRF token. For either approach, you need to make sure your application layout includes the CSRF meta tag in the <head> of the document:

<%= csrf_meta_tags %>

This prints out a meta tag that looks like this:

<meta name='csrf-token' content='THE-TOKEN'>

When making an Ajax request, read the meta tag content and add it to the headers as the X-CSRF-Token header.

const csrfToken = document.querySelector('[name="csrf-token"]').getAttribute('content');
fetch('/posts', {
  method: 'POST',
  body: JSON.stringify(body),
  headers: {
    'Content-Type': 'application/json'
    'X-CSRF-Token': csrfToken
  }
).then(function(response) {
  // handle response
});

Webhooks/APIs

Sometimes there are valid reasons to turn off the CSRF protection. If you expect to receive incoming POST requests to certain URLs in your application from third parties, you won’t want to block them on the basis of CSRF. You might be in this position if you are building an API for third party developers or if you expect to receive incoming webhooks from a service.

You can turn off CSRF protection, but make sure you are whitelisting the endpoints you know don't need this kind of protection. You can do so in a controller by skipping the authentication:

class ApiController < ApplicationController
  skip_before_action :verify_authenticity_token
end

If you are accepting incoming webhooks, you should be able to verify that the request came from a trusted source in place of verifying the CSRF token.

4. Net::ReadTimeout

The Net::ReadTimeout is raised when it takes Ruby longer to read data from a socket than the read_timeout value, which is 60 seconds by default. This error can be raised if you are using Net::HTTP, open-uri or HTTParty to make HTTP requests.

Notably, this doesn't mean that an error will be thrown if the request itself takes longer than the read_timeout value, just that if a particular read takes longer than the read_timeout. You can read more about Net::HTTP and timeouts from Felipe Philipp.

There are a couple of things we can do to stop getting Net::ReadTimeout errors. Once you understand the HTTP requests that are throwing the error you can try to adjust the read_timeout value to something more sensible. As in the article above, if the server you are making the request to takes a long time to put together a response before sending it all at once, you will want a longer read_timeout value. If the server returns the response in chunks then you will want a shorter read_timeout.

You can set read_timeout by setting a value in seconds on the respective HTTP client you are using:

with Net::HTTP

http = Net::HTTP.new(host, port, read_timout: 10)

with open-uri

open(url, read_timeout: 10)

with HTTParty

HTTParty.get(url, read_timeout: 10)

You can't always trust another server to respond within your expected timeouts. If you can run the HTTP request in a background job with retries, like Sidekiq, that can mitigate the errors from the other server. You will need to handle the case where the server never responds in time though.

If you need to run the HTTP request within a controller action, then you should be rescuing the Net::ReadTimeout error and providing your user with an alternative experience and tracking it in your error monitoring solution. For example:

def show
  @post = Post.find(params[:slug])
  begin
    @comments = HTTParty.get(COMMENTS_SERVER, read_timeout: 10)
  rescue Net::ReadTimeout => e
    @comments = []
  @error_message = "Comments couldn't be retrieved, please try again later."
    Rollbar.error(e);
  end
end

5. ActiveRecord::RecordNotUnique: PG::UniqueViolation

This error message is specifically for PostgreSQL databases, but the ActiveRecord adapters for MySQL and SQLite will throw similar errors. The issue here is that a database table in your application has a unique index on one or more fields and a transaction has been sent to the database that violates that index. This is a hard problem to solve completely, but let's look at the low hanging fruit first.

Imagine you've created a User model and, in the migration, ensured that the user's email address is unique. The migration might look like this:

class CreateUsers < ActiveRecord::Migration[5.1]
  def change
    create_table :users do |t|
      t.string :email
      t.timestamps
    end
    add_index :users, :email, unique: true
  end
end

To avoid most instances of ActiveRecord::RecordNotUnique you should add a uniqueness validation to your User model too.

class User < ApplicationRecord
  validates_uniqueness_of :email
end

Without this validation, all email addresses will be sent to the database when calling User#save and will raise an error if they aren't unique. However, the validation can't guarantee that this won't happen. For a full explanation you should read the concurrency and integrity section of the validates_uniqueness_of documentation. The quick description is that the Rails uniqueness check is prone to race conditions based on the order of operation for multiple requests. Being a race condition, this also makes this error hard to reproduce locally.

To deal with this error requires some context. If the errors are caused by a race condition, that may be because a user has submitted a form twice by mistake. We can try to mitigate that issue with a bit of JavaScript to disable the submit button after the first click. Something a bit like this is a start:

const forms = document.querySelectorAll('form');
Array.from(forms).forEach((form) => {
  form.addEventListener('submit', (event) => {
    const buttons = form.querySelectorAll('button, input[type=submit]')
    Array.from(buttons).forEach((button) => {
      button.setAttribute('disabled', 'disabled');
    });
  });
});

This tip on Coderwall to use ActiveRecord's first_or_create! along with a rescue and retry when the error is raised is a neat workaround. You should continue to log the error with your error monitoring solution so that you maintain visibility on it.

def self.set_flag( user_id, flag )
  # Making sure we only retry 2 times
  tries ||= 2
  flag = UserResourceFlag.where( :user_id => user_id , :flag => flag).first_or_create!
rescue ActiveRecord::RecordNotUnique => e
  Rollbar.error(e)
  retry unless (tries -= 1).zero?
end

ActiveRecord::RecordNotUnique might seem like an edge case, but it's here at number 5 in this top 10, so it is definitely worth considering with regard to your user experience.

6. NoMethodError: undefined method 'id' for nil:NilClass

NoMethodError appears again, though this time with a different explanatory message. This error usually sneaks up around the create action for an object with a relation. The happy path—creating the object successfully—usually works, but this error pops up when validations fail. Let's take a look at an example.

Here's a controller with actions to create an application for a course.

class CourseApplicationsController < ApplicationController
  def new
    @course_application = CourseApplication.new
    @course = Course.find(params[:course_id])
  end
  def create
    @course_application = CourseApplication.new(course_application_params)
    if @course_application.save
      redirect_to @course_application, notice: 'Application submitted'
    else
      render :new
    end
  end
  private
  def course_application_params
    params.require(:course_application).permit(:name, :email, :course_id)
  end
end

The form in the new template looks a bit like this:

<%= form_for [@course, @course_application] do |ca| %>
  <%# rest of the form %>
<% end %>

The problem here is when you call render :new from the create action, the @course instance variable wasn't set. You need to ensure that all the objects the new template needs are initialised in the create action as well. To fix this error, we'd update the create action to look like this:

  def create
    @course_application = CourseApplication.new(course_application_params)
    if @course_application.save
      redirect_to @course_application, notice: 'Application submitted'
    else
      @course = Course.find(params[:course_id])
      render :new
    end
  end

Check out this article if you are interested in learning more about the problems with nil in Rails and how to avoid them.

7. ActionController::ParameterMissing

This error is part of the Rails strong parameters implementation. It does not manifest as a 500 error though—it is rescued by ActionController::Base and returned as a 400 Bad Request.

The full error might look like this:

ActionController::ParameterMissing: param is missing or the value is empty: user

This will be accompanied by a controller that might look a bit like this:

class UsersController < ApplicationController
  def create
    @user = User.new(user_params)
    if @user.save
      redirect_to @user
    else
      render :new
    end
  end
  private
  def user_params
    params.require(:user).permit(:name, :email)
  end
end

The params.require(:user) means that if user_params is called and params does not have a :user key or params[:user] is empty, ActionController::ParameterMissing will be raised.

If you are building an application to be used via a web front end and you have built a form to correctly post the user parameters to this action, then a missing user parameter probably means someone is messing with your application. If that is the case, a 400 Bad Request response is likely the best response as you don't need to cater to potentially malicious users.

If your application is providing an API, then 400 Bad Request is also an appropriate response to a missing parameter.

8. ActionView::Template::Error: undefined local variable or method

This is our only ActionView error in the top 10 and that's a good sign. The less work the views have to do to render templates the better. Less work leads to fewer errors. We’re still left with this error though, in which a variable or method you expect to exist simply doesn't.

This crops up most commonly in partials, probably due to the many different ways you can include a partial with local variables on a page. If you have a partial called _post.html.erb that contains a blog post template and an instance variable @post set in your controller, then you can render the partial like this:

<%= render @post %>

<%= render 'post', post: @post %>

<%= render partial: 'post', locals: { post: @post } %>

Rails likes to give us plenty of options to work with, but the second and third options here are where confusion can creep in. Trying to render a partial like:

<%= render 'post', locals: { post: @post } %>

<%= render partial: 'post', post: @post %>

will leave you with an undefined local variable or method. To avoid this, stay consistent and always render partials with the explicit partial syntax, expressing the local variables in a locals hash:

<%= render partial: 'post', locals: { post: @post } %>

There is one other place you can slip up with local variables in partials. If you only sometimes pass a variable to a partial, testing for that variable is different within a partial to regular Ruby code. If, for example, you update the post partial above to take a local variable that tells you whether to show a header image in the partial, you would render the partial like so:

<%= render partial: 'post', locals: { post: @post, show_header_image: true } %>

Then the partial itself might look like this:

<h1><%= @post.title %></h1>
<%= image_tag(@post.header_image) if show_header_image %>
<!-- and so on -->

This will work fine when you pass the show_header_image local variable, but when you call

<%= render partial: 'post', locals: { post: @post } %>

it will fail with an undefined local variable. To test for the existence of a local variable inside a partial, you should check whether it is defined before you use it.

<%= image_tag(@post.header_image) if defined?(show_header_image) && show_header_image %>

Even better though, there is a hash called local_assigns within a partial that we can use instead.

<%= image_tag(@post.header_image) if local_assigns[:show_header_image] %>

For variables that aren't booleans, we can use other hash methods like fetch to handle this gracefully. Using show_header_image as an example, this scenario would also work:

<%= image_tag(@post.header_image) if local_assigns.fetch(:show_header_image, false) %>

Overall, watch out when you are passing variables to partials!

9. ActionController::UnknownFormat

This error, like the ActionController::InvalidAuthenticityToken, is one that could be caused by careless or malicious users rather than your application. If you've built an application in which the actions respond with HTML templates and someone requests the JSON version of the page, you will find this error in your logs, looking a bit like this:

ActionController::UnknownFormat (BlogPostsController#index is missing a template for this request format and variant.
request.formats: ["application/json"]
request.variant: []):

The user will receive a 406 Not Acceptable response. In this case they’ll see this error because you haven't defined a template for this response. This is a reasonable response, since if you don't want to return JSON, their request was not acceptable.

You may, however, have built your Rails application to respond to regular HTML requests and more API-like JSON requests in the same controller. Once you start doing this, you define the formats you do want to respond to and any formats that fall outside of that will also cause an ActionController::UnknownFormat, returning a 406 status. Let’s say you have a blog posts index that looks like:

class BlogPostsController < ApplicationController
  def index
    respond_to do |format|
      format.html { render :index }
    end
  end
end

Making a request for the JSON would result in the 406 response and your logs would show this less expressive error:

ActionController::UnknownFormat (ActionController::UnknownFormat):

The error this time doesn't complain about a lack of a template—it’s an intentional error because you have defined the only format to respond to is HTML. What if this is unintentional though?

It’s common to miss a format in a response that you intend to support. Consider an action in which you want to respond to HTML and JSON requests when creating a blog post, so that your page can support an Ajax request. It might look like this:

class BlogPostsController < ApplicationController
  def create
    @blog_post = BlogPost.new(blog_post_params)
    respond_to do |format|
      if @blog_post.save
        format.html { redirect blog_post_path(@blog_post) }
        format.json { render json: @blog_post.to_json }
      else
        render :new
      end
    end
  end
end

The error here is raised in the case of the blog post failing validations and not saving. Within the respond_to block, you need to call render within the scope of the format blocks. Rewriting this to accommodate for failure would look like:

class BlogPostsController < ApplicationController
  def create
    @blog_post = BlogPost.new(blog_post_params)
    respond_to do |format|
      if @blog_post.save
        format.html { redirect blog_post_path(@blog_post) }
        format.json { render json: @blog_post.to_json }
      else
        format.html { render :new }
        format.json { render json: @blog_post.errors.to_json }
      end
    end
  end
end

Now all of the formats are covered and there won't be any more unintentional ActionController::UnknownFormat exceptions.

10. StandardError: An error has occurred, this and all later migrations canceled

This last item on our top 10 disappoints me slightly. StandardError is the base error class that all other errors should inherit from, so using it here makes the error feel very generic, when in reality it is an error that has happened during a database migration. I would prefer to see this error as a descendent of the ActiveRecord::MigrationError. But I digress…

There are a number of things that can cause a migration to fail. Your migrations may have gotten out of sync with your actual production database, for example. In that case, you're going to have to go digging around to find out what has happened and fix it.

There is one thing that should be covered here though: data migrations.

If you need to add or calculate some data for all the objects in a table you might think that a data migration is a good idea. As an example, if you wanted to add a full name field to a user model that included their first and last name (not a likely change, but good enough for a simple example), you might write a migration like this:

class AddFullNameToUser < ActiveRecord::Migration
  def up
    add_column :users, :full_name, :string
    User.find_each do |user|
      user.full_name = "#{user.first_name} #{user.last_name}"
      user.save!
    end
  end
  def down
    remove_column :users, :full_name
  end
end

There are a lot of problems with this scenario. If there is a user with corrupt data in y x xour set, the user.save! command will throw an error and cancel the migration. Secondly, in production you may have a lot of users, which means the database would take a long time to migrate, possibly keeping your application offline for the entire time. Finally, as your application changes over time, you might remove or rename the User model, which would cause this migration to fail. Some advice suggests that you define a User model within the migration to avoid this. For even greater safety, Elle Meredith advises us to avoid data migrations within ActiveRecord migrations completely and build out temporary data migration tasks instead.

Changing data outside of the migration ensures you do a few things. Most importantly, it makes you consider how your model works if the data is not present. In our full name example, you would likely define an accessor for the full_name property that could respond if the data was available. If it’s not, then build the full name by concatenating the constituent parts.

class User < ApplicationRecord
  def full_name
    @full_name || "#{first_name} #{last_name}"
  end
end

Running a data migration as a separate task also means the deploy no longer relies on this data changing across the production database. Elle's article has more reasons why this works better and includes best practices on writing the task as well.

Conclusion

The most popular Rails errors can come from anywhere within the application. In this article we've seen common errors manifested in the model, the view, and the controller. Some of the errors aren't necessarily anything to worry about and are just protecting your application. Others should be caught as soon as possible and stamped out.

Nevertheless, it’s good to track how often these errors happen. This leads to better visibility into problems that are affecting your users or application security, so you can fix them quickly. Otherwise, these error messages will be shown to the user, but the engineering and product management teams will have no idea until users complain to support.

We hope you learned something new and are better equipped to avoid these errors in the future. However, even with the best practices, unexpected errors do pop up in production. It's important to have visibility into errors that affect your users, and to have good tools to solve them quickly.

Rollbar gives you visibility to production Ruby errors, which offers more context to solve them quickly, including form validation errors, person tracking, boot errors and more. Check out Rollbar’s full list of features and Ruby SDK documentation.

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of impactful Rails errors. :-)

5 Surprising and Sneaky Ways Errors Impact Your Business

Mon, 16 Apr 2018 00:00:00 GMT

Every app is imperfect and generates errors, but do you know how errors affect your users and business? When you look at logs for a large application, you might see thousands of errors per day and not know where to start. Even worse are errors that are not tracked by your application. They could be buttons that stop working, input that isn’t saved, or any number of other issues. Without proper error tracking, you are flying blind to these user experience problems.

Unfortunately, developers who are deep in the code are not expected to understand how these errors impact users, and ultimately, their company’s finances. Even if they know of an error that should be fixed, it can be difficult to articulate why they should be fixed instead of developing a new feature. Here are five ways developers can advocate to management about the importance of fixing errors.

User Experience

Customers don't report the vast majority of errors they encounter in your product or website. That means you could miss a large percentage of problems if you aren’t tracking and addressing them proactively. If a customer experiences a critical error during a trial period or onboarding you might lose that customer before they even get started. Established customers may be more understanding, but repeated errors over time can lead to death from a thousand cuts. Knowing who was affected by a problem and proactively reaching out can turn a frustrated user into a devoted one.

Clubhouse co-founder and CTO Andrew Childs explains, "We not only fix the error, but we contact the user directly and say, ‘We noticed that you’re trying to do this one thing... It’s fixed now.’ It lets us be proactive."

We not only fix the error, but we contact the user directly and say, ‘We noticed that you’re trying to do this one thing... It’s fixed now.’ It lets us be proactive.

NPS & CAC

Customer satisfaction as measured by Net Promoter Score (NPS) lets you know whether customers or prospects are happy with your apps and likely to refer friends. Word of mouth from people that customers trust is the best way for them to learn about your product.

A low score could be due to repeated frustrations with your product. Tools like Wootric can survey your users to determine your score and collect feedback on why they scored you that way. If technical difficulties or their downstream effects are lowering your score, you may benefit from an error monitoring solution like Rollbar.

Furthermore, your net promoter score (NPS) can affect your customer acquisition cost (CAC) and even impact your ability to raise capital. If the cost of acquiring customers is too high because they are frustrated with your product, your business may have a harder time raising capital to grow. It's better to invest in a great user experience early on so that you can grow your business more profitably.

Revenue Loss or Churn

In transactional businesses like online stores, errors directly impact your revenue because you don’t get paid for failed transactions or purchases. In subscription businesses, repeated dissatisfaction and problems can result in downgrades or even cancellation.

According to Baymard Institute, visitors abandon shopping cards 20% of the time because of website errors or crashes. In the worst cases, these errors aren’t obvious to the customer and it appears as if everything worked as expected. A customer would later check on their order to find it never even went through. Similarly, errors in order entry or processing can result in the customer getting billed incorrectly or being shipped the incorrect products. Customer Support has to step in to fix the problem, and you may have to go an extra mile to win that customer back.

An error monitoring solution can give you an early indicator that something is wrong so your team can fix it before more customers are impacted. It's especially important to keep an eye on revenue impact after deployments and infrastructure changes.

Development Velocity

If your application is brittle, the team may not feel comfortable deploying frequently, let alone continuously. This means you aren’t able to ship fixes or new features as quickly. Rob Zuber, CTO of CircleCI explains, "Without Rollbar giving us visibility into exceptions in production, we just wouldn’t be confident and we’d ship more slowly."

Additionally, if errors are too difficult or time consuming to investigate, your team may not want to devote engineering time and effort to address them quickly. If errors go ignored for too long, they can result in technical debt, which must be paid when you add new features, slowing your ability to react to the market changes.

While it may be counterintuitive, the best way to help your team develop features faster is to reduce the time they spend fixing problems. If these errors are fixed when the feature is first created, there will be fewer problems to deal with down the road for both your customers and your engineering team.

Employee Morale and Turnover

Repeated errors can affect employee morale when engineers have to work extra hours to fix problems. A few after hours work sessions may not cause any long term damage, but time away from home and family can mount up and make the work less enticing. When errors are difficult to diagnose and resolve, employees and their managers can feel unappreciated because they aren’t being successful in their jobs and the results eventually show in performance reviews and employee turnover.

Software developer Michael Lynch left his job at Google after being continuously disappointed by doing important work that didn’t gain him recognition he was promised, saying, "The problem, as I discovered at promotion time, was that none of this was quantifiable. I couldn’t prove that anything I did had a positive impact on Google." Recruiting new employees can cost employers tens of thousands of dollars, plus months of onboarding before they’re fully up to speed. Investing in a good development process is a cost-effective way to attract and retain great developers.

The problem, as I discovered at promotion time, was that none of this was quantifiable. I couldn’t prove that anything I did had a positive impact on Google.

As Rob Zuber, CTO of CircleCI explains, "There are so many emotional pains that developers and operators have from all kinds of hideous errors they have shipped. What if you could make that go away? That’s what Rollbar does."

There are so many emotional pains that developers and operators have from all kinds of hideous errors they have shipped. What if you could make that go away? That’s what Rollbar does.

Using The Right Tools

Each of the situations above can be addressed by having a good understanding of how errors impact your customers. It’s then necessary to deliver those insights to engineering teams quickly and with enough detail so they can address them with minimum guesswork.

Rollbar presents a clear overview of the errors impacting your customers so you can quickly prioritize fixes and retain those customers. In the screenshot below, you can see a list of the top errors from your app, how many times each happened, how many users are affected, and more. Your engineering team will be able to identify and focus efforts on the most important errors first.

It also gives you extra context to identify what caused each error so you can fix it fast. When your engineering team spends less time fixing problems, they have more time to develop new features.

Rollbar allows us to go from alerting to impact analysis and resolution in a matter of minutes. Without it, we would be flying blind"

Arnaud Ferreri, Director of Engineering Instacart.

With Rollbar, you’ll have better visibility into customer experience which can help you satisfy and retain more customers. Your engineering team will be able to fix problems faster, leaving more time to enhance your product and accelerate your business.

Take a deeper look at how Rollbar's product helps developers and what its customers have to say about the product. Then, sign up for a free trial and get started in minutes. You owe it to your customers, your team, and your business.

If you haven’t already, sign up for a 14-day free trial of Rollbar and stop flying blind in production.

Meet the Rollbar Team - David Karapetyan (Engineering)

Sun, 15 Apr 2018 00:00:00 GMT

Originally from Armenia, David Karapetyan is a software engineer who enjoys improving and streamlining systems. Since joining Rollbar in January of this year, David has been working to make things simpler, faster, and cheaper for our backend systems.

David was born in Yerevan, Armenia, and moved to Burbank, California when he was 12. After high school, he went to UC Davis, where he majored in Math. He continued on to get his M.A. in Math at USC, and was working on a Ph.D. in Math as well, but decided to leave the program. As he puts it, he "realized the economics didn't make sense" and decided to become a programmer.

He is a self-taught programmer, and learned how to program with Ruby and Smalltalk. David's first job in tech was writing test software for a solar power plant. Starting with that job and every job since, David has gravitated towards working in DevOps. At Rollbar, his main focus is DevOps, working to instrument and simplify things in a way as to make them accessible to everyone in the company so that he's not a bottleneck. One of the things he's looking forward to working on here is making some of the backend servers dynamically scalable based on load patterns like incoming event rates and CPU load. Ultimately, he says, the goal is to automate himself out of work.

David describes the culture here at Rollbar as "friendly and mature at the same time, which is a rare combination. People in general also seem to have lives outside of work and programming which is always nice."

We're glad to have David on the Rollbar team!

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your web and mobile application errors.

Under-the-hood with Scout: a look at a New Relic alternative

Thu, 05 Apr 2018 00:00:00 GMT

This is a guest post by Derek Haynes from Scout, an APM that integrates with Rollbar.

When New Relic launched ten years ago, web applications had a tendency to fail hard and in more obvious ways:

Today, it's easier to build resilient apps, but they fail in more complex, unique, and subtle ways. These issues are time-consuming to track down. While several niche New Relic alternatives have appeared, they've focused on a lighter feature set versus solving these increasingly hard performance problems.

Unlike existing "New Relic Lite" alternatives, Scout approaches performance monitoring much like Rollbar's best-of-breed error monitoring. Scout is designed to drastically reducing the investigation time for these complex performance issues.

Let's investigate a modern-day performance issue with Scout and compare the experience to New Relic.

A sudden increase in response time

You hop into Scout and see this chart:

There's a clear increase in overall response time for this Rails app, largely from time spent in the Controller layer. To understand the incident, we need to answer the following:

Is this isolated to a single web endpoint or across the app?
What triggered this problem?
What area of code is responsible for this bottleneck?
Which developer on our team is best equipped to fix this?

Let's start with the first question.

Is this isolated to a single web endpoint or across the app?

Around 60 endpoints received traffic during this incident. Are response times increasing across many endpoints or just one? In New Relic, you'd need to navigate to the "transactions" area, click on each transaction, and identify which endpoints match the trend. This can be a laborious process.

With Scout there's no need to manually click on dozens - or sometimes hundreds - of different endpoints and inspect charts.

Click-and-drag over the spike in response time to reveal additional details:

Scout analyzes the response time of each endpoint for you, identifying those with the greatest latency over the highlighted timeframe. There's only one endpoint that mirrors the overall response time trend - AlertsController#list.

We've dramatically reduced the time to identify the scope of the impact with Scout. This issue is limited to one endpoint.

What triggered this problem?

Clicking on the AlertsController#list link, we can see that there's a large spread in the response time of transaction traces collected by Scout:

It's harder to identify the dramatic spread in response time with New Relic as only traces for slow requests are captured:

Do these very slow requests share something in common?

We can explore the transaction traces collected by Scout via the Crossfilter feature. This lets us dynamically filter transaction traces across many dimensions (ex: the session's user email, IP, request path, etc). Let's inspect just the slowest AlertsController#list transaction traces:

We can see all of the slow requests are coming from the same request path and user email. We've identified the needle-in-the-haystack conditions triggering the incident. This quick inspection of trace data isn't possible with New Relic.

Now that we know the triggers, what section of the code is the root cause?

What area of code is responsible for this bottleneck?

When you inspect a transaction trace with New Relic, you see this:

Almost all of the time is spent in your custom code. You could guess where to add custom instrumentation, but that's a long process with a slow feedback loop. Local profiling may not help if you don't have the ability to reproduce the production dataset.

With Scout, you gain visibility into your application code without custom instrumentation. ScoutProf, a production-safe profiler, automatically analyzes your custom code:

ScoutProf identified two Ruby bottlenecks. Additionally, Scout identified a database query that returned nearly 21k rows (this information isn't available in New Relic).

We're almost there. Your team is composed of several developers. You aren't familiar with this code and are unlikely to be efficient at addressing this issue.

Which developer on our team is best equipped to fix this?

Scout integrates with GitHub, overlaying git blame output on top of application code. With that, we can identify the developer who most likely knows about this bottleneck:

It looks like we're seeing these very slow request because current_app has far more alerts than other apps. Since we're using Ruby to filter database rows, this is slow. The solution? Move the filtering conditions to the SQL query.

Language support?

Scout supports a growing number of languages. Today, Scout has performance monitoring agents for Ruby / Rails, Python, and Elixir.

Pricing?

In most cases, Scout is more affordable than New Relic. You can easily tune the volume of data you send to Scout for analysis by sampling or ignoring some transactions.

TL;DR

New Relic was built in the time of large-scale, major outages. Scout takes Rollbar's best-of-breed approach to today's softer, but harder to investigate performance issues.

Resources

If you haven’t already, sign up for a 14-day free trial of Rollbar and stop flying blind in production.

Meet the Rollbar Team - Ivan Gomez (Sales)

Thu, 22 Mar 2018 00:00:00 GMT

Ivan Gomez is a people person. Since he joined us in May of 2016, Ivan has been an integral part of the team, and loves helping our users get the most value out of Rollbar as possible.

Ivan was born in Ventura, CA, and lived most of his life in Silverdale, WA. He actually went to the same high school as one of Rollbar's cofounders, Brian Rue, although they didn't know each other. After high school, Ivan went to the University of San Francisco and studied Economics. Upon graduation, Ivan worked in Seattle, WA, at Avalara, a tax SaaS company. This got him interested in selling software. Eventually, Ivan ended up in Boston, interviewing for other sales positions, when he saw that Rollbar was hiring.

Ivan joined Rollbar because "we operate in an exciting space that is growing really fast, and we have an opportunity to define how people make software. It’s not overly saturated." He really enjoys educating as part of the sales process, and says it feels better than just transactional sales. Ivan has learned a lot from customer questions about Rollbar, as well as going to developer conferences.

Since Ivan joined Rollbar, he’s taken a coding course in HTML/CSS, JavaScript, and Ruby, which he has said has been very interesting and helpful. He’s always been techy, and he also loves talking to people and focusing on the growth side of things. For him, sales at Rollbar is a great cross-section of all of these areas. Ivan says he "gets to talk to smart people every day and help them solve problems. Sales is similar to engineering in that way." He really enjoys communicating with people, and feels that the only way we can help engineers is by adding value for them. In the next few months, Ivan is looking forward to creating new content to help people get more value from Rollbar.

We're very glad Ivan is on the team here at Rollbar!

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your web and mobile application errors.

Error monitoring in ASP.NET MVC

Wed, 07 Mar 2018 00:00:00 GMT

ASP.NET MVC is a modern web development framework that combines the features of MVC (Model-View-Controller) architecture for better separation of concerns and the best parts of the ASP.NET platform.

We’ll show you an example of how to catch errors and exceptions in ASP.NET MVC using a global action filter. We’ll also show you how to track them in Rollbar’s error monitoring service. This will give you real time visibility into your errors in production. It also captures person data and other context from your app so you can solve errors faster.

Above, you can see that we've created an example app that triggers an an exception when the user clicks on a button. The error message is tracked in Rollbar, including a stack trace where you can see the line of code that caused the error.

Create a global action filter

To track all of our exceptions, we have multiple approaches in .NET. Using a global action filter is the easiest way to catch all the exceptions. It receives uncaught exceptions for your whole application, not just an individual controller. We’ll show you how to override it to create your own global action filter with exception tracking. Here are some simple steps to create global action filter in your application.

Open up your own ASP.NET MVC project or use our open source example on GitHub at Rollbar-Dotnet-Example. You can use this filter to add any error monitoring solution, but we will show an example of how to set up Rollbar in the next section.
Next, let’s create our own exception filter to catch exceptions. In your App_Start folder, and add a .cs file by extending IExceptionFilter. You need to override the OnException method, allowing you to catch the exception that is going to occur throughout the project. You can find the App_Start folder inside your application directory.
```
namespace RollBar.App_Start
{
 public class RollbarExceptionFilter : IExceptionFilter
 {
   public void OnException(ExceptionContext filterContext)
   {
     if (filterContext.ExceptionHandled)
     return;
   }
 }
}
```

Now we need to add a FilterConfig to add our custom filter to the GlobalFilterCollection. You can find the FilterConfig.cs file inside the App_Start folder.

namespace RollBar
{
 public class FilterConfig
 {
   public static void RegisterGlobalFilters(GlobalFilterCollection filters)
   {
     filters.Add(new RollbarExceptionFilter());
   }
 }
}

Finally, register it in the GlobalFilters in the Application_Start method. You can find the Application_Start method in Global.asax, which is in the root application directory.
```
protected void Application_Start()
{
 FilterConfig.RegisterGlobalFilters(GlobalFilters.Filters);
}
```

Add Rollbar error monitoring

Now that we have our global action filter wired up, we’re going to show you how to integrate Rollbar’s .NET SDK in your code.

Visit https://rollbar.com and sign up for an account if you haven’t yet. Next, create your project and select .NET from the list of SDKs. Save the server side access token that is generated for you. You’ll need this to configure Rollbar in the steps below.
Install the Rollbar using NuGet Package Manager. You can install Rollbar using the Package Manager Console or in Manage NuGet Package for Solution. You can find both options in Visual Studio under Tools->NuGet Package Manager. Alternatively, to install from the console, run the command below:
```
Install-Package Rollbar
```

Configure the Rollbar SDK with the access token you received earlier. Add it in the Global.asax file and inside the Application_Start method.

protected void Application_Start()
{
 const string postServerItemAccessToken = "Your Post Server Access Token";

 RollbarLocator.RollbarInstance.
 Configure(new RollbarConfig(postServerItemAccessToken)
 {
   Environment = "production"
 });
}

Now add the Rollbar error tracking inside the global action filter we created earlier. Add it to the OnException method to send the crash log to Rollbar.
```
RollbarLocator.RollbarInstance.Error(filterContext.Exception);
```

You can also use Rollbar to track caught exceptions and other items using the same Rollbar instance. Learn more about the full API in our .NET documentation.

Test with an Example App

To test that it’s working, lets create a page that will generate an error message. In the example below, you’ll be able to generate an error by clicking the "Throw an error" button.

First, we’ll create a new form with a button to call CreateErrorController action on the server.

@using (Html.BeginForm("GenerateError", "CreateError", FormMethod.Post))
{
  <input style="height:50px;width:200px" type="submit" value="Generate an error"   name="name" />
}

When you click the "Generate an error" button, it will trigger the GenerateError method. In this method, we have added a bug which is throwing an exception with the text “This is test exception.”

namespace RollbarDotnetExample.Controllers
{
  public class CreateErrorController : Controller
  {
    // GET: CreateError
    public ActionResult Index()
    {
      return View();
    }

    [HttpPost]
    public ActionResult GenerateError()
    {
      throw new Exception("This is test exception.");
    }
  }
}

Since we wired up our global action filter to report the error to Rollbar, we should see the error show up on Rollbar’s item page. Below, you can see that this error last occurred 2 minutes ago.

The item page shows us a full stack trace including the line number where the error occurred. Using the tabs above the track back, you can see more contextual data to help determine the cause of the problem. For example, you can get extra information on which people were affected, which browsers they were using, which IP address they have and more.

Now you know how to catch exceptions in ASP.NET MVC applications! You also saw how easy it is to set up Rollbar for error monitoring. Learn more in our documentation for .NET, and sign up for a free trial today!

If you haven’t already, sign up for a 14-day free trial of Rollbar and stop flying blind in production.

Meet the Rollbar Team - Megan Anderson (Engineering)

Mon, 05 Mar 2018 00:00:00 GMT

Megan Anderson is a software engineer who enjoys helping developers solve technical problems. That's what led her to join Rollbar in May of 2017 as a support engineer, and pushed her into moving into a core engineer role a few months ago.

One of the rare Bay Area natives, Megan went to UCLA and majored in Physiology. She then worked in a genetics lab doing organ transplant matching research for awhile, handling more human spleens than 99% of people on the planet. She decided to apply for a graduate degree in Physical Therapy, and realized that wasn't the career path for her after all. A friend of hers went to a coding bootcamp and taught Megan to code. She liked coding so much that she decided to go to a programming bootcamp as well, and after eight years in LA, Megan moved back to the Bay.

Megan originally joined Rollbar as a support engineer, drawn to the role because she got to "help developers that have interesting questions and problems with a technical product." Megan found the engineering work here at Rollbar especially engaging. She really enjoys solving and debugging engineering problems, and this lead to her moving into the core software engineer role she is in now.

In the next few months, Megan would like to use her knowledge of how users use the product and her engineering skills to add value to Rollbar for our users. She's looking forward to making it clear and easy to use, not just technically interesting. She's especially looking forward to getting to improve things that frustrate her.

Megan describes the culture here at Rollbar as supportive. "When you have questions, people are happy to answer. People are very intelligent, and they assume that you’ll keep up. People want to help you develop professionally while the company succeeds. Our CEO cares about everyone’s opinions."

We're excited to have Megan on the team here at Rollbar!

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your web and mobile application errors.

Send Rollbar Error Alerts to Cisco Spark

Thu, 01 Mar 2018 00:00:00 GMT

Cisco Spark is a enterprise collaboration platform from Cisco that includes messaging, video conferencing, desk phones, and digital whiteboards to mention a few.

If your organization uses Cisco Spark, using Rollbar with it has just got easier. Depending on your setup, you can now receive Rollbar alerts not just in your messaging apps but also on that giant screen in the meeting room.

As a Cisco Spark customer you've invested a lot in enabling enterprise collaboration. With this new Rollbar integration, you can further extend the value of the platform for your engineering teams.

You can now get notified directly in Cisco Spark when certain situations occur, such as when a new error is detected by Rollbar, or when an error occurs at a very high volume within a specified time period.

If you've enabled deploy tracking in Rollbar, you can also notify Cisco Spark of deploys.

Rollbar alerts in Cisco Spark:

As always, you can customize what alert notifications Rollbar should send. You can also configure which space in Cisco Spark the alerts should go to.

Cisco Spark integration settings in Rollbar:

For more instructions on how to enable this integration for your account, check out our Docs page.

Please feel free to reach out if you have any questions!

If you haven’t already, sign up for a 14-day free trial of Rollbar and reduce production issue resolution times.

Meet the Rollbar Team - Francesco Crippa (Director of Engineering)

Tue, 27 Feb 2018 00:00:00 GMT

Our Director of Engineering, Francesco Crippa, has been making waves since he joined us in January. Originally from the north of Italy, Francesco worked in Ireland before coming to San Francisco almost five years ago. In addition to programming, he also builds light installations and dabbles in photography.

Francesco began working professionally as a programmer after his first year of studying software engineering in college; first at small startups, and then he founded his own. This eventually led to working in Rome at the National Bank of Italy. The day he moved to Rome is the first day he ever even visited Rome. It took three and a half years there before Francesco realized he simply didn't fit in Italy and decided to seek adventure outside his home country. He didn't speak English well, so he did freelance consulting around the world, traveling constantly to many different places (including some unusual countries like Saudi Arabia, Lichtenstein, and Albania, to name a few).

Eventually, Francesco figured it was time to leave Italy for good, and began working for EMC in Ireland, living in Cork. Because of cultural and linguistic barriers, Francesco describes that first year in Ireland as one of the toughest years of his life (and the Irish weather didn't help either). After three years he was ready to move on and almost accepted an offer for a job in Finland. But at that point Michelle, an Irish woman he met in Cork, entered the picture. The day before he was supposed to fly to Finland, Francesco got a call from a recruiter for Cisco, in Galway. It took less than 24 hours for him to accept the job with Cisco in order to stay in Ireland, and they eventually ended up getting married.

Francesco moved out to San Francisco to help Cisco build its own new cloud collaboration platform (Cisco Spark) from the ground up. After many years spent working in the enterprise market (along with a brief adventure in the consumer space with Zillow) he decided it was time to look for a small, fast growing and thoughtful company, where he could get back to one of his passions: building products to help every engineer to get better and be more effective every day. This led him to Rollbar, and the rest is history.

"I can't believe we have tools, devices and apps that make our life so easy and smooth, but when it comes to debugging the software that powers almost every aspect of our life we're left with old debug tools and log files... The tech industry empowered so many people in getting better and reinventing the way we work. It's time we help our engineers and the industry itself do the same."

We're very excited to have Francesco on the team here at Rollbar!

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your web and mobile application errors.

Announcing Our New .NET Error Monitoring SDK

Mon, 12 Feb 2018 00:00:00 GMT

Rollbar is happy to announce our official .NET error monitoring SDK. This new .NET SDK is more rreliable, supports all of the popular .NET platforms (NET Standard, .NET Framework, .NET Core, ASP.NET MVC, UWP, Mono, Xamarin), easy to configure, multithreading friendly, and more.

Rollbar helps you monitor errors in production applications. It provides you with a live error feed from your applications, including complete stack traces and contextual data to debug errors quickly. It also lets you easily understand your user experience by tracking who is affected by each error. Check out our features page to learn more about how it can help you.

.NET error monitoring, as seen in Rollbar

Improvements

Multi-target support including .NET Standard 2.0 and up, .NET Framework (or Full Framework) 4.5 and up, and .NET Core 2.0 and up. It also supports UWP, Mono, and Xamarin.
Native asynchronous (“fire-and-forget”) implementation. Due to its async nature the SDK has very little impact on its host at runtime. A blocking (synchronous) implementation of the same methods is also available.
Rate limiting and buffering of all messages.
Automatic retries in case of network failure.
Person information is now a part of a notifier configuration object, including field scrubbing for privacy.
Improved interface by separately defining ILogger and IRollbar. ILogger also now supports chaining calls.
Multiple instances of the notifier can be separately and differently configured, including different access tokens.
Easier configuration including support for app.config and appsettings.json files.
Subscribe to receive any internal operational/communication event or an error happening within the notifier.
More usage samples and sample apps.

Upgrading

The public API of our notifier v1 is not backward compatible with the pre-v1 releases. However, the conversion/upgrade process is very straightforward by using find-and-replace tools across your code base.

You should change your config from the old version:

Rollbar.Init(new RollbarConfig("POST_SERVER_ITEM_ACCESS_TOKEN"));

RollbarLocator.RollbarInstance
.Configure(new RollbarConfig("POST_SERVER_ITEM_ACCESS_TOKEN") { Environment = "test" };

Additionally, anywhere in your code that you were sending error reports via

Rollbar.Report(new Exception("trying out an error", new NullReferenceException()));
Rollbar.Report("basic info log example.");

Will need to be replaced with something like this:

RollbarLocator.RollbarInstance.Error(new Exception("trying out an error", new NullReferenceException()));
RollbarLocator.RollbarInstance.Info("basic info log example.");

Learn more in our documentation for .NET.

If you haven’t already, sign up for a 14-day free trial of Rollbar and stop flying blind in production.

Meet the Rollbar Team - Ali Shakiba (Software Engineer)

Sat, 10 Feb 2018 00:00:00 GMT

Ali Shakiba is an invaluable asset to our engineering team. Originally from Ahvaz, Iran, Ali came to America for graduate school and then joined us here at Rollbar last November.

Ali taught himself how to program in high school, learning BASIC to create his own video games. This led to him majoring in Software Engineering at Isfahan University of Technology in Isfahan, Iran. After college, he decided to get his Master's in Computer Science in America. After graduation, Ali came to San Francisco and eventually joined Rollbar.

Ali loves that here at Rollbar he gets to build a tool that helps other developers build things. Prior to joining Rollbar, Ali had been working on some open source developer-focused tools of his own, and he's excited to continue getting to work on helping developers build their products. He decided to join Rollbar because "it is very exciting that we are at center of the developer community and closely connected to them", and because it feels almost like you get to "be a member of all engineering teams that use Rollbar to make better and more reliable products."

In the future, Ali plans to work with engineering, product, and design to extend Rollbar's reporting and analytics functionality in order to provide users deeper and wider insight into the data that they report to Rollbar.

We're delighted to have Ali on the team here at Rollbar!

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your web and mobile application errors.

New Integration - Auto-create Clubhouse stories with error data from Rollbar

Fri, 09 Feb 2018 00:00:00 GMT

At Rollbar, we use Clubhouse to manage our software projects. One of the many ways we use it is for tracking issues in product releases. We'd have an epic in Clubhouse for each release, and create stories for bugs associated with the release that we need to fix. This is especially useful when we're working on service packs.

If you're not familiar, Clubhouse is a project and product management software designed for agile software teams. We've written about how Clubhouse uses Rollbar and published an interview with Andrew Childs, their CTO and Co-founder, on our blog. 🚀

With this integration, you can now automatically:

Create a story in Clubhouse for a Rollbar error, based on rules you set
Include in the story the stack trace and a link to the error details data in Rollbar
Mark the story resolved in Clubhouse when it's resolved in Rollbar
Change the status of a Clubhouse story when an error is re-activated

Clubhouse integration settings in Rollbar:

Clubhouse story from a Rollbar error:

To get started, check out this integration guide from Clubhouse.

Let us know if you have any questions, or tell us what you think!

If you haven’t already, sign up for a 14-day free trial of Rollbar and stop flying blind in production.

Introducing the Account Dashboard 📉

Wed, 07 Feb 2018 00:00:00 GMT

We are excited to release our new Account Dashboard! The Account Dashboard gives your engineering team insight into errors across multiple Rollbar projects. This insight provides an overview of how each application and service is performing, so you can quickly identify and drill down into problem areas and error trends.

Your browser does not support the video tag.

In today’s microservices world, applications consist of multiple services that interact together to deliver the full user experience. There is the client side of the application that the user typically sees, which may be a webpage or mobile app. Behind that, there may be an application server and multiple API services, often owned by separate development teams. This makes the job of monitoring more complex because you need to monitor all the services at once. Errors or a loss of availability in any one service can result in cascading failures to downstream services, and ultimately the user.

Check out what Jason Kozemczak, Tech Lead at Instacart had to say about the Account Dashboard.

The Account Dashboard makes it even easier than before to understand the overall health of the Instacart product as well as individual teams/systems that make up Instacart. It also makes it dead simple to know where our engineering time has the highest leverage regarding issues affecting our customers and shoppers.

The Account Dashboard displays the number of error occurrences over time for each project. This allows you to see if problems are primarily affecting a single Rollbar project or multiple projects. If two projects display a correlated pattern, they may be related to the same root issue. You can also compare each project to the previous 24 hours to see if there was a recent change, such as after a recent code deployment.

You can narrow down problems by filtering on specific Frameworks, Environments, or error Levels. You can also see more detail on the total number of Occurrences, Activations of new errors, as well as the number of code Deploys made.

Account Dashboard, as seen in Rollbar

Account Dashboard is available on the Growth and Premium plans. We hope that you and your team will save time in your day-to-day debugging efforts with the new Account Dashboard. We appreciate your feedback. Email us and let us know what you think.

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your software application errors.

Debugging JavaScript with Source Maps

Tue, 06 Feb 2018 00:00:00 GMT

One of the frustrating situations I often encounter when debugging JavaScript, is tracking down JavaScript errors to line 23 col 63475. I felt as though I was right on the edge of seeing the offending code and being able to fix it. And then, reality came crashing down. I realized that I’ve managed to debug myself right into the middle of a minified JavaScript file 😞.

There is a better way - Source Maps. JavaScript source maps are the key to taking what you’ve narrowed down in the minified code, and then being able to map them back to the source code so that you can view and resolve the problem without having to figure it out in the minified code.

Source maps are generated when JavaScript code is minified. By using the source map in production, you can trace your problems back to the exact line of source code. It also allows you to use your browser's developer console to step through the source code when debugging problems. We’ll look at how to generate a source map, and then we’ll look at how they work, and how we can use them to make our lives easier.

You can download or clone the source for the following demo from https://github.com/echovue/javascript-calculator. The source includes the original JavaScript file, the minified JavaScript file, and the source map. If you already know how to generate a source map or just want to use the generated file, you can skip down to How Do Source Maps Work?

Generating a Source Map

For this example, I’m going to be using a simple JavaScript application. Keeping it simple will help keep the demo manageable, and the concepts apply to any application, no matter the size.

The first thing that we’ll need is a tool to minify the code. I’m going to use UglifyJS, but most tools should support the ability to generate a production source map as part of the minification process. You can install UglifyJS with npm if you have that installed on your workstation.

$ npm install uglify-js -g

You can then validate installation by executing the following:

$ uglifyjs --version
uglify-js 3.2.0

In some cases, you may have to add the installation folder to your path.

Now that we have uglifyjs installed and we’ve verified that it works, let’s minify our code. If you’re using the example project, this command will overwrite the existing minified file and source map.

From the JS folder of the demo project, enter the following command.

$ uglifyjs calculator.js --compress --mangle --source-map --output calculator.min.js

This command takes our JavaScript file calculator.js and converts it to a minified version calculator.min.js, along with a source map calculator.min.js.map. Even with this small file, the minification process reduces the size of the file from 4KB to 1KB.

How do Source Maps Work?

Let’s open up the source map file to see what it has in it. I used a JSON parser to format it for easy reading, and shortened some of the lines with ellipses as well.

{
  "version" : 3,
  "sources" : [calculator.js],
  "names" : [resultNum,operator,el,element,charAt,document,querySelector,...],
  "mappings" : CAAC,WACC,aAyGA,IAAK,IAvFHA,EACAC,EAhBEC,EAAK,SAASC,GAChB,MAA0B,MAAtBA,...
}

The file specifies the mapping version used and identifies the source files and names of parameters. The useful part is the mappings, although unfortunately since they are in Base 64 VLQ, they’re not very useful for the human brain.

If you’d like to understand more about how the mappings work, and how they translate the minified code back to the source, I’d recommend reading How do source maps work. For now, let’s look at how to leverage the source map in production to make debugging easier.

If you open the index.html in Chrome or your browser of choice, you’ll notice that there is a JavaScript error which is thrown on the first line of our minified file. Let’s make it easier to identify and resolve this.

Figure 1. Error within the minified JavaScript file

Note: To complete the next set of steps, you will need to have the JavaScript-calculator web application running on a publicly accessible web server. For this demo, I created an AWS instance, installed an Apache Web Server, and served the web application from there.

Working with Source Maps in Production

When you’re debugging a production application, it gets trickier since production servers often don’t serve source maps. Do you want to make it easier for people you don’t know to see your raw source code? Fortunately, Rollbar supports using source maps to get meaningful stack traces while still using minified JavaScript in production.

Rollbar provides real-time production error monitoring with support for most programming languages and frameworks, including JavaScript, Angular, Node, React, etc. Because rollbar,js supports source maps, you can see the line of exact code where each error originated along with the stack trace. Let’s dive into an example of how it works.

Once you have created an account and your first project, you’ll be given a client-side access tokens. Add the script included under the Quick start browser section inside the <HEAD> tag in index.html.

You can upload your source maps via the API. Usually, we would use a script do this automatically at deployment time, but we will do it manually for this tutorial. From the root folder of the web application project, execute the following curl command after updating the access token and the minified_url to your token and URL:

$ curl https://api.rollbar.com/api/1/sourcemap \
-F access_token=8888888888888888888888888 \
-F version=0.0.1 \
-F minified_url=http://ec2-52-43-138-168.us-west-2.compute.amazonaws.com/javascript-calculator/js/calculator.min.js \
-F source_map=@js/calculator.min.js.map \
-F calculator.js=@js/calculator.js

Once that is done, clicking on an error should take you to a page with the stack trace. Here we can see the proper source code with files and line numbers.

Figure 2. Stack trace showing the original source code

When you click the source file link, it will open to the file and line number in GitHub, BitBucket, or Gitlab. There, you can use the tools to see what changes were made and when. To learn more, check out the source maps documentation for additional details and config options.

Tips for using source maps in production and debugging with Rollbar

Don't forget to update the version number when you update your JavaScript. If you don't, the filename, line and column numbers will be incorrect.
The value of minified_url must be the full URL of the minified file. This should start with http: or https:, which we'll strip off.
Make sure you're not missing one or both of the config params in the on-page JavaScript snippet. Set both payload.client.javascript.source_map_enabled and payload.client.javascript.code_version.
If you're using the upload method, check to be sure that the code_version used in the on-page snippet matches the version provided in the upload call.
If you're using the download method, make sure your source map file or minified JavaScript source files are on a host that's reachable from the public internet and are not gated behind an authorization wall.
If the JavaScript error that you are expecting to be un-minified does not have column numbers, and you haven't enabled guess_uncaught_frames, we won't be able to apply the source map. We need column numbers to be able to apply the source map without guessing.
If your source map file combines multiple sub-maps into "sections" within the top level map, we, unfortunately, don't yet support this source map format (but we are planning to soon).

TL;DR

In conclusion, source maps help you debug production code right in your browser's developer console. They tell you the exact file and line number when there is an error, and make it natural to use the debugging feature within your browser to step through the code. This makes it much easier to find the root cause of problems and fix them quickly. When monitoring production systems, make sure to choose solutions like Rollbar that support source maps and make debugging production super easy.

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of impactful JavaScript errors.

Meet the Rollbar Team - Zachary Collins (Software Engineer)

Mon, 05 Feb 2018 00:00:00 GMT

We're happy to introduce software engineer Zachary Collins, who joined Rollbar in January. Originally from Conway, Arkansas, Zachary lived in Japan for two years before making his way back to San Francisco and eventually joining Rollbar.

Zachary taught himself how to program at the age of 10, in BASIC, in order to create his own video games. Finding he was quite adept at programming, he took the GED at 14 and began studying computer science and music at Hendrix College. He's a talented percussionist as well as programmer, focusing mainly on keyboards such as the marimba or vibraphone. Zachary still composes music for the piano occasionally, although software engineering is his main focus now.

After college, Zachary moved to Palo Alto, California, and began working for Entity MCL. There, he was mostly writing in Python, programming an interface for wireless access for Windows XP.

Zachary also has a deep interest in Japan, and followed that interest to work at a small software company in Japan. He is fascinated by Japanese art and food, stating "everything comes with an old world presentation element that is fun and magical." While living in Japan, he also built the authentication systems for the G8 Summit in Hokkaido in 2008.

Zachary decided to join Rollbar due to the mature and relaxed culture, as well as the ability to mentor more junior software engineers. Both of his parents are teachers, and he also enjoys passing on his knowledge. He's looking forward to optimizing Rollbar's ETL pipeline, and dreams of reaching Rollbar "zero" (no exception occurrences) for our backend systems someday soon.

We're excited to welcome Zachary and glad he decided to join us at Rollbar!

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your web and mobile application errors.

Meet the Rollbar Team - Jaee Apte (Product Designer)

Wed, 31 Jan 2018 00:00:00 GMT

We're excited to introduce Jaee Apte, one of the newer members of our team. Jaee (pronounced "Zaa-ee") started at Rollbar as a Product Designer in October, and we are very happy to have her. She is also a talented visual artist, and enjoys product design because "it's a good bridge between being completely analytical and a crazy artist."

Jaee is originally from Mumbai, India, although she moved around a lot growing up. After completing her Bachelor's in Computer Science at the University of Pune, India, she worked at Microsoft doing product research and development. At Microsoft, she was on a small team working on cloud services for Visual Studio Team Services, working on building infrastructure for product development. Jaee was working mostly as a software developer, although she got to do a bit of product research and product management. This led her to realize that, while she wanted to stay in tech, she wanted a more creative role.

Jaee paints a lot and wanted to be able to bring that passion for creativity into her daily work. This desire to blend technology and design brought her to the University of Michigan, where she completed her Master's in Human-Computer Interaction (HCI). Armed with a newfound knowledge and a desire to build meaningful products for the user, she moved to San Francisco in search of the next opportunity. That's when she found Rollbar.

Aurora Borealis - Northern Lights by Jaee Apte

Jaee joined Rollbar because of the domain and the users. Since she comes from an engineering background, she knows what a niche set of users Rollbar serves. In her own words, "I'm excited about tapping into their brain and designing a product that makes their lives easier." She's made waves so far here at Rollbar, helping to design upcoming features around user onboarding and the dashboard, as well as helping to put a design system in place to enable better communication and transparency between engineering and design. In the future, she's looking forward to setting up a user-centered design process in place and improve the Rollbar experience through regular user feedback.

Starry Sunflowers by Jaee Apte

Jaee googles her name from time to time to make sure it's still unique. So far the only other Jaee on the web is a super talented Jazz musician and a fast-food restaurant in Rio di Janeiro.

We are excited that Jaee decided to join us in our mission of making software development easy!

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your web and mobile application errors.

Top 10 JavaScript errors from 1000+ projects (and how to avoid them)

Wed, 24 Jan 2018 00:00:00 GMT

To give back to our community of developers, we looked at our database of thousands of projects and found the top 10 errors in JavaScript. We’re going to show you what causes them and how to prevent them from happening. If you avoid these "gotchas," it'll make you a better developer.

Because data is king, we collected, analyzed, and ranked the top 10 JavaScript errors. Rollbar collects all the errors for each project and summarizes how many times each one occurred. We do this by grouping errors according to their fingerprints. Basically, we group two errors if the second one is just a repeat of the first. This gives users a nice overview instead of an overwhelming big dump like you’d see in a log file.

We focused on the errors most likely to affect you and your users. To do this, we ranked errors by the number of projects experiencing them across different companies. If we looked only at the total number of times each error occurred, then high-volume customers could overwhelm the data set with errors that are not relevant to most readers.

Here are the top 10 JavaScript errors:

Each error has been shortened for easier readability. Let’s dive deeper into each one to determine what can cause it and how you can avoid creating it.

1. Uncaught TypeError: Cannot read property

If you’re a JavaScript developer, you’ve probably seen this error more than you care to admit. This one occurs in Chrome when you read a property or call a method on an undefined object. You can test this very easily in the Chrome Developer Console.

This can occur for many reasons, but a common one is improper initialization of state while rendering the UI components. Let’s look at an example of how this can occur in a real-world app. We’ll pick React, but the same principles of improper initialization also apply to Angular, Vue or any other framework.

class Quiz extends Component {
  componentWillMount() {
    axios.get('/thedata').then(res => {
      this.setState({items: res.data});
    });
  }

  render() {
    return (
      <ul>
        {this.state.items.map(item =>
          <li key={item.id}>{item.name}</li>
        )}
      </ul>
    );
  }
}

There are two important things realize here:

A component’s state (e.g. this.state) begins life as undefined.
When you fetch data asynchronously, the component will render at least once before the data is loaded – regardless of whether it’s fetched in the constructor, componentWillMount or componentDidMount. When Quiz first renders, this.state.items is undefined. This, in turn, means ItemList gets items as undefined, and you get an error – "Uncaught TypeError: Cannot read property ‘map’ of undefined" in the console.

This is easy to fix. The simplest way: Initialize state with reasonable default values in the constructor.

class Quiz extends Component {
  // Added this:
  constructor(props) {
    super(props);

    // Assign state itself, and a default value for items
    this.state = {
      items: []
    };
  }

  componentWillMount() {
    axios.get('/thedata').then(res => {
      this.setState({items: res.data});
    });
  }

  render() {
    return (
      <ul>
        {this.state.items.map(item =>
          <li key={item.id}>{item.name}</li>
        )}
      </ul>
    );
  }
}

The exact code in your app might be different, but we hope we’ve given you enough of a clue to either fix or avoid this problem in your app. If not, keep reading because we’ll cover more examples for related errors below.

2. TypeError: ‘undefined’ is not an object (evaluating

This is an error that occurs in Safari when you read a property or call a method on an undefined object. You can test this very easily in the Safari Developer Console. This is essentially the same as the above error for Chrome, but Safari uses a different error message.

![Screenshot of TypeError: ‘undefined’ is not an object (undefined-is-not-an-object.png)

3. TypeError: null is not an object (evaluating

This is an error that occurs in Safari when you read a property or call a method on a null object. You can test this very easily in the Safari Developer Console.

![Screenshot of TypeError: null is not an object (null-is-not-an-object.png)

Interestingly, in JavaScript, null and undefined are not the same, which is why we see two different error messages. Undefined is usually a variable that has not been assigned, while null means the value is blank. To verify they are not equal, try using the strict equality operator:

![Screenshot of TypeError: null is not an object (null-is-not-an-object1.png)

One way this error might occur in a real world example is if you try using a DOM element in your JavaScript before the element is loaded. That’s because the DOM API returns null for object references that are blank.

Any JS code that executes and deals with DOM elements should execute after the DOM elements have been created. JS code is interpreted from top to down as laid out in the HTML. So, if there is a tag before the DOM elements, the JS code within script tag will execute as the browser parses the HTML page. You will get this error if the DOM elements have not been created before loading the script.

In this example, we can resolve the issue by adding an event listener that will notify us when the page is ready. Once the addEventListener is fired, the init() method can make use of the DOM elements.

<script>
  function init() {
    var myButton = document.getElementById("myButton");
    var myTextfield = document.getElementById("myTextfield");
    myButton.onclick = function() {
      var userName = myTextfield.value;
    }
  }
  document.addEventListener('readystatechange', function() {
    if (document.readyState === "complete") {
      init();
    }
  });
</script>

<form>
  <input type="text" id="myTextfield" placeholder="Type your name" />
  <input type="button" id="myButton" value="Go" />
</form>

4. (unknown): Script error

The Script error occurs when an uncaught JavaScript error crosses domain boundaries in violation of the cross-origin policy. For example, if you host your JavaScript code on a CDN, any uncaught errors (errors that bubble up to the window.onerror handler, instead of being caught in try-catch) will get reported as simply "Script error" instead of containing useful information. This is a browser security measure intended to prevent passing data across domains that otherwise wouldn’t be allowed to communicate.

To get the real error messages, do the following:

1. Send the Access-Control-Allow-Origin header

Setting the Access-Control-Allow-Origin header to * signifies that the resource can be accessed properly from any domain. You can replace * with your domain if necessary: for example, Access-Control-Allow-Origin: www.example.com. However, handling multiple domains gets tricky, and may not be worth the effort if you’re using a CDN due to caching issues that may arise. See more here.

Here are some examples on how to set this header in various environments:

Apache

In the folders where your JavaScript files will be served from, create an .htaccess file with the following contents:

Header add Access-Control-Allow-Origin "*"

Nginx

Add the add_header directive to the location block that serves your JavaScript files:

location ~ ^/assets/ {
    add_header Access-Control-Allow-Origin *;
}

HAProxy

Add the following to your asset backend where JavaScript files are served from:

rspadd Access-Control-Allow-Origin:\ *

2. Set crossorigin="anonymous" on the script tag

In your HTML source, for each of the scripts that you’ve set the Access-Control-Allow-Origin header for, set crossorigin="anonymous" on the SCRIPT tag. Make sure you verify that the header is being sent for the script file before adding the crossorigin property on the script tag. In Firefox, if the crossorigin attribute is present but the Access-Control-Allow-Origin header is not, the script won’t be executed.

5. TypeError: Object doesn’t support property

This is an error that occurs in IE when you call an undefined method. You can test this in the IE Developer Console.

This is equivalent to the error "TypeError: ‘undefined’ is not a function" in Chrome. Yes, different browsers can have different error messages for the same logical error.

This is a common problem for IE in web applications that employ JavaScript namespacing. When this is the case, the problem 99.9% of the time is IE’s inability to bind methods within the current namespace to the this keyword. For example, if you have the JS namespace Rollbar with the method isAwesome. Normally, if you are within the Rollbar namespace you can invoke the isAwesome method with the following syntax:

this.isAwesome();

Chrome, Firefox and Opera will happily accept this syntax. IE, on the other hand, will not. Thus, the safest bet when using JS namespacing is to always prefix with the actual namespace.

Rollbar.isAwesome();

6. TypeError: ‘undefined’ is not a function

This is an error that occurs in Chrome when you call an undefined function. You can test this in the Chrome Developer Console and Mozilla Firefox Developer Console.

As JavaScript coding techniques and design patterns have become increasingly sophisticated over the years, there’s been a corresponding increase in the proliferation of self-referencing scopes within callbacks and closures, which are a fairly common source of this/that confusion.

Consider this example code snippet:

function clearBoard(){
  alert("Cleared");
}

document.addEventListener("click", function(){
  this.clearBoard(); // what is “this” ?
});

If you execute the above code and then click on the page, it results in the following error "Uncaught TypeError: this.clearBoard is not a function". The reason is that the anonymous function being executed is in the context of the document, whereas clearBoard is defined on the window.

A traditional, old-browser-compliant solution is to simply save your reference to this in a variable that can then be inherited by the closure. For example:

var self=this;  // save reference to 'this', while it's still this!
document.addEventListener("click", function(){
  self.clearBoard();
});

Alternatively, in the newer browsers, you can use the bind() method to pass the proper reference:

document.addEventListener("click",this.clearBoard.bind(this));

7. Uncaught RangeError: Maximum call stack

This is an error that occurs in Chrome under a couple of circumstances. One is when you call a recursive function that does not terminate. You can test this in the Chrome Developer Console.

It may also happen if you pass a value to a function that is out of range. Many functions accept only a specific range of numbers for their input values. For example, Number.toExponential(digits) and Number.toFixed(digits) accept digits from 0 to 20, and Number.toPrecision(digits) accepts digits from 1 to 21.

var a = new Array(4294967295);  //OK
var b = new Array(-1); //range error

var num = 2.555555;
document.writeln(num.toExponential(4));  //OK
document.writeln(num.toExponential(-2)); //range error!

num = 2.9999;
document.writeln(num.toFixed(2));   //OK
document.writeln(num.toFixed(25));  //range error!

num = 2.3456;
document.writeln(num.toPrecision(1));   //OK
document.writeln(num.toPrecision(22));  //range error!

8. TypeError: Cannot read property ‘length’

This is an error that occurs in Chrome because of reading length property for an undefined variable. You can test this in the Chrome Developer Console.

You normally find length defined on an array, but you might run into this error if the array is not initialized or if the variable name is hidden in another context. Let’s understand this error with the following example.

var testArray= ["Test"];

function testFunction(testArray) {
    for (var i = 0; i < testArray.length; i++) {
      console.log(testArray[i]);
    }
}

testFunction();

When you declare a function with parameters, these parameters become local ones. This means that even if you have variables with names testArray, parameters with the same names within a function will still be treated as local.

You have two ways to resolve your issue:

Remove parameters in the function declaration statement (it turns out you want to access those variables that are declared outside of the function, so you don’t need parameters for your function):

var testArray = ["Test"];

/* Precondition: defined testArray outside of a function */
function testFunction(/* No params */) {
   for (var i = 0; i < testArray.length; i++) {
     console.log(testArray[i]);
   }
}

testFunction();

Invoke the function passing it the array that we declared:

var testArray = ["Test"];

function testFunction(testArray) {
  for (var i = 0; i < testArray.length; i++) {
     console.log(testArray[i]);
   }
}

testFunction(testArray);

9. Uncaught TypeError: Cannot set property

When we try to access an undefined variable it always returns undefined and we cannot get or set any property of undefined. In that case, an application will throw “Uncaught TypeError cannot set property of undefined.”

For example, in the Chrome browser:

If the test object does not exist, error will throw “Uncaught TypeError cannot set property of undefined.”

10. ReferenceError: event is not defined

This error is thrown when you try to access a variable that is undefined or is outside the current scope. You can test it very easily in Chrome browser.

If you’re getting this error when using the event handling system, make sure you use the event object passed in as a parameter. Older browsers like IE offer a global variable event, and Chrome automatically attaches the event variable to the handler. Firefox will not automatically add it. Libraries like jQuery attempt to normalize this behavior. Nevertheless, it’s best practice to use the one passed into your event handler function.

document.addEventListener("mousemove", function (event) {
  console.log(event);
})

Conclusion

It turns out a lot of these are null or undefined errors. A good static type checking system like Typescript could help you avoid them if you use the strict compiler option. It can warn you if a type is expected but has not been defined. Even without Typescript, it helps to use guard clauses to check whether objects are undefined before using them.

We hope you learned something new and can avoid errors in the future, or that this guide helped you solve a head scratcher. Nevertheless, even with the best practices, unexpected errors do pop up in production. It's important to have visibility into errors that affect your users, and to have good tools to solve them quickly.

Rollbar gives you visibility to production JavaScript errors and gives you more context to solve them quickly. For example, it offers additional debugging features like telemetry which tells you what happened on the user’s browser leading up to the error. That’s insight you don’t have outside of your local developer console. Learn more in Rollbar’s full list of features for JavaScript applications.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you take control of impactful JavaScript errors. :-)

Continuous Delivery with Jenkins and Rollbar

Tue, 09 Jan 2018 00:00:00 GMT

Continuous delivery (CD) helps reduce the cost, time and risk of delivering changes by allowing for fast incremental updates to applications in production. However, it’s essential to monitor your application after each deployment. You need to be notified immediately if something is wrong or users are having a poor experience.

Rollbar is a leading solution for error monitoring in production. It alerts you when new errors occur after a deployment. It can also tell you which deployment the error was first seen in, and which code changes were likely responsible. This can dramatically speed up your troubleshooting time and help you fix problems faster.

In the screenshot above, you can see that Rollbar has automatically identified the suspected deployment where the error was first seen (indicated by the second red circle). Additionally, it has identified the code change where the error was introduced (indicated by the third red circle). You can click on these links to go straight to the code change in GitHub or Bitbucket.

Jenkins is a popular open source server to automate continuous integration and continuous delivery. We'll show you how to configure Jenkins to notify Rollbar when a new version is deployed. Then, Rollbar will tell you the suspected deployment for each error.

Our pet clinic example

Rollbar and Jenkins should work with pretty much all applications but we chose the Java pet clinic project to demonstrate the integration capabilities. It is freely available and simple to run or modify. The source code is available on the spring-petclinic-angularjs repository on Github. Once you understand the example, you should try setting up your own application.

The pet clinic application is a Java and open source Spring Boot application. It depicts the functionality of a pet clinic with simple REST resources or entities like Pet, Owner, Visits, etc. The application consists of an AngularJS front-end UI module and a back-end REST API module. You need to have basic Java knowledge and don’t need to know AngularJS or Spring Boot technologies to complete this tutorial.

To demonstrate error monitoring, we will add a URL to the pet clinic that throws an error. We'll then show you how to trigger the error, view it in Rollbar, and identify the suspected change.

Integrate Jenkins with Rollbar

First, we are going to show you how to set up Jenkins for our pet clinic application. Then we are going to send deployment notifications to Rollbar so it can track when deployments are made.

Set up the Jenkins application

Step 1:

Create a Jenkins job for a project of your choice (we'll pick freestyle project for our example).

Step 2:

Configure your source code repository such as git or Bitbucket so that Jenkins has credentials to pull the code and build the project. Add the URL where Jenkins can pull the latest code.

Step 3:

Add the following scripts to build and deploy the service into the Jenkins server.

mvn clean package
sudo chmod 777 spring-petclinic-server/target
cd  spring-petclinic-server/target
java -jar petclinic-*.jar  & echo $! > ./pid.file &

The script will build the repository and package the jar file with Maven, then deploy it to the Jenkins server. (Advanced users: You can also deploy it to your choice of server but change the APIs in the scripts accordingly.) The successful deployment writes the process ID into the pid.file which can be used later to stop the process.

Send deployment notifications to Rollbar

Once the application build and deployment is successful, it is time to notify Rollbar of the successful deployment along with the deployed version. Rollbar can distinguish the code changes between the revisions of the deployments, and in the case of errors, it points to the newly introduced code for error resolution. It also points to the suspected deployment revision under the error.

Rollbar provides a RESTful deploy notification API to notify the deployments to Rollbar from any source using a simple REST service call. Please refer to the documentation to know more about this deploy API.

Add another build step to notify Rollbar after deployment of the service. Refer to the description on the input parameters below.

access_token: The destination project token on Rollbar. This token gets generated when a project is created on Rollbar.
environment: The deployment environment where the service is deployed. We can configure different environments such as development, staging and production.
revision: The deployed application version. This can be the jenkins BUILD_NUMBER, Maven project artifact version, or repository commit ID. Rollbar will create a link to the repository commit source code if the supplied version is the commit ID.
local_username: The user who deployed the build to server.

echo 'Notifying Rollbar of deployment and build version'
curl https://api.rollbar.com/api/1/deploy/ \
  -F access_token=obce2f27e4b84332b7873ce78ecb8f86 \
  -F environment=production \
  -F revision=${GIT_COMMIT} \
  -F local_username=jenkins_user

Rollbar now shows the deployment version history.

The revision number will be a link to repository source code pointing to the commit revision, as shown in the picture below. The revision is circled in blue.

We have successfully set up continuous delivery with Jenkins and are now sending deploy notifications to Rollbar.

Integrate your app with Rollbar

In previous sections, we learned how to set up the continuous delivery job and integrate with Rollbar. Now we will learn how to integrate the pet clinic application with Rollbar to monitor application errors. If handled or unhandled exceptions are thrown in the application, we will catch them and notify Rollbar with details about the error.

Add the Rollbar notifier SDK

Step 1:

Prerequisites:

Create an account with Rollbar https://rollbar.com.
Create your project and select Java from the list of notifiers.
Select the server side access token that is generated when the project is created.

Step 2:

Add a Maven dependency in the pom.xml file for the Rollbar-web notifier in the project.

<dependencies>
  <dependency>
    <groupId>com.rollbar</groupId>
    <artifactId>rollbar-web</artifactId>
    <version>1.0.0</version>
  </dependency>
</dependencies>

For Java based applications, Rollbar provides various other modes of integration inlz. For more information please visit Rollbar’s Java documentation.

Step 3:

We will create a custom utility class called ServerProvider which extends a Provider interface from the Rollbar library. The ServerProvider class provides a Rollbar ecosystem to look up and link classes with errors, including the source branch and version that caused the error and the base package of the source code. In the code snippet below, the root is the base package path where the source code is located.

Source Code:

public class ServerProvider implements Provider<Server> {
  @Override
  public Server provide() {

    return new Server.Builder().branch("master")
              .root("org.springframework.samples.petclinic")
              .build();
  }
}

Note: When the root is provided, Rollbar will give a link on the class name in the error traceback to the class in your source code repository. This will save you time because you don’t have to navigate your source code to find the location yourself.

Step 4:

We will be using the rollbar-web package to notify Rollbar of errors. Let’s start with the OwnerResource which creates a REST interface for managing a pet owner. To track errors in Rollbar we need to configure it with our access token and the ServerProvider we just created. To test it, we can trigger an error with a method throwError() method as shown below. When the arithmetic exception is thrown, the catch block will handle the exception and notify Rollbar with a rollbar.error(e) method call.

@RestController
public class OwnerResource extends AbstractResourceController {

  private static Rollbar rollbar = null;
  static {
    ConfigBuilder configBuilder = withAccessToken(ROLLBAR_SERVER_ACCESS_TOKEN);

    Config config =configBuilder.environment("production")
                      .handleUncaughtErrors(true)
                      .server(new ServerProvider()).build();
    rollbar = new Rollbar(config);
  }


  @GetMapping("/throwerror")
  public String throwError() {
    System.out.println("Introducing an error code here...");
    try {
      Int sum = 0;
      int avg = 1234 / sum;
    } catch (Exception e) {
      System.out.println("Error Thrown notifying to rollbar...");
      rollbar.error(e);
    }
  return "error-thrown";
  }
}

Add source control integration

Now, we would like Rollbar to integrate with our source control repository so that when an error occurs, it can automatically link us to the proper line of code. It can also tell us when there are code changes that break the deployment.

Step 5:

Make sure to have the project root configured in Rollbar as shown below during the project integration with SCM; it’s the source code location where the Java root packages are placed. If this is not configured properly, Rollbar will not be able link the source code to an error. In this example, the project root is /spring-petclinic-server/src/main/java.

Step 6:

Enable the Rollbar integration plugin or add-on in your source repository. For Bitbucket, it will be under account -> integrations -> rollbar -> add.

Step 7:

Make sure the pet clinic project is built locally and tests are running. Then, commit changes to the source repository to begin the CD process.

Run the test cases to generate errors

Now we will test all the pieces that we just put together. We will execute our build on Jenkins, run the application, then generate a test error.

Log in to the Jenkins server or the server where you deployed the application and stop the Spring Boot process. In the earlier steps we started the pet clinic service using Jenkins, which needs to be stopped before re-deploying the new revision of the code. Otherwise, the service will not be able to start when another process is already using the same port (8080).

pkill -f ‘java -jar petclinic*’

After the application restarts, you can generate the error using the curl command from your terminal:

curl http://localhost:8080/throwerror

That’s it! You just generated your first error.

Quickly find the root cause of errors

Check the Rollbar dashboard for the error logging on the Items menu tab which will show the error details along with the link to the source code file that caused the error. The error will show the suspected deployment version along with the piece of code causing the error as well as the code checked-in user and a hyperlink to source code on the Bitbucket.

Clicking on an error item in the list, we see the item detail page. Next to the trace back, we can see a link to the source code causing the error.

When you click on the link it will directly open the class in the bitbucket repository and highlight the suspected piece of the code, depicted in the next screenshot:

A second way to find the root cause of errors is by using the Suspect Deploy tab. It shows the revision where the error was first observed, along with changes that were made to the code since the previous deployment. We can see the commit message indicating that an error code was introduced.

Clicking on the link with the revision signature shows us a diff where we can see an additional error was introduced. Both versions result in a divide by zero error. In a real-world scenario, this would tell us what change caused the error, who made it and when.

We have successfully generated an error, tracked the error with Rollbar, and identified the root cause of the problem quickly with Rollbar.

Conclusion

Rollbar helps you monitor production deployments and notifies you when new errors occur. It can trace errors, then pinpoint the suspected deployment and code version. This will help you troubleshoot errors faster because you can quickly see what change caused it, when, and by whom. This will help you quickly identify and fix errors before they affect more customers.

Now that you understand how to configure Jenkins with Rollbar, try setting up your own application. To learn more about how Rollbar helps power continuous delivery, check out our ebook Low-Risk Continuous Delivery.

Monitoring tools for serverless environments and AWS Lambda

Mon, 08 Jan 2018 00:00:00 GMT

Serverless computing platforms like AWS Lambda represent a new computing paradigm. Over the past decade, we’ve been trying to abstract the application layer from the infrastructure layer. VMs started this by virtualizing hardware servers, and Docker containers extended this by packaging just the application code separate from the host system.

The next step in this process is to completely remove the host from the equation, and simply focus on the application code—which is what serverless computing is all about. But as with most things new, serverless forces you to change how you used to do things, including the way you monitor functions.

Below, I explain how monitoring changes in a serverless environment. I focus on Lambda in particular, although most of these observations apply to any type of modern serverless platform.

How monitoring is different with serverless

In traditional client-server apps, you typically monitor the performance of your servers, network latency, and more. However, with serverless platforms like Lambda, these metrics are irrelevant. This is because the vendor manages the underlying servers and plumbing, leaving you to focus exclusively on your application code.

This means you don’t need to worry about how much compute power your servers have available to execute your code. Lambda automatically scales the available compute capacity to ensure your code is executed (there’s a caveat, but more on that later). You don’t have to worry about load balancing across multiple servers, or optimizing network latency. AWS takes care of this, too.

Serverless metrics to monitor

Yet while all this is hidden from you, and handled by AWS, there are other metrics you should monitor.

The most important element within your control is your application code. With Lambda, you upload your application code as a function, and AWS handles the execution of this code. Errors in any line of your code will result in the function not executing as expected.

Although Lambda handles provisioning of resources to execute your functions, it has limits to the amount of memory and concurrent executions it can allocate to functions. The maximum allocated memory is 1536MB, and the maximum number of concurrent executions varies by region—some can run 500 events per minute, while others like US West can run 3,000 events.

If a function exceeds the limits on memory and concurrency, Lambda stops executing the function and throws an exception. If, for example, you experience latency with any of your functions, you should check the memory and concurrency rate. This is different from monitoring traditional server performance, and takes some getting used to. You may need to remove some functions, or make some functions smaller and more simple. However, this is still much easier than having to manage the infrastructure yourself and keep scaling resources according to demand.

Another common cause of errors in Lambda is access and permissions. If your Lambda function is supposed to access data stored in another AWS service, but doesn’t have the necessary permissions set in AWS IAM, your code will not execute.

Amazon’s built-in monitoring for Lambda

AWS primarily uses CloudWatch to monitor Lambda performance. CloudWatch tracks metrics like the number of functions executed, latency in execution, and errors during execution. By default, these metrics are recorded in one-minute intervals. If you want to go beyond these, you can setup custom metrics in CloudWatch using the AWS CLI or API. Custom metrics are more powerful as they can be recorded in intervals as low as one second. However, these high-resolution metrics come with a fee, unlike the default metrics which are free of charge.

CloudWatch also records errors to logs. The errors in CloudWatch logs are another crucial source of insight when troubleshooting serverless application problems.

Additionally, AWS provides scanning for application performance on Lambda using X-Ray. This service tracks the progress of a request in an application across various AWS services. So, if a Lambda function is triggered by an API Gateway, and during execution accesses data from an S3 bucket, X-Ray can trace the progress of this function and display it visually. This helps with root cause analysis.

Once you’ve identified the root cause and you’d like to dig deeper, you can look into CloudTrail logs to get additional details on errors. CloudTrail logs all API calls within Lambda. It includes details such as the source IP address, time, frequency of occurrence, and more.

Error Monitoring for Lambda

While AWS has provided many options to monitor Lambda performance, you still need to monitor your application logic for errors. Logical errors can cause your service to give failure responses to clients in the best case, or fail silently in the worst case. If you use a tool like Rollbar for end-to-end monitoring of your system, you can send your application errors and uncaught exceptions to Rollbar for holistic analysis. You can also leverage Rollbar’s unique monitoring features, like de-duping of alerts, viewing of detailed stack traces, and gauging the impact of errors with user tracking to fix errors sooner.

Conclusion

As you work with Lambda, there’s a lot to adapt to. Monitoring Lambda is different from monitoring traditional applications. There isn’t much lower-level infrastructure to monitor, but there’s a new way to monitor application performance at the upper layers of the stack. You need to leverage AWS’ built-in monitoring services and features like CloudWatch, X-Ray, and custom metrics. Your monitoring can also be more effective by adding on error monitoring tools like Rollbar which give you a deeper view into logical errors.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you take control of impactful production errors. :-)

What We Shipped in 2017 📦

Thu, 28 Dec 2017 00:00:00 GMT

At Rollbar, 2017 has been a year of exciting growth and development for the product and the team. We've grown the total number of users to over 100,000, processing billions of events per week, and the team has more than doubled in size. I know it's cliche, but, it genuinely does feel like we're just getting started. ;-)

Before we put this year behind us and speed into 2018, I thought we'd look back, and share some of the highlights of what we shipped in 2017.

Error Notifier SDKs

PHP SDK 1.0+ [IMPROVEMENT]
JavaScript SDK 2.0+ [IMPROVEMENT]
Lambda support for Python and Node [NEW]
Laravel SDK [NEW]
Java and Android SDK 1.0+ [IMPROVEMENT]
React Native SDK (Alpha) [NEW]
iOS SDK 1.0+ [IMPROVEMENT]

Product Features

JavaScript Telemetry [NEW]
JavaScript Source Maps
- Major JavaScript source map updates [IMPROVEMENT]
- Un-minify JavaScript method names [IMPROVEMENT]
- Automatic JavaScript source map processing [IMPROVEMENT]
Error Merging and Grouping
- Error merging and unmerging [NEW]
- Edit and merge multiple errors at once [NEW]
Export RQL results as CSV [IMPROVEMENT]
View errors in each deploy [NEW]

Security and Compliance

Compliance
- Compliant SaaS (HIPAA and ISO 27001) [NEW]
Security
- Encryption at rest [NEW]

Integrations

GitLab source control and issues [NEW]
GitHub Marketplace [NEW]

Love and use Rollbar? We value your feedback. What do you think of our new features and improvements in 2017? What sort of things would you like to see shipped in 2018? Please leave a comment below this post or connect with us at hi@rollbar.com.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you take control of impactful production errors. :-)

Most Popular Java Web Frameworks

Tue, 19 Dec 2017 00:00:00 GMT

As Java has evolved over the years, multiple attempts have been made to simplify development for various use cases. From official standards like Java Enterprise Edition, to community-driven frameworks, Java is continuing to prove itself to be adaptable and viable.

Our top list is based on usage from Hotframework.com's Java ranking and several other sources including blog posts and GitHub download numbers.

The top three are:

Spring
JSF
GWT

Other notable Java Web Frameworks:

Java Frameworks that are popular but not for the Web (We don’t want to forget them):

Hibernate (Data-focused)
Maven (Build-focused)
Apache Ant with Ivy (Build-focused)

Spring

Project Site: https://spring.io/

Primary Sponsor: Pivotal Software

Spring is more than just a web framework. It is a complete programming model that is built on and with Java, starting with Spring Boot, which is a way to get a spring application up and running with minimal configuration and no application server required. At the other end of the spectrum is Spring Cloud, which is a combination of components that allows developers to build resilient and reliable cloud-native applications that leverage the latest distributed patterns like a microservices architecture — two examples include application security and batch processing.

There are many use cases for Spring, and with the introduction of Spring Boot, it is a great solution for companies that are moving towards containers as it greatly simplifies the components required to support the running application.

Getting started with Spring is as simple as going to Spring Initializr and selecting the build framework you desire and any and all the Spring projects you want included in the initial application. It will create the Maven or Gradle configuration and all the basic spring configuration required to start.

Creating a simple web application starting with Initializr (Figure 1):

… which will create a Zip file with the following files in it:

./mvnw.cmd
./pom.xml
./.gitignore
./.mvn/wrapper/maven-wrapper.properties
./.mvn/wrapper/maven-wrapper.jar
./mvnw
./src/test/java/com/example/demo/DemoApplicationTests.java
./src/main/resources/application.properties
./src/main/java/com/example/demo/DemoApplication.java

You need a Controller — src/main/java/com/example/demo/DemoController.java:

package com.example.demo;

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;

@Controller
public class DemoController {

    @RequestMapping("/hello")
    public String hello() {
        return "hello";
    }

}

And a template file — src/main/resources/templates/hello.html

<!DOCTYPE HTML>
<html>
<head>
    <title>Hello World</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
    <p>Hello World</p>
</body>
</html>

JSF (Java Server Faces)

Project Site: http://www.oracle.com/technetwork/java/javaee/javaserverfaces-139869.html

Primary Sponsor: Oracle (soon to be a separate foundation)

JSF is a specification for displaying web user interfaces that is defined as part of the Java Platform, Enterprise Edition (JEE). JSF 1 was released in 2004, incorporated into JEE 5 and uses Java Server Pages (.jsp) as its templates. JSF 2 was released in 2009 as part of JEE 6, and leverages Facelets for templating and supports AJAX calls with a browser to allow modern web application lifecycles. JSF is component-based, allowing it to be expanded with additional components. IceFaces and MyFaces are examples of popular add-on components.

As JSF is part of the Java standard, it is popular with development teams that want to stick to published standards for increased portability across platforms. JSF also allows existing backend Java code to be extended with a web interface without having to refactor the base application by introducing a new framework.

A simple JSF application requires a Managed Bean, Facelet, and mapping the servlet.

helloworld.java

package helloworld;

import javax.faces.bean.ManagedBean;

@ManagedBean
public class HelloWorld {

    final String world = "Hello World!";

    public String getworld() {
        return world;
    }
}

helloworld.xhtml

<html lang="en"
      xmlns="http://www.w3.org/1999/xhtml"
      xmlns:h="http://java.sun.com/jsf/html">
    <h:head>
        <title>Facelets Hello World</title>
    </h:head>
    <h:body>
        #{hello.world}
    </h:body>
</html>

web.xml

<servlet>
    <servlet-name>Faces Servlet</servlet-name>
    <servlet-class>javax.faces.webapp.FacesServlet</servlet-class>
    <load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
    <servlet-name>Faces Servlet</servlet-name>
    <url-pattern>/faces/*</url-pattern>
</servlet-mapping>

GWT (Google Web Toolkit)

Project Site: http://www.gwtproject.org/

Primary Sponsor: Google

GWT is much like JSF in that it is strictly focused on building web interfaces. It is more popular than native JSF as it makes it easy to maintain complex JavaScript user interfaces with Java code. GWT has lost some of its popularity over the last couple of years as more development teams are pushing Java to the backend and having it expose REST APIs which are consumed by both native mobile apps and user interfaces built in Node.js, using frameworks like Angular.

A tutorial on how to build a simple GWT application can be found on its project site: Getting Started building a GWT app.

Conclusion

In summary, there are many viable Java Web Frameworks that can be used to address your needs. None of the top three are bad choices—It comes down to personal preference. Just be aware that once you commit to a framework and start to leverage its features, switching to another framework is not an insignificant amount of work.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you take control of impactful Java errors. :-)

Spring MVC Exception Handling and Monitoring

Mon, 27 Nov 2017 00:00:00 GMT

The Spring Famework is the most popular framework for Java according to hotframeworks.com. It provides a model view controller (MVC) architecture and readily available components to develop flexible and loosely coupled web applications.

If you are new to Rollbar, it helps you monitor errors in real-world applications. It provides you with a live error feed from the application, including complete stack traces and request data from the browser to debug errors quickly. It lets you easily understand user experience by tracking who is affected by each error. Learn more about our Java error monitoring product features.

While Rollbar’s notifier works with any Java application, we’re going to show you how to set it up with Spring and how to try it out yourself with a working example app.

Create a global exception handler

To track all of our exceptions in Spring, we’ll be making use of a global exception handler. This receives uncaught exceptions for your whole application, not just an individual controller. Spring offers two main approaches:

1. ControllerAdvice

When you create a class annotated with @ControllerAdvice, it will handle exceptions created by all your controllers. Each controller advice defines a method with a @ExceptionHandler annotation which becomes the default handler. You can insert your custom code to print or track errors there.

ControllerAdvice is only available in Spring 3.2 and above. We won’t be covering this approach in detail but you can see our working example on GitHub. You will need to uncomment the annotation at the top to run it.

2. HandlerExceptionResolver

Spring Framework also provides a HandlerExceptionResolver interface that you can implement to create a global exception handler. Since it has been around since early releases, it should be compatible with both old and new code. The Spring Framework makes it easy to set up by providing a SimpleMappingExceptionResolver. We’ll show you how to override it to create your own global handler which will track exceptions.

Warning: Be careful about mixing both approaches in the same application. Most applications use one approach, and using two may result in unexpected behavior.

The example below shows you how to override the SimpleMappingExceptionResolver. It allows you to create a custom method to build a log message and to return a view to the user with a more friendly error page. If you want to run this example yourself, check out our project Rollbar-Example-Java on GitHub.

public class MyMappingExceptionResolver extends SimpleMappingExceptionResolver {
    String accessToken = "8e194f5f31db4ff1b4e3e0951a40c936";
    Rollbar rollbar;

    public MyMappingExceptionResolver() {

        setWarnLogCategory(MyMappingExceptionResolver.class.getName());
    }

    @Override
    public String buildLogMessage(Exception e, HttpServletRequest req) {

        System.out.println("Exception : " + e.toString());
        RequestProvider requestProvider = new RequestProvider.Builder().userIpHeaderName(req.getRemoteAddr()).build();
        rollbar = Rollbar.init(withAccessToken(accessToken).request(requestProvider).build());
        rollbar.error(e);
        return "MVC exception: " + e.getLocalizedMessage();
    }

    @Override
    protected ModelAndView doResolveException(HttpServletRequest req, HttpServletResponse resp, Object handler,Exception ex) {

        ModelAndView mav = super.doResolveException(req, resp, handler, ex);
        mav.addObject("url", req.getRequestURL());
        return mav;
    }

}

In order to make use of this class, you must configure it in your bean configuration file. Here we map in a default error page called "error". We also pass in the exception attribute, which will give our view access to the exception object for reporting.

<bean id="simpleMappingExceptionResolver" class="com.in28minutes.controller.MyMappingExceptionResolver">
    <property name="exceptionMappings">
        <props>
            <prop key="java.lang.Exception">error</prop>
        </props>
    </property>
    <property name="defaultErrorView" value="error" />
    <property name="exceptionAttribute" value="ex" />
</bean>

Adding Rollbar's Java error monitoring SDK

Now that we have a simple exception handler wired up, we’re going to show you how to integrate Rollbar’s Java error logging SDK in your code.

Visit https://rollbar.com/signup/ and sign up for an account if you haven’t done so yet. Next, create your project and select Java from the list of notifiers. Select the server side access token that is generated for you. You’ll need this to configure Rollbar in the steps below.

Add a dependency for the rollbar-web notifier in your chosen package management system. For Maven, add the dependency below in your pom.xml file:

<dependencies>
   <dependency>
       <groupId>com.rollbar</groupId>
       <artifactId>rollbar-web</artifactId>
       <version>1.0.0</version>
   </dependency>
</dependencies>

For Gradle:

compile('com.rollbar:rollbar-web:1.0.0')

Now we will add Rollbar tracking inside the MyMappingExceptionResolver that we created earlier. It should go in the buildLogMessage method. Add the access token that we got in step one. We are also creating a request provider object to give us more context for debugging problems later.

@Override
public String buildLogMessage(Exception e, HttpServletRequest req) {

   RequestProvider requestProvider = new  RequestProvider.Builder().userIpHeaderName(req.getRemoteAddr()).build();
   rollbar = Rollbar.init(withAccessToken(accessToken).request(requestProvider).build());
   rollbar.error(e);
   return "MVC exception: " + e.getLocalizedMessage();
}

You can also use Rollbar to track caught exceptions, warnings, and other items using the same rollbar object. Learn more about the full API in our Java documentation.

Test with an example app

To test that it’s working, let’s create a page that will generate an error message. In the example below, you can generate an error by clicking the "Throw an error" button. To run the code, just check out Rollbar-Example-Java on GitHub. Clone the project and replace the access token with your own access token in the MyMappingExceptionResolver class.

This form adds a button which will post to the URL /spring-mvc/createException when you click it.

<form action="/spring-mvc/createException" method="POST">
    <center>
        <input style="height:50px;width:200px" type="submit"  value="Throw an error" />
    </center>
</form>

It will then trigger the throwException method. In this method, we have added a bug which attempts to call a method on a null object.

@RequestMapping(value = "/createException", method = RequestMethod.POST)
public String throwException(ModelMap model) {

    System.out.println("Error : here....");
    String exception = null;
    exception.toCharArray();
    return "error";
}

Since we wired up our MyMappingExceptionResolver to report the error to Rollbar, we should see the error show up on the Rollbar’s demo error live feed.

Grouped Spring MVC exceptions, as seen in Rollbar

Click on that item to see extra detail on the error. This provides more context to help you find the cause of the error. The Occurrences tab will show the request object that we provided, including the IP address of the client, which browser and operating system they were using, and more. The People tab will show which user was affected and how many times.

Spring MVC error and stack trace, as seen in Rollbar

It’s pretty easy to get error tracking across your entire application thanks to Rollbar and Spring’s global exception handlers. It only takes a few minutes to set up, and you will have way more context to track and debug problems faster in the future.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you take control of impactful Java errors. :-)

Announcing Our New Java Error Monitoring SDK

Fri, 17 Nov 2017 00:00:00 GMT

Rollbar is proud to announce the first official 1.0 release of our Java notifier SDK for error monitoring. Java continues to be one of the world’s most popular programming languages. It’s the most sought-after language for employers and has the second most pushes on GitHub according to Stackify’s 2017 rankings.

Rollbar helps you monitor errors in real-world applications. It provides you with a live error feed, along with stack traces and contextual data to debug errors quickly. It's important to select an error monitoring solution that supports all the languages your company uses for a clear picture across your whole stack from the front end to the back end. Some competing solutions just don't give you that full visibility. Learn more about our product features for Java.

This new version of our Java notifier SDK supports many frameworks including web apps, Android, Scala, Kotlin, Spring MVC, and more. In web applications, it captures data about the request including the user's IP address and browser so you can narrow the problem down. It helps you identify root cause quickly, by giving you contextual data on what changed and when from your source control and deployment systems. You can also react proactively by seeing which users were affected and prioritize fixes.

Grouped Java exceptions, as seen in Rollbar

Additional improvements in our 1.0 release include:

Support for both Gradle and Maven build systems.
Synchronous and asynchronous senders, so that your application does not block sending events to Rollbar.
A disk buffer queue so that you do not lose events even during network outages.
A new listener API so that you have the option of monitoring payloads.
The Android notifier has been unified with the main Java notifier.
An API that allows you to provide the notifier more custom information.
A GitHub examples folder for different integrations.

If you are upgrading from our previous 0.5.4 release of the Java notifier, see this guide for upgrading to 1.0. Since this is a major version release with new features, it will require some adjustments to your code to maximize all of the new functionality.

You can learn more in our Java notifier SDK documentation.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you take control of impactful Java errors. :-)

Laravel error reporting now available

Mon, 30 Oct 2017 00:00:00 GMT

We're happy to introduce our latest error reporting SDK, for Laravel, which tracks and reports errors that happen in your Laravel applications. Laravel is a popular and powerful MVC framework for PHP, designed for developers who need a simple and elegant toolkit to create full-featured PHP web applications.

In real-world production applications, it’s important to monitor errors so you understand your user’s experience and can fix issues before more are impacted. Rollbar helps by providing you with a live error feed from your application, including complete stack traces for instant visibility. To give you more context to debug problems, we also track the environment the error is coming from (prod or staging), the server that generated the error, and even the user’s session. Learn more about Rollbar’s product features.

Below, we'll show you how to add Rollbar error reporting to your Laravel apps and even give you a code example that you can try yourself.

Laravel error reporting with Rollbar

Here are some simple steps describing how to integrate Laravel SDK in your code. You can find more details in the Laravel Documentation.

Sign up for a Rollbar account if you haven’t done so yet.
Create your project and select Other from the list of notifiers. Copy the server side access token that is generated for you. You’ll need this to configure Rollbar in the steps below.
Open command prompt in your project directory and type following command to install the Laravel notifier SDK.
```
composer require rollbar/rollbar-laravel
```
Add the project access token from step 1 in your .env file. You can find .env file in your project directory.
```
ROLLBAR_TOKEN=[your Rollbar project access token]
```
Add the Rollbar service provider in the providers array config/app.php. You can find the config directory under your project.
```
Rollbar\Laravel\RollbarServiceProvider::class,
```
If you only want to enable Rollbar reporting for certain environments you can conditionally load the service provider in your AppServiceProvider. You can find it in the app/providers folder under project directory.
```
public function register()
{
    if ($this->app->environment('production')) {
        $this->app->register(\Rollbar\Laravel\RollbarServiceProvider::class);
    }
}
```
Add the following snippet in the config/services.php. You can find the config directory under the main project directory.
```
'rollbar' => [
        'access_token' => env('ROLLBAR_TOKEN'),
        'level' => env('ROLLBAR_LEVEL'),
    ],
```

The level variable defines the minimum log level at which log messages are sent to Rollbar. If not specified, the default is debug. For development, you could set this either to debug to send all log messages, or to none to send no messages at all. For production, you could set this to error or warn so that info and debug messages are ignored.

Testing Rollbar with an example Laravel application

To test that it’s working, let’s create a page that will generate an error message. In the example below, you will be able to generate an error by clicking the “Create an error” button.

Laravel example application

Check out our project on GitHub called Rollbar-Laravel-Example. To run it, clone the project and replace the access token with your project access token. You can find the access token inside .env file under your project directory.

When you click the “Create an error” button it will generate an error message. It triggers the MessagesController@submit action which has mapped to the home/submit url. In the submit method of MessagesController, we attempt to access the foo field on x, which is empty. This results in the error “ErrorException: Create default object from empty value”.

<?php
namespace App\Http\Controllers;
use Illuminate\Http\Request;

class MessagesController extends Controller
{
    public function submit(){
       try{
           $x = null;
           $x->foo = 5; // Null field access
       }catch(Exception $ex){
           echo "Caught exception";
       }
       return "SUCCESS";
   }
}

Viewing and reporting Laravel errors in Rollbar

Next open Rollbar to see what these Laravel errors look like in your account’s error occurrences page. The error we just generated should be called “ErrorException: Create default object from empty value”.

Laravel errors as, seen in Rollbar

Get more details by clicking on the error. You can now see a traceback showing you the exact source code file, method, and line number that generated the error.

Laravel error stack trace, as seen in Rollbar

For errors that impact many customers or are difficult to troubleshoot, Rollbar gives you additional context to solve them faster. With source code integration, you can click on the file to view in GitHub and see what changes were recently made. You can also see when the problem started, which deployment is suspected, and even which browsers and people are affected. I can also set alerts when the problem happens again. This will help you quickly determine the business impact, fix the problem, and proactively respond in the future.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you take control of Laravel errors. :-)

Custom error handling for Angular

Tue, 24 Oct 2017 00:00:00 GMT

Angular 2+ and AngularJS (version 1) are popular open-source JavaScript MVC frameworks that let you build highly structured, testable and maintainable front-end applications. Angular is most commonly used on single-page applications. Stable and reliable single-page applications depend on solid client-side error monitoring tools and techniques. But getting the right exception data and context isn’t always easy. We’re going to dive into how to capture, handle and debug Angular errors.

Error handling in vanilla JavaScript consists of using try, catch and finally statements. You can also use these statements in Angular modules. However, Angular has a special logic to handle uncaught exceptions. We’ll show you how to create custom error handlers for Angular that you can override to add your own functionality.

Error logging in Angular

The ErrorHandler class in Angular 2+ provides a hook for centralized exception handling. The default implementation of ErrorHandler prints error messages to the console. This service is very simple by design. To intercept the error handling we need to write a custom handler.

On the other hand, uncaught exceptions in AngularJS are all funneled through the $exceptionHandler service. When unmodified, $exceptionHandler sends all uncaught exceptions to the $log.error service. The $log.error service passes the error through to the client’s console.

Here’s how you can create your own error handler:

Angular 2+
AngularJS 1.X

class ErrorHandler {
  constructor() {}
  handleError(error: any): void;
}

$exceptionHandler(exception, [cause]);

In Angular 2+ the handleError(error: any): void method allows you to implement your own code to do something with the exception, such as recover your app gracefully or report it to an error monitoring service.

In AngularJS 1.X, when we call $exceptionHandler, we can pass two arguments:

exception: the exception associated with the error the application encountered.
cause: additional context about the cause of the exception.

The second argument, cause, is entirely optional. It can be helpful when you’re writing exception handlers or adding external handling functionality. $exceptionHandler will simply run on its own, and logging uncaught exceptions to the console in production isn’t very useful. Fortunately, Angular has a handful of ways to extend $exceptionHandler to take more action and gather more context when errors occur.

Basic configuration for handling the error in Angular

Let’s understand how error handlers work in both frameworks. In Angular 2+, error handling can be done by creating a CustomErrorHandler which implements ErrorHandler. In AngularJS 1.X we can extend $exceptionHandler to take more action and gather context when errors occur.

import {ErrorHandler} from '@angular/core';

@Injectable()
export class CustomErrorHandler implements ErrorHandler {
  constructor() {}
  handleError(error) {
    // your custom error handling logic
    console.log(error);
  }
}

angular
  .module('exceptionOverride', [])
  .factory('$exceptionHandler', function() {});

Now in our AppModule we have to add our CustomErrorHandler as a provider. In AngularJS can extend the module by using .factory(). This is an alternative to $provider.factory() inside a chained .config() on the module.

import {NgModule, ApplicationRef, ErrorHandler} from '@angular/core';
import {BrowserModule} from '@angular/platform-browser';
import {CustomErrorHandler} from './custom-error-handler';
import {AppComponent} from './app.component';

@NgModule({
  declarations: [AppComponent],
  imports: [BrowserModule],
  bootstrap: [AppComponent],
  providers: [
    {
      provide: ErrorHandler,
      useClass: CustomErrorHandler,
    },
  ],
})
export class AppModule {}

angular
  .module('exceptionOverride', [])
  .factory('$exceptionHandler', function() {
    return function(exception, cause) {
      exception.message += 'Angular Exception: "' + cause + '"';
      throw exception;
    };
  });

Now any exception in our application will be caught in our custom error handler.

Production error monitoring with Rollbar

In real-world production applications, it’s important to monitor errors so you understand your user’s experience. It’s best to monitor errors so you’re the first to know and can fix the issue before more customers are affected.

Rollbar tracks and analyzes what’s broken and why by using rich contextual data, including detailed stack traces, request params, telemetry on user behavior, affected users and more. This helps developers to quickly identify and fix errors. Learn more about Rollbar’s JavaScript features.

By including the official Rollbar{:target="_blank"} notifier SDK for Angular 2+ or the popular community notifier ng-rollbar for AngularJS 1.X in your Angular application, errors are automatically captured, grouped and reported—and with much more contextual data than the console provides.

The first thing you’ll want to do is sign up for a Rollbar account. Once you’re signed up, you’ll create a new project and select your Rollbar SDK.

Creating an Angular 2+ Project, as seen in Rollbar

Next, you'll see installation instructions and a client-side access token which you’ll need to use to send events to Rollbar.

Installing Rollbar

To install Rollbar in Angular 2+, simply run the command given in the Angular 2+ tab example below. In Angular JS 1.X, you need to follow some simple steps, shown in the AngularJS 1.X tab example.

Angular 2+
AngularJS 1.X

npm install rollbar --save

bower install ng-rollbar --save

// Then, include the SDK in your application with this script tag:

<script type="text/javascript" src="bower_components/ng-rollbar/ng-rollbar.min.js"></script>

Configuring Rollbar with your Angular app

In Angular 2+, you’ll need to update your application’s CustomErrorHandler; we will rename it to RollbarErrorHandler. Remember to add your own access token which you received earlier! You’ll also notice that we added a function called rollbarFactory in the example. This creates a named factory that is necessary when using ahead-of-time (AOT) compilation. If you want to follow along yourself, you can find our working example project on GitHub{:target="_blank"}.

In AngularJS 1.X, you’ll need to inject the library into your app and then add the RollbarProvider to your config. Remember to add your own access token, which you received earlier, as well as your environment name.

import * as Rollbar from 'rollbar';
import {BrowserModule} from '@angular/platform-browser';
import {
  Injectable,
  Injector,
  InjectionToken,
  NgModule,
  ErrorHandler,
} from '@angular/core';
import {AppComponent} from './app.component';

const rollbarConfig = {
  accessToken: 'YOUR_ACCESS_TOKEN',
  captureUncaught: true,
  captureUnhandledRejections: true,
};

@Injectable()
export class RollbarErrorHandler implements ErrorHandler {
  constructor(private injector: Injector) {}

  handleError(err: any): void {
    var rollbar = this.injector.get(RollbarService);
    rollbar.error(err.originalError || err);
  }
}

export function rollbarFactory() {
  return new Rollbar(rollbarConfig);
}

export const RollbarService = new InjectionToken('rollbar');

@NgModule({
  imports: [BrowserModule],
  declarations: [AppComponent],
  bootstrap: [AppComponent],
  providers: [
    {provide: ErrorHandler, useClass: RollbarErrorHandler},
    {provide: RollbarService, useFactory: rollbarFactory},
  ],
})
export class AppModule {}

var app = angular.module('myApp', ['tandibar/ng-rollbar']);

app.config(function(RollbarProvider) {
  RollbarProvider.init({
    accessToken: 'YOUR_ACCESS_TOKEN',
    captureUncaught: true,
    payload: {
      environment: 'YOUR_ENVIRONMENT',
    },
  });
});

Testing the sample application

You can test if everything was set up correctly by opening a console window and entering:

window.onerror(
  'TestRollbarError: testing window.onerror',
  window.location.href
);

or throwing an error in an application like this:

throw new Error('Hello World, Error Message');

Once everything is installed correctly, you should see the TestRollbarError in the Rollbar dashboard as shown below.

Dashboard for Angular errors, as seen in Rollbar

Manually handling errors with Rollbar

You may decide to manually handle errors. This gives you extra control over the information passed to Rollbar. Just inject Rollbar into the corresponding Angular component. The code looks something like this:

@Component()
export class MyComponent {
  constructor(private rollbar: Rollbar) {}
  manualHandle() {
    this.rollbar.error(this.errorMessage);
  }
}

myApp.controller('MainCtrl', function($scope, Rollbar) {
  $scope.onSomeEvent = function() {
    Rollbar.error('this is an error with special handling');
  };
});

You may flag errors by severity in both frameworks so that the most critical errors are seen and handled first. You can also pass extra data to help with debugging.

// Rollbar severities
rollbar.critical('some critical error'); // most severe
rollbar.error('some error');
rollbar.warning('some warning');
rollbar.info('some info');
rollbar.debug('some debug message'); // least severe

Debugging Angular errors with Rollbar

Now that you have Rollbar integrated into your Angular app, any errors that occur automatically get captured, grouped and reported to your new project in Rollbar. You’ll be able to quickly and easily see which errors are occurring and how often they occur, as well as the full context and analytics into all of your exceptions. You can set alert notifications on any of the errors and have them delivered to your email. Rollbar collects a wide variety of context data for each error, including detailed stack traces, request params, URL, environment, affected users, browser, location, host and much more. I hope this was a helpful introduction to error handling in Angular that shows how easy it is to get started.

[6]: https://github.com/mostlyjason/angular4-rollbar

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you take control of pesky Angular errors. :-)

Error Boundaries in React 16

Wed, 11 Oct 2017 00:00:00 GMT

React recently announced the release of version 16 with long standing feature requests including error boundaries, fragments, improved server-side rendering and more. Error boundaries are especially useful so that an error in one small part of the UI doesn’t break your entire application. Instead, it’s better to contain those errors to the affected components and recover gracefully.

You can try using error boundaries yourself in our working Codepen example. We forked it from Facebook's initial version to show how to add Rollbar's JavaScript error logging. When you open this page, you can activate an error boundary by clicking at least five times on the counter. You will then see the error message “Something went wrong.” Let's dive deeper into this example to learn how error boundaries work.

Codepen example, React error boundaries

What are error boundaries?

Error boundaries are React components that catch JavaScript errors anywhere in their child component tree. They can log errors and display a fallback UI to replace the component tree that has crashed. They are similar to try-catch statements in your code, but surround page elements instead.

An error boundary is a React component with a componentDidCatch(err, info) method. Any errors occurring in a component tree get reported up to the nearest error boundary’s componentDidCatch function. Note that error boundaries only catch errors in the components below them in the tree. For simple apps, you can declare an error boundary component once and use it for your whole application. For more complex applications with multiple components, you can have independent error boundaries to gracefully recover each part.

You can also report these errors to an error monitoring service like Rollbar. This will give you the ability to track how many users are affected by errors, find what caused them, and ultimately improve your user experience. We’ll show you a working example of how to do this at the end.

How to implement React error boundaries

Here are some simple steps describing how to implement them in your code:

Create a component class that extends a React Component and passes the props.

Add a componentDidCatch and render method as shown in the below example. The componentDidCatch method allows you to catch the error, and change how the component is rendered. For example, you can display an error message like “Something went wrong!”

class ErrorBoundary extends React.Component {
 constructor(props) {
   super(props);
   this.state = { error: null, errorInfo: null };
 }

 componentDidCatch(error, errorInfo) {
   // Catch errors in any components below and re-render with error message
   this.setState({
     error: error,
     errorInfo: errorInfo      
   })
 }

 render() {
   if (this.state.errorInfo) {
     return (
       <div>Something went wrong!</div>
     );
   }
   return this.props.children;
 }
}

Add your new component in HTML, surrounding the parts you would like to include in your error boundary. In this example, we are adding an error boundary around a buggy counter component.
```
<ErrorBoundary>
 <BuggyCounter/>
</ErrorBoundary>
```

Adding Rollbar to React error boundaries

For real-world production applications, it’s important to monitor errors so you understand your user’s experience. If your application is breaking, you don’t want to lose customers or have them complain on app reviews or social media. It’s much better to monitor those production errors so that you’re the first to know, and can fix the issue before more customers are impacted.

Thankfully, it’s pretty easy to add Rollbar tracking to your React error boundaries. We will show you how:

Open our previous Codepen example which has error boundaries configured.

Notice the first part of our JavaScript code, which configures and loads Rollbar. You can find the full details on how it works in our JavaScript documentation.

var _rollbarConfig = {
  accessToken: "3cda8fbafbfe4a6599d1954b1f1a246e",
  captureUncaught: true,
  captureUnhandledRejections: true,
  payload: {
    environment: "production"
  }
};

!function(r){function o(n){if(e[n])return e[n].exports...

Add Rollbar error tracking to the error boundary’s componentDidCatch method.

componentDidCatch(error, errorInfo) {    
 Rollbar.error(error); // Send it to Rollbar!
}

Load the page and trigger the error by clicking 5 times. Then, see what these errors look like in our public demo account’s Items page. You don’t need to sign up since this account is open to everyone.

Live error feed, as seen in Rollbar

Get more details by clicking on the item near the top of the list that says “Uncaught Error: I crashed!” You can now see a traceback and telemetry recording what happened before the error occurred. This provides more context to help you find the root cause of errors.

Error details page, as seen in Rollbar

We highly recommend you add error boundaries to your React applications. They are very easy to use, and not only help you provide a graceful recovery, but also report those errors to Rollbar so you can quickly fix errors before they impact more users.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you take control of React errors. :-)

Quickly edit and merge many error items at once

Wed, 20 Sep 2017 00:00:00 GMT

Rollbar just added a new feature that allows you to quickly edit and merge hundreds or thousands of items at once. In the past, you could only edit or merge items on a single page, which is limited to 30 items at a time. We added a new link that allows you to apply the change to all items in your view or matching a search. This feature provides a similar user experience to how Gmail lets you select all conversions, and then update or delete them.

This is useful to perform bulk updates, such as assigning many items from a single component to a developer to fix. You might want to merge multiple errors that are similar but come from different parts of your application. You could also resolve all your errors before doing a deployment so that you have a fresh view with the new version.

Thanks to our intelligent fingerprinting rules you only need to merge items once. New occurrences will remain in the merged item. This makes it much easier to manage the items in your account by keeping your view clutter free, and giving you accurate metrics on how often errors occur. As a result, you can focus your attention on new and high-impact errors.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you take control of impactful production errors. :-)

Introducing AWS Lambda Support for Node.js and Python

Thu, 07 Sep 2017 00:00:00 GMT

We’ve just updated our Rollbar.js and Python libraries, making it easy for you to monitor errors on AWS Lambda. If you’ve been considering building apps with serverless architectures on Lambda, we’ve got the exception tracking covered so you can rest easy.

Serverless architectures

Serverless architectures have taken resource abstraction to the next level.

We've now gone from having servers hosted and managed in the cloud, to having servers that require zero touch and are ephemeral in nature - they're spun up automatically only when certain events are triggered.

They also scale out automatically with usage, and whereas before you paid for compute resources you allocated, now you only pay the resources your app actually uses.

AWS Lambda

In AWS Lambda, currently the most popular service for building serverless apps, how much you use is based on how many times you trigger functions and how long it takes for those functions to execute.

Lambda functions come with limits, such as the concurrency execution limits. This is set at account-level and when you hit those limits, throttling kicks in. As a result, your functions don’t get executed and you get errors. This is just one of many things to consider and monitor when architecting your app to run on Lambda.

Monitoring errors in Lambda

Lambda uses Cloudwatch for monitoring and logging, providing metrics such as number of invocations, execution duration times, throttles, logs, and number of errors from failed invocations.

For richer and more granular information on errors that helps you debug them quickly, we recommend you use a dedicated error monitoring tool like Rollbar.

With Rollbar, you get complete stack traces, replay requests, track events by browser, IP address, affected users tracking, and much more.

Rollbar support for Node.js and Python on Lambda

Our Rollbar.js library now includes a new lambdaHandler function. Check out the docs here for more details, and get the latest version of Rollbar.js here.

For Python, our pyrollbar library has been updated to support Lambda as well. You can read the docs here for more details, and get the latest library here.

Give it a try and let us know what you think!

If you encounter any problems or have improvement requests, we welcome you to open an issue in Github. Since the notifier is open source, we always welcome your contributions.

If you haven’t already, sign up for a 14-day free trial of Rollbar and start monitoring for errors in your Node.js app on Lambda.

Introducing JavaScript Telemetry

Wed, 16 Aug 2017 00:00:00 GMT

We're excited to introduce JavaScript telemetry, which provides a timeline of events in the browser leading up to when an error occurred. This helps you find the root cause of JavaScript errors faster by providing critical information on user behavior, network activity, and more. You might also think of this timeline of events as a trail of breadcrumbs or a black box recorder which tells you what happened just before a crash or error.

In the screenshot below, we see a timeline showing a complete story of how the user encountered an error. First they loaded the page, typed their email address into the sign up form, validated the email, navigated to an onboarding page, and then the error occurred. This gives clear context on what caused the error and which component needs to be fixed.

JavaScript telemetry data, as seen in Rollbar

Troubleshooting can be difficult and time-consuming

If you’re a web developer, you know it can be difficult to reproduce unusual errors and problems reported by users. There are many variables including inconsistent browser environments, unexpected user behaviors, network calls, state changes in the application, and more. Reproducing problems costs your team time in development, QA and support. Unfortunately, if your team cannot reproduce the problem it’s often put on the back burner which can leave users hanging with no resolution.

Having enough contextual information can save you time because you can quickly see the cause of the problem without having to manually test different scenarios or debug your code. It also improves time to resolution because you don’t have to ask for the user for clarification about what they were doing, the complete context will be tracked automatically and available instantly.

How telemetry helps you fix impactful issues, faster

Rollbar helps you fix impactful issues faster by giving you additional context, including the following data:

Item	Description
Page load events	Tells you when the DOM content loaded and the full page loaded, to help distinguish errors before and after page load
User behavior	What users clicked on, typed into input boxes, or navigated routes in single page apps
Console messages	Log events and error messages that occurred just before the error
Network activity	XHR and fetch calls to APIs on own server or a third party’s
Rollbar exceptions and messages	If other exceptions or messages occurred leading up to this one, we'll include links to them in the telemetry timeline

An example of how to use custom Rollbar messages is to you can track is the full response data from API calls. This is particularly useful for seeing the status code from the server, which could indicate a problem processing the request due to rate limiting, authentication problems, and more.

Additionally, if you track a unique identifier such as a transaction ID you will be able to trace the error to a specific request on the server side. This provides you additional information to debug server-side problems with database connections, business logic, etc.

Unlike with logs, you don’t have to worry about having too much debug information tracked through breadcrumbs because the information is only sent in the case of an error. Providing enough debug information may help you fix edge cases even if you can’t easily reproduce it in your own browser.

Getting started with JavaScript telemetry

The first step to getting telemetry data is to make sure you’re using the latest version of the JavaScript SDK, rollbar.js. Version 2.2.0 or higher sends telemetry data back to Rollbar. It’s backwards compatible so you should be able to just drop the new version in. At this time, we only support telemetry data for client-side JavaScript so you only need to update your client-side code. If you’re using our client-side snippet or CDN library, it should update automatically. If you’re using bower or npm you will need to run the update command and then deploy the new version.

npm update --save rollbar

You can see the telemetry data in the Rollbar application on the error item view. The first tab is the Traceback tab, and the telemetry data will be shown at the bottom when data is available for the error item. You will be able to see more by clicking on the Occurrences tab. Seeing commonalities between multiple occurrences can help you identify what common conditions could be associated with the root cause.

JavaScript Telemetry, available within the Traceback tab, in Rollbar

To learn more about it please see the telemetry documentation. We also provide options to fine tune which types of telemetry data are sent in the rollbar.js documentation.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you take control of impactful JavaScript errors. :-)

Purpose-Built Error Monitoring Alternative to New Relic

Tue, 08 Aug 2017 00:00:00 GMT

One of the more frequently asked questions we hear is: "Doesn't my existing Application Performance Management (APM) solution, such as New Relic, monitor and track errors in our application?"

The short answer is - it’s not enough. They are very complementary and most Rollbar users use both (we do) for various debugging scenarios. APM tools are great at telling you what's slow, when and where exceptions occur. Rollbar provides a lot more contextual information to help you determine what's broken, why they occur, who is affected, and how to fix them. It saves you a substantial amount of time dealing with errors and leaves more time for resolving issues and improving your product experiences.

Let’s see how New Relic and Rollbar compare when it comes to monitoring and managing application errors.

Features	Rollbar	New Relic
Error tracking	✓	✓
Error analytics	✓	✓
Stack traces	✓	✓
Error frequency heatmap	✕	✓
Infrastructure monitoring	✕	✓
Transaction tracing	✕	✓
Notifications and alerting	✓	✓
Issue tracker integration	✓	✓
Filter by host	✓	✓
Data retention > 8 days	✓ (180 days)	✕
Custom rate limits	✓	✕ (100 events per minute)
Live error view	✓	✕
Git repo integration	✓	✕
Map errors to deployments	✓	✕
Raw payload per error	✓	✕
Replay requests	✓	✕
People affected	✓	✕
Telemetry	✓	✕
Local variable values	✓	✕
Custom error grouping rules	✓	✕

Overview of New Relic's Error Analytics

The New Relic APM and Browser products offer a high-level view of performance issues over time and where they are coming from. Error Analytics allows you to see when there is an increase in errors, and which transactions are generating them.

Error Analytics, as seen in New Relic

You can then drill down into the details for each error to see the exception message, stack trace, details about the request, and which host the error was generated on. These clues may be enough to determine the root cause, but more complex problems require more information to debug.

Error stack trace, as seen in New Relic

Furthermore, New Relic is good at monitoring leading indicators in your infrastructure or application performance that "may" eventually lead to an error. This can help you be more proactive in addressing issues before they affect your customers. For example, if a server has a high CPU or memory load it may eventually lead to errors in your application. Performance problems in your code can also lead to errors as timeouts. This is where the strength of APM really shines.

However, there are a few shortcomings when it comes to monitoring errors in large production environments.

New Relic’s Error Analytics only supports 7 days of retention of trace data. This severely limits the ability for your developers to troubleshoot problems that happened in the past, such as waiting for a new sprint to address them, bugs that were on the backlog, and more. Fortunately, Rollbar offers 180 days of retention.
New Relic caps your data at 100 events per minute per agent instance. The fact is that today's production services can process hundreds or thousands of transactions per minute. A single repeated error could drown out key events with clues to the root cause of the problem. That's why Rollbar allows you to customize your data caps for each service, and we store each occurrence without sampling.

How Rollbar takes error monitoring a step further

Rollbar gives you a live view of your errors up to the last few seconds, helps you debug problems faster, automatically tracks whether those problems are resolved, and helps you follow up with your customers to ensure amazing product experiences.

That’s why it’s become an essential part of the DevOps tool belt. A modern DevOps organization functions best when it has a solution for application performance management, error tracking, and log analysis. This extra context is especially important when using Continuous Development. Since you may be deploying changes many times per day, you need to be able to monitor for errors continuously, and quickly determine which change caused them.

Rollbar doesn’t just help developers. It also tells you who was affected so your product management team can prioritize fixes for issues that impact many customers or high-value customers. It also helps your customer support team proactively reach out to improve customer relationships.

✓ Monitor application errors live, in real-time

When critical errors occur in production you need to know quickly so you can solve them before more customers are impacted. Rollbar’s live error view will show you errors occurring in real time. Below, we can see the last error occurred only 37 seconds ago. This is a great page to watch during and immediately after deploys to see if the latest changes broke anything. It's also useful when interactively debugging an issue or verifying that it’s been fixed. For example, you may want to replay the command to see if they error occurs, and you can watch to see if it shows up in real time on this page.

Live error feed, as seen in Rollbar

We also give you a high-level summary of the top errors in the last day. This table is sorted to show critical errors at the top ranked by how often they occur, followed by less serious levels of errors below. What’s unique on this page is that we automatically recognize which errors are new or have reactivated. this lets you know whether the error was caused by recent code change, or whether it’s a regression on a previously resolved issue.

Dashboard view, as seen in Rollbar

✓ Debug production errors faster

Rollbar also gives you access to a variety of features that help you debug problems faster. Let’s start by looking at the detailed view of the same error message we were just looking at earlier in New Relic. You can see how often it happens over time, see a detailed traceback, a history showing when they error began, and what changes have been made since then. You can correlate the errors to browsers, operating systems, users, IPs, and more. It also integrates with your issue tracking system so you can create a new bug report or resolve an existing one. With compatible notifier SDKs, you also get access to local variables and telemetry, which we will describe in more detail below.

JavaScript telemetry data, as seen in Rollbar

✓ See local variables when your exception was thrown

When you're debugging an exception, you can use our local variables feature for insight into their values when the exception was thrown. In the past, you'd have to reproduce locally in your debugger, or add an additional rollbar.report_message() to log each variable in production. But now you can just press "locals" for the stack frame and see it instantly. This feature is available for compatible notifier SDKs like Python.

Local variables in the stack trace, as seen in Rollbar

✓ Telemetry to send events that occurred prior to your error

Our JavaScript telemetry provides a timeline of events in the browser leading up to when an error occurred. This helps you find the root cause of errors faster by providing critical information on user behavior, network activity, and more. You might also think of this timeline of events as a trail of breadcrumbs or a black box recorder which tells you what happened just before a crash or error.

JavaScript Telemetry data, as seen in Rollbar

✓ Connect Git repos and link stack traces to broken code

Rollbar helps you find the root cause of issues faster by letting you drill down into the code where this error was generated. It works because Rollbar integrates with your source repository. By clicking on the link in the traceback, it opens the code in GitHub, Bitbucket, or GitLab and highlights the line where the error occurred. This helps you see the root cause almost immediately. New Relic doesn’t offer this kind of deep linking of error stack traces to Git repositories.

Linked stack traces to git repos, as seen in Rollbar

✓ See which code deployment caused your error

Rollbar also shows you the deployment that we suspect has caused this error to occur, and the changes that have been made since the previous deploy. This is especially important to see who made the changes and whether there is a potential interaction when multiple changes happen together. If you need to roll back your deployment your product management, it links you directly to the pages where you can do so quickly.

Deployment tracking, as seen in Rollbar

Clicking on one of the changes will open up the diff view in your source repository. Here we can see the exact commit where the error was added into the code.

Deployment tracking, as seen in Rollbar

✓ Drill down into the raw payload for each error occurrence

For more complex errors, you may need to see the raw payload to see what additional information was passed to Rollbar when reporting the error. This is really useful because you can attach additional debug information in your code, or even entire objects or request payloads. This can help you debug issues where there are unusual request parameters or processing sequences. Rollbar also stores every individual instance of the error and payload, whereas New Relic will only provide you a sampled summary. If you are debugging a problem that only happens occasionally or with specific parameters, it’s important to have every instance recorded.

Raw JSON error payload, as seen in Rollbar

✓ Replay the cURL command and quickly reproduce errors

If you want to reproduce the error, you can also replay the occurrence by issuing a new HTTP request with the same parameters and header information. This is useful when debugging the problem because you can repeat it and then analyze what happens interactively, especially when there are multiple interconnected services that are difficult to track on a single box. It’s also useful after you’ve patched the problem and want to verify that it’s working correctly.

cURL replay, as seen in Rollbar

✓ Resolve errors, track regressions and affected users

Rollbar automatically resolves errors and give you the peace of mind that it will be tracked and reopened if your fix does not work. You can be automatically notified of regressions. You can also track which customers are affected by errors, which empowers your customer support and product management teams to deliver a good customer experience.

After you’ve fixed an error, you’ll want to make sure that the error is resolved in production. Also, you'll want to make sure that it doesn’t come back again. Rollbar can automatically mark errors as resolved through its integration with your source control and deployment systems. Just include a reference or link to the error in your commit message, then we will track when it’s been deployed and reactivate the error if it reoccurs.

Below, you can see the dashboard page which displays the top 5 new and reactivated errors. This saves you time because you don’t have to make a reminder to manually go back and check for each individual error after your code is deployed. Rollbar will automatically reactivate errors if they are not fixed after the deployment.

Workflows, as seen in Rollbar

✓ See who is affected, how many times, by what errors

Customer support and operations teams may want to know which customers were affected by production issues. This can help them communicate proactively and offer additional support. It also lets them see how many different issues each customer has been impacted by, to get a good idea of the customer’s experience over their lifetime with the company.

This can also help your product management team prioritize which bugs to fix, either by the number of people affected, or by prioritizing errors that affect high-value customers. It’s better to rely on analytics than on customer support tickets or social media complaints. The data is more complete because only a small percent of customers will complain. Also, support and social media channels create extra delay identifying and fixing the problem. Rollbar will help you find the root cause and fix it proactively before more customers are impacted.

People tracking, as seen in Rollbar

✓ Custom error grouping rules

Grouping allows you to see a summary of the errors in your application instead of being overwhelmed by each individual occurrence. This makes it easier to see when new errors pop up, how often they happen, and when they reactivate after being resolved. Most APM's have default rules for grouping errors, but they offer you limited flexibility in changing them. This can create problems because if the groups are too coarse-grained, you will lack visibility needed to solve the errors. Conversely, if they are too fine-grained, you can be overwhelmed with data while lacking insight. Fortunately, Rollbar allows you to customize your grouping rules so they are perfectly tailored for your application’s needs. You can create these custom rules using a simple JSON configuration.

Custom error grouping rules, as seen in Rollbar

We both have our unique strengths, which is why we encourage our customers to use both tools together. New Relic is a great infrastructure and performance monitoring solution, which can tell you when and where problems have occurred. Rollbar gives you the extra context to determine why the problem happened, who is affected, and make sure that it’s resolved.

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take your application error monitoring a step further. :-)

Security matters: Rollbar now offers advanced encryption at rest

Wed, 19 Jul 2017 00:00:00 GMT

Are your services secure?

In today’s world, you can hardly go a week without reading in the news about security breaches, malware, and more. We’ve already had headline news this year for Wanna Cry, and now there are dozens of copycat malware programs taking advantage of out-of-date systems. Think of all the services that your company uses from error monitoring to logging and APM. Some of them may be delivered by vendors and others set up by internal teams. Did your IT team evaluate these services to determine how secure they are? If not, you might want to reconsider the services you use or who can best deliver them.

Unfortunately, many companies only do the bare minimum due to time constraints, limited budget, lack of expertise, etc. Real security is hard because threats are constantly evolving, and they target the weakest link across all layers of your organization. Companies make good targets because they concentrate data from potentially billions of customers in one place. Their infrastructure can have millions of lines of code that no one person fully understands. They often rely on a complex and poorly monitored network of third-party service providers, offshore contractors, platforms, modules, etc. These all present opportunities to attack.

Sophisticated companies evaluate the security of their service providers when making a purchase, and they hold internal services to the same standards. They typically have a checklist of security requirements covering certifications, security audits, data management processes, etc. Companies in the EU or US-regulated industries like finance and healthcare may have additional legal requirements like PCI-DSS and HIPAA. They require their service providers to be complaint, particularly when sensitive customer data is being handled. Even if your company is not regulated in this way, these certifications can signal that a company takes security seriously.

Rollbar's advanced encryption at rest

Rollbar provides strong security to all our accounts through encrypted data transmission, secure authentication, regular penetration testing, and more. Additionally, we deliver a higher level of security to customers to who sign up for our Compliant SaaS solution. This solution complies with security and privacy standards like ISO 27001, CSA STAR, US-EU Privacy Shield, and HIPAA for healthcare data. Our data center meets the stringent SOC 2 Type 2 standard, among others. We also implement a number of procedures for risk management and communication with our customers. Compliance is important because you don’t need to take our word for it, we have passed rigorous audits to prove it.

One key feature that lets us deliver better security is our encryption at rest. This means the data is encrypted when it’s stored inside our data center. Many other services do not offer encryption at rest, and store customer data in plaintext. This allows hackers to download the data if they are able to get access to one of the servers. This is how hackers were able to access millions of customers’ data in the Sony data breach.

Some companies often confuse encryption at rest with disk-based encryption. This encrypts the hard disk so that if someone were to physically steal the hard disk they would not be able to access the contents. However, most attacks target computers while they are running and the disks are unlocked. That means malicious code or hackers could gain privileges using a vulnerable service on the computer and then read files in plaintext while the computer is running. Also, a breach could compromise all customers whose data on is on the disk.

Rollbar goes above and beyond to offer a more advanced form of encryption at rest. We encrypt in the application layer and use unique keys for each customer project. This prevents attackers from reading files in plaintext while the computer is running. It also separates customers and projects to limit the scope of an attack. We encrypt using the AES-256 GCM algorithm, the same one used by Amazon Web Services (AWS). Furthermore, we encrypt our customers’ historical data, in addition to data that’s newly coming in.

See it in action

You can see the status of encryption at rest under Account Settings –> Encryption. You can see when the encryption was activated and when the historical data encryption is complete.

Encryption at rest as seen in Rollbar

Encryption at rest is currently available to customers of our Compliant SaaS solution. For more information, contact us to talk to a sales representative.

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your application errors.

Merging versus Custom Fingerprinting

Fri, 23 Jun 2017 00:00:00 GMT

With the recent announcement of support for manual merging, there are now multiple ways to combine occurrences of errors into a single item.

Before merging was an option, if the default fingerprinting algorithm didn’t combine occurrences the way you wanted, then you needed to define custom fingerprinting rules. Custom fingerprinting rules require you to learn our JSON-based rule syntax, and that could be a deterrent against setting them up.

Now that you can easily merge errors via the UI, is there still value in setting up custom fingerprinting rules? Absolutely, and this blog post will explain why!

Rollbar’s Default Fingerprinting Algorithm

To understand merging vs. custom fingerprinting rules, it helps to first look at how Rollbar combines occurrences of errors.

Minus a few tweaks we’ve made over the years to optimize results, our default algorithm works as follows:

Combine the filenames and method names from all of the stack frames
Append the exception class name
Take the SHA1 hash of the result

The resulting SHA1 hash is what we call the ‘fingerprint’ of the error. All occurrences that have the same fingerprint will be combined into a single item.

The default behavior can be changed to include the exception message, or any other occurrence data, when calculating the fingerprint too. This will result in more unique errors in your Rollbar projects, so use it carefully.

Custom Fingerprinting Rules

The default fingerprinting algorithm is a good start and may be sufficient for many projects, but often you’ll want to tweak the behavior. Some of the most common reasons for creating custom fingerprinting rules include:

You want to combine all occurrences of a certain exception class into one item, regardless of the stack trace.
For some (but not all) types of errors, you want to have more unique items created (e.g. include the exception message or the server hostname).

Custom fingerprinting rules allow you to specify precise logic for combining occurrences and generating fingerprints using a JSON-based syntax:

{% assign body = site.data.docs.custom-grouping.body %}

[
  {"condition": {"path": "body.trace.exception.class", "eq": "TimeoutError"},
    "fingerprint": "timeout-error"
  },
  {"condition": {"path": "body.trace.exception.class", "eq": "IndexError"},
    "fingerprint": "index-error-{{body.trace.exception.message}}"
  }
]

In the example above, all occurrences of TimeoutError will be combined into a single item while occurrences of IndexError will be combined only if they have the same exception message.

Custom fingerprinting rules allow for very sophisticated fingerprinting behavior, but also require a bit of learning to work with the JSON syntax. In addition to the learning curve, custom fingerprinting rules have the following limitations:

Custom fingerprinting rules only apply to future occurrences, not those which already exist in your project.
Once occurrences are combined by a custom fingerprinting rule, they cannot be split into separate items.

Merging

If there are errors in your project that were not combined by the default fingerprinting algorithm nor your custom fingerprinting rules, then you can manually merge them together.

In addition to the simplicity of merging via the UI, it also has the advantage of allowing you to un-merge in case you made a mistake or change your mind about how to organize your errors.

For more information about merging, check out our announcement blog post and our merging docs.

When to use custom fingerprinting rules vs. when to merge

Now that you can merge errors in the UI, why go to the trouble of creating custom fingerprinting rules?

When errors are merged via the UI, all past and future occurrences that have the same fingerprint as one of the member items will be added to the resulting group item.

Suppose that you want all occurrences of OperationalError to be tracked as a single item. If you just merge all the existing OperationalError items, then any future OperationalError that has a different stack trace is going to be treated as a different error because the default algorithm assigns a different fingerprint, and you’ll need to continually merge every new OperationalError with the old ones.

If you create a custom merging rule to assign the same fingerprint to all OperationalError occurrences, then you will not have to have to merge any future occurrences because the system will do it automatically.

Suppose you’ve already got several different OperationalErrors, and you want to track all past and future occurrences as a single item. The steps to do this are:

1. Merge all your existing OperationalErrors into one merge item.

2. Create a custom fingerprinting rule to combine the OperationalErrors and use the fingerprint of the group item (In this case #311) created in the previous step.

[
  {"condition": {"path": "body.trace.exception.class", "eq": "OperationalError"},
   "fingerprint": "group-item-311"
  }
]

Once this custom fingerprinting rule is saved, every occurrence of an OperationalError will be added to item #311

This is just one among many use cases that illustrate why custom fingerprinting rules continue to be useful along with merging.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you take control of your application error monitoring. :-)

Technical details and challenges of building Error Merging

Mon, 19 Jun 2017 00:00:00 GMT

Hopefully you've had the chance to try out our latest feature, error merging. We've heard a lot of positive feedback from our users. They're especially excited to be able to easily merge and un-merge related errors. We thought it would be useful to share how the Rollbar team made this happen from a technical standpoint. If you're interested in the nitty-gritty of how we implemented error merging, read on.

I interviewed Todd Dampier, one of the engineers here at Rollbar who was instrumental in making error merging possible, about what was involved in engineering this feature.

First, let's start with why we created error merging in the first place. Our users were asking for the ability to merge (and unmerge) their errors via our UI, as opposed to having to create custom fingerprinting rules. Prior to creating error merging, if you wanted to merge two items together, you created a custom fingerprinting rule, which doesn't apply retroactively. You also could never unmerge items that had been merged together. As Todd explains it, "custom fingerprinting is extremely static and only applies when the rule is in existence. You can look at an occurrence of an item and extract the fingerprint, and it is always associated with the item. Item merging is very dynamic. You never change the fingerprint of a given occurrence, and that fingerprint could even come from custom fingerprinting. You can change your mind over and over. Merging is a tool for managing complexity. You can take your dashboard from a zillion items to a tenth of that."

In order to delve into the details of how our engineering team implemented error merging, let's define some terminology we'll be using:

|-------------------|-------------| | Occurrence | An instance of an error item. | | Native Item | An aggregation of occurrences that all share a distinct fingerprint. This fingerprint is computed by either custom fingerprint rules or Rollbar's default fingerprinting algorithm. | | Group Item | An item comprising one or more Native Items that have been merged together under user direction. | | Item | Either a Native Item or a Group Item. | | Merge | The process of creating or altering a group item by adding one or more Native Items to its constituents. | | Un-merge | The process of altering (or altogether destroying) a Group Item by removing one or more Native Items from its constituents.| | Merge Transaction | A unique identifier created whenever a user merges or un-merges items. |

Implementing error merging was tricky, since we needed to be able to merge and un-merge every Group Item without changing any behaviors of the Native Items. To do this, the team created dual identities for items: Native Items and their associated fingerprints, and other aspects that are merged into an umbrella item (the Group Item). A Group Item, under the hood, "both is and isn’t an item in its own right," according to Todd. It is a single point of visibility of the constituent Native Items, where they can be managed and controlled. All the Native Items in a Group Item can become activated, muted, and resolved together. In order to do this, we needed to tease apart every aspect of item behavior and decide whether to approach this in a top-down (the property is defined by the Group Item) or bottom-up (it is defined by the Native Item) fashion. For example, the team needed to decide if you should be able to search for a Native Item's original title and find the Group Item or not (we decided you should).

The team implemented error merging as a logical view on top of Native Items. All Native Items are still captured using the current fingerprinting algorithm, as they were previously. We now also store a mapping from a Group Item to each merged Native Item it comprises in our database. This allows constituent Native Items to be un-merged at any time, and their original item counts will still be accurate. One interesting thing to note about Group Items is that they can be merged with other Group Items, yet the data structure we maintain is flat. We discard the intermediate Group Items in such cases, keeping only the final resulting Group Item. What this means is if two different Group Items are merged, the Native Items forming those two original Group Items will now constitute one new Group Item. One of the two previous Group Items will be 'archived' — that is, no longer exist as an active object.

In order to track these Native Items to Group Items mappings, the team created an entirely new table in our database. In this table there is a unique transaction ID associated with every merge and un-merge action, so there is a complete historical record of every merge or un-merge a Native Item undergoes. Since items are the meat of what Rollbar deals with, error merging touches a lot of the tables in our database. We added some new columns into our item table to delineate whether or not a given item is a Group Item or a Native Item. If a Native Item, the status column notes if the item is active, resolved, muted, or part of a Group Item. If the Native Item is part of a Group Item, the status of the item is determined by the status of the Group Item. When two items are merged, a new Group Item is created in our item table and the constituent Native Items are updated (but we do not notify you of a new item according to your New Item notification settings). When we update all the tables that are effected when a Group Item is created or changed, we lock the Group Items and Native Items respectively to isolate the calculation and database updates and avoid race conditions.

There are top-down properties and bottom-up properties that are associated with an item. Top-down properties of a Group Item are set on the Group Item and affect the Native Items that constitute the Group Item. These include the level of the item, the status, the assigned user, timestamps denoting when the item's status was last set to active/muted/resolved, and the users who are subscribed to the item. Bottom-up properties of a Group Item are determined by the Native Items in the Group Item. Some bottom-up properties of a Group Item include the IDs and timestamps for the first occurrence, last occurrence, and activating occurrence, as well as the total number of occurrences. When a Native Item is un-merged from a Group Item, all of the properties that were top-down now must be set on the Native Item. This includes users who were subscribed to the Group Item; all subscribers are thus subscribed to the Native Items that once constituted the Group Item.

Let's walk through what all of this means in practice. What happens when an item hits our pipeline? For any occurrence, we first store it in our database, generate a fingerprint, and find a row associated with that fingerprint in our item table. If it's an occurrence of an unmerged item, that row correlates to a Native Item. The code then checks if that Native Item is associated with a Group Item or not. Since this is an unmerged item, it isn't associated, and we continue down our processing pipeline as normal. For an occurrence of a merged item, we will find a reference to a Group Item ID. We then increment the total occurrences for both the Native and Group Items, and update other tables as needed.

While we have been discussing the back end of how error merging works, it's important to note that implementing the user experience side of this was no easy feat either. We needed to create new views for Group Items, including all of the associated functionality that a user would expect, such as the ability to comment on the item. Additionally the team implemented restrictions on a user's ability to edit a Native Item once it has been merged into a Group Item. A wide variety of user interface considerations were discussed, including how to delineate visually the difference between a Native Item and a Group Item on a user's items view page. Some new user interface components specific to error merging were created, such as a popup that allows you to toggle between editing and merging items.

Given how complex implementing all of this was, I asked Todd what his favorite part of working on error merging was. "The relational database aspect of it. You can always go to the database and cut through the layers of code. The database ultimately offered a way out of the stickiest corners of the functionality."

To learn more about how to merge your items, check out the docs here.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you take control of your application error monitoring. :-)

How Clubhouse uses Rollbar to monitor JavaScript errors

Wed, 14 Jun 2017 00:00:00 GMT

I'm eager to share an insightful interview our friends at Changelog recently did with Andrew Childs, CTO at Clubhouse and Rollbar power-user. We're big supporters of the Changelog podcast and we asked them, to help us produce a handful of interviews with our customers. It's a fun project that lets us pull back the curtain and learn more about our customers processes for handling errors and deploying code. Read. Listen. Enjoy!

Featured in this interview: Adam Stacoviak, Founder & Chief Editor at Changelog, a podcast on software development and open source. Andrew Childs, CTO of Clubhouse, an easy-to-use project management tool for software teams.

Adam: Andrew, let's start off with you telling me a bit about Clubhouse.

Andrew: Clubhouse is a software company based in New York, building project management software, specifically for software teams that are looking for something simple and flexible but gives them a little bit more visibility into what the whole team is doing and where the team is going at a high level.

Adam: How important is error tracking to Clubhouse?

Andrew: It's really important, because our software isn't perfect. You could see if you looked at our Rollbar, we have a constant stream of errors coming in that need to be fixed. People are leaving these windows open for weeks and they've got all these third-party extensions loaded, and those extensions are doing things to the page, rewriting things and rewriting stuff on DOM and it's like you're in a very hostile environment and you can only do so much to guard against that locally when you're testing things yourself.

You can only do so much to guard against errors locally when you're testing things yourself.

Live Error Feed as seen in Rollbar

Adam: What's your policy on errors? Do you have a no-error policy?

Andrew: Not really. The front-end team here is a small team and we don't have a lot of formal processes in place yet.

Adam: Does Rollbar help you organize that better then? Since you have a small team, Rollbar's awesome because it helps you visualize those things and categorize things and keep track of those errors better than maybe having a slightly larger team where you can squash bugs faster?

Andrew: Yeah, absolutely. It does a good job of combining similar JavScript errors. Like, if multiple people are having the same issue, it will combine all those. There's a good overview of how many people an error is affecting. If it's just one single who's on Linux, running an old version of Firefox, then we know; or if it's something that's affecting everybody.

Person Error Tracking as seen in Rollbar

Adam: It helps you power test too, I assume, which is important at your stage.

Andrew: Yeah, absolutely. Whenever we get a new error that comes in, it goes to our Slack channel so we're notified right away whenever we have a new issue. The page on Rollbar that has the error information is nicely laid out. Since they've got source-map support, we can actually see exactly where in the code - even though our code is minified, we can see exactly which line is causing the issue.

Whenever we get a new error that comes in, it goes to our Slack channel so we're notified right away whenever we have a new issue.

Rollbar Notifications in Slack (Deploy, New Error, Resolved Error)

Adam: Does it deminify your code then?

Andrew: It does. It's great! Rollbar's source-map feature takes the minified JavaScript code, it goes line by line and says. Yeah, it basically deminifies the code for really accurate error detection and mapping.

Source Maps as seen in Rollbar

Adam: The help of Rollbar is obviously in tracking errors, wondering what's going on; having an application that actually works is the entire point, right? So what was the process like for monitoring, tracking, fixing bugs before you started to work with Rollbar? Did you use something else? What were the tools that you might have used?

Andrew: We started out using actually Google Analytics to basically track bugs, but Google Analytics isn't built for error tracking at all, really; it's built for counting pageviews and unique visitors, conversions and things like that. That didn't work, so we were looking around, we found a tool called Sentry which did a little bit better. It gave us some stack traces and it gave us a little bit more information, but it was not exactly working very well for us. So we started looking around and we found Rollbar, switched over to that in probably about an hour.

Adam: What were some of the challenges you faced in general and how did it just not work right for you, and ultimately what was the thing that made you feel like "I need to find another solution because this isn't working"?

Andrew: Previous tools provided some information about the exact error that we were getting, but we weren't able to really drill down and see who it was affecting and how often, and there were just some usability issues around the other products we tried.

Adam: It sounds like there was a lack of visibility to the details. You need access to specifics, "Hey, if you're gonna tell me my application is broken in this area, then give me on this information. Who else has had this issue? Is this issue reoccurring? What code is tied to it?", things like that.

Andrew: Actually I just found the story that I had from November of 2014 to switch over from Sentry to Rollbar. You can basically send anything to Rollbar, you can send arbitrary data, and I think at the time at the other tools where lacking and not as robust.

Adam: Right. So not only just tracking the error, but tracking things like customer name, location in the world, things that matter to you when you track errors. Kind of like customizability.

Andrew: Yeah, I mean the thing is that we could not only send errors, but also track user interactions and start looking at a specific person to see what chain of interactions lead up to this specific error happening.

Adam: What were some of the challenges you were facing related to error tracking? You said you've got a small team, so those are the ones that are kind of tracking errors, it seems, unless I'm wrong. But talk to me about some of the challenges faced in general when trying to track errors and trying to monitor for errors, and ultimately trying to build a better product.

Andrew: It's a small team working on the web app, and on the client-side we don't have a QA team, so it's us doing all the QA and all the firefighting when errors happen. There is definitely a tension between having a rigorous integration test suite and having test-run everything, and then on the hand having a barebones integration test suite and the fixing things quickly in production. Obviously, you don't want errors to be happening all the time, you don't want errors to be happening... Basically, every error that happens, you lose credibility in the face of your customer, and what Rollbar let us do is to very quickly react when a error actually happens, and keep errors from becoming huge issues for us as a company.

Basically, every error that happens, you lose credibility in the face of your customer, and what Rollbar let us do is to very quickly react when a error actually happens, and keep errors from becoming huge issues for us as a company.

Adam: No matter how good your engineers are, no matter how many cups of coffee you drink that day, no matter how alert you are, you're gonna have errors. Everyone writes errors, so it's important to not only find errors and fix them fast, but be able to really understand why the error came about. Can you talk a bit about the Rollbar interface, how you can look at an error, see different customers it might have interacted with, maybe actually unfurl the actual code there - can you talk to me about that piece there?

Andrew: Let's see I'm looking at a specific error - I can see the exact stack trace for that error; because we had source-maps enable, we can see exactly what line caused the issue, not just line 6000, column 20. We can actually look at the stack traced and know exactly what the problem is, which is great. On top of that, we can see who it has affected, how often it's happened over the last 60 days (which is great) and with any alert we can pass in arbitrary data, so we can pass in basically a list of interactions that lead up to that point in time. User clicked on story, open story, update the owner, and if that was the last thing in the action, then we know it's possible that it was related to that.

Because we had source-maps enable, we can see exactly what line caused the issue, not just line 6000, column 20. We can actually look at the stack trace and know exactly what the problem is, which is great.

Adam: So it's definitely given you a trail of breadcrumbs to follow, much better than obviously just knowing an error occurred and the piece of code it touched. You actually have some other information that you're able to send to Rollbar through the way you interface with it as you see fit. It's something that is customizable to your needs.

Andrew: Right. And error reporting in JavaScript can be extremely difficult and extremely opaque. You'll see a lot of random type errors and unknown script errors that if you were just looking at the name that JavaScript gave you, it's basically useless. And without source-maps enabled, the stack traces also can be useless as well, because it's just showing you some minified and concatenated file.

Adam: You've got a small team, right? Speed matters, so being able to reduce the amount of time it takes for you to investigate an error, maybe even fix the error that day and ship out to production and solve the problem once and for all... Talk to me a bit about examples of reduced time and how important that is. Do you have any particular examples where because of Rollbar you were able to track the error and then see the code quickly and fix the error that day, potentially?

Andrew: Absolutely. It actually just happened last week, we did a deploy and a couple minutes later everybody got a new alert in Slack, and because we had source-maps enabled, I knew exactly what the problem was and we had it fixed and deployed within five minutes. It's not like that every time, but it happens often enough that it's a huge benefit for us.

Last week, we did a deploy and a couple minutes later everybody got a new alert in Slack, and because we had source-maps enabled, I knew exactly what the problem was and we had it fixed and deployed within five minutes.

Adam: Talk to me about how important it is to have a tool like that available to you. Like you said, it doesn't happen every time, but it's a possibility that it can not only track the error, but it can also give you the insight to be able to use your own brain as an engineer - look at the code, figure it out and actually within five minutes solve a potentially big problem. How important is that to you?

Andrew: It's really important. Actually we not only fix the error, but we can contact the user directly and say, "We noticed that you're trying to do this one thing... It's fixed now." Rollbar lets us be proactive and fix impactful issues that could be a disaster for us in terms of how people perceive us and the quality of our product. It basically guards us against that.

Rollbar lets us be proactive and fix impactful issues that could be a disaster for us in terms of how people perceive us and the quality of our product.

Adam: I imagine a small team like your is pretty customer-focused because you're a startup. You're obviously counting every user experience you get, wanting that user experience to be positive. How has that specific thing where, since you're able to send arbitrary data to Rollbar and actually which customer interfaced with this error, had this error, got hit with this error - how has that changed how you interface with customers on a support level, even if they haven't said, "Hey, I have this error" because Rollbar told you, they didn't tell you through support, or something like that. How has that changed how you focus on and deliver customer happiness?

Andrew: Yes, it can be really difficult to investigate and figure out what is going on on a customer's screen, and pretty much the first thing that we do is actually going to Rollbar and take a look at the error feed. Since Rollbar lets us drill down and see specific people, we can see what if anything that user's been doing to kind of give us an idea of what possibly could have gone wrong. So it's definitely helped with the initial conversation that we have, where we don't have to say "What browser are you using?" - we know that already. We know exactly what version of the code they're looking at, which is great, because we can do a deploy, but a user might have that tab open for two weeks and is running two-week-old code.

I hate to admit how many errors we've had to fix, but there've been many cases where we've been able to fix whatever issue the user reported within minutes, if not before the report of the error, because they were still looking at the old version of the page, so all we had to do was say "Just refresh your page, it's already been fixed." Their response is almost always like, "Wow, you guys are amazing."

It's pretty great to be able to take what should be a crappy situation where we have a error and turn it into a success story where the customer is actually really happy about us.

Adam: Now be honest with me... Don't say this because I'm asking you to, but based on this entire conversation, could you build a successful version of Clubhouse, keep customers happy, do all the things we've talked about in this conversation without Rollbar?

Andrew: It's very safe to say that we wouldn't be where we are right now if we weren't using Rollbar.

Adam: That's awesome. Thanks so much for your time today, Andrew, I appreciate it.

Thank you to Andrew and the team at Clubhouse for sharing this level of insight and for leading by example when it comes to error tracking and working to maintain error-free experiences for their users.

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your client-side and server-side application errors.

Turning Errors Into Action: How and Why Resolving Errors Matters

Tue, 30 May 2017 00:00:00 GMT

Why resolving errors matters?

After fixing a bug, who is responsible for making sure if it really resolves the customer's problem? A lot of companies take a fire-and-forget mentality where the developer makes a code change, and they never think of the problem again until someone complains. Developers often assume that the fix will be deployed with the next release, that the fix will behave the same in production as it does in their development environment, and that it resolves every case uniformly. Only for the most urgent problems will they wait for the fix to hit production and then verify the improvement on the customer's side.

Let's face it, verifying the resolution of issues in production is a chore and takes away from developer productivity. Often, the developer has already moved on to other problems by the time fixes are deployed. In many companies that don't have continuous deployment, developers have little control over when releases get shipped. If you're using an issue tracking system, the developer's job usually ends when the code is written and merged. Even if developers wanted to go back and verify the fix works, they may not have the proper tracking tools to do so.

Who is accountable for ensuring that bugs are fixed on the customer side? The support or product management teams will complain if it doesn't work as expected, but why should they be responsible? They have better things to do than follow up on dozens of bugs that are filed every week.

You're forcing customers to report unresolved problems if no one else is checking. This results in more support tickets, bad experiences and lower satisfaction scores, which is only the tip of the iceberg. Only a small percentage will bother filing a support ticket and most suffer in silence. If problems happen consistently, your customers will be less likely to recommend your product to others, and may eventually churn out. While checking costs you time, making your customers do it can cost way more.

Resolving issues doesn't have to be painful or take a lot of time. Rollbar has created a better way which automatically monitors production code changes and can track each error to resolution. It can save you time and lets you deliver a better customer experience.

Here's 6 ways to quickly resolve errors in Rollbar:

1. Resolve errors and automatically track regressions in the app

Rollbar allows you to mark errors as resolved several different ways, but the simplest is by clicking the resolve button in the web app. You can see the resolve button in the screenshot below, which is colored green.

The real intelligence behind this feature is that it will continue to track this item in the background to ensure that it is really resolved in production. If this item occurs again, Rollbar will reactivate it and can notify you immediately. errors can become reactivated if you thought you had fixed the error in production, but failed to deploy the change, you missed a use case in your code, or there was a later regression that caused it reactivate.

The saves your team time because you don't have to go back and monitor whether your code change really works in production. You just have to deploy it, resolve the item, and trust Rollbar to notify you if the error returns.

2. Resolve errors in a code version

What if you've made a code change to fix an error, but it hasn't been rolled out yet? If you resolve that item now, it will just reactivate. Instead, you want to resolve that item when the change is live.

If your application is a mobile app or single page application, this feature is a must-have because different clients could be running different versions at the same time. Without it, the item will keep getting reactivated when the error happens to users who haven't updated to the latest version of the app. This also applies to large server-side systems, where it's common for deployments to gradually roll out across multiple nodes or environments, so you may be running two versions in parallel.

Rollbar has a little bit of magic that allows you to resolve an error in a code version. It will wait for the fixed version to go live before notifying you again. To use this feature, just enter the version that contains the fix when you resolve an item in the web app. This will prevent the item from becoming reactivated in earlier versions.

Rollbar will know when your fixed version is live by looking at the code_version parameter that you pass when you set up your notifier. Here is an example of JavaScript code passing the version. It happens to use the SHA from the GitHub commit.

var _rollbarConfig = {
    # ... other configuration
    payload: {
        environment: "production",
        version: "3da541559918a808c2402bba5012f6c60b27661c"
    }
};

This saves you more time because you don't have to keep track of when a code change goes live in order to monitor whether it's been fixed. Rollbar will automatically track these changes and notify you if the error occurs again when the new code version is live.

Learn more in the resolve-in-version docs

3. Resolve in a commit message

An even easier way is to resolve the item when you commit code that fixes the issue. This way, you don't have to manually resolve or paste the version number in Rollbar's webapp. Rollbar can integrate with your source control system to get all the commits when you deploy. For example, in your commit message, just add the “resolve” keyword and then paste the link to the item:

resolve https://Rollbar.com/demo/demo/errors/315/

When you do a deployment, make a call to Rollbar's REST API to notify it of the revision that you are deploying. The revision should be the git SHA for the latest commit. Rollbar will get a list of all the commits that have been made prior to your latest revision. It will then resolve any errors that have been referenced in the commit messages. For example, if you deploy using bash scripts, you could add the following commands to notify Rollbar:

ACCESS_TOKEN=POST_SERVER_ITEM_ACCESS_TOKEN
ENVIRONMENT=production
LOCAL_USERNAME=`whoami`
REVISION=`git log -n 1 --pretty=format:"%H"`

curl https://api.rollbar.com/api/1/deploy/ \
  -F access_token=$ACCESS_TOKEN \
  -F environment=$ENVIRONMENT \
  -F revision=$REVISION \
  -F local_username=$LOCAL_USERNAME

You only have to set up this bash script once, and from then on, you can easily resolve in commit messages. Similar to resolving errors in the web app, Rollbar will not reactivate errors for clients using an older code version. This way, you know that if an item reactivates in your Rollbar dashboard, it is a real regression and is worthy of your attention.

Learn more in the resolve-via-commit docs

4. Merge errors that are duplicates

If an error or item in your item screen is a duplicate of another, you should merge them. You can even merge errors that have different messages but the same root cause. This prevents overload on your dashboard page and alerts. By default, rollbar will automatically merge or group occurrences of errors that have the same fingerprint. You can change the algorithm used to group these occurrences. Alternatively, you can manually merge them on the errors screen. Just select the duplicates and then click on the merge button as shown in the screenshot below.

Learn more in the grouping docs

5. Mute errors that you can safely ignore

I know what you're thinking, “It's great to be notified about errors I care about, but what about all the other noise?” We know you don't want to be distracted by unimportant information. That's why we added the ability to mute errors you don't care about. You can see the mute button in the screenshot above to the right of the resolve button.

Muting errors will hide them from your dashboard and item view, and Rollbar will not send you notifications. Furthermore, thanks to our grouping rules you only have to mute one occurrence of an error and all future occurrences will also be muted. Just remember to use this power wisely! People tend to fix what they see, so if you intend to fix it, keep it visible!

6. Auto-resolve old errors for a clean view

We always want to prioritize and focus our efforts on the most critical issues. What about all the one-off errors that build up over time? These could be rare edge cases, operational issues that have since gone away, or errors that were fixed but the team forgot to mark them as resolved. We don't want to distract our team or spend a lot of time clearing them out, so we can ask Rollbar to automatically resolve them.

Rollbar can automatically resolve errors under two conditions. The first is automatically resolving errors that have not been seen for several days:

The second is automatically resolving errors when a deploy is made. This is useful when you only want to track issues that are happening in your latest production build.

Rollbar still tracks the history of each of these errors so if they come back in the future, you'll still have a record. This prevents less important errors from distracting your team, so they can focus on the most critical issues.

Reactivation notifications and dashboard give confidence

When you resolve an item, you want to have confidence that it is fixed and will not return. Unfortunately, not all of us have psychic abilities to know what problems will happen in the future. Fortunately, Rollbar can monitor each resolved item and notify you if it is reactivated. This saves you a lot of time because you don't have to go back and check whether the problem is fixed, you can just rely on Rollbar to notify you.

Rollbar offers many channels for notification including email, Slack, HipChat, and PagerDuty. Additionally, you can see all the new and reactivated errors on your dashboard. It's broken down into errors from the last day and the last week. If you just did a deployment earlier today, check the last day category. If you do weekly sprints, you can see errors from the last sprint in the last week category.

Follow up with affected customers to build trust

Following up with customers is probably one of the most valuable and most overlooked parts of resolving problems. It's particularly important for high-value customers who took the time to report a problem, or for large problems that affect many customers. Customers understand that companies have problems from time to time, and they are willing to forgive if you show that you listen and take them seriously. Also, if you develop a good relationship where you have an open door, they will be more willing to share feedback because they see a return on the investment of their time, and they may even become an advocate or spokesperson for your product.

Of course, following up takes time, but it's much easier now than it used to be. Without Rollbar, you might see a bunch of errors in your logs, or a dip in your New Relic or business metrics dashboard. That's because the sources don't record the customer's information, and may sample the data so you don't have a complete picture.

Rollbar gives you a complete record of every customer affected by each error. You can quickly query and export a list of all affected customers, and then copy them into your email automation tool. If it was a particularly disruptive problem, your marketing team can even offer something special to turn it from a negative to a positive experience. You'll know you did a good job when you see the “thank you” responses come in.

Get to “Rollbar zero”

Some of our best customers make it a point to review and resolve all the errors in their Rollbar dashboard on a regular basis. The reason is because when new errors pop up, they want to have instant visibility so they can react quickly and resolve them before more customers are affected. They don't want to have unimportant errors clogging their view, nor do they want to automatically resolve and potentially miss issues. One customer makes it fun by throwing a “Rollbar party”. They get their developers in a room and knock out errors together.

Conclusion

The net is that Rollbar helps you keep your customers and make them happy. You'll have better visibility into errors affecting customers, and you'll be able to track that they are actually resolved in production. It also streamlines the workflow for your development team because Rollbar will automatically track resolution without requiring anyone to remember and follow up later. When you mark a critical item resolved in Rollbar, you can trust that it improves customer experience.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you take control of impactful production errors. :-)

New and improved JavaScript notifier SDK - rollbar.js 2.0

Sat, 20 May 2017 00:00:00 GMT

We are excited to announce a major update to our JavaScript notifier SDK in version 2.0. This new version adds support for isomorphic or universal applications, which can run on both the client and the server. It standardizes the configuration and logging across environments, and makes it consistent with notifiers for other languages. Here are the big changes:

Single library for both client and server side

The JavaScript ecosystem has evolved a tremendous amount over the past several years. During that time, server side JavaScript usage has grown to the point where it is quite common to be using JavaScript throughout an application's stack. Applications now are often designed to run both in the client browser and on the server side using Node. These are commonly referred to as "isomorphic" or "universal" applications. The benefits of isomorphic applications include faster loading times, better support for legacy or mobile browsers, and easier search engine indexing.

With the line between client and server blurring, it is desirable to have a unified view of exceptions across both environments. To make this possible, we unified the separate notifier libraries so that exceptions are logged uniformly. One require to rule them all.

Consistent interface for configuration and logging across multiple environments

We now offer a consistent interface for configuration and logging for both client and server side. Furthermore, it is consistent with our notifier libraries for other languages. This will make it much easier for developers since they only need to use one pattern.

The new way to configure Rollbar is to create a configuration object. Regardless of where you’re running, the schema for the configuration object is exactly the same in both environments. You can choose to initialize Rollbar based on what’s most convenient in your environment. If you’re running within a browser environment, Rollbar can automatically read from the _rollbarConfig variable on the window when loading.

<script>
var _rollbarConfig = {
    accessToken: "POST_CLIENT_ITEM_ACCESS_TOKEN",
    captureUncaught: true,
    captureUnhandledRejections: true,
    payload: {
        environment: "production"
    }
};
// Load Rollbar snippet or library
...

// log a generic message and send to Rollbar
rollbar.log('Hello world!');
</script>

If you’re using a module loader or running in a server environment, you can pass this configuration block to the constructor.

var Rollbar = require('rollbar');
var rollbar = new Rollbar({
  accessToken: 'POST_SERVER_ITEM_ACCESS_TOKEN',
  handleUncaughtExceptions: true,
  handleUnhandledRejections: true
});

// log a generic message and send to Rollbar
rollbar.log('Hello world!');

Additionally, we have configuration examples for frameworks like Angular, Express, Hapi, and more. You can see these examples in the JavaScript SDK documentation.

Multiple Rollbar instances

Rollbar is no longer treated as a singleton, but can have multiple instances. The benefit of instances is that you can have different loggers with different configuration options in your application. This is helpful, for example, in pages with components tracked in separate Rollbar projects. For convenience and backwards compatibility, we still instantiate a window.Rollbar object.

Upgrading from the old `node_rollbar` library

The upgrade path from node_rollbar version 0.6.4 to version 2.0.0 of this library is not automatic, but it should be straightforward. While it is backwards compatible, we encourage you to update to the new syntax to take advantage of future functionality. The main changes are to library initialization and usage.

When you’re initializing the library, we now offer instances with a constructor instead of just a singleton as mentioned previously. Also, when using the library, methods like reportMessage/reportError/… have been deprecated in favor of log/warning/error/… These methods are more consistent with our notifier interface in other languages. They can accept a variety of parameters, including strings, errors, callbacks, and more.

Old:

var rollbar = require("rollbar");
rollbar.init("POST_SERVER_ITEM_ACCESS_TOKEN");
rollbar.reportMessage("Hello world!");

New:

var Rollbar = require("rollbar");
var rollbar = new Rollbar({
  accessToken: 'POST_SERVER_ITEM_ACCESS_TOKEN'
});
rollbar.log("Hello world!");

Additionally, you now enable logging of unhandled exceptions and rejections through the configuration object instead of making a separate method call. More details are available in the JavaScript SDK documentation.

To make the library backwards compatible and make migrating easier, we provided a variety of helper functions. For example, the old style was to always pass the access token as the first parameter. We permit this style for convenience when no other options are necessary, but for new code one should use an object as the only argument. Furthermore, the reportMessage/reportError/… methods are deprecated, but they are still in the code and just delegate to the log/warning/error/… methods. We encourage you to update to the newest logging methods.

If you have any issues or requests, we welcome you to file them in GitHub. Since the notifier is open source, we welcome your contributions and help improving the library.

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you [take control of annoying JavaScript errors](/error-tracking/javascript/].

Instacart, leader in grocery delivery, relies on Rollbar for production error monitoring

Mon, 15 May 2017 00:00:00 GMT

Instacart, a leader in the on-demand marketplace, provides one-hour grocery delivery to users of their app and employs thousands of shoppers across the US to support order fulfillment.
Of the one trillion dollar grocery industry, only 1% of market share currently comes from online grocery sales. As Instacart increasingly captures more of this market, they turn to Rollbar for continuous monitoring of their service’s health.
With a promise of one-hour delivery, and a shopper workforce relying on their apps, Instacart’s services must be up at all times. Rollbar’s proactive alerting and granular error forensics facilitates the continuous integration and deployment pipeline at the heart of Instacart’s service.

Rollbar allows us to go from alerting to impact analysis and resolution in a matter of minutes. It's fully ingrained into our development cycle and monitoring. Without it we would be flying blind.

Arnaud Ferreri, Engineering Lead for the Consumer Team at Instacart.

The Challenge

While many industries have worked out how to participate in the on-demand marketplace, the one trillion dollar grocery industry is one of the last holdouts. Tight margins may be making some investors skittish. Retraining customer perception about grocery quality and freshness with app-ordered deliveries might be at play. Building and scaling the technology layer that acts as the on-demand engine has been a daunting challenge. It’s this last barrier to entry that Instacart has already dismantled.

Since its inception in 2012, Instacart has been wooing grocery shoppers away from retail spaces and onto smartphones, to place grocery orders through their app. Instacart currently has presences in about 26 markets, partnering with large chains such as Whole Foods, Costco, and Safeway to offer users of the app in-store prices on groceries. The only extra cost to the app user is a simple delivery charge. The big selling point though is the promise of one-hour delivery. Instacart employs thousands of shoppers across the U.S. to fulfill orders, and processes upwards of 20,000 requests per minute.

At this magnitude and pace, Instacart finds itself publishing new code releases around 30 times a day. Tests happen along several layers and stages in the production process before release but the final and most important test is “actually having real customers use our product and use our API endpoints,” says Arnaud Ferreri, Engineering Lead for the Consumer Team at Instacart. “At our stage, if anything goes wrong, we have to be alerted very, very quickly, and that’s what Rollbar has solved for us.”

At our stage, if anything goes wrong, we have to be alerted very, very quickly, and that’s what Rollbar has solved for us.”

The Solution

At its current scale (and still scaling), Instacart does a lot of meta level monitoring. They are hosted by AWS so they use CloudWatch; they view full logs in PaperTrail; they run build tests on their staging servers through CircleCI; and they use New Relic to track performance. But for proactive alerting and for granularity on error forensics, they turn to Rollbar.

“The fact that you get the exact stack trace that links into your code base; the fact that you have all parameters of a given request so you can easily reproduce the issue; the fact that you have information about the customer who triggered the error so you can easily see if it’s the same customer repeating the same error again… All of these make it very easy to understand the scale and complexity of a given problem at a given time,” says Arnaud.

Growing along with the popularity of the app, the Instacart engineering team has scaled pretty rapidly over the past two years, going from five engineers to 70. Working in Rails, JavaScript, and Python, the team has appreciated the way Rollbar integrates seamlessly with these languages, as well as with other tools they use such as Github, PagerDuty, and Slack. At this point they are in a continuous deployment cycle for a lot of their services, with any new commits in their code base going live to production. “Because of that, we can’t have engineers looking at production every minute to see if everything is going right. We need to be proactively alerted when something goes wrong,” says Arnaud.

We can’t have engineers looking at production every minute to see if everything is going right. We need to be proactively alerted when something goes wrong.

The Instacart engineers work in their own branches, and those branches get merged back to the master project. With each merge a new build release gets pushed to their staging servers, then to their beta servers and finally the release hits production. At all stages of this process, the team gets pinged proactively by Rollbar about any new errors that come in.

For a time, the Instacart engineering teams adopted an ambitious, power user goal called ‘Rollbar Zero’, shooting for zero Rollbar errors at any given time on any open project. As Arnaud explains it, this involved getting to “a level of health that we were not seeing errors that we were unaware of, or that we were able to control." This exercise not only boosted coding performance but also helped to brilliantly clarify workflows for the teams.

For Instacart, power usage of Rollbar isn’t limited to their consumer app. Since thousands of shoppers across the U.S. rely on their shopper apps for work, Instacart carefully goes with staged releases of these apps. First they release to a subset of beta shoppers, then to one city, then more cities, until finally they release across the board. The engineering teams have recently added Rollbar into these shopper apps to get the same centralized information they utilize for their web projects.

The Results

It’s an exciting time for the grocery industry, and Instacart is poised to change grocery shopping habits permanently. Of course, none of this would be possible without the continuous integration and deployment pipeline driving Instacart’s code ever forward. “The fact that we’re ramping up continuous deployment for a lot of our services - I think that’s only doable because we have Rollbar integrated into our alerting system,” says Arnaud.

Since code errors can potentially affect thousands of real customers and working shoppers, the ability for Instacart to prioritize error resolution according to the size of the problem is vital. “We’re using Rollbar to its maximum to make sure that we get alerted for the right reasons, and that we alert the right people for the right severity of problem,” says Arnaud. One designated person is pinged if a new error pops up, whereas, if a known error is happening multiple times a minute, a team gets pinged. So instant reaction to issues is guaranteed, in proportion to the size of the issue.

Arnaud says that Rollbar has helped his engineering teams “dramatically improve the way we do our detective work, understanding when there’s an actual issue, where it’s coming from, at what volume, how many customers it’s affecting, and what action needs to be taken.” Without Rollbar, they wouldn’t be able to solve issues as quickly as they need to since they depend on the fine, granular context on errors that Rollbar provides.

Rollbar is so ingrained into Instacart’s development cycle that Arnaud admits he and his team don’t even see it as a third-party tool: “Rollbar’s so tightly coupled into the way we work, it seems part of our system as a whole.”

Rollbar’s so tightly coupled into the way we work, it seems part of our system as a whole.

Thank you to Arnuad and the team at Instacart for sharing this level of insight and for leading by example when it comes to continuous delivery and working to maintain error-free experiences for their users.

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your application errors.

Improved PHP error reporting with our latest rollbar-php 1.0 updates

Thu, 04 May 2017 00:00:00 GMT

We are excited to announce a significant upgrade to our PHP notifier SDK. This new version will make it much easier for customers to integrate their PHP apps with Rollbar. It uses the latest standards like Composer, proper namespaces, and is more reliable. Here are some of the highlights:

Our first major 1.0 release

This major release of our PHP SDK represents a significant update in architecture, style, and quality for our PHP notifier. In addition, many outstanding issues were fixed so 1.0 is more reliable and better tested than our prior versions.

Our prior version, 0.18.2, was written in an old style of PHP with most of the code living in a single file. This was due to our desire to support the oldest version of PHP possible. Due to the changing landscape over the past couple years, it became reasonable to drop support for PHP 5.2 and this opened the door for many language features which benefit both our code and yours.

While the interface has undergone a redesign, it is still backward compatible with 0.18.2 via convenience functions.

Support for Composer

Our SDK is now properly supportive of composer which has become the de facto dependency management tool in PHP. Just add the following configuration to your composer.json file:

{
    "require": {
        "rollbar/rollbar": "~1.0"
    }
}

Standardized namespace

We are now using PHP namespaces, a feature introduced in 5.3. At the top of your file, you can delete require_once 'rollbar.php'; and instead add the following:

use \Rollbar\Rollbar;

Standardized logging method

You can now call a single log method with the ability to set the level and include all of the same data as before, instead of different method calls for report_exception, report_message, etc. This makes it consistent with our notifier libraries for other popular languages.

use \Rollbar\Payload\Level;

// Message at level 'info'
Rollbar::log(Level::info(), 'testing info level');

Current users of the PHP notifier can quickly update to the new version because it’s backwards compatible. Just remove the old PHP notifier file, and add the new dependency in composer. While it is optional, we do recommend that you use the new logging interface, since it lays the groundwork for future improvements.

If you previously tried our PHP error tracking library, but found it difficult to use, give the new one a try since it is much easier to set up and is more reliable.

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your PHP errors.

Meet the Rollbar Team - Todd Dampier

Mon, 01 May 2017 00:00:00 GMT

Meet Todd Dampier, another member of the Rollbar team! Todd started in late 2016 and we are more than thrilled to have him. Todd came from mLab where he was the founding CTO and has years of experience in his field as a Software Architect and CTO. Todd spent time at MIT, where he completed his undergrad and Masters in Computer Science.

I asked Todd about his thoughts on Rollbar and joining the team. Here are his answers in his own words.

What excited you about Rollbar or attracted you to the company?

When I started looking for what to do next professionally, I cast a fairly broad net. As a generalist with highly eclectic interests, the only thing I knew for certain was that I was done for a while with the field of MongoDB-as-a-Service.

From my very first conversation with Rollbar, what struck me most was the caliber of the people. The recruiting team was flawlessly professional, direct, and no-nonsense, as well as remarkably warm and personable. Cory and Brian impressed me as founders: their maturity, experience, technical depth, and product perspective gave me tremendous confidence in Rollbar as a team and as a business. As I met more of the team, that confidence just grew — and I’m happy to say the trend continues unabated today.

I also love the product. Coming from mLab, I’m a staunch believer in the value of farming out an annoying-but-necessary piece of functionality to a service backed by people who focus on that problem and do it right.

And as a technologist, I enjoy building and delivering services that I would want to use myself. It's a special kind of satisfaction that comes with developing products for other developers.

At Rollbar, we get to make the error tracking platform that we would — and do! — trust to monitor our own software.

What are you looking forward to working on (technically)?

Even in a seemingly simple service, there are myriad details to consider. There's always more to do, and you can always do it better.

I’m looking forward to working on features that make our toolset more valuable to our users — ones that afford greater utility and less friction when brought to bear on the problems our users are tackling.

Building a public service instead of an in-house solution tends to take all the technical design questions to the next level — such as scale, availability, reliability, and security, to name a few. I’m looking forward to working on these back-end issues as well, to making our service as robust and maintainable as we possibly can. Most immediately, we’re taking a thorough look at security throughout our platform. I’m also looking forward to some retooling we’ve been contemplating in our event-processing pipeline.

How would you describe the culture?

It probably sounds corny, but I've found Rollbar to be full of real, genuinely good people. From the top down, the culture is one of hard work, intellectual honesty, and doing the right thing for the customer — all of which you might reasonably expect in a well run business. More unusual, I think, is the high degree of mutual respect and human decency that permeates the organization. People seem to feel safe speaking their minds and appropriately prioritizing their personal lives & families, while still showing up and delivering what the organization needs. This comfortable pragmatism dovetails beautifully with a team that is geographically distributed, too: open and honest communication that’s as synchronous or asynchronous as it needs to be. People here have healthy sense of humor as well, which is crucial. Personally, I’m really happy and honored to be part of this team.

What's your vision for your contribution to Rollbar? What would you hope to achieve in the next 6 months?

To summarize what I said above, it basically boils down to some stuff you can see and some stuff you can’t see (unless we mess it up). At a high level, my vision is simply to make sure Rollbar is hands-down the best service in its category.

The value of a system that monitors, collects, and tracks errors is obvious to anyone who’s built a distributed software deployment.

There’s no rational reason, however, why you should need to develop your own error monitoring system — and if you feel like you want to anyway, you should probably come work at Rollbar!

Fun fact about yourself that you're willing to share?

Believe it or not, I didn’t always think computers were going to be the way I made my living.

My seventh grade class did a career unit, and one component of it was to pick an adult who possessed one of these “career” things and then follow him or her around for a day to see what having a career was all about. I hardly knew where to start.

When I was little, I’d spent a lot of time around judges. They were pretty cool people, and their job seemed kind of cool, too. So I wrote a letter to the Chief Justice of the Supreme Court of the United States, asking if he would help me with this “career” exercise. Someone on his staff wrote me a very polite reply, explaining that Chief Justice Burger was much too busy to have a random middle schooler underfoot all day. I was crushed! But that disappointment just served to strengthen my bond with my Apple //e.

We are excited that Todd took an interest in Rollbar and that he decided to join us to help on our mission of making developing software easy!

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your web and mobile application errors.

Introducing hassle-free, compliant SaaS error monitoring

Wed, 26 Apr 2017 00:00:00 GMT

I'm very excited to announce that today, Rollbar is launching the first and only compliant SaaS error monitoring solution on the market. Our compliant solution meets the standards of HIPAA and ISO 27001, thanks to our team's hard work over the past 6+ months on our infrastructure, application, and policies.

From the early days of Rollbar, I've been motivated to help developers build better software, faster, because I believe that empowering software developers is one of the most impactful (if not always visible) ways to improve all of our lives. We've seen customers use Rollbar to build and maintain just about everything you can imagine, from consumer apps to enterprise, healthcare, government, scientific research, and beyond. As we've found our way into applications that power more and more critical functions of modern society (like healthcare and finance), we realized that we would need to be able to meet those organizations' security and compliance requirements--somehow.

Our first iteration was Rollbar On-Premises (which we do still offer)--passing the compliance burden off to customers--and a handful of eager customers signed on. Over time, however, we learned that not only is it difficult for us to maintain and support a self-hosted product, it's a huge pain for customers, too. Forgive me for quoting a competitor who seems to be learning the same thing:

We went back to the drawing board to ask: why do customers need to self-host? And we realized that in almost all cases, they didn't. What compliance-conscious customers need is compliance, and compliant SaaS can be done.

The solution we're announcing today is for every developer who wants to build great software inside an organization that needs strong security or compliance guarantees [1]. If you're in healthcare, we can sign BAAs; if you're a large enterprise, we can go through your security review; if you have specialized needs, we're interested--and you can focus on shipping code, not worrying about how to monitor your error monitoring service.

Get in touch and we'd love to help you start running Rollbar on your whole stack. You can also learn more by reading the Compliance Overview and our Security and Compliance docs.

[1] Rollbar's compliant SaaS solution meets the standards of HIPAA and ISO 27001. Our facility and operational controls have been evaluated by independent third parties and comply with AICPA SOC 2 Type 2, EU-US Privacy Shield and CSA STAR.

Security and Compliance Needs?
[Contact us](sales@rollbar.com?subject=Requesting more info) for more information regarding our secure, compliant error monitoring solutions. Also, If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of distracting application errors.

Major JavaScript source map updates

Thu, 20 Apr 2017 00:00:00 GMT

We’re excited to introduce a major update to JavaScript source map support in Rollbar! This update introduces several features unique to Rollbar (source map search, source map failure logs, inline warnings in stack traces) that are critical to anyone using client-side JavaScript in a large-scale, continuous deployment environment.

Why Use Source Maps?

If your application uses client-side javascript, then you’re probably minifying your code to reduce file sizes and improve page load times. The benefits of minified Javascript are clear, but minification also introduces the need for source maps in order to debug errors that were detected in minified code. Without a source map, it's nearly impossible to determine where an error occurred in your Javascript code.

Due to the need for source maps, debugging minified code will always be inherently problematic, but we’ve just introduced several improvements to Rollbar’s source map handling that will make troubleshooting much, much easier.

Something didn’t work right…

When a Rollbar stack trace includes a minified file that doesn’t have a matching source map, you’ll now see an information message that links you to your project’s source map screen where you can further troubleshoot why the source map wasn’t available.

View failed source map uploads and downloads

More often than not, you look at your source maps because something didn’t work as expected. You can now see when Rollbar fails to access a source map, including detailed information about what happened.

Filter and search your source maps

On large projects with frequent deployments, you may have a lot of source maps generated from a lot of different code versions. With our new search capability, you can find source maps based on code version or filename.

Easier pruning of old source maps

Finally, we’ve fixed a bug that many of you told us about - When deleting a source map, you will remain on the same page you were previously viewing. This makes it much easier to clean out old source maps that are no longer needed in your project.

Want to learn more about source maps?

Check out our previous blog post Using JavaScript Source Maps to Debug Errors and our detailed source map docs.

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your JavaScript errors.

Upcoming Events - Meet the Rollbar team IRL

Tue, 18 Apr 2017 00:00:00 GMT

Come meet some of the Rollbar team members this Spring! We'll be attending these upcoming events below and we'd love to meet with you in person! We'd love to hear your thoughts on Rollbar and answer any questions you may have. You may want to share your projects with us, get to know the team or pickup some Rollbar gear. Please drop by and make sure to say Hi!

Angular Attack

Where: Global Online Hackathon / When: Fri, April 21 to Sun, April 23

Angular Attack is a weekend long hackathon featuring Angular. It's an online, virtual competition with contestants worldwide. Compete over two days to build the best Angular app with your team, and win prizes and glory. Hope to see you there! We're giveing away 6 months of Rollbar for free and a Star Wars BB-8 Droid to the winning team.

RailsConf

Where: Phoenix, Arizona / When: Tues, April 25 to Thurs, April 27

RailsConf is the largest gathering of Rails developers in the world, not to mention the largest gathering of Rubyists. We'll be sponsoring and exhibiting, so stop by and say hi, learn about updates to our ruby gem and grab some of our new Rollbar swag.

PyCon

Where: Portland, Oregon / When: Thurs, May 18 to Sat, May 20

PyCon is the largest annual gathering for the community using and developing the open-source Python programming language. Organized by the awesome folks in the Python community. We are sponsoring and exhibiting for the first time at PyCon. Please stop by our booth and say hi, talk shop and grab some new Rollbar swag.

GlueCon

Where: Broomfield, Colorado / When: Wed, May 24 to Thurs, May 25

Gluecon is a developer oriented conference that is focused on how Serverless Architectures, Containers, Microservices, APIs, DevOps, Mobile, Analytics, Performance Monitoring, and cutting edge developer platforms and tools are changing the technology landscape. We're excited to sponsor this event for the first time and our very own Andrew Weiss will be giving a talk on "Exception Handling Across Time and Space". Sounds like a party!

Schedule an in-person meeting

We’re making [office hours](sales@rollbar.com?subject=I'd like to meet at [add event you are attending here]") available during these events. Schedule some time with us to sit down with a Rollbar expert to show you how you can level-up your app monitoring and error handling. Visit us at our booth or email [sales@rollbar.com](sales@rollbar.com?subject=I'd like to meet at [add event you are attending here]) to get something scheduled. 📅

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you defeat annoying errors in your applications. :-)

Error alert notifications + how to use them for better monitoring

Tue, 04 Apr 2017 00:00:00 GMT

Rollbar offers an impressive array of error alerting and notification options for you to choose from, which is awesome! But sometimes we hear from our users that they aren't quite sure how to leverage their notifications to get the best results. What do notifications here at Rollbar do? How do they work? How should you interpret them? And perhaps most importantly, what are the best practices? Let's explore the answers to all these questions today.

Rollbar supports many different messaging and incident management tools, so you and your team can get notified about errors and important events. These notifications can be anything from a message in a Slack channel to an alert in PagerDuty. There are some common notification types that we offer, no matter what tool you're using to be notified.

Notification Type	Triggered when...
New Item	An error/ message is seen for the first time.
Every Occurrence	Every time an error/ message occurs ( use wisely ).
10^th Occurrence	10th, 100th, 1,000th, 10,000th, ... occurrence
High Occurrence Rate	`{x}` occurrences seen in `{y}` minutes ( configurable ).
Item Resolved	An error/message is marked `Resolved` .
Item Reopened	An error/message is marked `Active` by a user.
Item Reactivated	An error/message occurs again after being marked `Resolved` .
Deploy	A new deploy is reported.
Daily Summary	(Available in email only) Summary of daily error/message activity in a project

So now that you know what types of notifications you can get, how about customizing them? You can filter your notifications to adjust the circumstances under which you receive a notification. For instance, you can control what team the notification is sent to, or even which individual users should be notified. You may wish to only receive a notification when the source is a certain notifier, or the title of the item matches a specific string. You can find all of the myriad filters available to you for notifications (and further documentation of how to configure these filters) **[in the docs](/docs/filtering-notifications/)**.

Once you've set up the specific conditions under which a notification is triggered, you might want to customize the message that gets sent to you and your team. That's where the different notification variables Rollbar offers come in handy. You can have your notification alert an entire channel in Slack to a new item in Rollbar, for instance. Check out the docs to see exactly what variables are available, and then come back here for some tips and tricks on how to use filters and variables to get the most out of Rollbar notifications.

Routing your notifications effectively

Here at Rollbar, we sometimes hear from users that they aren't sure exactly how to route their notifications so that the right people see the right ones. The best way to do this is to use our filters! You can customize your notifications so that each one goes to a specific person or team, based on specific criteria. For instance, let's say you have an Item Reopened rule, and you want to alert your entire engineering team right away in Slack when an item is reopened, but only if the item's level is Error, and it occurred in production. Just use notification filters in your Slack notification rules, and have it post to your #engineering channel, and voila! All your engineers will see when an item is reopened that matches that criteria.

In essence, the best way to ensure the right people see the right notifications at the right time is to set filters on the environment, filename, title, or other criteria as needed, and send it to the specific emails, teams, or channels that need to be alerted in a timely manner. The more specific you can get with your matching criteria, the less notification clutter you'll get. You can set up multiple rules for the same type of notification, so you can customize exactly who sees what notification to reduce the clutter for everyone on your team. Want the Python developers on your team to get all the Python errors and nothing else? Filters are your friend! The best way to get started with filters is to check out the docs on notification filters.

There are two main types of tools that Rollbar users have questions about when it comes to properly setting up their notifications, and maximizing the information included in the notifications: messaging apps, and incident management tools. Let's take a look at messaging apps first.

Notifications for messaging apps

When setting up notifications for messaging apps such as Slack or HipChat , you want to make sure the message is succinct but helpful and informative. A basic Slack message is [{{"{{project_slug"}}}}] {{"{{username"}}}} deployed revision {{"{{revision"}}}} to {{"{{environment"}}}} {{"{{link"}}}}, which gives a good amount of information in one sentence for a deploy notification. Rollbar supports variables in notifications using {{VARIABLE_NAME}} syntax. Different variable values are available depending on the type of event that triggers the notification, which you can check out in the docs here. Any data value sent in the JSON payload of an item or occurrence may be used as a variable, including custom data. Examples of usage are {{"{{body.request.url"}}}} and {{"{{body.server.host"}}}}. If your JSON payload includes the custom values {{"{{ handler: { key: process-job, id:100"}}}} then you can use the variables {{"{{body.handler.key"}}}} and {{"{{body.handler.id"}}}} in your notifications. To view the full set of available values, look at the "Params" values of an occurrence in your project. All params besides the ones listed in the above notification variables must be prefaced with "body".

Sometimes, our users want to be able to alert everyone in a channel using common Slack syntax such as @channel, @group, @here, or @everyone. You can do this with notification variables in Rollbar as well! Make sure to use the syntax <!channel>, <!group>, <!here>, or <!everyone>. So, for the earlier example, if you wanted to add @here to the beginning of the message, you would simply change the message to <!here> [{{"{{project_slug"}}}}] {{"{{username"}}}} deployed revision {{"{{revision"}}}} to {{"{{environment"}}}} {{"{{link"}}}}.

Here at Rollbar, we have six different notification rules for Slack:

As you can see, we set a high occurrence rate notification to alert us once there's been at least 100 occurrences in 5 minutes of an error in production, which ensures that we are notifed when there is a error occurring enough that it's immediately concerning. We also like to be alerted if an item that we thought we resolved is reopened, if an item is reactivated or resolved, if a deploy has happened, and especially if there is a new error. Here's how some of Rollbar's notifications look like in Slack:

Notifications for incident management tools

The type of information you want in a notification from incident management tools like PagerDuty, VictorOps or Datadog. is probably slightly different than for a messaging app. Typically, the alerts you're getting for an incident management tool deal with the possibility of an urgent problem that your team needs to address. You're not going to want to be alerted with these tools if a simple deploy is reported. You'll want to know what type of notification was triggered and why, as quickly as possible. Therefore, the basic incident management message we start with is [{{"{{project_slug"}}}}] {{"{{environment"}}}} - {{"{{trigger_description"}}}} {{"{{level"}}}}: {{"{{title"}}}}. All of the incident management tools we are currently integrated with, aside from Datadog, offer the same amount of customization for your notifications. Datadog has some extra options, which are detailed in the docs.

Notifications for email

Aside from messaging apps and incident management tools, Rollbar also offers email notifications, and automatically sends out a daily summary of each project's activity via email. This daily summary includes how many occurrences of each type of error occurred in the past day, broken down by error level and environment. You can specify the time it is sent out daily, as well as whether to send it only if there is new data. The summary can be a very helpful overview of what errors are occurring and where, without needing to log into the UI. We recommend using the configurable email notifications for alerting your team to items that aren't vital to be seen immediately.

Here at Rollbar, we have 11 email notifications configured:

This allows us to get notified via email for every occurrence with certain titles, if an item is resolved, reactivated, or reopened, and when a deploy happens. We also have two different high notification occurrence rates to be notified on, one of which goes solely to one person, so you can see there are quite a few configuration options for who gets alerted and under what circumstances. Here's how one of these emails looks in our inboxes and in the email itself:

Now that you've gotten an overview of Rollbar notification types, give them a spin for your project and you can be sure you'll never miss an important error again. If you need help setting up notifications for your project, don't hestitate to contact us at support@rollbar.com. We love hearing from you and we're happy to help!

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your web and mobile application errors (and notifcations).

Onboarding remote employees, on the other side of the world 🌎

Tue, 28 Mar 2017 00:00:00 GMT

When I first started full-time here at Rollbar, I was living in Malaysia for the month, starting a year-long journey around the world that I'm currently still on. While it was certainly fun and exciting to explore Kuala Lumpur, it was a bit more difficult to onboard in a new job when I was 15 hours ahead of the rest of the team. Luckily, everyone here at Rollbar is incredibly flexible and helpful, and we were able to get creative about the onboarding process. If you're considering onboarding a remote employee in a different time zone as the rest of your team, read on for some tips and tricks that should help speed up the process.

This is my third job in tech, and my second as a remote employee, so while I can't say I've seen it all in terms of onboarding, I have seen a few different methods. The most obvious (and easiest) method is to have your new team member come in to the office bright-eyed and bushy-tailed and have all the members of the team available to help make their onboarding a success. Unfortunately, if the team member you're onboarding doesn't live in the same area, that isn't really possible. What to do? Some companies get around this by requiring all remote employees come onsite for their onboarding. If you're an all-remote company, like the second place I worked, you've probably got the kinks figured out in terms of onboarding your new employees. In many ways it can be easier to onboard as a remote employee at a place where everyone is remote, rather than as one the few remote employees in an otherwise "traditional" office: you're all remote, and you all understand the challenges surrounding that, so you've worked out methods that get around this for the company as a whole. But how to onboard a remote team member when most people are in the office everyday? Here at Rollbar, we've figured out a few key steps.

Have a specific employee be in charge of training your newest team member

It's easy to have an onboarding document with a few tasks to complete and then start assigning more tasks to your new team member, telling them to feel free to ask anyone for help, but that is actually not necessarily the best way to onboard a remote employee. When your new team member is remote, you have to bear in mind that they aren't getting face time with all the other members of the company, and they may not be sure of who does what exactly in the company when it comes to more specific questions. Rather than having them individually message each person on Slack, trying to find the right person for each question, give them a specific point of contact on your team to help onboard them. For me, it was the other support engineer here, who also happened to be remote. For pretty much any question that I had, I asked him first, and it made me feel much more comfortable more quickly than I otherwise would have. I always knew that no matter what I needed to know, I had someone to ask who would respond as soon as possible, and was looking out to make sure I wasn't lost. Feeling comfortable and supported is key for onboarding a new employee, and having someone to act as your new employee's point of contact enables that.

Overlap, overlap, overlap

Having a specific employee as the point of contact for your newest team member is super helpful, but it can still be quite difficult to onboard someone 15 hours ahead of the rest of your team if there is no overlap in working hours. Here at Rollbar, I specifically worked at least half of the time for the first few weeks during hours that would overlap with the rest of the team. This meant that I was staying up until 2 or 3 in the morning sometimes, but since I knew it was only for the first couple of weeks, it really wasn't so bad. Plus, I wasn't languishing for a day waiting for a response to a simple question, which can be very demoralizing when you're first starting at a new job.

Set up a document for questions

It may not always be possible to have significant overlap between your remote employee's hours and the rest of your team. One of my favorite things we did here at Rollbar to get around this issue was to set up a daily questions document. Previously, I had been leaving my questions in our engineering Slack channel, which would sometimes get drowned out with other chatter, so I wasn't always getting the answers I needed. During a chat one day with Cory, our CTO, he came up with the brilliant idea of me having a daily questions document where I could leave any and all questions I needed help with for the other team members to check when they got online. This meant that I could get the answers I needed all in one place, instead of having to scroll through our chats and hope that everyone had seen my questions and answered them. In fact, we set up a great integration with Quip, where every time someone edits my questions doc, it posts to our engineering channel in Slack, so everyone is notified. Cory also came up with the idea of this document having two sections: unresolved and resolved questions. When a question is answered satisfactorily, I just move all the info into the resolved section, so if I ever run into that issue again, I have a quick reference point to check. I actually still use this questions doc, and when we onboarded another support engineer, my daily questions doc was an invaluable resource for getting him started as well.

Have a regular checkin with a higher up

Having regular checkins with your boss is important for any employee, but for remote employees, it's crucial. Especially if your employee is in a time zone that is very different from the rest of the team, finding a time for them to be clued in to what's going on in the company is invaluable. Cory and I meet once a week to discuss what I'm working on, what the rest of the team is working on, and answering any questions I may have. While this seems like something that could be easily overlooked, I credit these meetings the most with making me feel comfortable and valued as a team member here at Rollbar even when across the world from everyone else. It was in one of these meetings, where I was explaining how difficult it was to make sure I always got the answers I needed from other team members in Slack, that Cory came up with the daily questions document that I still use to this day. I look forward to our meetings every week since it keeps me clued in to the rest of the company, no matter where I am in the world.

If you have a good onboarding process for your remote employees, they'll be able to provide value to the rest of the team quickly, and they'll feel like a valued team member as well. Once you get a good remote onboarding process set up, your company can reap the benefits of having employees around the globe, such as support available during hours your team would normally be offline, or engineers who are on-call when the rest of your team is sleeping. While it can seem like an insurmountable challenge to onboard an employee on the other side of the world from the rest of your team, take it from us here at Rollbar - with a little creative thinking and some flexibility, it can be done!

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your web and mobile application errors.

Building the IoT Rollbar error alarm 🚨

Sat, 11 Mar 2017 00:00:00 GMT

Our friends at Losant wanted to share how they built an actual 'error-alarm' using the Rollbar and Losant API's. Enjoy!

When I envision a tech company's smart office, I see tons of dashboards and indicator lights that monitor everything. Efficient monitoring is a critical piece of today's technology stack, and there are always ways to improve. Rollbar already does an impeccable job at alerting you when errors are thrown in your application. But, to increase awareness, accountability, and awesomeness in the office, we can connect Rollbar to our smart office. In this tutorial, we are going to build an office error alarm powered by Rollbar and Losant.

Building the Alarm

Intel Edison - This is a small compute module that we will use to control the devices in our office.

WeMo Insight Switch - This is a Wifi enabled, smart plug that we can control remotely.

**Rotating Police Party Light - This is a regular party light.

Setting up the Light

Connect the WeMo Insight Switch to your office network using the WeMo app and plug in the light. You should be able to control the light with the WeMo app.

Setting up Intel Edison

The WeMo switch exposes a local API. There is a Node.js module that wraps the API so we can control the WeMo switch.

Since the API is local, the Intel Edison must be on the same network as the WeMo switch to connect to the device. Intel has a configuration tool we need to use to set up SSH and connect the Edison to Wifi.

Once the Edison is configured, using the credentials you just set, SSH into the device and download the source code for the alarm:

$ wget https://github.com/Losant/example-rollbar-alarm/archive/master.zip
$ unzip example-rollbar-alarm-master.zip
$ cd example-rollbar-alarm-master
$ npm install

Let's take a look at index.js:

'use strict'
const Wemo = require('wemo-client')
const Device = require('losant-mqtt').Device

const LOSANT_DEVICE_ID = 'DEVICE_ID'
const LOSANT_DEVICE_KEY = 'DEVICE_KEY'
const LOSANT_DEVICE_SECRET = 'DEVICE_SECRET'
const WEMO_DEVICE_URL = "http://192.168.86.135:49153/setup.xml"

let wemo = new Wemo()

/**
 * Construct Losant Device
 * @type {Device}
 */
let device = new Device({
    id: LOSANT_DEVICE_ID,
    key: LOSANT_DEVICE_KEY,
    secret: LOSANT_DEVICE_SECRET
})

/**
 * Connect to Losant
 */
device.connect()

wemo.load(WEMO_DEVICE_URL, (deviceInfo) => {
    console.log('Device Found: %s', deviceInfo.friendlyName)
    /**
     * Get the client for the found device
     * @type {Wemo Client}
     */
    var client = wemo.client(deviceInfo)

    /**
     * Listen for commands from Losant
     */
    console.log('Listening for Losant command...')
    device.on('command', function(command) {
        console.log('Command from Losant received: %s', command.name)

        if (command.name == 'rollbar-wemo-switch') {
            console.log("Setting State to : %s", command.payload.state )
            client.setBinaryState(command.payload.state)
        }
    })
})

In index.js , we are configuring our app to use a device in Losant. Then, load the WeMo device and listen for a command from Losant called rollbar-wemo-switch. The rollbar-wemo-switch command will include a parameter on the payload called state. state is a boolean. Lastly, state is passed into the client.setBinaryState function to turn the light on/off.

To connect to the WeMo device's API, we need to get its setup URL. In the code, it is represented as WEMO_DEVICE_URL. WeMo has a discover feature that we can use to find all of the devices on the network and retrieve their setup URL. There is another file in our source called find-devices.js we can run to do this for us:

After you find your device, update the WEMO_DEVICE_URL constant. It’s important to note: Because of DHCP, the IP of your WeMo device could change. It’s recommended to set a static IP address on the WeMo device using the DHCP reservation feature of your wireless router. You can also use the serial number of the device, like in this example.

Before we can run index.js , we need to configure the constants: LOSANT_DEVICE_ID, LOSANT_DEVICE_KEY, and LOSANT_DEVICE_SECRET. We get these values from Losant.

Setting up Losant

Losant makes it simple to interact with hardware. Using the Losant platform and SDK, we can collect data from devices and send commands back to devices.

First, create an account or log into Losant. Next, create an application for this project and create a device to obtain a device ID.

From the navigation menu in Losant, we can set up an access key/secret:

Now, you have a device ID, access key, and access secret. Be sure to update index.js with these values.

Connecting Rollbar and Losant

To connect Rollbar and Losant, we are going to use webhooks. Rollbar can send webhooks based on rules, such as the 10^nth occurrence of an error. In our use case, our CTO can define the rules for the alarm within Rollbar based on his specific, per-project, requirements.

Create a webhook in Losant. Then, copy the webhook URL into Rollbar.

In Rollbar, webhooks are configured per-project. In the Rollbar dashboard, pick a project and go to Dashboard -> Settings -> Notifications -> Webhook. Paste the Losant webhook URL there, like so:

Building a Workflow

In Losant, workflows handle communication and logic between services and devices. All workflows start with a trigger, like a webhook. All workflows end with an output, like a device command. In between, Losant has tons of nodes that can accomplish tasks, like decisions based on logic or connect to other services like Twilio.

To start, Let’s create a workflow. Once created, we will start with the webhook node as our trigger. Each node has its configuration in the right panel of the dashboard. For the webhook node, we need to select the webhook we configured for Rollbar.

Once we have received the webhook trigger, we are going to send a rollbar-wemo-switch command to the device with a state of 1 to turn the alarm on.

We do not want the alarm to continue forever; it could get annoying. So, twenty seconds is long enough for someone to see the warning and act upon it. To do this, we can use the delay node to pause the workflow for twenty seconds. Then, send a rollbar-wemo-switch command to the device with a state of 0 to turn the alarm off.

Testing the Light

In your terminal, log into the Edison, go to the directory of your project and run:

npm start

You should see your device’s name. You can also use a process monitor, such as PM2, to keep the app running forever. Also, Intel has an IDE that makes it easy to deploy apps to the Edison.

In Rollbar, we can send a test notification. Once pressed, if everything worked, the workflow in Losant will trigger and your light will turn on for twenty seconds. 🚨

What's Next?

There is no limit to building a smart office, only your imagination. Here are some ways to extend this project:

Attach a button to serve as an “ack” to turn off the button.
Attach a speaker for a sounding alarm.
Connect multiple lights to control each by a different Rollbar trigger.

To create and learn more, Losant has compiled a list of great tutorials and projects you can use for inspiration for more projects. Any feedback or questions? Let us know in the comments!

Thank you to Taron and the team at Losant for sharing this awesome tutorial and literally and physically :-) leading by example when it comes to being alerted to errors and exceptions. Bravo!

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your application errors.

Women at Rollbar

Wed, 08 Mar 2017 00:00:00 GMT

Happy International Women's Day from all of us at Rollbar! To commemorate the day, the women of Rollbar thought it would be great to discuss some of the experiences we have had as women in tech.

_{Image via Recode}

Currently Nellie, who handles all office- and employee-related matters, is the only woman in the Rollbar office full-time. Aside from her, there is one other woman who works full-time, Rivkah, our Support Engineer/Developer Advocate. We recently just met for the first time at our quarterly on-site, despite working together for six months. We have a unique distributed team at Rollbar, which is why the two of us hadn't met in person yet. We are definitely outnumbered by men in our workplace.

There has been a lot of attention in the media recently about Susan Fowler, the woman who wrote a blog post on her time at Uber. We both found it disturbing, and yet sadly familiar. We have also experienced similar situations as Susan Fowler in previous workplaces.

We recently discussed potential topics for blog posts, and the topic of women in the workplace came up. We both discovered we had experienced being the only female or one of a small percentage (< 3%) at multiple companies. We both also had encountered similar situations as Susan Fowler: unwelcome and highly inappropriate behavior which was reported to HR and not appropriately resolved. Reading Susan's story, we were both surprised and unsurprised, as the double standard of disbelieving women and questioning whether certain behavior was warranted has been a long-standing hot topic of debate and is unfortunately extremely common in our industry.

Rollbar having mostly men (aside from the two of us) is the most extreme ratio of men to women in both of our experiences, especially since Rivkah is remote and Nellie is in the office daily. Interestingly enough, at Rollbar, the vibe is one of the warmest and most welcoming we have experienced, and the team is one of the smartest and kindest we have known. We believe that begins from the top. At Rollbar, the founders are both responsible for this, along with creating core values that mirror the kind of culture they want to build and the values they believe in.

Despite all of the recent news about unacceptable behavior that goes without consequence, there are companies that exist which consist of mainly men and yet defy those old paradigms. Rollbar has a zero tolerance for behavior that would be considered questionable, unprofessional, or unethical. For that, we are grateful! Nellie does get asked often what it is like to be the only woman in an office full of men in a sympathetic tone, however here it's actually not something she really notices. It feels refreshing to say that despite the number of men or women, this is a team we have greatly enjoyed being part of.

If you're interested in chatting about being a woman in tech, or you'd like to work with us here at Rollbar (we're hiring!), please don't hesitate to reach out to either of us! Nellie can be reached at nellie@rollbar.com, and Rivkah's email is rivkah@rollbar.com.

Up your incident management alerts game with Rollbar and VictorOps

Thu, 02 Mar 2017 00:00:00 GMT

Resolving on-call errors can be faster and easier than ever before when you integrate VictorOps and Rollbar. VictorOps is a fantastic on-call management tool that provides incident notifications to engineers to help teams resolve incidents faster and minimize downtime. We're excited to announce that you can now automatically send your Rollbar items to VictorOps, and get alerted to new errors quickly - before your users notice.

Linking Rollbar and VictorOps

To automatically turn your Rollbar items into Alerts in VictorOps, read on.

Configuration is per-project in Rollbar.

Head to the Notification settings page for a project: Dashboard -> Settings -> Notifications -> VictorOps.
In VictorOps, go to Settings (or Complete Setup) -> Alert Behavior -> Integrations, and choose Rollbar.
Click "Enable Integration" if it's not already enabled.
Copy the API key from VictorOps. In Rollbar, enter the API and routing keys, and click "Enable VictorOps Integration."

Congrats! You have now integrated Rollbar with your VictorOps account. Now, when a new error or higher occurs in Rollbar, it will create an Alert in VictorOps, which will be automatically resolved when it is resolved in Rollbar. If you want, you can customize the default rules by editing, adding, or deleting them. By bringing your Rollbar data into VictorOps, you can streamline your monitoring, minimize downtime, and resolve incidents faster.

For more information, check out our integration docs for VictorOps. We also have an integration with another incident management alerts provider, PagerDuty; check out the docs here. To see if Rollbar integrates with any other tools you’re using, see our full list of integrations here.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you defeat errors in production. :-)

Using JavaScript source maps to debug errors

Mon, 13 Feb 2017 00:00:00 GMT

Some of the most common questions we get here at Rollbar deal with source maps:

What are source maps and what do they do?
How can you enable source mapping?
Why aren't your source maps working properly?

Let's explore Javascript source maps together, starting with the basics.

Existing Rollbar users, go to our JavaScript source maps documentation to get started.

As web developers, we have two major goals. First, we want a highly performant website. Second, we want that website to be easy to debug and maintain. Unfortunately, these goals are often at odds with one other. If we minimize our JavaScript, we can achieve some of these goals. Minimizing JavaScript reduces the download size, and the smaller the payload for our website users, the better. If we combine our JavaScript files, we can also reduce the number of http requests.

For a highly performant website, we want to reduce the number of requests the browser has to make in order to fully load, render, and process the page. Furthermore, with transpilers, we can start using new JavaScript features before they are implemented by the browsers.

Unfortunately, minimizing JavaScript also comes with some downsides: it's hard to debug, and we lose original filenames, line numbers, and column numbers. Luckily, source maps solve this problem. A source map is a file that provides a mapping from the minified line and column numbers back to the original file, line, and column. Source maps use Base64 VLQ encoding to reduce the size of the file, while still maintaining all required information.

Using JavaScript source maps in Rollbar

Rollbar supports using source maps to get meaningful stack traces while still using minified JavaScript in production. Once you enable source mapping, you'll see the original source filename, line number, method name, and code snippet in your stack traces, and your error grouping should be more resilient to code changes. In order for the minified-to-source translation to work, we need at minimum a stack trace with column numbers, and a source map file. You'll get a stack trace with column numbers if you give Rollbar an Error object, in a browser that provides column numbers. Your source map file can optionally include any or all of the original source files referenced by the source map. You can include them using the sourcesContent key.

If Rollbar receives an occurrence with certain keys set in it, we will use any source maps that you have provided to us to decode your stack trace. The values that need to be set in your configuration are as follows:

var _rollbarConfig = {
  // add these new params:
  payload: {
    client: {
      javascript: {
        source_map_enabled: true,
        code_version: "some version string, such as a version number or git sha",
        // Optionally have Rollbar guess which frames the error was thrown from
        // when the browser does not provide line and column numbers.
        guess_uncaught_frames: true
      }
    }
  }
};

When we receive an occurrence that has these values, we will look up the source map by filename of the JavaScript file and the code_version specified. Then we will step through all the frames of the stack trace, and convert from the minified location to the original location. We store the source map package by project+version+minified URL. If you upload it more than once, each subsequent upload will overwrite the previous data.

You have two options for providing your source map files to Rollbar: automatic download, or uploading your source maps. We recommend uploading your source map files, and you have two options for how to do this. One is to manually upload your source maps through our UI. Go to your project settings page, and click Source Maps. There, you'll be able to manually select the file you wish to upload. However, our recommended way to upload your source map files is to upload via our API at the beginning of your deploy script (before the new code is in production). Here's an example cURL command for how to do this:

curl https://api.rollbar.com/api/1/sourcemap \
  -F access_token=aaaabbbbccccddddeeeeffff00001111 \
  -F version=version_string_here \
  -F minified_url=http://example.com/static/js/example.min.js \
  -F source_map=@static/js/example.min.map \
  -F static/js/site.js=@static/js/site.js \
  -F static/js/util.js=@static/js/util.js

In the above example, our website is http://example.com, we have a minified JavaScript file at http://example.com/js/example.min.js, and we have a source tree like this:

example/
example/static/js/example.min.js
example/static/js/example.min.map
example/static/js/site.js
example/static/js/util.js

If you want to have Rollbar automatically download your files, you need to provide a comment at the bottom of your JavaScript files that points to your source map file (for example: *//# sourceMappingURL=URL_TO_SOURCE_MAP*). How does automatic downloading work? If we receive a JavaScript error, source maps are enabled, and we don't already have the source map for the current code version, we will schedule an attempt to download it. For each stack frame, we'll first download the minified source file and look for a sourceMappingUrl comment. If it has one, we'll try to download that file and save it as the source map. Then for future errors, we'll use the source map to translate the minified frames back to original frames. This is the easiest method, but not suitable for all cases (such as if you don't want to expose your source map or code to the public web). It is also less reliable than the upload method, since the source map won't be downloaded yet when the first few errors come in. Because of this, we recommend the upload method for production use.

However, if you're using the automatic download method, you can notify our API to trigger a download for each of your minified files. Doing this will reduce the number of errors that the source map isn't applied to, since we'll have a greater chance of downloading the source map before the first error after a deploy occurs. Here's a sample command for this:

curl https://api.rollbar.com/api/1/sourcemap/download
  -F access_token=aaaabbbbccccddddeeeeffff00001111 \  
  -F version=92429d82a41e930486c6de5ebda9602d55c39986 \  
  -F minified_url=http://example.com/static/js/example.min.js

Tips and tricks for using source maps with Rollbar

This all seems simple enough, but there are some common problems our users have run into.

Don't forget to update the version number when you update your JavaScript. If you don't, the filename, line and column numbers will be incorrect.
The value of minified_url must be the full URL of the minified file. This should start with http: or https:, which we'll strip off.
Make sure you're not missing one or both of the config params in the on-page JavaScript snippet. Set both payload.client.javascript.source_map_enabled and payload.client.javascript.code_version.
If you're using the upload method, check to be sure that the code_version used in the on-page snippet matches the version provided in the upload call.
If you're using the download method, make sure your source map file or minified JavaScript source files are on a host that's reachable from the public internet, and are not gated behind an authorization wall.
If the JavaScript error that you are expecting to be un-minified does not have column numbers, and you haven't enabled guess_uncaught_frames, we won't be able to apply the source map. We need column numbers to be able to apply the source map without guessing.
If your source map file combines multiple sub-maps into "sections" within the top level map, we unfortunately don't yet support this source map format (but we are planning to soon).

May all your errors be easy to fix, and if you run into any issues, feel free to contact Customer Support.

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of pesky JavaScript errors.

Meet the Rollbar Team - Andrew Ledvina

Fri, 10 Feb 2017 00:00:00 GMT

We're excited to introduce Andrew Ledvina as the newest member of the Rollbar team and Lead SDK Engineer. The timing couldn't have been more perfect for Andrew to start during our quarterly all-hands last week! He was able to have 1:1 meetings with the team, meet our remote team members in person, join for company outings and happy hours, and get an overall feel for the team!

As our Lead SDK Engineer, Andrew brings years of experience and extensive knowledge in many fields to the team. With his undergraduate studies completed at CU Boulder, Andrew holds a BS in Applied Mathematics with a Finance Emphasis, a BA in Economics and a BA in Chemistry. After graduation, Andrew then went on to receive an MA and PhD in the Department of Operations Research & Financial Engineering at Princeton University.

Prior to Rollbar, Andrew was the Director of Engineering at Flexport and the company's first engineering hire. Andrew has also spent time working at companies like Gigster, Facebook, and even helped a friend with his health insurance startup Alan, based in Paris, France.

I was able to ask Andrew a few questions and let him share his thoughts on Rollbar.

Q&A with Andrew Ledvina:

What excited you about Rollbar or attracted you to the company?

I was working at a different startup and recognized the need to track, understand, and ultimately fix errors that always exist in production code. Searching around for different solutions to this problem I came across a few different companies that had a solution for this problem. I had a few criteria in mind when choosing between alternatives: support for JavaScript errors in the browser as well as errors on our backend servers, simplicity of getting started based on the documentation, and pricing that started free so I could try out the service before naturally having to pay as my traffic scaled. Rollbar fit all of my criteria better than the alternatives and we never really looked back as it helped us fix errors quickly.

As a software engineer, I understand that errors are unavoidable when building systems in an effort to get features out to users as quickly as possible. There will always be bugs. Good tooling is therefore essential to having a product that is both high quality and able to move forward at a reasonable velocity. I see Rollbar as an essential part of every company's toolbox whether they currently know that or not. As someone who writes code for a living, it excites me to help others who write code to be more efficient. Rollbar is a utility in the best sense of that word.

What are you looking forward to working on (technically)?

My role is spread across all of the different SDKs that Rollbar both currently has and soon will offer. I am excited about writing code in a large variety of different languages and paradigms, from imperative Python to functional Clojure and everything in between. There are both architectural challenges that span all of the SDKs as well as technical challenges in each particular language and framework that we want to support. I am looking forward mostly to working on the cross cutting problem of how best to interact with Rollbar from your code independent of what language or framework you choose to use.

How would you describe the culture?

The culture at Rollbar I would describe first and foremost as engineering driven. I was surprised during the interview process as well as after I started by the technical depth at every level and position that I have dealt with. In hindsight this fact is obvious considering the product, but nonetheless the quality bar is quite high. An important feature of building any good product is being able to empathize with the end user. At Rollbar, we are the same as our customers, we use our own product to monitor our own systems. That idea of building something you would want to use is a core part of the culture and permeates the discussions around features and the quality of the product.

What's your vision for the SDKs at Rollbar? What would you hope to achieve in the next 6 months?

Our SDKs are the main bridge between our client's code and our service, and as such are vitally important. My main goals for the next 6 months revolve around bringing consistency and feature parity to all of the notifiers we support. This means unifying the API used to interact with Rollbar from configuration through to sending errors. Along with this, I would like to get to a state where we have extremely high quality code that is thoroughly tested. Anytime you include a third party library in your code you are trusting someone else to have put in the effort to produce as error free code as possible. We want people to use our SDKs because they are rock solid and easy to use. I think that is true in some languages but not in others.

Being a bit of a language nerd, I am also excited to go more in depth in some languages and frameworks that I have only used tangentially. Languages like Go and Elixir are up and coming for a reason, and I would love to see official Rollbar support for them sooner rather than later.

Fun fact about yourself that you're willing to share?

My favorite animal is the Rock Hyrax, colloquially known as a dassie, which I discovered during a month spent in South Africa. The closest living relative to the hyrax is the elephant, which is quite surprising considering dassies are only about 20 inches long and weigh around 8 pounds. I spend my free time reading, running, and jumping out of airplanes.

As culture and core values are extremely important at Rollbar, Andrew's vision and personal perspective align with the company's mission to provide the best error monitoring solution for developers. Andrew's unique blend of engineering and data science, along with his interest in SDKs, make him a rare addition to the already diverse team.

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your web and mobile application errors.

Rollbar all together, quarterly all hands

Thu, 09 Feb 2017 00:00:00 GMT

This week was our first quaterly all-hands of 2017 for the full Rollbar team. This is exciting, as Rollbar are a small and distributed team with members in Barcelona, Fort Worth, Las Vegas, Santa Monica and abroad. Most of us haven't all officially met aside from Slack, Google Hangouts or phone calls, so while we feel like we know one another from working together for some time, this has been long awaited!

To add to the excitement, we also had a new hire start on Monday. We are extremely happy to welcome Andrew Ledvina to our team as Lead SDK Engineer. As a remote team member, Andrew is based in Santa Monica. We're thrilled to have him start during our quaterly all-hands and add his expertise and energy to the team!

Rivkah Standig, our awesome Support Engineer/Developer Advocate, joins us from her travels abroad, flying in from Iceland. As the one team member that most of us had not yet met, we are happy she could make this on-site during her year of remote work and travels! Our other remote team members flew in from near and far: Jon de Andres (Engineering) came from Barcelona, Chris Barmonde (Engineering) from Las Vegas and Mike Smith (Growth) from Fort Worth. With focus on our remote team, it should also be mentioned that our San Francisco team is pretty amazing as well.

Some of the week's highlights (photos below) include a team lunch, a joint birthday happy hour and a group Thai cooking class in the Mission!

The Thai cooking class was a hands-on team activity, where we split into teams and created elements of a 3 course meal together. The instructor was very informative; she added a lot of energy to the experience! Once we prepared our meal, we were able to sit and chat with each other over it. The setting was in a restored church loft in the Mission and provided cozy refuge from the rainy winter day. There was a lot of cooking, chatter, and laughter throughout the event. The team enjoyed it and left with the recipes in hand to try at home.

On a final note, this quarterly all hands makes for a memorable one! We conclude the week of team meetings, gatherings and outings having a better sense of each other, our company mission, and are ready to continue the new year strong.

How GorillaStack used Rollbar to level up logging with the Serverless framework

Tue, 07 Feb 2017 00:00:00 GMT

Our friends at GorillaStack wanted to share how they set up Rollbar with the Serverless framework, and made a handy tool so you can do the same.

Here at GorillaStack, we are big lovers of the Serverless framework. By default, the Serverless framework uses CloudWatch logs to store any system log messages and output from your lambda code. Pretty quickly, we found ourselves needing to escalate log messages such that we could be notified of application errors and act on them more proactively.

Enter Rollbar. I have been using their service on many side projects since discovering their free tier some years ago. We also robbed them of almost all their swag at the last re:Invent conference. Rollbar is a great system to use for escalating errors and messages from your applications. Best of all, there are options for adding it as a transport for popular logging solutions. As users of the node module 'winston', we found a transport available for Rollbar. Unfortunately, it wasn’t quite up to date, and didn’t format certain argument orders correctly, so we forked it ourselves. :)

https://github.com/GorillaStack/winston-rollbar

Using this integration with your application logging library, messages of all levels will still be logged to CloudWatch logs, but those at or above a configured log level will be escalated to Rollbar, which will then notify you of application errors as they occur.

As we usually do, we decided to publish an open source example to demonstrate our approach. To get started, the only thing you will need is a Rollbar API token. Simply export this into your environment before deploying our example repository to see it in action.

export ROLLBAR_SERVER_ACCESS_TOKEN= <_access_token_here_>
git clone git@github.com:GorillaStack/serverless-rollbar-example.git
npm install
sls deploy
sls invoke -f happyEndpoint
sls invoke -f unhappyEndpoint

You will receive an email after invoking the unhappyEndpoint with a summary of the error log message. If you check your CloudWatch logs, you will see log messages of all levels.

This is a simple example of how to very simply level up your lambda logging using Rollbar. Have you achieved this in a simpler way? Let us know in the comments.

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your application errors.

Connect Rollbar to your source code and issues in GitLab

Tue, 31 Jan 2017 00:00:00 GMT

Good news for GitLab users! You can now deep link error stack traces in Rollbar directly to the code in your GitLab repos. Rollbar now supports GitLab's native issue tracking features, allowing you to seemlessly debug and manage new issues as they happen in your existing workflow. We’ve rolled out support for source control for both GitLab.com and self-hosted. Issue tracking only supports GitLab.com, self-hosted support is coming soon.

Linking source code in GitLab to Rollbar stack traces

Whenever Rollbar detects errors in your application, you receive a detailed stack trace. By connecting your GitLab repos, you can also click through from the stack trace in Rollbar directly to the underlying code that broke in GitLab. It's like having x-ray vision for your application errors.

Turn errors in Rollbar into trackable issues in GitLab

Rollbar now supports GitLab issue tracker. It's a fairly straightforward integration: When viewing an error in Rollbar, simply click the "Create GitLab Issue" button to send the error details to GitLab. You can also link an error in Rollbar to an existing GitLab Issue, by copying and pasting the URL for the GitLab Issue you would like to link.

When you connect Rollbar and GitLab together, your debugging and error monitoring will become much more efficient. For more information, visit our GitLab documentation or press release for details.

Level up your issue and error tracking workflow when you connect Rollbar to GitLab. GitLab users, signup and get 100,000 errors monitored for free and let us help you take control of annoying application errors.

Rollbar integration for the Ionic framework

Fri, 06 Jan 2017 00:00:00 GMT

Our friends at Cuttlesoft wanted to share how they use Rollbar to detect errors in Ionic built applications. Enjoy!

At Cuttlesoft, we use Rollbar's excellent full-stack error monitoring service for pinpointing and fixing tricky bugs. Our team loves Rollbar for its integrations with other popular services (we get our error notifications via Slack so we’re constantly in the know). For building hybrid mobile and progressive web apps, we generally rely on Ionic. Ionic is an open-source framework for hybrid mobile app development maintained by Drifty. Built with AngularJS and Cordova, Ionic is a popular tool for mobile developers everywhere. To combine these two, we've developed a method for integrating Rollbar error tracking with the Ionic stack.

We find it so useful that our co-founder and CTO Emily Morehouse wrote a tutorial for using our method of integration so that you too can harness the bug-squashing power of Rollbar + Ionic. Using just a few tools, we'll show you how to make Rollbar work in perfect sync with Ionic. Since Ionic is a hybrid framework, there are a few different services where Rollbar needs to be plugged in:

Native iOS
Native Android
JavaScript

To do this, we'll use a few different plugins that help monitor each OS and stack layer. We'll use Emily Morehouse's fork of the Cordova Rollbar plugin to catch native errors, and the ng-rollbar AngularJS plugin to catch our Ionic application errors. Now, let's dive in. The first (and easiest) plugin to set up is the Cordova plugin, so let's knock that out first. We're assuming that you've already created an account and project on Rollbar's site but if you haven't, go do that first.

Catching Native Errors

Simply install the plugin:

cordova plugin add https://github.com/emilyemorehouse/cordova-plugins-rollbar.git --variable ROLLBAR_ACCESS_TOKEN="" --variable ROLLBAR_ENVIRONMENT=""

Then, add the following code, probably in your run function and definitely after the device is ready. To use the Ionic platform ready trigger:

$ionicPlatform.ready(function() {
    window.cordova.plugins.Rollbar.init();
});

Or to use the Cordova deviceready listener:

document.addEventListener('deviceready', function () {
    window.cordova.plugins.Rollbar.init();
});

Either works fine, but we're a fan of the former. for a variety of reasons, it's generally not kosher to use variables through command line arguments (security of course, but also that running ionic platform reset can completely mess up your project setup as variables are not always persisted in your configuration files), so be aware of the caveats with this approach. And that's it! All native crashes and errors will now be reported. It should be noted that native crashes are typically not caused by your application, but are super helpful if you're developing or using a plugin that could have issues.

Catching Javascript errors

To catch Javascript errors (the most important ones and ones that you're usually the most responsible for), you have a bit more setup and a lot more control over when and how these errors are caught. First off, let's get the dependencies installed:

bower install ng-rollbar --save

Now, let's get them included (we have bower set up so it installs to lib, so adjust as necessary:

Remember to include the module:

angular
.module('myApp', [
	'ionic',
    'tandibar/ng-rollbar'
]);

Initialization must occur in your config function, though we find that it would make more sense to keep it in the run function as that's where every other initialization occurs. But the earlier the initialization is called, the better - since any exceptions that occur prior to initialization won't be caught and configuration happens before running. Place this in your config function:

myApp.config(['RollbarProvider', function(RollbarProvider) {
    RollbarProvider.init({
    	accessToken: '',
        captureUncaught: true,
    	payload: {
            environment: ''
    	}
	});
}]);

That's it! Any error that's thrown will now be caught. This does not, however, work automatically with "console.error"s. If you'd like to manually throw a Rollbar exception, you can use any of the following severities as long as you inject Rollbar:

// Rollbar severities
Rollbar.critical("some critical error");
Rollbar.error("some error");
Rollbar.warning("some warning");
Rollbar.info("some info");
Rollbar.debug("some debug message");

Another trick we like to use is to simply override the console.\* functions and replace them with the Rollbar functions. Toss this into your run function if you want to convert all console logs into Rollbar messages, though we only use the error override in production and all others in development:

console.error = Rollbar.error;
console.warn = Rollbar.warning;
console.info = Rollbar.info;
console.debug = Rollbar.debug;

// Duplicated use of Rollbar.info for console.log since an equivalent does not exist
console.log = Rollbar.info;

There you have it, Rollbar support for the Ionic Framework. If you're a fan of this tutorial, visit cuttlesoft.com/blog for other helpful guides, open-source projects, and blog posts about all aspects of software development. Big thanks to Rollbar for building such a great service, and for letting us share our tutorial with you.

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your application errors.

Join Rollbar at AWS re:Invent

Mon, 21 Nov 2016 00:00:00 GMT

Join us this year at Amazon Web Services annual conference, re:Invent! Rollbar will be participating along side of Atlassian and other Atlassian ecosystem partners like StatusPage, Splunk, Puppet, Xmatters and Workato. We will all be in the Atlassian booth demonstrating how our solutions work better together. Come by and check it out - booth 2446.

Come join us at the event and see how integrating Rollbar’s error monitoring with Atlassian’s products provides teams with:

Instant error grouping and aggregation: Errors are grouped by root cause and linked to your source code within Bitbucket.
Real-time notifications: Intelligent alerting rules configured for HipChat and JIRA.
Easy workflow and prioritization: Discover the most impactful errors with data and create issues in JIRA instantly with rich data, contextual details and error analytics.
Faster time to resolution: Debug, deploy fixes and track regressions much faster when your error data moves through HipChat, JIRA and Bitbucket automatically.

We'd love to chat with you in person at AWS re:Invent. Make plans to stop by booth 2446 and say hi. We look forward to meeting you.

Deep link stack traces to Bitbucket and track Pipeline deploys

Mon, 14 Nov 2016 00:00:00 GMT

Good news for Bitbucket users! You can now deep link error stack traces in Rollbar directly to the code in your Bitbucket repos. Rollbar also now integrates with Bitbucket Pipelines.

Deep links to your source code in Bitbucket

Configure Pipelines to notify Rollbar of new deploys

Once Pipelines is configured, every deploy of your app will automatically be tracked in Rollbar. Making it easier to identify which code changes led to critical production issues.

Rollbar integrates with the following Bitbucket features, Bitbucket Issue Tracker, and now Bitbucket repos and Bitbucket Pipelines.

Don't forget Rollbar also integrates with other Atlassian products, like JIRA, HipChat and of course Bitbucket.

Dealing with errors can be time consuming and often an interruption in your everyday development workflow. It's important that we continue to work to improve this process and that we extend the capabilities and insights of your current toolset and workflows. We hope that you enjoy the new integration options!

Supercharge your issue and error tracking workflow when you connect Rollbar to Bitbucket. Sign up for a 14-day free trial of Rollbar and let us help you take control of your annoying production errors.

Join us for a panel discussion on DDoS attacks, DNS and more [EVENT]

Tue, 08 Nov 2016 00:00:00 GMT

Join us next week for CircleCI's November Office Hours!

Cory Virok, Co-founder and CTO of Rollbar, along with others from the startup community will be participating in a panel discussion on October's DDoS attack that took down a lot of the services we rely on in our everyday workflows.

When: Wednesday November 16th @ 6pm PT

Where: Heavybit 325 9th Street San Francisco, CA

Agenda:

6:00 - 6:30 pm: Networking, food and drinks.
6:30 - 7:00 pm: Downtime Panel: October’s DDoS Attack Decoded.
7:00 - 8:00 pm: Q+A and Networking.

Panelists and Moderator:

Scott Klein, Co-founder, StatusPage.
Cory Virok, Co-Founder and CTO, Rollbar.
Lev Lazinskiy, Head of Technical Services, CircleCI.
Jim Rose (moderator), CEO, CircleCI.

RSVP for this event using this link.

This will be a fun and informative event. There will be food, drinks and free Rollbar swag. :-) Bring a friend or your entire team, the more the merrier. Hope to see you there!

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you defeat annoying errors in your applications. :-)

Correlate errors to feature rollout tests with Rollbar and Split

Thu, 03 Nov 2016 00:00:00 GMT

Our friends at Split wanted to let you know that you can now integrate Rollbar with Split. And take control of your error handling and feature rollouts - together. Enjoy! :-)

Rollbar is an excellent error monitoring service that works across all of the languages Split supports. It’s great at telling you what’s going wrong and where, but doesn’t necessarily have all the details on why. That’s where Split comes in, with our new Rollbar integration.

We built Split to help teams take control of their deployments by slowly delivering features to targeted groups of users. In this way you can limit the scope of trouble when things do go wrong, killing the feature until it’s fixed, then slowly rolling it out again. Our new integration brings rollout metadata from features in Split into the Rollbar interface, so teams can quickly correlate errors against the latest deployments.

When it looks like a recently-released feature was responsible for the degradation of service, Rollbar users can click-through to Split to quickly take action, saving valuable time in the remediation process.

Here's a a quick demo on how Rollbar and Split work together to make your controlled feature rollouts more productive.

Like you, we care deeply about giving your end-users the best experience possible, and Split’s controlled rollouts paired with Rollbar’s robust error notification make delivering on that promise easy. Our Rollbar integration is free to customers of both services, and you can always try **[Split free for 30 days](https://split.io)**.

To learn more, visit our integration documentation for Split. To see if Rollbar integrates with other tools you’re using, see our full list of integrations here. If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you defeat errors in production.

Postmortem from outage on October 21, 2016

Tue, 25 Oct 2016 00:00:00 GMT

We had a significant outage this past Friday, October 21. Many customers were not able to reach rollbar.com or api.rollbar.com, some data was lost, and a few customers experienced cascading issues caused by our API outage. The root cause was a Distributed Denial of Service (DDoS) attack on our DNS provider.

We weren't the only service that had issues on Friday, but that is no excuse—we know that our customers rely on Rollbar as a critical monitoring service and it's all the more important that we are up when everything else is down. We'd like to share some of the details about what happened and what we're doing to prevent this kind of issue from happening again.

Outage Timeline and Resolution

All times PDT.

Approximately 4:00 a.m.: DDOS began against Dyn, Rollbar's DNS provider
4:00 a.m. - 6:00 a.m.: A few Rollbar customers noticed problems and contacted our support team
6:00 a.m.: First attack is mitigated by Dyn
8:00 a.m.: We became aware of the first attack, but noted that it had ended.
9:15 a.m.: We noticed timeouts reaching rollbar.com and that a second wave of the attack was under way.
9:23 a.m.: We updated our status page about the issue
9:40 a.m.: We began migration to an alternate DNS provider (AWS Route 53)
10:40 a.m.: Configuration changes for the DNS migration were complete; however, we now had to wait for the nameserver change to propagate
11:07 a.m.: We updated our status page, as well as Twitter and Facebook, regarding the migration and posted static IP addresses that customers could use in the meantime.
Over the next 6-12 hours, DNS traffic moved over to AWS and all customers were able to resolve rollbar.com and api.rollbar.com

Customer Impact

During the outages:

Many customers were not able to reach rollbar.com
Many customers' applications (servers, web and mobile clients) were not able to reach api.rollbar.com to send data, leading to data loss as those requests failed and, in some cases, application problems caused by the slow requests

What We Learned

The problems that this outage exposed were:

DNS was a single point of failure. While Dyn has been an extremely reliable provider over the past 4+ years, Friday's outage showed that they are not infallible.
Using DNS for geographic load balancing and failover made us more susceptible to an outage like this, since we use short TTLs and have a generally more complex DNS setup.
Our monitoring systems did not detect this kind of outage. Traffic was normal and all internal systems were normal. We only learned about it through customer reports and manual testing.
Some Rollbar client libraries do not handle failed requests well and are susceptible to resource starvation caused by retries. We're investigating two reports of this in the Sidekiq backend for the Ruby library, and are investigating one other that we believe is in a custom Java library.
We had some problems communicating with customers during the outage:
- Our status page was hosted on the same domain as our website, so customers who could not reach rollbar.com likely couldn't reach status.rollbar.com either
- Twitter, our backup channel, also uses Dyn for DNS and was affected by the same issue.

What We're Doing

In response to Friday's events and what we've learned since:

We are re-evaluating our DNS setup so that we will be resilient against failures on this tier, likely by using multiple DNS providers. Since we use advanced DNS features like dynamic routing, this is not quite as simple as it may sound and we're doing some research first.
We are adding monitoring coverage for DNS.
We're moving our status page from status.rollbar.com to rollbarstatus.com, which will use a different DNS provider than our primary domain.
We will be investigating how our client libraries handle network failures and audit them to ensure that the impact to the host application is minimal in the case of a network outage.

For any customers who are looking for an immediate solution to this problem, we'd like to refer you to rollbar-agent to move the network request out of the application and into another process that reads queued events from disk. This also has the benefit that requests will eventually succeed after a network outage. This model is powerful, but currently fairly difficult to set up. We'll be scheduling some improvements to rollbar-agent soon.

Other Resources

There has been a lot of chatter around the web about this outage. Here are a few links that we found helpful or interesting:

Conclusion

We know our customers depend on Rollbar as a critical, reliable piece of infrastructure, and we're sorry that we did not meet this standard on Friday. If you have any questions, don't hesitate to contact us at support@rollbar.com.

A better error monitoring workflow Rollbar + Atlassian

Wed, 12 Oct 2016 00:00:00 GMT

Dealing with errors can be time consuming and often an interruption in your everyday development workflow. It's important that we at Rollbar continue to work to improve this process and that we extend the capabilities and insights of your current toolset and workflows.

Atlassian users can now rejoice! 🎉 Debugging errors just got a whole lot easier and faster. With our latest improvements and integrations with Atlassian products you can now keep tabs on your application directly within the tools you're already using to build it. No more switching between multiple tools and services to debug and deploy code. Creating a seamless user experience and unified workflow around production errors.

Integrating Rollbar’s error monitoring with Atlassian’s products provides teams with:

Instant error grouping and aggregation: Errors are grouped by root cause and linked to your source code within Bitbucket.
Real-time notifications: Intelligent alerting rules configured for HipChat and JIRA.
Easy workflow and prioritization: Discover the most impactful errors with data and create issues in JIRA instantly with rich data, contextual details and error analytics.
Faster time to resolution: Debug, deploy fixes and track regressions much faster when your error data moves through HipChat, JIRA and Bitbucket automatically.

Watch our short video for an overview of how Rollbar and Atlassian's products work together:

Take your production error monitoring to the next level and maintain your existing workflow. By integrating Rollbar with JIRA, HipChat and Bitbucket you can detect, diagnose and resolve errors in less than 2 minutes!

By integrating Rollbar with JIRA, HipChat and Bitbucket you can Detect, diagnose and resolve errors in less than 2 minutes!

[Click to Tweet]

Below are a few examples of how to manage and fix errors that occur in your software using these new integrations:

Get real-time error alerts in HipChat

Send real-time notifications about application exceptions and code deployments to your rooms in HipChat. Reduce alert fatigue and customize error grouping rules to get alerted to issues that matter most. Stay on top of code deployments and fixes as they happen. See HipChat documentation for more details.

Deep links to your source code in Bitbucket

Rollbar integrates with Bitbucket repos, Issue Tracker, and Pipelines. Create deep links to your source code in Bitbucket to help you diagnose and fix issues faster. Link stack traces in Rollbar directly to lines of code in your Bitbucket repos. Configure Bitbucket Pipelines to notify Rollbar of new deploys and see the commits that were included with each code deployment. See Bitbucket documentation for more details.

Add rich error data to JIRA issues

Connect Rollbar to JIRA and enhance issues with detailed error data like stack trace, error status, total occurrences, unique IPs affected, first seen date, last seen date and the date the error reoccurred and regressed. All conveniently displayed, linked and accessible within JIRA. Set notification rules in Rollbar to automatically open new issues in JIRA. Create and assign JIRA issues with a few clicks. With bidirectional syncing, statuses update in both JIRA and Rollbar. See JIRA documentation for more details.

Rollbar has integration support built for HipChat, JIRA Issue Tracker, a native JIRA Add-on (new), Bitbucket source control (new), Bitbucket Issue Tracker, native Bitbucket Add-on and Bitbucket Pipelines (new).

Attending Atlassian Summit?

Visit us at booth B19 at Atlassian Summit this week and sit down with a Rollbar expert to show you how you can level-up your app monitoring and error handling processes with our latest JIRA, HipChat and Bitbucket integrations. To learn more visit us at our booth, or go to https://rollbar.com/summit and receive the first month of Rollbar free.

On the road again, where Rollbar will be this fall

Tue, 11 Oct 2016 00:00:00 GMT

Rollbar team members will be at the following events in the next few months, and we'd love to chat with you in person! Whether you have some specific questions about Rollbar, want to tell us about your projects, or just want some Rollbar gear, we'll be hanging out at these events eager to to talk shop. If you see us at any of these places, make sure to say "hi"! 👋

Atlassian Summit

Where: San Jose, California / When: Weds, Oct 12 to Thurs, Oct 13

The Atlassian Summit is all about the various Atlassian products (such as JIRA, Hipchat, or Bitbucket), and has talks from both Atlassians and Atlassian customers about how to use Atlassian tools better and more effectively. We'll be at booth B19 at the Summit, and would love for you to visit us there and chat about our new and improved JIRA, Hipchat, and Bitbucket integrations.

FSTOCO (Full-Stack Toronto)

Where: Toronto, Canada / When: Mon, Oct 17 to Tues, Oct 18

FSTOCO is a full-stack web conference for all web professionals. With two days of talks running the gamut from UX, project management, front end, and back end, there's something for everyone. We're excited to be sponsors of FSTOCO this year, and hope to see you there!

Ruby Rampage Hackathon

Where: virtual, global hackathon! / When: Sat, Oct 22 to Sun, Oct 23

Do you love hackathons and Ruby, and not having to leave your apartment? If so, Ruby Rampage is for you! Compete online over two days to build the best Ruby app with your team, and you could win a prize. We're proud sponsors of Ruby Rampage, and would love to "see" you there.

Thunder Plains

Where: Oklahoma City, Oklahoma / When: Thurs, Nov 3 to Fri, Nov 4

We're sponsoring Thunder Plains, a web and mobile development conference focusing on JavaScript and related technologies. With only 225 tickets, it promises to be an intimate and inspiring event.

RubyConf 2016

Where: Cincinnati, Ohio / When: Thurs, Nov 10 to Sat, Nov 12

RubyConf is the main international annual gathering for the Ruby community. We are sponsoring RubyConf and would love to talk shop with you there.

Node Knockout Hackathon

Where: virtual, global hackathon! / When: Sat, Nov 12 to Sun, Nov 13

If you love online hackathons but aren't a Rubyist, Node Knockout might be right up your alley. Compete online over two days to build the best Node.js app with your team, and win prizes and glory. Hope to see you there!

Node Interactive

Where: Austin, Texas / When: Mon, Nov 28 to Wed, Nov 30

Node Interactive is the conference for all things Node.js! With talks ranging from front end, to back end, to the Internet of Things, Node Interactive has something for every Node developer. We're sponsoring Node Interactive this year and can't wait to chat with you there.

AWS re:Invent

Where: Las Vegas, Nevada / When: Mon, Nov 28 to Fri, Dec 2

AWS re:Invent is the largest gathering of the global Amazon Web Services community. Rollbar will be participating along side of Atlassian and other Atlassian ecosystem partners in booth 2446.

Schedule an in-person meeting

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you defeat annoying errors in your applications. :-)

Rollbar add-on now available for JIRA

Mon, 10 Oct 2016 00:00:00 GMT

We’ve recently released a new JIRA Add-On to help extend your application error monitoring workflow. JIRA is one of the leading issue tracking solutions used by agile teams. Enabling them to plan, track, release and support building better software.

Send rich error data to JIRA

Connect Rollbar to JIRA and enhance issues with detailed error data like stack trace, error status, total occurrences, unique IPs affected, first seen date, last seen date and the date the error reoccurred and regressed. All conveniently displayed, linked and accessible within JIRA.

Here are a few of the ways you can improve your monitoring workflow when you connect Rollbar with JIRA and install the new Rollbar for JIRA add-on:

Link and display root cause exception data in issues.
Set notification rules to automatically open new issues.
Create and assign JIRA issues with a few clicks.
Bidirectional syncing, update statuses in both Rollbar and JIRA.

PRO TIP / REMINDER: Automatically (and manually) create JIRA issues from Rollbar errors when you configure JIRA as a Notification Channel in Rollbar. Supercharge your issue tracking when you use JIRA Notications + Rollbar for JIRA Add-On together. Check out our JIRA documentation for details.

To install the new Rollbar for JIRA add-on:

In your JIRA account go to Admin > Add-ons
Search for "Rollbar"
Click "Install" on the Rollbar for JIRA add-on

Go to our Atlassian Marketplace listing for JIRA
Click 'Get it Now'

We hope that by bringing Rollbar data into JIRA you can streamline your issue tracking and monitoring efforts and save a lot of time in your day to day debugging efforts.

To see if Rollbar integrates with other tools you’re using, see our full list of integrations here. If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you defeat errors in production. :-)

How CircleCI uses Rollbar to level up their continuous delivery

Thu, 22 Sep 2016 00:00:00 GMT

I'm excited to share a fun and insightful interview our friends at The Changelog recently did with Paul Biggar, Founder at CircleCI (and Rollbar super-user). We're big supporters and fans of The Changelog and we asked their host and master interviewer Adam, to help us produce a few short interviews with our customers. It's a fun project that lets us pull back the curtain and learn more about how our customers monitor their applications and processes for handling errors and deploying code. Enjoy!

Featured in this interview: Adam Stacoviak, Founder & Chief Editor at The Changelog, a podcast on software development and open source. Subscribe via iTunes or RSS. Paul Biggar, Founder of CircleCI, a leading continuous integration platform.

Adam: Hey there. Adam here, editor-in-chief of Changelog. I'm here with Paul Biggar, founder of CircleCI (circleci.com) and we're gonna talk about how important Rollbar is to him and his team to help them deliver on their brand promise to ship better coder faster. Paul, tell me about CircleCI. What types of services do you provide?

Paul: CircleCI is a continuous integration and continuous delivery platform. Our customers are the developers in an organization. Developers rely on us heavily as part of their deployment workflows.

Adam: I guess deploying obviously is important. You've got the phrase 'ship it' for a reason, so it's a critical piece to an ops or developer organization to be able to ship code fast, ship code reliably.

Paul: Right. The idea is to do continuous delivery, where you're constantly delivering value to your customers. Every time a developer pushes code, the CircleCI test will run and they'll validate that that code works, whether it's on a branch or whether it's on the master branch, and then they deploy the code to Heroku, to AWS, to wherever your hosting is.

Adam: Break down the problem a little bit deeper for those out there who might not be that informed about CI or CD services and what the value is that you actually do. Some people might still be rsync-ing or doing different things to deploy their code, and they don't have workflows and things like that. Can you break down the problem a little bit more, and how CI and CD solves that?

Paul: You can always just run tests on your own machine, and you can always just deploy from your own machine. It's hard to really describe how much better it is for it to be part of an automated, continuous process, but it's like night and day. When our customers switch to a continuous delivery model from deploying on their machine whenever, it completely changes the velocity of their team and of their organization. If you're doing continuous delivery, you literally change how your company delivers code and the speed at which it delivers code. I would never work at a company that didn't do continuous delivery.

If you're doing continuous delivery, you literally change how your company delivers code and the speed at which it delivers code.

Adam: So clearly you're like a part of the machine that should not/could not break. If you break, you're losing trust, you're losing value, and ultimately you're not delivering your brand promise. So tracking errors, tracking bugs, keeping your team informed about breaks, keeping your team informed in terms of like - it might even be different parts of the service, but you have a microservice architecture versus a monolith, and you've got various teams plugging into CircleCI and how things are built... Let's talk about the obvious question here, which is how do you use Rollbar? And not just how do you use Rollbar, but why do you use Rollbar?

Paul: I was talking about doing continuous delivery there, and one of the key parts about doing continuous delivery, you don't just have to test your software, but you have to constantly keep track of it. If you're going to be doing deploys ten times a day or twenty times a day, and you have to know that each deploy works, and the way to do that is to have really good production monitoring. Rollbar is the thing that you need to do that monitoring. You need to make sure that every time you deploy, you're gonna get an alert if something goes wrong, and that's exactly what Rollbar does for CircleCI, for our infrastructure.

If you're going to be doing deploys ten times a day or twenty times a day, and you have to know that each deploy works, and the way to do that is to have really good production monitoring.

Rollbar is the thing that you need to do that monitoring.

Adam: Let's talk about what you did before Rollbar. Before Rollbar, I'm sure your team had processes for monitoring, tracking, fixing errors... Help me understand what it was like for your team before Rollbar. What kind of processes did you have, how did you deal with these things prior to introducing Rollbar to your team?

Paul: Before we used Rollbar we used a different crash reporting service and we were shopping for a new one. We did the tour and looked at Rollbar and all of its competitors, and it was really the feature set of Rollbar that was super impressive and that made us go there, in particular the people tracking I think is really... It's not just a great feature, but it also kind of speaks our language because we're very focused on making sure that the customers are happy and we want to make sure that we have an individual understanding of what happens to each customer. The fact that we're able to click on "This customer is experiencing a lot of errors", and to be able to follow the progression of errors that they've been experiencing is very valuable to our team.

The fact that we're able to click on "This customer is experiencing a lot of errors", and to be able to follow the progression of errors that they've been experiencing is very valuable to our team.

Paul: If we get an e-mail from a customer and the customer says, "Your website keeps glitching on me," and being able to go to Rollbar and to say, "Okay, this individual customer, this is how they're experiencing the site"... Because otherwise you have to give an overall state of things, and overall things are looking good, because if they weren't, we'd be dealing with it.

Person Tracking as seen in Rollbar

Adam: I think Rollbar to me, just in general error monitoring, seems to be engineer-focused, but I've gotta imagine, based on what you've just said there, that it supports whomever is doing support for you; maybe it's actually engineers who do support for you. Can you talk a bit about that crossover, where it's an informing tool not just to engineers, but also to other parts of the business?

Paul: Everyone at CircleCI is empowered to look into whatever they need to look into. If you're in support, and our support people are engineers - have an engineering background, they will often need to dive into what exactly a customer is experiencing, so being able to see... We use a couple of monitoring tools, we also use Datadog, so being able to see there's an increase in errors on Rollbar, there's a decrease in reliability on the API - you would see that in Datadog - this is stuff that you need to be able to say, "Why is the customer experiencing this problem?". Rollbar is really used by the dev-ops and the engineering team, but it's an invaluable tool for the support team too.

Rollbar is really used by the dev-ops and the engineering team, but it's an invaluable tool for the support team too.

[Click to Tweet]

Paul: And when we were first looking at why is Rollbar gonna be valuable, it was a time when we didn't have a support team. All of us were doing support; we were either alternating by day or by week at that point, and it was part of the tool set. It was just a great place to start looking at where a customer experienced a certain problem, and that was what won Rollbar over for us.

Adam: It's obvious that with those kinds of insights and that kind of informing across teams - not just engineers, but other parts of the business - you've gotta be able to make customers happy pretty quickly. Can you give me any particular experience where something was happening, an error was happening, and you were able to track down who it was affecting? Maybe you found that before they even reported it to the support, and you won the day, you won a customer for life.

Paul: I think it's a lot more mundane than not, and I think intentionally so. People don't tell dramatic stories about their monitoring software if their monitoring software is good. When Rollbar is working the way that you know it's working, it's because you don't have any dramatic situations. You get alerts about new things that are happening, we get it shipped into Slack, and if Rollbar is telling us that something is happening, we immediately look into it, and on the website and with the customers nothing happens; everything stays the same. We fix the bug, we figure out why it's happening. We either mute the alert or we fix it, and it doesn't get to the point where a customer has to be dramatically saved, or anything like that. The site just keeps working.

We fix the bug, we figure out why it's happening. We either mute the alert or we fix it, and it doesn't get to the point where a customer has to be dramatically saved, or anything like that. The site just keeps working.

Rollbar Notifications in Slack (Deploy, New Error, Resolved Error)

Adam: You talked before about your choice of Rollbar and their feature set. Can you give me an example - and you may have already done this, but can you give me an example of particular features that you were excited about, and which features they had that you dreamed about and were able to put in use? Can you walk me through what those were?

Paul: I think deploy tracking is probably not an intuitive feature, but it completely changes how you use it. Knowing which is the first deploy that introduced this problem allows you to immediately solve the problem. "This was first introduced at this commit" - then you go back to that commit, you look at the 20 to 50-line diff between that and the previous deploy, and you're like, "Oh, this is what we changed; this is what the exception looks like. It's obvious what the problem is."

Adam: How often does something like that happen for you?

Paul: All the time. When you're looking at monitoring, "How did it start happening?" is as close to a root cause analysis as you can get.

"How did it start happening?" is as close to a root cause analysis as you can get.

[Click to Tweet]

Adam: I'm just thinking, since it happens - let's not say all the time, because that sounds bad, let's just say often... And this is a feature that you sought out. Previously, this superpower was not available to you and your team; you signed up for Rollbar and suddenly just this single feature is a super power for you to help you catch things.

Paul: After we added deploy tracking on our end and during our deploy process something pings Rollbar, it completely changed how we were able to track down errors. Bugs and errors and exceptions and all that are just a constant. It's not like we deploy perfect software and it works all the time. We're constantly triaging, "Are these exceptions important enough to fix or are they just noise? Are they handled by the dev or is it just noise that comes from having a distributed system?" and the ability to say when that particular error started happening is just like a night and day feature. If you don't use that feature, once you have it you immediately realize how useful it is and how valuable it is. You can get to the root cause immediately.

After we added deploy tracking on our end and during our deploy process something pings Rollbar, it completely changed how we were able to track down errors.

Deploy Tracking as seen in Rollbar

Adam: Can you give me an example of another feature that you use that you're totally in love with that isn't very clear that it's available to you, that you use quite often?

Paul: The ability to configure when something alerts you. The gap between a hundred occurrences and a thousand occurrences of something is quite big. Being able to have Rollbar say "This thing just became a big deal" is important.

Adam: So you can basically say, "If this type of thing has happened before a hundred times, that's okay. We're familiar with that, we're aware of that bug, we're on that bug, our team is tracking that bug." But when it blows up to a thousand or two thousand, clearly it's gone from dwarf to monster.

Paul: Right, because it's very rare to suddenly have a new bug that we've never seen before. What usually happens is the bug will happen in a very rare situation. Maybe someone triaged it at some point and said, "That's low down the list of priorities," and then all of a sudden, a bug that we've seen dozens of times, hundreds of times before, once-a-day sort of thing, suddenly we an error a thousand times in an hour, or ten thousand times in an hour. That's when something becomes super serious, and that's the things that we want to get alerted about.

suddenly we see an error a thousand times in an hour, or ten thousand times in an hour. That's when something becomes super serious, and that's the things that we want to get alerted about.

Adam: The brand promise of CircleCI is "Ship better code faster", which in my opinion - and maybe the listeners will feel this way, too - seems pretty confident. You're confident in what you do, you're confident in what your software promises to deliver to its customers. Can you talk a bit about how Rollbar has helped you deliver on that brand promise?

Paul: The most important thing in delivering things quickly to your customers is having a level of confidence that the code works. You can't perfectly test something before it goes out, you can't perfectly validate it. You're gonna put it in front of the customers and something weird and new is gonna happen, and you have to know when that goes wrong. You have to have something like Rollbar to tell you something just went wrong and to alert you about it immediately. Otherwise you would just be paralyzed by fear. You'd be unable to ship anything because you wouldn't know when it would go wrong and what the downsides would be.

You can't perfectly test something before it goes out, you can't perfectly validate it. You're gonna put it in front of the customers and something weird and new is gonna happen, and you have to know when that goes wrong.

Adam: Let's assume anyone who listened to this is someone who needs to use Rollbar. Someone needs to know about this tool, needs to know about this product, needs to know how it has changed how you do business because of it. I'd like them to know how important this tool is to you. A better question might even be, "Could you have done what you're doing with CircleCI, the level of morale for the team, the happiness level - all these different things that play into this, without Rollbar's help?"

Paul: We operate at serious scale, and the first thing we do when we create a new service is we install Rollbar. We need to have that visibility, and without that visibility it would be impossible to run at the scale we do. Certainly with the number of people that we have. We're a relatively small team operating a major service, and without the visibility that Rollbar gives us into our exceptions it just wouldn't be possible. If there's people out there who ship code without Rollbar, I can only imagine the pain that they're going through.

the first thing we do when we create a new service is we install Rollbar. We need to have that visibility, and without that visibility it would be impossible to run at the scale we do.

If there are people out there who ship code without Rollbar, I can only imagine the pain that they're going through.

[Click to Tweet]

Adam: That's awesome. Thank you so much for your time today, Paul. I really appreciate you taking the time to speak with me. For those tuning in, if you haven't yet, head to rollbar.com to learn more. Cheers!

Thank you to Paul and the team at CircleCI for sharing this level of insight and for leading by example when it comes to continuous delivery and working to maintain error-free experiences for their users. Bravo!

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your application errors.

Improved ruby error reporting with our latest gem updates

Wed, 14 Sep 2016 00:00:00 GMT

Recently, we released version 2.13.0 of the rollbar-gem. This update is full of new features and some minor bug fixes. The full release notes can be found here, Release 2.13.0. Here are a few of the highlights in this update:

1. Allow overriding configuration

Many customers have asked to implement a way to override the default configuration for a specific block of code. Some of them use the same process to send reports to our API for different projects, some need to change the environment, and others want to use one async handler for a single block of code (or none at all).

So, we've added a new method called Rollbar.with_config to do this. It receives a Hash object with the configuration overrides you want to use for the given block. The configuration options can be found at Configuration. The Hash passed to with_config should be formatted like {environment: 'specific-environment'}. For example:

Rollbar.with_config(use_async: false) do
  begin
    # do work that may crash
  rescue => e
    Rollbar.error(e)
  end
end

This method looks similar to Rollbar.scoped, and Rollbar.with_config uses it internally. Now Rollbar.scoped can receive a second argument with the configuration overrides for the given block of code. So if you need to set a new payload scope and new config for a code block, you can write:

scope = {context: 'foo'}
new_config = {framework: 'Sinatra'}

Rollbar.scoped(scope, new_config) do
  begin
    # do work that may crash
  rescue => e
    Rollbar.error(e)
  end
end

In the example above, we are defining a new payload scope and overriding the framework configuration for the reported errors inside the given block.

2. Code and context

The data we send for each frame in a backtrace is the filename, code line number, and the method name. In the occurrence detail you can see the formatted backtrace showing that data. For example: https://rollbar.com/Rollbar/rollbar-gem/items/5979/.

However, Rollbar can give you more detailed information about the line of code for each frame. In fact, we can display the exact text for each line of code in the backtrace. Previously, the gem wasn't sending this information, so we've added it as a feature to this new release. You can configure whether or not to send the extra data per frame, send it only for lines of code related to your project files, or send it for all the backtrace lines. The new configuration option is send_extra_frame_data and can have the values :none, :app or :all. Example:

Rollbar.configure do |config|
  config.send_extra_frame_data = :all
end

An example for the configuration above can be found in https://rollbar.com/Rollbar/rollbar-gem/items/6800/. You can see that for each line of code, its text code is now available.

3. Async with Resque

We've integrated the implementation for the Resque async handler from resque-rollbar in the gem, so you don't have to install the resque-rollbar gem anymore. Other async handlers were already available in the gem, such as DelayedJob, GirlFriday, Sidekiq, SuckerPunch, or the basic threaded one. We also wanted to have Resque in the gem, since we know it's used by many of our customers.

4. Content-Type and Content-Length

When debugging errors, it's important have as much useful information as possible, and we were missing the headers Content-Type and Content-Length. Now you'll see them in your occurrence details, so it will be easier to identify common errors with empty JSON bodies or wrong content types.

5. GET, POST, and JSON body in the right places

When pushing the request data into our API we were merging all the GET, POST, and JSON body parameters, and (if defined) Rails route parameters into the params attribute. We implemented it this way so that in a Rails controller you'd have everything merged into the params value. However, we wanted to separate them so it will be easier to find the correct data and identify what exactly is in the GET, POST, JSON body, or route params. Now, you'll find all that data is separated out in the occurrence details.

6. Bug fixes

We've pushed a few bug fixes for the gem, some of which were reported by our customers. Thank you to everyone who reported bugs! Some important bug fixes:

Fix DelayedJob serialization on some backends
Unable to send Rake reports at some scenarios
Validation reports with ActiveModel objects are fixed

I hope you take full advantage of the updates to the ruby gem. Please feel free to make pull requests, if there's anything you feel we could do better!

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you defeat annoying ruby errors in your applications. :-)

5 ways to reduce noise when logging your JavaScript exceptions

Tue, 16 Aug 2016 00:00:00 GMT

Developing and maintaining user facing software is a challenge and a very distracting one at that. :-) Often times it can be difficult trying to stay focused on what matters most. It can be hard to tell what's really broken and why, with dozens of alerts notifying you every other minute. Volatile... The client-side being one of the most volatile of them all.

When we attempt to capture errors in this environment we can very quickly get overwhelmed by lots and lots of noise. This noise is typically generated from many different places. Some examples would be old outdated browsers, browser extensions, third-party scripts, bots, spiders, etc. Rollbar's JavaScript error monitoring supports many different ways of reducing this noise so you can be more proactive in what and how you're collecting your JavaScript exceptions.

1. Ignore noisy errors on the client-side

Rollbar.js supports the ability to ignore errors on the client-side. This option is really great because you have access to the entire payload and can filter by any value in it. By doing this the error will never be sent to the Rollbar API. This is supported via the checkIgnore configuration option.

var _rollbarConfig = {
    ...
    checkIgnore: function(isUncaught, args, payload) {
        // Code here to determine whether or not to send the payload
        // to the Rollbar API
    }
    ...
};

Let's break down the function value. First the parameters passed to the function.

isUncaught: This is true if the error bubbled up to window.onerror or false if this came from one of the Rollbar.js logging methods
args: This is the args passed to the Rollbar.js logging method. If the error is uncaught and is from an unhandled rejection, the args parameter contains the Promise object.
payload: This is the payload that will be sent to the Rollbar API. You can use anything in the payload to conditionally filter these errors out.

To have Rollbar.js ignore the payload here, return true from the function, otherwise return false to continue processing and have the error sent to the Rollbar API.

2. Whitelist specific domains

You can configure Rollbar.js to only accept errors from your own domains. This option works really well when you use a lot of third party scripts and only want to see errors coming from domains that you have control over.

var _rollbarConfig = {
    ...
    hostWhiteList: ['domain1.com', 'domain2.com']
    ...
};

When hostWhiteList has been configured, we will enumerate over the stack frames in each error. At least one of the frames must contain a filename that contains at least one of the strings in this configuration option. The items in the array are compiled into a Regex that is used to compare against the filenames. This can be useful if you use a lot of third party scripts which could be generating errors that you can not do anything about and have little control over.

3. Ignore certain types of messages

Rollbar.js has support for ignoring specific messages. This is configured under the ignoredMessages key. This method of cleaning up noisy data works really well when you have a small sample of messages that you don't care about.

var _rollbarConfig = {
    ...
    ignoredMessages: ["Can't find Clippy.bmp. The end is nigh."]
    ...
};

When this configuration key is set, rollbar.js looks through the values in the setting, and if any of them match the exception message, the error will be discarded. Again these will be compiled into a regex that is compared against the exception mesasge.

4. Custom error grouping options

The first three options (ignoring errors on the client-side, whitelist accepted hosts, ignore messages) are all specific to the client-side and work to prevent the error from ever being sent to and recorded in Rollbar. Custom Grouping however is when you still want the error data sent to Rollbar and viewable in your dashboard for data metrics and reporting purposes, but you want to reduce the some of the 'noise' in the Items list. You can setup custom grouping rules to group your errors together in a way that makes sense for you. This is the best option to use when you still want to see the error occurrences in your dashboard, but want full control over how they are grouped together. Here is a really in depth blog post on how to improve your error grouping in Rollbar.

5. Ask for help :-)

Email us at support@rollbar.com. We'd be happy to help you set things up for your Projects and unique situation.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you defeat annoying client-side javascript errors in your applications. :-)

SAML-based Single Sign-On (SSO) now available

Mon, 18 Jul 2016 00:00:00 GMT

If your team uses Google Apps for Work or Okta, you can now access your Rollbar account using SAML-based single sign-on (SSO). SSO via Google Apps and Okta is available on all paid plans, and can be setup in minutes by an admin.

Once SSO is enabled, users can access your Rollbar account with just a click from the Google App Drawer or Okta My Applications screen.

Google App Drawer

![]({{ site.cdnurl }}/blog/images/2016/2016-07-18-saml-single-sign-on-is-now-available/screenshot2016-07-08at50539pm_480.153799.l.png)

Okta My Applications

If Google or Okta-based SSO is enabled on your account, then SSO options will show up on the login screen whenever you are prompted.

Ready to set up SSO for your account? Check out our step-by-step documentation.

Two-Factor Authentication now available for all users

Thu, 23 Jun 2016 00:00:00 GMT

We’re excited to introduce Two-Factor Authentication (2FA) in Rollbar as an optional extra layer of security on every user’s account. 2FA reduces your risk of having your account hacked through phishing, credential exploitation or other remote attacks.

We highly recommend enabling Two-Factor Authentication for your entire team. It’s easy to setup (and free)!

To enable 2FA, go to your user settings page and click Enable under Two-Factor Authentication. Then scan the QR code, and enter in your two-factor authentication verification token from an authentication app of your choice. One of the more popular authentication apps is Google Authenticator (available free for iOS and Android). Once you're all setup, you’ll be prompted for your token whenever you log in.

2FA is available for all users in all plans at no additional charge. To learn more about Two-Factor Authentication in Rollbar, check out our documentation.

Exception monitoring for production environments

Mon, 06 Jun 2016 00:00:00 GMT

Tools like Rollbar have changed the way development teams are recording and managing their exceptions. What used to be a very personal developer-by-developer activity can now be a team-wide tool for greater transparency, and increased application quality.

But many still treat exception monitoring as a developer activity, and they are not leveraging its benefits across all environments, from development to stage and integration, to systems testing and production. In this post and another on QA environments, I will review why exception monitoring in all environments is so beneficial, and some best practices for setting it up.

We are trying to standardize with Rollbar for exception monitoring across environments and clients. It helps our clients have visibility and thus better input into the application and development processes, and it’s a good way for us to ensure quality prior to delivering releases to customers.

But even after release, the tool has been extremely useful for the following reasons:

1. It is needed to support CD and canary releases:

More and more we are asked to consider using continuous delivery (CD) and canary release processes. While in many cases it is not possible or a good fit, when we do get an opportunity to implement CD, exception monitoring is the only way to support it, because code goes from developer to source repo and directly to prod as long as the basic tests show up green. We know very little about the code, and do not have the eyeballs on exceptions that we normally would. This way, an exception in prod is just one more trigger to let us know that a release should be rolled back. We do the same for our server monitoring, so of course we should as well with code. We also use it to help with more supporting data in A/B testing of releases.

2. We cannot always ensure environment parity:

We do our best to ensure parity between all environments, but it’s simply not possible. Different applications have different requirements in terms of infrastructure. Our development environment is fairly static, and it is local. Production can be on different configurations, and even different clouds. So we are dealing with a hybrid scenario where for each client the production environment is unique. These variables can easily cause issues where the differences in the production environment compared to dev causes issues in code—things such as a mismatch on frameworks and other artifacts. When this happens, sometimes exception monitoring is the only way to know.

3. It is a Key Performance Indicator - accountability all the way to production:

I believe that all developers have increasing responsibility for what happens in production. But in our case, because we own the development processes of our clients, there is no question our developers are accountable for code quality all the way up to production. And the analytics we receive from Rollbar help us gauge how well our Dev groups are doing with code quality, where it matters the most—the user. This allows us to quantify the impact directly and leave no question as to how what happens in Dev impacts users.

4. It helps QA and Ops communicate with developers:

If there is a bug in production, often QA, Support and Ops teams are left with little information to pass on to the Dev team. In reality, what usually happens is that it is not reported at all, not until the error is noticed in earlier environments much later on. With Rollbar we are able to show details about exceptions. Even if Support, QA and Ops do not understand the details, they are able to communicate more effectively with developers and get them pointed in the right direction faster. It helps avoid the “worked on my machine” problem, and helps relate application issues with potential infrastructure issues. Plus, it reduces the amount of time that it takes for issues to be resolved. And because we have monitoring of all environments, we can see if the exception was already showing up in earlier environments. (If so, something is broken in our process.)

5. It helps QA and Ops communicate with developers:

Using the People data feature in Rollbar, we can, via support or email, engage directly with customers, before they potentially submit a ticket, or even worse, reach out on social media about the issue they encountered. This is a tailored user experience, and something that only happens in production.

6. Web Jobs:

We do a lot of web jobs that run specific serverless functions. In Azure, we use WebJobs, Azure Functions and AWS Lambda. Because these scripts are not apart of our applications and our applications’ error reporting, it is not easy to know what is going on, or to debug them. In both AWS and Azure when a script breaks, sometimes the only indicator you get is that the job stops, or it perpetually restarts. Calls to Rollbar in our exception blocks in some cases is the only way of knowing if something breaks. When we use Rollbar in our web jobs, we make sure that the web job is set up as its own environment so we can report on each one.

We do get exceptions and errors in production, even for small applications. See below:

As long as that is happening, we need exception monitoring in production—because as we move from the development environment to production, the cost of each exception goes up. The consistency across all environments means that our knowledge does not drop off a cliff, and we have greater transparency, consistency, and quality of product.

Using Rollbar to unravel existing Laravel applications

Tue, 24 May 2016 00:00:00 GMT

Read our new blog Announcing Laravel error monitoring with Rollbar!

About 6 months ago I inherited a project (let’s call it Project Mayhem) that was grossly neglected by the original developer. I won’t go into too many details, but let’s just say that I’ve seen spaghetti with more order than this codebase had. No unit tests, no documentation, illogical architecture, and an expecting client… I felt like I was literally living in one of those nightmares where I’m late for the final exam in a class I passed a decade ago.

The icing on this terrible cake was that I also inherited the hosting and was graciously provided with absolutely no specs in order to replicate the previous production environment. I was flying blind, and if it weren’t for Rollbar's php error logging tools, this project would have crashed and burned long before I could make any meaningful changes to it. I was able to manage and prioritize exceptions as they happened, which gave me the information I needed to build out a proper hosting architecture and quickly fix existing issues in the codebase.

Suffice it to say, I am a big fan of Rollbar.

Luckily for me, Project Mayhem was built on Laravel, an increasingly popular PHP framework with an impressive array of third-party plugins; one of which just so happens to be made specifically for Rollbar. Getting Rollbar setup on Laravel was as simple as running just a few commands, but what if the project wasn’t built on Laravel? What if it was a “bespoke” PHP application?

Since this blog was written, we've released a new Laravel Notifier.

Well… Rollbar thought of that. They’ve built out a PHP client for their API that allows developers to quickly and easily integrate Rollbar with their applications. To get started, all you have to do is require "rollbar/rollbar": "~0.15.0 in your composer.json file and (like all Composer plugins) you’re good to go!

{
    "require": {
        "rollbar/rollbar": "~0.15.0"
    }
}

The interesting thing about the way the Rollbar PHP client works is that (unless otherwise specified) it will automatically catch your exceptions; you do not have to send them up manually. No muss, no fuss. All you have to do is add the following code to your application’s entry point (typically a bootstrap file or index file):

$config = array( 'access_token' => 'POST_SERVER_ITEM_ACCESS_TOKEN' Rollbar::init($config);

What this little piece of code does is install an exception handler in PHP which catches every exception thrown and sends it up to Rollbar. I should note here that the configuration can be much more in-depth, but I’m only showing the bare-bones to get you started. While nothing else needs to be done in order to start enjoying the benefits of Rollbar in your PHP application, you can still manually send up data by simply calling the report_method() function.

Rollbar::report_message(’Stuff and things.', Level::WARNING);

The beauty of this method is that it gives you the ability to send supplementary, non-critical data up to Rollbar. Warnings, internal errors, bad data, attempted security breaches. You can easily send any useful data up to Rollbar with minimal effort, and save yourself hours of headache.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you defeat annoying laravel errors in your PHP applications. :-)

Streamline monitoring with Rollbar & Datadog

Mon, 25 Apr 2016 00:00:00 GMT

Good news! We’ve recently released a new integration with Datadog to help extend your error monitoring options. Datadog is a leading cloud monitoring solution that brings metrics from all of your apps, tools & services (like Rollbar) into one place. Now Rollbar and Datadog users can syndicate exceptions, errors and code deployments as 'Events’ within Datadog (example below).

Here's a few of the ways you can improve your monitoring when you connect Rollbar and Datadog:

Get notified of exceptions, errors, code deployments in your event stream
Filter notifications by severity, environment, host, users and more
Search for exceptions in your graphs Discuss exceptions with your team
Most important - spend less time debugging issues

We hope that by bringing your Rollbar data into Datadog you can streamline your monitoring efforts and reduce some of the noise in your day to day workflow.

For more information, check out our integration docs for Datadog. Also, to see if Rollbar integrates with other tools you’re using, see our full list of integrations here.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you defeat errors in production. :-)

6 ways to improve error grouping in Rollbar

Mon, 11 Apr 2016 00:00:00 GMT

You're two weeks into using Rollbar. You've watched in amazement as issue after issue comes in without a single customer complaint to accompany them. How did you ever find errors before!?

Now that your unresolved errors have drastically decreased, you've started to notice a handful of Rollbar items that all seem to be exactly the same issue. Maybe you've been notified that your UI has exceeded the maximum call stack when calling a particular function. And in one case you found out that your database is actually missing several columns which got grouped into a single error.

What's a new Rollbar user to do? Here's 6 steps to help you improve your error grouping in Rollbar:

1. Learn how Rollbar groups items.

The Rollbar grouping algorithm attempts to be as smart as possible by grouping items by root cause. Error type items are grouped by a combination of platform, environment, error class, and stack trace information. Message type items are grouped by message text after stripping out certain number-like and date-like portions of the message.

2. Customize your error grouping.

Custom grouping enables you to decide exactly how Rollbar will group your items. You can alter the title, change the fingerprinting (and therefore the grouping), and use any data you send to us to make it work!

3. Make sure you're setting the server root.

Before grouping by filename, Rollbar will strip off whatever you're sending in the server root segment. This means that you can host the code from varying locations, but still get the same grouping, no matter where on the system you host your code. Each notifier allows setting this in its own way. Check out the docs for your notifier to learn how!

4. Consider customizing your stack trace before sending.

The rollbar.js, rollbar-gem, and several of our community notifiers have the ability to alter the error payload being sent to Rollbar just before sending them. Almost all of them can send custom data. If you have a specific case where you know how to make the payload get grouped correctly (by stripping out repeated stack frames in a recursive function, or normalizing a dynamically named function) you can alter the stack trace before it's sent to get it to work with our default grouping algorithm. If the alteration is just too complex, consider sending a custom data field and using custom grouping to group on that when present. There are tons of ways to get this exactly right. Don't settle for anything less.

5. Send a fingerprint.

Sometimes you know exactly how an item should be grouped in Rollbar, and there's no simple way to express that in a custom grouping rule. In this case all our official notifiers, and most community ones, have a way to send a fingerprint that is the field we group by to determine what item an occurrence gets grouped with. The field can be up to 40 characters long. If you go over, that's fine! Our system will hash it to ensure it fits. This will almost never result in a collision, so feel free to have a fingerprint like "the-error-that-occurs-when-we-send-a-bad-id-to-stripe-for-customer:BIGNAME". We'll take care of making it fit!

6. Ask for help!

Email us at support@rollbar.com. We'd be happy to help you set things up for your unique situation.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you defeat annoying errors in your applications. :-)

Meet the Rollbar team IRL at these upcoming events

Fri, 25 Mar 2016 00:00:00 GMT

Rollbar is hitting the road. Catch one (or more) or our team members at the following events in the next few months. We welcome the opportunity to talk to you about your projects, answer your questions and share some Rollbar swag. If you see us at an event, please stop by and say "Hi.”

Lone Star PHP

Where: Dallas, Texas / When: Fri, April 8 to Sat, April 9

The Lone Star PHP Conference brings together great PHP speakers and topics into a fantastic two-day event that is 'clap, clap, clap' deep in the heart of Texas. Catch me, Mike Smith (and local Texan) handing out Rollbar swag and making friends. ;)

EmberConf

Where: Portland, Oregon / When: Tues, March 29 to Wed, March 30

EmberConf is the flagship event of the Ember community. It’s two days of sessions, talks, and good times with the Ember Core Team and community members from around the world. Our CEO and Co-founder Brian Rue will be there, so say hi if you happen to see anyone in a Rollbar shirt.

Ruby On Ales

Where: Bend, Oregon / When: Thurs, March 31 to Fri, April 1

We’re ending our tour of Oregon at this years Ruby on Ales event. Ruby on Ales is a two-day single track Ruby conference, with a super cool name. It is two full days of presentations, networking and good ole fashion community fellowship.

RailsConf

Where: Kansas City, Missouri / When: Wed, May 4 to Fri, May 6

RailsConf is the largest gathering of Rails developers in the world, not to mention the largest gathering of Rubyists. We'll be sponsoring and exhibiting, so stop by and say hi, learn about updates to our ruby gem and grab some of our new Rollbar swag.

PyCon

Where: Portland, Oregon / When: Mon, May 30 to Wed, June 1

PyCon is the largest annual gathering for the community using and developing the open-source Python programming language. Organized by the awesome folks in the Python community. We are sponsoring and exhibiting for the first time at PyCon. Please stop by our booth and say hi, talk shop and grab some new Rollbar swag.

Want a free ticket to PyCon?

We have one free pass to PyCon we are giving away. Click here to tweet us and we’ll randomly select the winner on Mon, April 4th.

Schedule an in-person meeting

We’re making [office hours](team@rollbar.com?subject=I'd like to meet at [add event you are attending here]") available during these events. Schedule some time with us to sit down with a Rollbar expert to show you how you can level-up your app monitoring and error handling processes. Visit us at our booth or email [team@rollbar.com](team@rollbar.com?subject=I'd like to meet at [add event you are attending here]) to get something scheduled.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you defeat annoying errors in your applications. :-)

Query Rollbar data directly from our API

Mon, 15 Feb 2016 00:00:00 GMT

Good news for RQL users. Until recently RQL usage was limited to the Rollbar interface. That's no longer a problem. Now you can access RQL directly from our API.

We like to believe that our interface provides just about everything you need to discover, investigate and resolve errors in your software. That being said, we know that everyone's needs are not the same. So, we created the Rollbar Query Language (RQL for short) to give Rollbar users more controls over their data. Many of our users have grown to depend on RQL for getting their day to day work done.

RQL is a familiar SQL-like language for querying your Rollbar data. Here's a few ways that RQL can be useful:

Want to find out the number of 500s your users experienced broken down by the url they were on at the moment?
Search items using more complex filters than is possible on the Rollbar website.
Search for items that occurred between two specific deploys

Extend the power of RQL and make the perfect dashboard or tool with your queried error data directly from the API. For help, checkout our API documentation for RQL.

For an example that runs on the data from our Live Demo check out the API-Examples repository on GitHub.

New to Rollbar? Signup for a 14-day free trial and get instant insights into your application errors.

Rollbar add-on now available for Bitbucket

Thu, 11 Jun 2015 00:00:00 GMT

On the heels of Atlassian's announcement of Atlassian Connect for Bitbucket yesterday, we're excited to introduce our very own Rollbar Add-On for the Bitbucket community.

With the new Rollbar Add-On for Bitbucket and our recent Bitbucket Issue Tracker integration, you can now monitor and keep tabs on your errors and exceptions within Bitbucket. No more switching between multiple tools and services to debug and deploy code. Atlassian Connect for Bitbucket provides an integration architecture that embeds add-ons right within the their UI creating a seamless user experience and unified workflow.

How to install the Rollbar Bitbucket add-on?

If you're currently a user of Bitbucket, click on your avatar, select "Manage Account", and simply install the Rollbar add-on by selecting "Find new add-ons" from the left menu.

Once the add-on is installed, Rollbar data will be accessible per your Bitbucket repo dashboard(s). Note: Atlassian Connect for Bitbucket uses fine-grained permissions to grant add-on access to repositories, issues, accounts, teams, snippets and pull requests.

We are thrilled to be a part of Atlassian Connect for Bitbucket and for the opportunity to build and maintain a valuable add-on for the Bitbucket community and Rollbar users to find and fix errors even faster!

What's next? We're currently working toward full support for Bitbucket which includes Issues, Source Control and Authentication. Stay tuned.

Leave a comment or send a note to support[at]rollbar.com and let us know if you have any feedback or questions. Happy to help.

Easier way to get your team on Rollbar

Tue, 19 May 2015 00:00:00 GMT

Joining and getting your team on Rollbar is now easier.

Rollbar is better together and our latest account setting makes it easier to get everyone together. I'm happy to introduce Email Domain Whitelists.

Previously the only way to join your team on Rollbar was being invited via email. Admins had to send email invites one by one. This can be tough for large organizations and fast growing engineering teams.

With Email Domain Whitelists, now you can set an email domain whitelist, so that anyone with an address at a specified domain can join your account (once confirmed). No more one off invitations. Making the workflow for admins and new team members easier and less confusing.

PROBLEM:

As we have grown we've seen more and more new users who are employees at companies that are already using Rollbar get lost trying to join their company’s account. The scenario looks something like this:

You see a notification in your company's Slack, HipChat, or an issue in JIRA, GitHub etc.
Click link to view more details
Hit the Rollbar login page (confusion ensues)
Decide to signup and create a new separate account that is orphaned from your company’s account
Ask Admin to add/invite you to their Team on Rollbar
The new account you created gets abandoned :(

SOLUTION:

With the new email domain whitelist, anyone who signs up (from specified domains) automatically gets added to your account once they confirm their email address. Go to /settings/accounts/Name/ to setup (need to be on the Owners team to enable).

Few items of note. Invites are on and email whitelists are off by default. Currently it only works on pages that would normally require login (i.e. link to project, error item, deploy). Here's how it works:

Next time someone clicks a link in Slack, HipChat, JIRA, GitHub they will prompted to login or join a team ![]({{ site.cdnurl }}/blog/images/2015/2015-05-19-easier-way-to-get-teams-on-rollbar/rollbar-teams-setting-page.146385.l.png) ![]({{ site.cdnurl }}/blog/images/2015/2015-05-19-easier-way-to-get-teams-on-rollbar/rollbar-teams-signup.146491.l.png)
Confirmation ![]({{ site.cdn_url }}/blog/images/2015/2015-05-19-easier-way-to-get-teams-on-rollbar/confirmation-teams-rollbar.146387.l.png)

Interested? Have your account admins enable the setting, send your team links to your projects and get everyone onboard.

This is our first solution to solving this problem. We're evaluating how we can extend this to direct signups (that match a whitelist) and sharable links while maintaining the highest privacy and security standards.

PS - It doesn't cost you more to add additional team members. So if you have 1 trillion members on your team, it's all good. :)

Edit and rename error item titles

Thu, 26 Mar 2015 00:00:00 GMT

You can now rename/edit your error titles. Fix ugly long titles. Hover over, click, edit, and save.

Enjoy!

Connect Rollbar to Bitbucket Issue Tracker

Tue, 17 Mar 2015 00:00:00 GMT

New integration now available - Bitbucket Issue Tracker

Supercharge your issue and error tracking workflow when you connect your Rollbar and Bitbucket accounts. New Items in Rollbar will instantly create Issues in your Bitbucket repo, or you can create and link Issues with the click of a button within Rollbar.

Here's how:

Go to your project's Settings, then Notifications, and select Bitbucket Issues from the list of channels.

Click 'Connect with Bitbucket” to grant Rollbar access to your account.

From here, you can choose which repository, and add/edit/remove rules for Issues to be created automatically.

Like magic, your Rollbar error items and details now show up in your Bitbucket repo. Success!

Create Bitbucket Issues manually

Prefer to create Issues by hand? You can create an Issue directly from the error Item page in Rollbar, or link with an Issue that already exists. You can use this alongside the automatic rules; or, remove the rules for full manual control.

What's next?

We're working toward full support for Bitbucket, like we have for GitHub - Issues, Source Control and Authentication. I know Rollbar users who rely on Bitbucket in their workflows are rejoicing. :)

Let us know if you have any feedback or questions. We're here to help.

Deploy and enjoy!

Daily, Hourly, New Errors and Trend graphs are now clickable

Tue, 10 Mar 2015 00:00:00 GMT

Yes, that's correct.

Daily, Hourly, New Errors, and Trend graphs are now clickable. You can find and fix errors even faster, and in less clicks. :D

Common usability feedback we get from our users:

Sure would be nice if I could click the dashboard bar graphs and sparklines to quickly see what caused a spike in error events etc.

Couldn't agree more. We love aggregating data and we love it clickable. So we enabled it!

The following are now clickable in the project Dashboard:

Hourly Error/Critical Occurrences
Daily Error/Critical Occurrences
Daily New/Reactivated Items
Trends (24 hour and 7 day)

Trends are also clickable on the Items page. For reference Trends are these guys ![]({{ site.cdn_url }}/blog/images/2015/2015-03-10-error-tracking-data-now-clickable/trend.144545.o.png 'error tracking trends') also called 'sparklines'.

When viewing a specific error item, the Last 60 Minutes, Hours, and Days are now clickable and aggregate error data by your selection.

We're excited to get this features out the door. It reduces a lot of friction in navigating Rollbar. One of many UI and UX improvements to come. :)

Deploy and enjoy!

Using Logstash and Rollbar Together

Mon, 02 Mar 2015 00:00:00 GMT

The infrastructure behind most modern web applications includes an assortment of tools for collecting server and application metrics, logging events, aggregating logs, and providing alerts. Most systems are made up of a collection of best-in-class tools and services, selected and deployed over time as team members arrive and depart, needs change, the system grows, and new tools are introduced. One of the challenges web development and operations teams face is collecting and analyzing data from these disparate sources and systems and then piecing together what’s happening by looking at multiple reports and dashboards.

Two common pieces in this puzzle are Logstash and Rollbar.

Logstash (and the Kibana web interface, both of which are heavily supported by and integrated with Elasticsearch) lets you collect and parse logs, store them in a central location, search and explore the data via the Kibana UI, and output events to other services. Logstash provides a powerful tool for taking logs in many different formats, converting them into JSON events, then routing and storing those events.

Rollbar collects errors from your application, notifies you of those errors, and analyzes them so you can more efficiently debug and fix them. With a few lines of code or config changes to your application, you can make errors, complete stack traces, trends and affected user reports accessible via your Roller dashboard. Like Logstash, Rollbar collects and analyzes events represented in JSON.

By connecting Logstash and Rollbar, you can not only centralize and analyze your system and application logs, but also improve error tracking and simplify debugging by providing context to developers looking at errors generated by their code.

Most Logstash users have tried to configure Logstash to parse multi-line exception messages, or have tried to convince a development team to adopt standards for application debugging and error message. For code your team controls, it’s likely much simpler to install the Rollbar notifier for the language and framework you’re using. You can send errors from your Ruby, Python, or PHP application or browser JavaScript to Rollbar and the service will parse stack traces automatically and update your dashboard in real time. You can see the values of local variables when the error occurred, and these errors are associated with other errors of the same type.

For Rollbar users, Logstash allows you to collect errors from external applications and ship them to Rollbar, where they'll appear on the same dashboard as your application errors. Database and web server errors, for example, can be passed along to Rollbar to help developers determine whether the error is due to a bug, database performance issue, or operational issue with the web server.

To get started...

If you're Logstash user who'd like to get started with Rollbar, you can sign up for a free account.
If you're a Rollbar user who'd like to try Logstash, see "Getting Started with Logstash"
For instructions on configuring Logstash events to Rollbar, see "How to Integrate Logstash with Rollbar".

Increasing max-open files for beanstalkd

Sat, 28 Feb 2015 00:00:00 GMT

Quick tip: If you are running out of file descriptors in your Beanstalkd process, use /etc/default/beanstalkd to set the ulimit before the init script starts the process.

e.g.

# file: /etc/default/beanstalkd
BEANSTALKD_LISTEN_ADDR=127.0.0.1
BEANSTALKD_LISTEN_PORT=11300
START=yes
BEANSTALKD_EXTRA="-b /var/lib/beanstalkd -f 1"

# Should match your /etc/security/limits.conf settings
ulimit -n 100000

Lots of resources online tell you to update your /etc/security/limits.conf and /etc/pam.d/common-session* settings to increase your maximum number of available file descriptors. However, the default beanstalkd installation on Ubuntu 12.04+ uses an init script that starts the daemon process using start-stop-daemon which does not use your system settings when setting the processes ulimits. Just add this line to your defaults and you're good to go!

Get notifications every time an error occurs

Thu, 26 Feb 2015 00:00:00 GMT

You can now setup notifications every time an error occurs. Previously specific error Notifications were only available for New Items and 10^th Occurrences. Notification Rules are available for all Channels (Email, Slack, HipChat, Trello, PagerDuty).

Assign errors to your team

Thu, 26 Feb 2015 00:00:00 GMT

Ever wanted to assign error items to other team members in Rollbar? Of course you have. Now you can. It is a pretty straight forward enhancement, but here is an overview.

On the error ‘items’ details page, there's an “Assigned to" dropdown with the members of your team. Once assigned, we’ll shoot an email to that team member letting them know you assigned that specific item to them, including link and details. They'll be automatically added as a 'watcher' for that specific item and will receive notifications about any comments and updates.

Assignment events will be listed in the item history section, so you can see who assigned it to whom, when.

To quickly find items assigned to yourself or others on your team, search 'assigned:me', ‘assigned:username’, or 'assigned:unassigned' on the Items page.

We're excited to get this out into the wild. Especially for some of the larger teams using Rollbar. Let us know what you think and how we can make it better for you and your team.

Debugging Node.js Apps in Production with PyCharm

Fri, 19 Dec 2014 00:00:00 GMT

Node.js has a built-in debugger that you can start in running processes. To do this, send a SIGUSR1 signal to the running process and connect a debugger. The one, big caveat here is that the debugger only listens on the local interface, 127.0.0.1.

The following are instructions for debugging Node.js applications running in your company's private network from your laptop, through a bastion host.

SSH into the production host that is running the Node.js app
- Put your production app into debug mode. prod-host $> kill -s SIGUSR1
- As root, start an SSH tunnel to connect your private network with localhost. prod-host $> ssh -N -q -L :8585:localhost:5858
On your laptop
- Start an SSH tunnel to the production host, through your bastion host. laptop $> ssh -N -q -L 5858::8585 @
Open PyCharm and create a remote debugging configuration.
- Run → Edit Configurations
- Click the + button on the top-left of the window and select “Node.js Remote Debug”
- Set the host to 127.0.0.1 using port 5858, name it and save.

Run the new Debug configuration.
- Run → Debug...
- Select the new configuration.

At this point your laptop will have connected to your local SSH tunnel which will be connected to your production host's private network interface which will be tunneled to your production host's local network interface and your Node.js process.

PyCharm → local SSH tunnel → bastion host → production host private network → production host localhost → Node.js

Set some breakpoints in PyCharm and watch as your production process begins waits for you to step through your app.

Note: If you'd rather use the command line instead of PyCharm just run the node debugger from your laptop:

laptop $> node debug localhost:5858

Happy debugging!

Troubleshooting

Sometimes PyCharm will just not connect to the running process on your production machine. Try restarting each of the SSH tunnels.

Restart the SSH tunnel on the production machine.
Restart the SSH tunnel on your laptop.
Restart the PyCharm debugger.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you defeat annoying errors in your applications. :-)

RQL String Functions

Tue, 16 Dec 2014 00:00:00 GMT

RQL now includes a basic library of string functions. You can use these to slice and group your data in arbitrary ways. For example, "email domains with the most events in the past hour":

SELECT substring(person.email, locate('@', person.email)), count(*)
FROM item_occurrence
WHERE timestamp >= unix_timestamp() - 3600 AND person.email IS NOT NULL
GROUP BY 1
ORDER BY 2 DESC

The new functions: concat, concatws, lower, upper, left, right, substring, locate, length, charlength. The functions are implemented to be compatible with MySQL; see the RQL docs for details.

Processing Delay Postmortem

Fri, 05 Dec 2014 00:00:00 GMT

Yesterday from 2:20am PST until 10:22am PST, we experienced a service degredation that caused our customers to see processing delays reaching nearly 7 hours. While no data was lost, alerts were not being sent and new data was not appearing in the rollbar.com interface during this time.

We know that you rely on Rollbar to monitor your applications and alert when things go wrong, and we're very sorry that we let you down during this outage. We'd like to share some more details about what happened and what we're doing to prevent this kind of issue from happening again.

Overview of API Tier and Processing Pipeline

When data is received by our API endpoints (api.rollbar.com), it hits a load balancer which proxies to an array of "API servers". Those servers do some basic validation (access control, rate limiting, etc.) and then write the data to local disk. Next, a separate process (the "offline loader") loads these files asynchronously into our database clusters. Then, a series of workers process the raw data into the aggregated and indexed form you see on rollbar.com, and send alerts for any triggered events. This system is designed for reliability first and performance second.

When occurrence processing latency exceeds 30 seconds, we show an in-app notification that processing is behind. This is calculated as follows:

For each API server, calcuate ([timestamp of last occurrence received] - [for the last occurrence that was fully processed by the pipeline, the timestamp it was received on the API server])
Report the maximum delay of API servers as the processing delay

The API tier primarily receives three kinds of data: occurrences (the "item" endpoints), deploys, and symbol mapping files (source maps, Android Proguard files, and iOS dSYMs). Currently, all three of these are loaded by the same offline loader process, to different database clusters depending on the type of data.

Outage Timeline, Cause, and Resolution

At about 2:00am PST, a node in the database cluster that stores the symbol mapping files ran out of disk space. Unfortunately, this did not set off any alerts in our monitoring system because the disk space alert had been previously triggered and acknowledged, but not yet resolved.

At about 2:20am PST, the next symbol mapping file arrived on one of the API servers and since the database server was out of disk, it could not be loaded. This caused other files on that API server--containing occurrences and deploys--to not be loaded either. At this time, a processing delay first appeared in the Rollbar interface, and some (but not all) data was delayed. Over the next several hours, the delay continued to rise (as data on some API servers was not processed at all) and the percent of data that was delayed also rose (as more API servers enocuntered the same problem).

At 8:25am PST, a Rollbar engineer started work for the day and noticed a support ticket about the processing delay. He immediately escalated to a second engineer who began investigating. At 8:40am PST, a third engineer joined and updated status.rollbar.com to say that we are investigating the issue.

At 9:05am PST, we identified the immediate problem that the symbol mapping files were blocking occurrences from being loaded. We began mitigating by moving those files aside to allow the higher-priority occurrence data to load. This began the recovery process, but created a backlog at first level in the processing pipeline, causing all data to be delayed (instead of just some).

At 9:11am PST, we identified disk space as the root cause, and resolved this a few minutes later. At 9:35am PST, we updated status.rollbar.com to state that we had identified the issue.

At 9:55am PST, processing latency hit a peak of about 25,000 seconds. We updated status.rollbar.com with our estimate of 36 minutes to full recovery.

At 10:43am PST, processing was fully caught up. status.rollbar.com was updated a minute later.

Evaluating our Response

Once our team became aware of the issue, we were able to identify and fix it relatively quickly (40 minutes from awareness to identification, with fix immediately afterwards). Recovery was relatively fast as well, given the length of the backlog (1hr 38minutes to recover from 7hrs 45min of backlog).

It took far too long to for us to notice this issue, however, as our automated monitoring failed to alert us and we only discovered the issue via customer reports.

Improvements

We've identified and planned a number of improvements to our processes, tools, and systems to address what went wrong. Here are the highlights:

We're auditing our internal monitoring to ensure that checks are meaningful, needed checks are present, and alerts are functioning correctly
We've created a standing Google Hangout so we can more easily coordinate our response when we're not all in the same location
We're investigating whether we can automate the component status on status.rollbar.com, so that it doesn't need to be manually updated. (Side note: go there now and click "Subscribe to Updates"!)
We've identified and planned system-level improvements to decouple symbol mapping uploads from the critical occurrence processing pipeline
We've planned a project to improve our ability to recover more quickly from processing backlogs

In Conclusion

We're very sorry for the degradation of service yesterday. We know that you rely on Rollbar for critical areas of your operations and we hate to let you down. If you have any questions, please don't hesitate to contact us at support@rollbar.com.

October Release Roundup

Fri, 31 Oct 2014 00:00:00 GMT

Happy Halloween, everyone! Here's a roundup of what's new in Rollbar this month.

Ruby Upgrades

The rollbar gem for Ruby got a lot of attention in October. Early in the month, we released version 1.1.0, which added support for Ruby 2.1 exception causes, and a new 'failover_handlers' feature for more reliable asnyc reporting. Mid-month, we released version 1.2 which adds a new, much nicer and more powerful interface for sending the data you want into Rollbar.

In 1.2, you can do:

begin
  Rollbar.info("About to do_something")
  do_something
rescue => e
  # send a message and extra data along with an exception
  Rollbar.error("Something went wrong", e, :foo => "bar")

  # customize payload attributes, like the 'person' or 'fingerprint'
  Rollbar.scope({:fingerprint => "something"}).error(e)
end

More in the docs. It's available now on Rubygems (latest version is 1.2.7).

New Status Site

We've upgraded status.rollbar.com. We'll be using it to communicate about outages, so if you'd like to be notified, go there and subscribe to updates. The new status site also shows the current maximum latencies for the processing pipeline.

Link Rollbar Items with Existing 3rd-party Issues

You can now link a Rollbar item with an existing issue in your issue tracker:

Or if you have a Rollbar item that is already linked, you can now change or remove the link. This works with Asana, GitHub Issues, JIRA, Pivotal Tracker, Sprintly, and Trello.

Geolocation for IP Addresses

Rollbar now shows geolocation information on the IP address detail page:

You can get there by clicking on an IP address anywhere in the app. More features on this theme to come; stay tuned.

Basic stats on the Item page

The basic item statistics--when it was first and last seen, how many times it has occurred since being resolved, and how many total IPs have been affected--are finally available at a glance on the Item detail page.

Time Format setting

If you'd rather see timestamps use a 24-hour clock, head over to your project's Settings page:

Misc

Improved processing pipeline speed by ~33%
Added API methods to set rate limits
Improved support for code versions in rollbar-android 0.1.0
Fixed a bug with wrapped functions in rollbar.js 1.1.11
Added links to detailed usage reports from the Invoice and Usage pages
Fixed an issue where rate limits would over-limit when duplicate items were received
Deploy emails now include the deploy comment
RQL aggregate functions now support arbitrary expressions as parameters
Improved filename linking for vanilla ruby stack traces
Added the High Occurrence Rate trigger for HipChat notifications
Long param values are now displayed better on the occurrence detail page
The "viewing now" list no longer shows yourself in the list
Fixed a few edge cases with the Slack integration
Fixed a bug where the trace_chain section in the Raw JSON part of the occurrence detail page would sometimes be reversed
Fixed an issue where gravatar images would block the page load

RQL minor updates

Thu, 25 Sep 2014 00:00:00 GMT

A couple minor updates to RQL today:

IS NULL and IS NOT NULL are now supported
Fixed a crash in queries that contain a GROUP BY plus an ORDER BY on a column referenced only in the ORDER BY.

New "Reports" API calls

Wed, 20 Aug 2014 00:00:00 GMT

We've released two new API calls, exposing some of the data on the Dashboard via our JSON API.

Use the /reports/topactiveitems to fetch the same data as "Top 10 Active Items in last 24 hours". And use /reports/occurrence_counts to fetch the same data as "Daily Error/Critical Occurrences" and "Hourly Error/Critical Occurrences".

More details in the docs. If you give this a try, send us any feedback at team@rollbar.com.

Occurrence counts by minute

Thu, 31 Jul 2014 00:00:00 GMT

We've released an improvement to our Item Detail pages, adding a graph showing the aggregate occurrence counts per minute. It's live now for everyone and looks like this:

You can use this to see patterns that previously were hard to spot, like errors that occur on a regular, sub-hour interval (like the one shown above). It's also useful for quickly seeing how the occurrence rate changes after a deploy.

node_rollbar 0.3.11

Thu, 24 Jul 2014 00:00:00 GMT

We've released a new version of our Node.js library, version 0.3.11. It's available on npm and GitHub.

This release adds a new function to the public API, handleErrorWithPayloadData. The name's a mouthful, but it allows you to use (nearly) the full power of the Rollbar API when reporting errors. For example, to report an error as a "warning" and pass some additional data:

rollbar.handleErrorWithPayloadData(err, {level: "warning", custom: {someKey: "some value"}});

See more examples in the docs for caught exceptions.

More stack trace filename search filters

Tue, 22 Jul 2014 00:00:00 GMT

We've released several new search filters to make it easier to find errors by the filenames in their stack trace. If you've ever wanted to find client-side Angular JavaScript errors that only occurred within your own specific domain, or exceptions originating in a specific part of your code, keep reading.

You've always been able to search for errors where any filename contains a string, but now you can also search for errors where:

all filenames contain a string (allfiles:myproject)
no filenames contain a string (nofiles:evilproject)
the top stack frame's filename contains a string (topfile:mydomain.com)
the bottom stack frame's filename contains a string (bottomfile:some_important_library)
the number of filenames is within a range (minfiles:1 and/or maxfiles:10)
any filename contains a string (file:.rb)

These filters can all be combined in a single query. If you forget which filters exist, mouse over the ? next to the search box to see the help text.

Missing daily summary emails

Mon, 21 Jul 2014 00:00:00 GMT

We just rolled out a fix for missing daily summary emails. The bug was introduced last week when we refactored a bunch of our email code. As a result, some projects did not receive their daily email for the previous 24 hours.

We don't want to spam users with old summaries so please contact support@rollbar.com if you'd like to get yours.

rollbar-gem v0.13.2

Tue, 08 Jul 2014 00:00:00 GMT

We've released a patch version (0.13.2) of our Ruby gem, fixing two issues.

The usesuckerpunch config option could cause timeouts when using Unicorn (#121) - thanks to Fabio Napoleoni for the report
Sidekiq payloads were being mutated, causing the sidekiq-failures UI not to display parameters correctly (#122) - thanks to @Ditchou and Dmitry Krasnoukhov for investigating and reporting this.

It's available now on Rubygems and GitHub.

Prettier Node.js stack traces

Tue, 01 Jul 2014 00:00:00 GMT

Small tweak of the day: we now render Node.js stack traces to look more like they do via console.log(err.stack). We also fixed an issue where the context lines would appear indented at random levels.

Before:

After:

Debugging by IP Address

Mon, 30 Jun 2014 00:00:00 GMT

We've released a first round of features around IP Addresses. Similar to how you can see the history by Person, you can now see the history by IP address. If you're tracking down an issue affecting logged-out users, this can be really helpful.

To get there, click on an IP address anywhere in the Rollbar interface:

You'll be able to see all events affecting that IP:

There's also a handy link to the WHOIS record for the IP.

Better support for Browserify and Webpack

Wed, 18 Jun 2014 00:00:00 GMT

Recently (actually last week), we released a new version of rollbar.js that has a few fixes to play nicely with Browserify and Webpack. It's being served from our CDN and the source is on GitHub.

We've also put together examples showing how to use rollbar.js with each. If you use Browserify or Webpack and want to include rollbar.js that way instead of using our standard snippet, this is your lucky day:

Thanks to @altano for helping with this.

Codeship + Rollbar

Tue, 17 Jun 2014 00:00:00 GMT

Our friends over at Codeship wrote a piece about how they use Rollbar to track deployments and fix exceptions. More on the Codeship blog: Tracking Deployments with Rollbar.

rollbar.js snippet update

Tue, 10 Jun 2014 00:00:00 GMT

We've released a new version of rollbar.js (1.0.0-rc9), which fixes a bug in some environments.

If you've seen "TypeError: Uncaught TypeError: Cannot read property '_wrapped' of null" in your project, you should update to the latest snippet. Grab it from the docs to update.

More Powerful Custom Grouping

Mon, 09 Jun 2014 00:00:00 GMT

{% assign defaultfingerprint = site.data.docs.custom-grouping.defaultfingerprint %} {% assign defaulttitle = site.data.docs.custom-grouping.defaulttitle %} {% assign body = site.data.docs.custom-grouping.body %} {% assign client = site.data.docs.custom-grouping.client %}

Our Custom Grouping feature just got a lot more powerful. It's now possible to:

group TimeoutErrors by the controller+action it appears in
group 404s by path
group all Android exceptions by the app version number

and much more, just by writing a simple JSON rule. This feature is live now for all accounts.

For the uninitiated: Custom Grouping allows you to tune Rollbar's grouping algorithm for the specifics of your application. If our default algorithm isn't grouping incoming occurrences like you want, you can define rules to customize. Rules consist of a condition, a title, and a fingerprint. If an incoming occurrence matches the condition, then it will be grouped with others that have the same fingerprint, and given the title title. (If it doesn't match, then the next rule is evaluated, or if there are no more rules, the default algorithm is applied.)

Previously, the title and fingerprint could only contain static text. Now, they're templates that can reference:

any part of the incoming occurrence
the title and fingerprint generated by our default algorithm

For example, to group all ActionController::RoutingErrors by the request path (which comes in as the exception message):

[
  {
    "title": "{{ body.trace.exception.class }} {{ body.trace.exception.message }}",
    "fingerprint": "{{ body.trace.exception.class }}-{{ body.trace.exception.message }}",
    "condition": {
      "path": "body.trace.exception.class",
      "eq": "ActionController::RoutingError"
    }
  }  
]

Or to have all Android exceptions grouped by app version:

[
  {
    "title": "{{ default_title }} in version {{ client.android.code_version }}",
    "fingerprint": "{{ default_fingerprint }}-androidversion-{{ client.android.code_version }}",
    "condition": {
      "path": "framework",
      "eq": "android"
    }
  }
]

See the docs for more details.

We've already seen this solve a diverse set of grouping problems and are really excited about how powerful this is. If you have any questions about how to get your errors grouped the way you want, drop us a line at support@rollbar.com and we'll be happy to help.

New to Rollbar? Sign up for a free account and start monitoring errors in 5 minutes.

Post-mortem for website assets outage

Fri, 06 Jun 2014 00:00:00 GMT

We had an issue from late last night through this morning where many users were not able to use the rollbar.com website because CSS and Javascript assets were not loading in some regions. This post will cover what happened, its cause, why we didn't notice it sooner, and the changes we're making going forward.

Problem and Cause

From 5:04am UTC until 4:21pm UTC today, users in/near Europe got 404 responses when trying to load our static CSS and JS assets for rollbar.com. This resulted in a visually broken or unusable site, depending on browser and CDN cache.

The cause was a misconfigured load balancer in our Amsterdam data center that was put into service at 12:05am UTC. The assets should have been present on that machine, but they weren't. For several hours, this didn't appear, since the assets were still cached in our CDN. When they fell out of the CDN's cache, requests for those assets from Europe-based CDN endpoints began to 404. Users hitting those endpoints would then get 404s as well.

Detection and Fix

We first noticed this via the last line of defense: customer support channels. IRC, Twitter, and our support inbox were full of reports (thanks everyone!). Unfortunately, since this happened during the middle of the night for our team (we're based in San Francisco), we didn't notice until 3:33pm UTC when I checked my phone on the way to the office. I immediately escalated to the rest of the team. At 3:36pm UTC, Cory started investigating.

The website appeared fine for us, so we suspected it may be some sort of CDN-related issue. Based on the time of the reports, we suspected it might be localized to Europe (later confirmed via Twitter). At 4:05pm UTC, we identified the load balancer addition as the most likely cause.

At 4:06pm UTC, we took the load balancer back out of service. The assets were still cached by the CDN as being 404s, though, so this wasn't enough to fix. At 4:16pm UTC, we deployed a change to the web tier to bust the cache; at 4:21pm UTC, the deploy was complete and the issue fixed. We confirmed the fix with several of the users who had reported the issue.

You can see the start, rise, and fix of this issue on this Rollbar Item that happens when jQuery doesn't load. This is normal to happen intermittently (all networks occasionally fail), but not this much:

Planned Improvements

While we'd of course prefer not to make configuration mistakes, the primary failure here was that our none of our automated monitoring detected this. That meant instead of being paged as soon as it happened, we didn't know anything was wrong for over 10 hours. "Whatever can go wrong that isn't monitored, is probably already broken."

Today, we'll be making the following changes:

adding a Nagios alert to monitor 404s on static assets on our load balancers (would have alerted us as soon as the issue started)
add a header to all requests to identify which load balancer served the request (would have made this easier to identify)

We're also considering:

a Nagios alert if there is a sudden large number of support tickets
a Nagios or SMS alert if there is a Rollbar Item affecting a large number of unique users (Rollbar actually already supports sending notifications like this, so it's just a matter of hooking it up for ourselves)

Security patch for the recent CCS Injection Vulnerability

Thu, 05 Jun 2014 00:00:00 GMT

For the security conscious folks out there - We just finished patching our load balancers with the latest security updates.

That is all. :)

More info can be found here.

Local variable values in stack traces

Thu, 05 Jun 2014 00:00:00 GMT

"This stack trace would be so much easier to debug if I knew what the value of that variable was" - said us, many many times. We finally scratched our own itch and built this into Rollbar.

It's really awesome. Take for example a bug I ran into with our new deploy emails feature:

"Hmmm, sure would be nice if I knew which variable was None--is it prev_deploy or deploy?" In the past, I would've had to reproduce locally, or add an additional rollbar.report_message() to log each variable in production. But now I can just press "locals" for the stack frame and see it instantly:

"OK, it's prev_deploy." That was all I needed: prev_deploy will be None when deploy is the very first deploy in this project. Easy two-line fix. Time taken from opening the Rollbar notification to committing the fix: 2 minutes. Rollbarred!

How it works

We've implemented this in Python (pyrollbar), and are actively exploring other languages. Ruby and PHP are looking promising. Here's how it works in Python.

When this feature is enabled and an exception is reported, we use the inspect module to collect:

all stack frames: names and values of all variables that are function arguments
in-project stack frames: names and values of all local variables

There are two main edge cases to deal with:

The variable could contain sensitive data. To deal with this, we use the same scrub_fields configuration used for scrubbing request data. If the variable name matches one of the field names to be scrubbed, the variable's value will be scrubbed. (Like access_token in the screenshot above.)
The variable could contain a circular reference, be extremely large, or otherwise hard to serialize. To deal with this, we use the repr module to safely convert the value to a string of reasonable length. We've been using this internally now in production for the past two weeks without issue.

How to Enable

If you're using Python, simply update to pyrollbar 0.8.0 or greater. Local variables are on by default; see the docs for the full config reference.

If you're using Erlang, you're in luck - erollbar already sends argument values. Check its docs for details.

Want this in a different language? Let us know -- email support@rollbar.com or open an issue on the GitHub repo for the appropriate Rollbar library.

Don't use Rollbar yet? Sign up for a free account, follow the setup instructions, and you'll be debugging production in power mode in no time.

Deploy emails show which commits were deployed

Wed, 04 Jun 2014 00:00:00 GMT

We recently added a feature I've wanted for a long time: our Deploy email notifications now show which commits were in that deploy. It looks like this:

This works by querying GitHub's API to find the differences between the commit used for the previous deploy and the commit you just deployed. (If this looks familiar, it's because we show the same information on the Deploys page). As long as one user in your account has GitHub enabled (and the token enabled for your account), all users will be able to see the commit list.

Want this? Sign up for a free account, connect with GitHub, and set up deploy tracking. Whenever you deploy, you and your team will get an email like the one above.

Daily summary email bugfix

Mon, 02 Jun 2014 00:00:00 GMT

Last week we fixed a bug in our daily summary emails that a customer reported on May 1st. The bug caused some of the emails to miss information on errors that occurred during certain parts of the day. This was a pretty serious bug. It affected almost all of our customers and had been around for a very long time.

The issue ended up being a miscalculation in our timezone handling code. Most developers who have worked with timezones know how difficult it can be to get it right. I'll explain how we did it and how Rollbar helped us quickly narrow in on the problem.

Debugging

The symptoms:

The occurrence counts in some of the daily summary emails were too low
Some new errors were not even in the daily summary emails

The first thing I did was to recreate the database query by hand. After reading the code and pulling out the relevant queries, I was able to recreate the daily summaries for production but in a development environment. At this point, it was clear that we were missing data but the query was correct.

Some possible culprits:

MySQL slave delay
Long-running MySQL transactions
Nebulous time-zone related bug in our code

I was able to verify that MySQL slave delay was not the cause by looking at our Scout graphs for the database host. Slave delay was nowhere near where it needed to be to cause this.

Next, I looked for long-running MySQL transactions. We have quite a few long-running connections so it was impossible to determine if we had long-running transactions by looking at just the output from MySQL's SHOW FULL PROCESSLIST.

I added some code to the script that sends the daily summary emails to print out the maximum id on a table in our database during each run of the script. If the maximum id never changed, it would mean the transaction in which the query was executing was never being closed, (since we use REPEATABLE READ by default on all connections.)

The ids were changing. This wasn't the problem...

At this point it was clear that the problem was in our code. I had pored over it for a couple of hours, testing everything and still could not find the problem. Then, I decided to add some Rollbar debugging to get more information about the code that was running in production.

After looking at the first occurrence it was clear what the problem was. The min_ts and max_ts were incorrect. This code path was pretty straight forward and it was easy to find the culprit - time.mktime().

The fix

The docs for time.mktime() say that it assumes the datetime object passed in uses the local timezone. Our code was passing in a datetime object for UTC which was getting ignored when we generated the timestamps for our query.

The fix was a one-liner. Instead of using time.mktime() we now us calendar.timegm() since it expects UTC.

Happy coding and watch out for timezone bugs!

rollbar-php v0.9.2 released

Fri, 30 May 2014 00:00:00 GMT

We've released another small update to rollbar-php, version 0.9.2. Get it on Packagist or GitHub.

This release contains a fix for an issue occurring in high-volume environments using rollbar-php with rollbar-agent. Previously, it was possible for relay log filenames to collide; now, the filenames include both the PID and the current time in microseconds (from microtime(true)).

Thanks to Ryan Fink for the pull request.

rollbar-php v0.9.1 released

Thu, 29 May 2014 00:00:00 GMT

We've released another small update to rollbar-php, version 0.9.1. Get it on Packagist or GitHub.

This release contains a fix for an issue when using rollbar-php with rollbar-agent, our server-side agent that allows asynchronous reporting. Previously, when used in agent mode, blank files would be created on each request, even if there was nothing to report. Now, these files are only created when needed.

Thanks to Ryan Fink from Red Ventures for the fix.

rollbar-gem v0.12.20 released

Thu, 29 May 2014 00:00:00 GMT

We've released a new patch version of rollbar-gem, version 0.12.20. Get it on Rubygems or GitHub.

This release contains a compatibility fix for Sidekiq < 2.3.2, as well as an improvement to the project_gems feature. If you have multiple gems that you want to be considered in-project code, you can now specify them with a regex (strings still work too).

For example:

Rollbar.configure do |config|
  config.project_gems = [/your_company/, "some_specific_gem"]
end

Thanks to Jonathan Cheatham for the pull request.

rollbar-php v0.9.0 released

Fri, 23 May 2014 00:00:00 GMT

We've released a new version of rollbar-php, version 0.9.0. It's available now on Packagist, or find the source on GitHub.

E_NOTICES NOW IGNORED BY DEFAULT

Rollbar can track ENOTICEs as 'info'-level events, but in practice this often turns into large amounts of data, even for relatively low-traffic projects. We've changed the defaults to now ignore ENOTICE errors.

If you do want E_NOTICEs to be sent to Rollbar, configure like this:

<?php
Rollbar::init(array(
  'access_token' => 'token here',
  // other current config...
  'included_errno' => (E_ERROR | E_WARNING | E_PARSE | E_NOTICE | E_CORE_ERROR | E_USER_ERROR | E_RECOVERABLE_ERROR)
));

MAXERRNO IS DEAD, LONG LIVE INCLUDEDERRNO

The max_errno configuration option has been removed and replaced by included_errno. includederrno takes a bitmask, like the PHP errorreporting directive. The default includes everything more severe than ESTRICT, except ENOTICE.

X-ROLLBAR-ACCESS-TOKEN HEADER NOW SENT

A few months ago, we added a feature to the API tier: the access token can now be provided in a header, instead of being in the body of the request. This makes things easier on our API workload, especially for requests that result in 429 or 403 response codes (because of rate limits, account limits, disabled access tokens, etc.). It should result in slightly faster response times in those cases.

rollbar.js v1.0.0-rc8 released

Thu, 22 May 2014 00:00:00 GMT

Today, we released rollbar.js version 1.0.0-rc8. This release includes a fix for a bug where the global variable i was used, which could cause strange behavior when interacting with other code that also references i as a global variable.

The change is live now and being served from our CDN - no action is required on your part for most customers. If you're self-hosting rollbar.js, we recommend upgrading your copy to the latest.

Processing Delay Postmortem

Fri, 11 Apr 2014 00:00:00 GMT

Yesterday from about 2:30pm PDT until 4:55pm PDT, we experienced a service degradation that caused our customers to see processing delays up to about 2 hours. While no data was lost, alerts were not being sent and new data was not appearing in the rollbar.com interface. Customers instead would see alerts notices on the Dashboard and Items page about the delay.

We know that you rely on Rollbar to monitor your applications and alert you when things go wrong, and we are very sorry that we let you down during this outage.

The service degradation began following some planned database maintenance, which we had expected to have no significant impact on service.

The Planned Maintenance

We store all of our data in MySQL in a master-master/active-passive configuration. Yesterday we needed to add partitions to our largest table - a routine procedure. Normally, this process takes about 15 minutes, during which time customers experience small delays in data processing. This process generally goes unnoticed by customers. However, this time something caused the database to load new data extremely slowly which, in turn, caused the outage.

Timeline

2:06pm - We began the planned maintenance by promoting the passive master to be the new active master.
2:29pm - The planned maintenance was complete.
- All connections to our old active master were closed and the new active master was getting new data and processing it.
2:40pm - It became apparent that new data was being loaded and processed very slowly.
- We turned off our data loaders to decrease any contention in the database.
2:41pm - We began profiling the slow worker.
2:47pm - We tested a theory that a single recurring item was causing most of the slow processing.
~2:50pm - We noticed that ping times to the new active master were 1-5 milliseconds - an order of magnitude slower than normal.
2:52pm - We turned off replication from the passive to the active database which seemed to help loading data, but not by much.
~2:50pm - 3pm
- The ALTER to add partitions completed on the passive database.
- Up to this point, there were no connections on the passive database.
- We decided to switch back to the previous active master but quickly reverted after finding that our passive database was missing a significant number of rows.
- MySQL replication was 0 seconds behind.
- It was unclear how the passive database thought it was caught up but was missing data.
~3:15pm - We identified a the slowest portion of the slow worker, which happened to be unused. We removed this code and deployed to all workers.
- This got processing back to normal speeds and allowed us to begin catching back up.
~3:30pm - We turned the data loaders back on.
4:30pm - The worker responsible to making new occurrences appear in the interface was caught up, but notifications were still delayed.
4:55pm - Everything was caught back up and we were back to 0 delay.

Follow-on Tasks

We have two open questions:

Why did the data loaders slow down when we switched to the new active master?
How did the databases get out of sync?

We have some theories as to why the data loaders slowed down so much but we are not sure. It could have been the amount of concurrent processes trying to load data into the same table. It could also have been something about the disk layout or cache on the new active master. We plan to investigate serializing loads in general and/or slowly ramping up loads after maintenance in the future.

To determine why our databases became out of sync, we wrote a script to tell us the exact moment when they diverged. Once it completes we will find the coordinates in the new active master’s binlogs that correspond with the point in time where the databases became out of sync, then restart replication on the passive master using those coordinates.

Conclusion

We take downtime very seriously and we want to be as transparent as possible when it happens.

We are sorry for the degradation of service and we are working on making sure it doesn’t happen again. If you have any questions, please don’t hesitate to contact us at support@rollbar.com

Heartbleed Bug Response

Tue, 08 Apr 2014 00:00:00 GMT

Updated 4/9 7:30pm

What is Heartbleed?

CVE-2014-0346, known as “Heartbleed”, is a bug in OpenSSL v1.0.1 through 1.0.1f that allows a remote attacker to access private memory on the target server. It has existed for almost 2 years. More info can be found here: http://heartbleed.com/

With this vulnerability, an attacker can:

Get your private key for your domain’s ssl cert
Decrypt all current and past SSL traffic to/from all affected machines

If this sounds bad, it is. Most sites on the Internet are affected.

Are you affected?

Probably. If your web server or load balancer is running on linux and you’ve updated your packages anytime in the last 2 years, you are more-than-likely affected.

To check your OpenSSL version, run openssl version -a

Check out http://filippo.io/Heartbleed/ to test your servers for the vulnerability.

How We Responded

We learned of CVE-2014-0346 at around 4:50pm on 4/7 and immediately began our response. We completed the most important fix (patching OpenSSL) within about an hour, and have been working over the past 24 hours on related issues.

Here is a timeline of what we’ve done since the vulnerability was announced:

4/7 - 3:01pm - Ubuntu Security Announcements email

Subscribe to this list here
4/7 - 4:50pm - Began updating our load balancers with the fix. All servers patched by 6pm.

We’re running nginx on Ubuntu 12.04. Updating is as simple as:
```
apt-get update
apt-get upgrade
openssl version -a # should show that it was built on April 7, 2014
service nginx restart
```
The above didn’t work for us on the first try because our servers were talking to a mirror that hadn’t updated to the latest packages (after all, they were only a couple hours old). Changing the domain in each line in /etc/apt/sources.list to archive.ubuntu.com and then running apt-get update again solved this.
4/7 - 11pm - rollbar.com and api.rollbar.com SSL certs were rekeyed

We use DigiCert for our SSL certs. The process was quick and easy.
4/7 - 11:10pm - Previous rollbar.com and api.rollbar.com SSL certs revoked

In order to prevent a possible man-in-the-middle attack we had Digicert revoke our old certs.
4/7 - 11:30pm - ratchet.io and submit.ratchet.io rekey requested

We still support our old domain, ratchet.io which use NetworkSolutions SSL certs
4/8 - 11:50am - All rollbar.com cookies were invalidated, forcing users to re-auth

Since an attacker could have accessed our customers’ cookies, we changed the secret key that we use to encrypt cookies. This invalidated all logged-in users’ sessions.
4/8 - 12:30pm - 2:25pm - All third-party tokens and keys were regenerated and deployed

We use services like Stripe, Mailgun, Heroku - All required new keys to be generated.
4/8 - 3:30pm - ratchet.io and submit.ratchet.io certs were rekeyed and deployed
4/8 - 5:30pm - Published this blog post and added in-app notifications to change passwords and cycle access tokens

Update 4/9

Thanks to this post on security.stackexchange, we additionally patched our application and compute servers (everything that can make outgoing HTTPS requests). This was started at 3:45pm and completed at 4:45pm. The attack surface here is much lower, as it requires creating a Rollbar account and setting up a webhook that points to the attacker’s malicious server. We audited our logs and confirmed that there has been no such suspicious activity.

Recommended actions for Rollbar Customers

Change your password
Cycle any access tokens you have used (create and start using a new one, then disable or delete the old one).
- For projects, go to the project dashboard, then Settings -> Project Access Tokens. Most customers will need to do this.
- For accounts, go to Account Settings -> Account Access Tokens. Most customers will not need to do this.

Note for Heroku Users

If you’re using Rollbar through Heroku, we’ve already started the process of cycling your access tokens. We’ve created new tokens and updated them in your Heroku config. You should update the token in any other locations (i.e. development environments, and anywhere it might be hardcoded) and then disable/delete the old tokens.

Closing notes

This was painful, but we’re thankful to the security researchers who discovered and responsibly disclosed this issue, and to the security teams at Ubuntu and elsewhere who quickly released patched packages.

If you have any questions, please don’t hesitate to contact us at support@rollbar.com

Connecting Rollbar with PagerDuty

Wed, 16 Oct 2013 00:00:00 GMT

Using Rollbar with PagerDuty is now a lot more seamless. PagerDuty provides SaaS IT on-call schedule management, alerting, and incident tracking. With our new integration, you can automatically send issues found by Rollbar into incidents in PagerDuty.

We have a few customers using it already. Here’s what Richard Lee, CTO at Polydice, a mobile development studio, has to say:

“With Rollbar’s integration of PagerDuty, we’re able to get notified as soon as errors detected, and avoid possible downtime to our customers. This powerful combination becomes a must have tool for us.” — Richard Lee, CTO at Polydice

Integrating Rollbar with PagerDuty is easy; just create a new Generic API System in PagerDuty, and then link it in Rollbar’s Notification settings. See our docs for detailed instructions.

Resolving Rollbar Items in Versions

Tue, 17 Sep 2013 00:00:00 GMT

We just rolled out a new feature to help track which versions/revisions errors are resolved in. When resolving items within Rollbar, you have the option of entering a revision or version number. If one is entered, it will appear in the item’s status history to let anyone looking at the item better understand specifically when it was fixed.

This version can be combined with a new code_version parameter set in the configuration options of the latest versions of our notifiers. This can be set to either a numerical value (eg. 1, 24, 300), a semantic version value (eg. 1.0.3, 2.9), or a git revision sha. Here are examples on how to set this parameter in our JavaScript and Ruby notifiers:

In the JavaScript snippet:

_rollbarParams = {
    // ... other configuration
    "client.javascript.code_version": "bdd2b9241f791fc9f134fb3244b40d452d2d7e35"
}

In your rollbar-gem configuration:

Rollbar.configure do |config|
    # ... other configuration
    config.code_version = 'bdd2b9241f791fc9f134fb3244b40d452d2d7e35'
end

The other notifiers have a very similar top-level code_version configuration settings. See the notifier readmes for more info.

If you resolve an item within Rollbar in a certain version and are also specifying a code_version for your code, we will use both of these values to decide whether or not to reactivate the item.

For example, say you have a bug in version 1.0 of your app. The bug is fixed and will be deployed to users in verision 1.1, but that won’t happen for a few days. You can just resolve the Rollbar item associated with this bug now, but also specify that the resolved version is 1.1. You will no longer get reactivation notifications for this item until occurrences of this item with a code_version >= 1.1 come in.

If you connect Rollbar with GitHub, this process will also work with Git SHAs. We’ll query the GitHub API to determine whether one commit is a parent of the other.

Auto-resolving items in GitHub commits

You can now also include Rollbar item tags in your GitHub commit messages to automatically resolve them in the correct revision when deploying. Just include one of the following strings in your commits for each item you want to resolve:

fix $ref
fixed $ref
fixes $ref
resolve $ref
resolved $ref
resolves $ref
close $ref
closed $ref
closes $ref

Where $ref is one of the following item tags:

Full item URL, eg. https://rollbar.com/item/123456789
Item ID, eg. rb#123456789
Short item ID, eg. rb#22 This appears at the top left of an item page.

Then execute a deploy by hitting the deploy API endpoint. The items referenced in any of the commit messages of the deploy will be resolved using the respective revision of that commit.

You can start tracking and resolving errors in Rollbar by signing up for free.

Ad-hoc error reporting with Rollbar CLI

Thu, 08 Aug 2013 00:00:00 GMT

We just coded up a quick tool to send Rollbar messages from the command line. It’s useful for quick, one-off monitoring scripts that you don’t have time to instrument with one of our notifiers.

To install, just pip install rollbar and you’re done.

e.g. Tracking all non-500s as WARNINGs from HAProxy

tail -f /var/log/haproxy.log | awk '{print $11,$0}' | grep '^5' | awk '{$1="";print "warning",$0}' | rollbar -t $ACCESS_TOKEN -e production -v

e.g. Watch failed login attempts

tail -f /var/log/auth.log | grep -i 'Failed password' | awk '{print "error user ",$11,"failed auth from ",$13}' | rollbar -t $ACCESS_TOKEN -e ops

More info on how to install and use it can be found here.

jQuery Error Instrumentation

Wed, 07 Aug 2013 00:00:00 GMT

Today we are releasing a new feature for our JavaScript notifier that should make tracking down errors much easier if you use jQuery 1.7 and above. The new functionality comes in a separate JS plugin snippet that should be placed right below where jQuery is loaded. Here is the first version of the plugin:

// <![CDATA[
!function(r,n,e){var t={"notifier.plugins.jquery.version":"0.0.1"};n._rollbar.push( {_rollbarParams:t});r(e).ajaxError(function(r,e,t,u){var o=e.status;var a=t.url; n._rollbar.push({level:"warning",msg:"jQuery ajax error for url "+a,jquery_status:o, jquery_url:a,jquery_thrown_error:u,jquery_ajax_error:true})});var u=r.fn.ready; r.fn.ready=function(r){return u.call(this,function(){try{r()}catch(e){ n._rollbar.push(e)}})};var o={};var a=r.fn.on;r.fn.on=function(r,e,t,u){ var f=function(r){var e=function(){try{return r.apply(this,arguments)}catch(e){ n._rollbar.push(e);return null}};o[r]=e;return e};if(e&&typeof e==="function"){ e=f(e)}else if(t&&typeof t==="function"){t=f(t)}else if(u&&typeof u==="function"){ u=f(u)}return a.call(this,r,e,t,u)};var f=r.fn.off;r.fn.off=function(r,n,e){ if(n&&typeof n==="function"){n=o[n];delete o[n]}else{e=o[e];delete o[e]} return f.call(this,r,n,e)}}(jQuery,window,document);
// ]]>

The source can be found on GitHub here.

The snippet wraps the ready(), on() and off() functions in jQuery to wrap any passed-in handlers in try/except blocks to automatically report errors to Rollbar. This lets us collect the full stack trace with line and column numbers for each frame, instead of just the last frame with only a line number. When combined with source maps, this makes debugging JavaScript errors much more doable.

The new snippet also adds a handler to ajaxError() to automatically report any jQuery AJAX errors such as 404s and 500s to Rollbar. If you don’t want this, add the following option to your base snippet’s _rollbarParams:

"notifier.plugins.jquery.ignoreAjaxErrors": true

You can start tracking errors in Rollbar by signing up for free. Or read more in the docs.

JavaScript and Source Maps in a Django App

Fri, 02 Aug 2013 00:00:00 GMT

It’s pretty well known that every web app needs frontend JavaScript these days to provide the best possible user experience. You are probably going to have a bunch of JavaScript files that need to be loaded by your users for that to happen, and since we all care about performance, minifiying and compressing these files is an absolute must. But what happens when it comes time to debug issues in these minified files? Stack traces will more or less be completely useless. How do we solve this problem?

JavaScript source maps solve this problem. They allow you to map a point in a minified file back to the unminfied source, making it possible to actually identify and fix issues encountered in a production app environment.

Below I have outlined a simple guide for setting up source map generation and usage in a sample Django app. You’ll learn how generate source maps for minified files, debug errors that happen in these files, and also a quick overview of what’s required to get this working for your production environments.

Local Debugging with Source Maps

Say you have a simple Django app with the following directory structure:

...
app/
    ...
    views.py
    static/
        js/
            site.js (containing various models and functionality used in your app)
            jquery.js (unminified)
            util.js
    templates/
        index.html

site.js would have the following code:

var aFunction = function() {
    var a = b;
}

App = {};
App.errorCausingFunction = function() {
    aFunction();
}

views.py would just contain one view that rendered index.html, and here is how index.html would look like:

...
<script type="text/javascript" src="static/js/site.js"></script>
<script type="text/javascript" src="static/js/jquery.js"></script>
<script type="text/javascript" src="static/js/util.js"></script>
....
<script type="text/javascript">// <![CDATA[
App.errorCausingFunction();
// ]]></script>
...

Let’s minify. Start by installing UglifyJS2:

npm install -g uglify-js

Here is an example command run from app/ that will generate minified Javascript:

uglifyjs static/js/site.js static/js/jquery.js static/js/util.js --output static/js/all.min.js

Here we are using uglifyjs to minify three JS files, site.js, util.js, and jquery.js, into all.min.js.

Update your index.html to include only all.min.js:

Now let’s try navigating to index.html and seeing what happens:

Not the most helpful stack trace. All the line numbers are 1, and if this were a much larger file with more generic function/variable names, it would be completely useless in helping you debug where things went wrong.

Let’s introduce source map functionality to our app.

Modify your minification command to look like this:

uglifyjs static/js/site.js static/js/jquery.js static/js/util.js --output static/js/all.min.js --source-map static/js/all.min.map --source-map-url /static/js/all.min.map

Here we are adding two new options, --source-map and --source-map-url. UglifyJS2 will now generate the resulting source map as static/js/all.min.map, and will append a comment to the end of the minified file containing the url to the source map on your website, in this case /static/js/all.min.map. Note, you may need to modify the comment in all.min.map to read //# instead of //@, as this is a recently new convention.

Now navigate to your app in Chrome with Developer Tools open. If everything is set up right, Chrome will automatically translate the frames in the stack trace to the unminified equivalents, like so:

Note that the filenames and line numbers now refer to the original source code, instead of the minified source.

Production Debugging with Source Maps

The above process is all fine and dandy for errors encountered on your local machine, but what if you want to keep track of errors encountered by your users in real-time?

Here at Rollbar, we have recently reworked our error processing pipeline to support the application of source maps on JavaScript errors. Here’s how you would get Rollbar hooked up and reporting from your production environment:

Create a Rollbar account
Follow the instructions to insert the Rollbar Javascript notifier snippet into your base template
Modify the snippet configuration to signal that source maps should be used:

js _rollbarParams = { // ... other params ... // set this to 'true' to enable source map processing "client.javascript.source_map_enabled": true, // provide the current code version, i.e. the git SHA of your javascript code. "client.javascript.code_version": "bdd2b9241f791fc9f134fb3244b40d452d2d7e35" }

Make sure your minified files link properly to publicly accessible source maps using the sourceMappingURL comment:

// ... minified js file contents ...
//# sourceMappingURL=

Now, when your app sends Rollbar an error report, Rollbar will automatically attempt to download source maps defined in your minified files and apply them to stack frames located in these files.

Here is an example of the source map application process in action with an unminified stack trace that you would see in Rollbar:

Notice the unminified source filenames with relevant line and column numbers.

Automating things

It’s a bit annoying to have to minify everything every time you change one of your Javascript files. We have a small script set up here in our dev environments that uses macfsevents to listen for Javascript file changes. Once such events occur, we check to see if only the Javascript files we care about are affected. If so, we run an uglifyjs command on all the Javascript files to generate minified sources and source maps.

You can even go one step further by making an API call to Rollbar to upload your source map as part of your deploy process. This API endpoint also accepts source file uploads for files referenced by the source map, giving us the ability to print out the unminified source code for each frame in the stack trace. For example:

Here’s a sample command you could use to upload a source map and source file to our API:

curl https://api.rollbar.com/api/1/sourcemap \
-F access_token=aaaabbbbccccddddeeeeffff00001111 \
-F version=bdd2b9241f791fc9f134fb3244b40d452d2d7e35 \
-F minified_url=http://127.0.0.1:8005/static/js/all.min.js \
-F source_map=@app/static/js/all.min.map \
-F app/static/js/site.js=@app/static/js/site.js

The last param is a mapping of source file path to actual source file contents. The path would need to match the one defined in your source map, generated by your minification tool.

More info

Check out the documentation for more info about integrating your JavaScript and source maps with Rollbar. Rollbar also integrates with your Python, Rails, PHP and Node.js based backends.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you defeat annoying django errors in your applications. :-)

Debug Production Errors in Minified JavaScript with Source Maps and Rollbar

Thu, 25 Jul 2013 00:00:00 GMT

Rollbar just got a much-requested feature: Source Maps support for Javascript. If you minify your Javascript code in production, this will make debugging production errors much easier. This feature is now live for all accounts.

Here's a link to a recently updated tutorial on using JavaScript source maps with Rollbar.

What Are Source Maps?

If you minify your Javascript code (i.e. using UglifyJS2 or the Closure Compiler), it gets harder to debug errors. Stack traces reference the line/column numbers in the minified code instead of the original source code.

Source Maps were designed to resolve this; they provide a mapping back from the minified line/column numbers to the original code. Chrome and Firefox have tools to use them in development, but what about errors that happen in production?

Source Maps and Rollbar

Rollbar can now map stack traces that reference minified code back to the original source files, lines, and column numbers. Here’s what a stack trace might have looked like before:

Here’s the de-minified version:

We’ll also use the de-minified stack trace in our grouping algorithm, which should result in more useful grouping.

Getting this set up

To get started, you’ll need to make a change to _rollbarParams in the on-page javascript snippet. Add the following two parameters:

_rollbarParams = {
  // ... existing params ...
  // set this to 'true' to enable source map processing
  "client.javascript.source_map_enabled": true,
  // provide the current code version, i.e. the git SHA of your javascript code.
  "client.javascript.code_version": "bdd2b9241f791fc9f134fb3244b40d452d2d7e35"
}

Next, either: - Add a sourceMappingUrl comment at the end of your minified file to point to the source map - Upload the source map (along with all source files) separately, as part of your deploy process

This second step is a bit more involved so please see our docs for more details.

Caveats

All of this relies on having a stack trace with line and column numbers. Unfortunately, browser support for column numbers is inconsistent. As of today, this will work in Chrome, Firefox, and IE10+, and only for caught errors reported like this:

try {
  doSomething();
} catch (e) {
  _rollbar.push(e);
}

Uncaught errors (reported via window.onerror) don’t have column numbers in any browser we’re aware of, so they aren’t able to be de-obfuscated. For best results, catch all your exceptions so you don’t fall back to the top-level error handler.

If you haven’t already, signup for a 14-day free trial of Rollbar and let us help you defeat annoying javascript errors in your applications. :-)

Async node.js API server testing

Fri, 12 Jul 2013 00:00:00 GMT

This post is about how we built our test suite for our API server at Rollbar and some of the tricks and gotchas we ran into along the way. We wanted to build a test suite that not only tested the API logic, but also the underlying code, namely the Express and the Connect middlewares we use. If our API server was going to break, we wanted to know before we deployed it to thousands of customers and millions of requests per day.

Testing is super important. If you don’t want to test, this probably won’t be very helpful or interesting.

We use Vows. Why not Mocha?

Mocha is, by far, the most widely used testing framework for Node.js apps. So, why didn’t we use it? The two main reasons were that Vows was the first thing I found when Googling “nodejs async testing” and the other is that the syntax of Mocha tests felt like another language and less like code. Mocha tests are more readable but the benefit of readability was overshadowed by the need to remember all of these new, special-case methods that Mocha injects.

//Mocha
[1,2,3].indexOf(5).should.equal(-1);

//Vows
assert.equal([1,2,3].indexOf(5), -1);

There’s something that bothered me about the former. I didn’t like how the library used a bunch of magic to enable something this small/strange.

Mocha has a lot of awesome features but none that were important enough for me to switch.

A simple Vows test

Vows works just as you’d expect it to, except when it doesn’t. More on that later…

var vows = require('vows');
var assert = require('assert');
vows.describe('testmodule').addBatch({
  "call username() with a valid user id": {
    topic: function() {
      var callback = this.callback;
      return username(42, this.callback);
    },
    "and verify username is correct": function(err, username) {
      assert.isNull(err);
      assert.isString(username);
      assert.equal(username, "cory");
    }
  }
}).export(module, {error: false});

The above test will make sure that the function username() calls its callback with (null, "cory").

Note that we use this.callback since everything is assumed to be async and we use {error: false} when we export the batch. More on those later.

Check out the Vows website for better examples.

Useful design patterns (I swear this will be short)

We’ve found a few idioms and conventions that have been super helpful. Without going too much into design patterns and architecture, here are a few tips that have made writing tests super-easy; almost enjoyable—almost.

Separate your view logic from your API business logic

Your server’s views should have one job, to marshall data from the request/socket/carrier pigeon and provide it to your API library.

Any error checking done in your views should be to make sure the types provided to your API library are correct.

Make every function you write use a callback

This is super-important for refactoring and adding new features. If you find yourself wanting to add a feature that requires i/o into a code path that was assumed to be completely synchronous, you’ll need to refactor the hell our of your code to make it work. Don’t bother. Make everything take a callback. Embrace async!

Make the first argument to every callback be an optional error

This is how the Node.js developers do it and I agree. It makes for a lot more boiler-plate code but it forces you to keep error handling in-mind when developing. Writing defensive code is more important than writing fewer lines of code.

This will also make testing much, much easier with Vows. How? Read on…

Testing the API server, for reals

Definitely write tests and exercise your API library directly but don’t stop there. Fork a process, start your API server up in it and start firing requests at it using Vows.

testcommon.js:

exports.initTestingAppChildProc = function(config, promise) {
  // ... Setup temporary config file
  // ... Get path to your main app.js
  // ... Initialize the api library

  // fork a child process to start the api server
  var args = [configPath, 'test'];
  var appProc = fork(appJsPath, args);

  // This is used to tell if our API server died during its
  // initialization.
  var pendingCallback = true;

  appProc.on('message', function(message) {
    if (message == 'ready') {
      pendingCallback = false;

      // This is how we know our API server is ready to
      // receive requests. The message is emitted in the
      // API server once it's ready to receive requests.
      promise.emit('success', null, appProc);
    }
  });
  appProc.on('exit', function(code, signal) {
    if (pendingCallback) {
      var msg = 'child process exited before callback';
      console.error(msg);
      promise.emit('error', new Error(msg));
    }
  });
  appProc.on('SIGTERM', function() {
    process.exit();
  });
};

In our API server:

// initialize the API and start the web server when it's ready
api.init(config, function(err) {
  if (err) {
    log.error('Could not initialize API: ' + err);
    process.exit(1);
  } else {
    // Start up the server
    var httpServer = app.listen(port, host, function() {
      log.info('API server is ready.');
      log.info('Listening on ' + host + ':' + port);

      // Use the "ready" message to signal that the server is ready.
      // This is used by the test suite to wait for the api server
      // process to start up before sending requests.
      if (process.send) {
        process.send('ready');
      }
    });
  }
});

tests/routes.project.js:

vows.describe('routes.project').addBatch({
  // Provides a reference to the api server child process
  'Start up an API server': {
    topic: function() {
      var promise = new events.EventEmitter();
      common.initTestingAppChildProc(config, promise);
      return promise;
    },
    teardown: function(err, childProc) {
      var callback = this.callback;
      var shutdown = function() {
        api.shutdown(callback);
      };
      if (childProc) {
        childProc.on('exit', function(code, sig) {
          shutdown();
        });
        childProc.kill();
      } else {
        shutdown();
      }
    },
    'and get a valid project': {
      topic: function(err, childProc) {
        common.apiGet(url('api/1/project/',
            {access_token: config.test.validEnabledReadAccessToken}), this.callback);
      },
      'returns 200 OK': common.assertStatus(200),
      'returns JSON': common.assertJsonContentType(),
      'fast local response time': common.assertMaxResponseTime(20),
      "returns a valid api response": common.assertValidApiResponse(),
      "has a result key in the JSON response": common.assertJsonHasFields(['result']),
      "there's no api error": common.assertNoApiError(),
      'all of the deploy fields are available': common.assertJsonHasFields(db.projectFields(),
          'result'),
      'cross-check account id with api.getAccount': {
        topic: function(err, resp, body) {
          var project = body.result;
          api.getAccount(body.result.account_id, this.callback);
        },
        'verify the account is not null': function(err, account) {
          assert.isNull(err);
          assert.isObject(account);
        }
      }
    }
}).export(module, {error: false});

There is a lot happening in these tests.

We use promises to notify our test when the API server is ready.
- Documentation for using promises with Vows can be found here.
- I’m not completely on-board with the Promise design pattern but it seemed like the easiest way to get this working. Mostly, I needed an event to be fired when there was an error that caused the API server process to shut down.
We use a Vows teardown function to shut down the API server process.
We use our API library to help test our API server.
- We cross-check our API server’s response by using our API library directly.
We use Vows macros for reusable tests on all API requests.
- We also make use of Vows contexts even though there are none in this example.
- Documentation for macros and contexts are here.

Gotchas

Never, ever, ever throw an uncaught exception in a Vows topic. It makes debugging impossible. I’ve wasted hours looking through my API library for a bug only to find that I had a silly bug in my topic.

Always use export(module, {error: false}) in your Vows batches. This option is not really described in the Vows docs. I had to find it in the source. Basically, if you don’t have this, Vows will inspect the first argument to each test to see if it’s an error. Vows will potentially call your test functions with a different set of arguments depending on if the first parameter to the topic’s callback is an Error or not. It’s completely strange and magical and confusing.

Testing without mock objects means that you need a real database which means you probably need real-ish data to test with. This is tough. We chose to maintain a DB SQL fixture that we have to update whenever the schema changes. It’s a bit clunky but it works. I’m open to suggestions for this if anyone knows of a better way.

Wrapping up…

We use CircleCI to run all of these tests and are really happy with their service. It’s fast and easy to set up. Also, it has all of the systems that our API server uses like MySQL, Beanstalkd and Memcache pre-installed. This gets us closer to testing in a production environment than would otherwise be possible.

Hopefully you were able to glean some useful tips from our experience at Rollbar. We love building tools for devs like you!

Add me on Twitter @coryvirok. Follow @rollbar for more updates.

Moment of zen

✓ OK » 497 honored (33.232s)

May Release Roundup

Wed, 29 May 2013 00:00:00 GMT

Here’s a roundup of what’s new at Rollbar in the month of May.

Big Features

We revamped our notifications system, and added integrations with a bunch of new services. Rollbar now works with Asana, Campfire, Flowdock, GitHub Issues, Hipchat, JIRA, Pivotal Tracker, and Trello, as well as any arbitrary system via a Webhook. See the announcement blog post for more details.

Small Features

You can now customize how occurrences are grouped. This first release allows you to define rules of things that should always be grouped together. See the documentation: Custom Grouping Rules. An in-depth post on how to use this is coming soon.
There’s now a “Download CSV” link at the bottom of the Items page, which will let you download a CSV of what you see on the page. Note that this information is also available via our API. ![]({{ site.cdn_url }}/blog/images/2013/2013-05-29-may-release-roundup/download-csv.133485.o.png)
You can now sort the Items page by Total Occurrences or Unique Users, in additon to Last Occurrence. Click on the column headers to change the sort. ![]({{ site.cdn_url }}/blog/images/2013/2013-05-29-may-release-roundup/item-list-sort.133486.o.png)
Links to files in GitHub are now linked to the appropriate revision, when this information is available. We’ll use one of the following (trying each in order):
- the value of server.sha
- the value of server.branch, if it looks like a SHA
- the revision from the last deploy before the first occurrence of the item

Library Updates

Ruby

We released rollbar-gem versions 0.9.11 through 0.9.14. The changes include a fix for use with Rails 4, a concurrency bugfix, better support for JSON requests, and the ability to include custom metadata with all reports. See the full changelog for details. To upgrade, change the rollbar line in your Gemfile to:

gem 'rollbar', '~> 0.9.14'

We also contributed a fix to resque-rollbar to force use of synchronous mode when reporting Resque failures (instead of async mode, which doesn’t play nicely with Resque).

Python

pyrollbar gained a feature and now at version 0.5.7. See the changelog for details.

Bug Fixes

Fixed an issue where pressing the back button would sometimes cause Chrome to render one of our JSON responses as if it were HTML
Fixed a bug where removed email addresses could not be re-added

Documentation Updates

Added guide on how our grouping algorithm works.
Cleaned up our Items API Reference and fixed a few outdated parts

More is on the way. Stay tuned! And don’t forget to send us any feedback: team@rollbar.com — we love hearing from you.

Rules Engine for Notifications, Plus Integrations with Campfire, Hipchat, JIRA, and Trello

Mon, 06 May 2013 00:00:00 GMT

Today we’re revamping the model for defining what you want to be notified about from Rollbar. Rollbar now integrates with Asana, Campfire, Flowdock, GitHub Issues, Hipchat, JIRA, Pivotal Tracker, and Trello, as well as any arbitrary system via a Webhook.

New Integration Channels

In addition to our existing channels (Email, Asana, GitHub Issues, Pivotal Tracker, and Webhook), we’re launching support for four more: Campfire, Hipchat, JIRA, and Trello. You can set up all of this in Settings -> Notifications.

Notification Rules Engine

Notifications are now configured per-project (instead of per-user-per-project), using a trigger-action model. There are triggers for the following events:

New Item (first occurrence of a new issue)
Reactivated Item (a previously resolved issue has occurred again)
10nth Occurrence (an issue has occurred for the 10th, 100th, etc. time)
Resolved Item (an item has been resolved by hand)
Reopened Item (an item has been reopened by hand)
Post-deploy (you’ve notified us that you deployed a new release)

Corresponding actions are available for most actions in most channels. If it would make sense, it probably exists.

Most actions can be configured as you’d expect (i.e. set which teams should receive an email, or which user to assign JIRA issues to).

Item-related triggers can be filtered by environment, level, title (exception class+message), and filename. Deploy triggers can be filtered by environment and comment. Our underlying tech supports much more than the UI exposes, so let us know what other filters you’d like to see.

Migration for existing customers

We’ve migrated existing customers’ settings to the new system, but there were a few aspects that didn’t map very well (i.e. per-user-per-project settings). We hope the new system is easier to use for most use-cases and still workable for complex setups, but let us know if there’s something you are having trouble doing.

Questions? Feedback?

Let us know if you have any questions about how to get the notifications you want. We look forward to your feedback. What other integrations do you want? Let us know in the comments, or email us at team@rollbar.com.

Taking UNIQUE indexes to the next level

Fri, 29 Mar 2013 00:00:00 GMT

You’ve probably seen unique constraints somewhere – either in Rails’ validates :uniqueness, Django’s Field.unique, or a raw SQL table definition. The basic function of unique constraints (preventing duplicate data from being inserted) is nice, but they’re so much more powerful than that. When you write INSERT or REPLACE statements that rely on them, you can do some pretty cool (and efficient) things that you would’ve had to do multiple queries for otherwise.

This post covers unique indexes in MySQL 5.5. Other versions of MySQL are similar. I’m not sure about Postgres or other relational databases but presume they’re similar-ish as well.

Primer: what is a unique index?

Pre-primer: data in a database is stored on disk somewhere. In a SQL database, the data is organized into tables which have rows and columns. An index is a way to look up particular rows, based on the values of one or more columns, without having to scan through the whole table. Instead, you look up those values in the index, which tells you where to find the matching rows.

Index lookups are typically faster than full table scans because they’re organized for fast searches on the indexed columns (usually using binary trees), and they’re also generally smaller than the original data.

A unique index is an index that also imposes a constraint: that no two entries in the index can have the same values. It can be comprised of one column or many columns. If many columns, then the entire tuple of columns is used for determining uniqueness. There can be other columns in the table that are not part of the index; these don’t affect the constraint.

Primary keys are a special case of unique index; we’ll cover this in more detail later.

Unique indexes can be created in a CREATE TABLE statement like this:

CREATE TABLE user (
  username varchar(32),
  password char(32),
  unique (username)
);

or using an ALTER TABLE statement like this:

ALTER TABLE users ADD unique (username);

What does a unique constraint affect?

A unique constraint prevents you from changing your data in a way that would result in having duplicate data in the index. For example, given the above user table, the following will happen if we try to insert duplicate data:

mysql> INSERT INTO user VALUES ('brian', PASSWORD('asdf'));
Query OK, 1 row affected, 1 warning (0.04 sec)

mysql> INSERT INTO user VALUES ('brian', PASSWORD('asdfjkl'));
ERROR 1062 (23000): Duplicate entry 'brian' for key 'username'

or if we try to get duplicate data with an update:

mysql> INSERT INTO user VALUES('sherlock', PASSWORD('123456'));
Query OK, 1 row affected, 1 warning (0.00 sec)

mysql> UPDATE user SET username = 'brian' WHERE username = 'sherlock';
ERROR 1062 (23000): Duplicate entry 'brian' for key 'username'

So we can see that unique indexes are a great way to maintain consistency of our data at the database level. If two people try to sign up with the same username, for example, the database will reject it and return a duplicate key error.

Taking it to the next level

MySQL provides several commands, all variations of INSERT, that can take advantage of unique indexes by specifying what to do (instead of erroring) when the insert would result in a duplicate. These are best illustrated by example.

INSERT … ON DUPLICATE KEY UPDATE

Let’s say we’re building a simple ad impression tracking system. Ads are served by web servers and the impression counts are tracked in a database. We just want to know the number of ad impressions each hour. So we make a table like this:

CREATE TABLE hour_impression (
  hour int unsigned not null,  -- number of hours since unix epoch
  impressions int unsigned not null default 0,
  primary key(hour)
);

Side note: here we’re using hour as the primary key, rather than having an auto-increment primary key like before in the user table. This guarantees that hour is unique (since primary keys are a subset of unique keys), and has a nice property of laying out the data on disk in hour-order.

A naive algorithm for recording each impression would be to:

Check if a row already exists for the hour
If not: INSERT INTO hour_impression (hour, impressions) VALUES (:hour, 1)
If so: UPDATE hour_impression SET impressions = impressions + 1 WHERE hour = :hour

But this exposes a race condition: what happens if two impressions happen at approximately the same time, on two different web servers? It’s possible that both will try to INSERT, and the second one is going to fail (because of the unique constraint).

What we want to do is combine the above algorithm into a single step. This is what INSERT … ON DUPLICATE KEY UPDATE is for:

INSERT INTO hour_impression (hour, impressions)
VALUES (379015, 1)
ON DUPLICATE KEY UPDATE impressions = impressions + 1

Now that it’s a single step, we can run as many of these statements in parallel as we want, and the database will take care of the concurrency issues for us. Sweet!

INSERT IGNORE

Now let’s say instead of counting the number of impressions in each hour, we just want to know which minutes any impressions at all were shown. So we create a table like this:

CREATE TABLE minute_impression (
  minute int unsigned not null,  -- number of minutes since the unix epoch
  primary key (minute)
);

Similar to before, a naive algorithm for recording which minutes had any impressions would be to:

Check if a row already exists for the minute
If so, do nothing
If not, INSERT INTO minute_impression (minute) VALUES (:minute)

This has the same kind of race condition as in the previous example. INSERT IGNORE exists to combine all of this into a single step:

INSERT IGNORE INTO minute_impression (minute) VALUES (22740922)

And as before, now we can run as many of these in parallel as we want and let the database take care of the concurrency.

More tricks

REPLACE

The opposite of INSERT IGNORE. Overwrites matching rows with the new data instead of discarding it.

Nullable unique indexes

Values in a unique index have to be unique, but there’s an exception: NULLs don’t count. For example, let’s say you let people pick their username after signup. You might have table like:

CREATE TABLE user (
  id int unsigned not null auto_increment,
  username varchar(32) default null,
  unique (username),
  primary key (id)
);

You can have as many users as you like who haven’t chosen a username (it’ll be NULL) while still preventing multiple users from having the same username.

VALUES() in the ON DUPLICATE KEY UPDATE clause

You can insert multiple rows in a single INSERT … ON DUPLICATE KEY UPDATE statement, and the UPDATE rule will apply for each row that would’ve been a duplicate. In some cases you’ll want the update statement to reflect the values of each particular row, and that’s not possible to do by hardcoding them in the statement.

For example, let’s return to the ad impression tracking problem from before, with this hour_impression table:

CREATE TABLE hour_impression (
  hour int unsigned not null,  -- number of hours since unix epoch
  impressions int unsigned not null default 0,
  primary key(hour)
);

But now instead of recording impressions one at a time, we’re batching them so that each INSERT increments impressions by a value 1 or higher. If we insert one of these batches at a time, we can do:

INSERT INTO hour_impression (hour, impressions)
VALUES (379015, 23)  -- 23 impressions during 12am 3/28/2013
ON DUPLICATE KEY UPDATE
impressions = impressions + 23

If we want to insert multiple rows in the same statement, there’s a problem – the amount in the UPDATE clause is hardcoded. We can fix this using VALUES() to reference the value from the would-have-been-inserted row:

INSERT INTO hour_impression (hour, impressions)
VALUES (379015, 23), (379015, 55)
ON DUPLICATE KEY UPDATE
impressions = impressions + VALUES(impressions)

Conclusion

Unique indexes are useful when used alone and become incredibly powerful when used in combination with INSERT … ON DUPLICATE KEY UPDATE and its variants. We make heavy use of this at Rollbar and it works great.

Questions? Corrections? Let me know in the comments.

References

INSERT … ON DUPLICATE KEY UPDATE syntax
INSERT IGNORE syntax (ctrl+f on the page)
REPLACE syntax

Improved grouping for Javascript errors

Thu, 21 Mar 2013 00:00:00 GMT

We’ve released an updated to how Javascript errors are grouped in Rollbar. The new update does a better job of separating different errors into different groups (“Items” in Rollbar parlance) while still recognizing the same issue in different browsers as the same. It’s now enabled for all new projects. Existing projects can enable it on the Migrations tab in Settings.

Now the longer version…

First some background: by default, exceptions in Rollbar are grouped using their stack traces. We take all of the filenames and method names in all of the stack frames, plus the exception class name, apply a number of heuristics to normalize them, and then combine everything together and take a sha1 hash. The result is a 40-character string used as the “fingerprint”; occurrences with matching fingerprints that also have the same project, environment, and platform are grouped together. The fingerprint can also be overridden at the API level for custom grouping.

This generally works pretty well:

Omitting the line numbers from stack frames means groups persist across code changes elsewhere in the file.
Using the whole stack trace, instead of just the very last frame, avoids conflating unrelated issues that happen to cause an exception on the same line of code.
Using just the exception class, instead of also the message, avoids including data in the fingerprint, and when we have a nice, long stack trace, that’s usually enough uniqueness.

Javascript uncaught errors are a different story though. They’re reported through window.onerror, which luckily is supported in all major browsers but doesn’t provide a lot of context: only the error message, filename, and line number. The Rollbar javascript library converts this into a stack trace with a single frame (using the filename and line number), and parses the error message to get a semblance of class and message.

The problem with this approach has been that since the grouping algorithm only uses filenames (not line numbers), there’s only one stack frame, since it only uses the exception class name (not message), there isn’t a whole lot of uniqueness. The updated algorithm improves this dramatically. Here’s how it works:

There aren’t that many different kinds of Javascript errors. We’ve built up a database of the error messages generated by all of them, in all major browsers.
When we process an error, we try to match it against the patterns we know about. Several of the patterns have data in them; for some of the patterns, we keep the data (i.e. if it’s a variable name), and in others we discard it (i.e. if it’s an out-of-range precision value).
If we aren’t able to match any of the known patterns, we fall back to the old algorithm.

We’ve been using this internally for the past couple weeks and so far it feels much better. A caveat: our pattern database currently is English-only, so errors from end-users with their language preferences set otherwise will be grouped using the old algorithm.

The new algorithm is now enabled for all newly-created projects. We aren’t automatically turning it on for old projects because none of the new groups will map to old groups; this will be a one-time event but could result in a lot of notifications and we don’t want to force the change. To enable it for an existing project, find the Migrations tab in the Settings for your project, and check the box next to “Browser JS Occurrence Grouping V2”.

Any questions? Let us know in the comments.

Rollbar collects and analyzes errors so you can find and fix them faster. Create a free account to get started today.

Launch and initial funding

Tue, 26 Feb 2013 00:00:00 GMT

Today we’re excited to announce the public launch of Rollbar. Rollbar tracks and analyzes errors in production applications, helping dev and ops teams diagnose and fix them.

Platform-agnostic API

Anything that can speak JSON and HTTP can talk to Rollbar. Our API accepts raw “items” (errors, exceptions, and log messages) and deploys as inputs, and aggregated items, occurrences, and deploys as outputs. We provide official libraries for Ruby, Python, PHP, Node.js, Javascript, and Flash; or you can roll your own.

Severity levels

Just because something raises an exception, doesn’t mean it should be treated as an “error”. Rollbar lets you utilize five severity levels (from “debug” to “critical”) to control visibility and notifications. Severity can be set in your code, or after-the-fact in the Rollbar interface.

Track users through your stack

Person tracking helps you provide great customer support by emailing affected users when you fix an error they hit. Or see the history for a particular user and link customer error reports to code problems, client- and server-side.

So much more

API endpoints on 3 continents. Resolving and reactivations. Real-time notifications for new issues. Graphs everywhere. Deploy tracking. Search by title, host, file, context, date, severity, status. Replay an issue by pressing a button. SSL everywhere. GitHub, Asana, and Pivotal Tracker integration.

We’ve built many of the pieces our beta customers have needed, and we really think you’re going to like it. Start a free trial now, or see pricing, features, or docs.

More firepower

We’re also excited to announce that we’ve raised an initial round of funding from some of the smartest people in the business. Mike Hirshland (Resolute.vc), Hiten Shah (KISSmetrics), and Arjun Sethi participated in the round. This funding gives us some extra firepower to grow the team and bring our vision to life.

We’re really excited and can’t wait to keep making Rollbar better!

Brian, Cory, and Sergei

Upgrading to the new Rollbar notifier libraries

Tue, 26 Feb 2013 00:00:00 GMT

We’ve updated all of our notifier library repositories to match the name change to Rollbar today. The old Ratchet.io repos have been deprecated and all further development will continue on the respective Rollbar versions.

Please note that the submit.ratchet.io endpoint and the existing libraries will continue to work for the indefinite future, so you don’t have to do anything right now. But we do recommend upgrading to take advantage of future updates.

Upgrading should be seamless and quick. Please contact support@rollbar.com if you run into any issues.

Here are links to the upgrade instructions for each:

Browser JS - update the JS snippet used on your site to the version shown here
pyratchet
ratchetio-gem
ratchetio-php
ratchet-agent
node_ratchet
flash_ratchet

Post-mortem from last night's outage

Fri, 11 Jan 2013 00:00:00 GMT

tl;dr: from about 9:30pm to 12:30am last night, our website was unreachable and we weren’t sending out any notifications. Our API stayed up nearly the whole time thanks to an automatic failover.

We had our first major outage last night. We want to apologize to all of our customers for this outage, and we’re going to continue to work to make the Rollbar.com service stable, reliable, and performant.

What follows is a timeline of events, and a summary of what went wrong, what went right, and what we’re doing to address what went wrong.

Background

First some background: our infrastructure is currently hosted at Softlayer and layed out like this (simplified):

That is:

our primary cluster of servers is in San Jose
all web traffic (rollbar.com / www.rollbar.com) is handled by lb2
all API traffic (api.rollbar.com) is handled by lb1
lb3 (in Singapore) and lb4 (Amsterdam) are ready to go but not in use yet (more on this below), providing failover and faster API response times to customers outside of the Americas.

We’ve been in the process of setting up lb3 and lb4, along with some fancy DNS functionality from Dyn, to provide redundancy and faster response times to our customers outside of the Americas. Each is running a stripped-down version of our infrastructure, including:

a frontend web server (nginx)
two instsances of our node.js API server
a partial database slave (for validating access tokens)
our offline loading process (soon to be open-sourced!), for doing async writes to the active master database.

Switching DNS to Dyn requires changing the nameservers, which can take “up to 48 hours”. At the start of this story, it’s been about 36 hours. To play it safe, after testing out Dyn on a separate domain, we configured it to have the same settings as we had before – lb3 and lb4 are not in play yet.

Timeline

Now the (abbreviated) timeline. All times are PST.

9:30pm: Cory got an alert from Pingdom that our website (rollbar.com) was down. He tried visiting it but it wouldn’t load (just hung). Remembering the pending DNS change, he immediately checked DNS propagation and saw that rollbar.com was pointing at the wrong load balancer – lb1 (the API tier), not lb2.

Cory and Sergei investigated. The A record for rollbar.com showed as correct in Dyn, but DNS was resolving incorrectly.

9:47pm: Cory and Sergei looked at @SoftlayerNotify and saw that there was an issue underway with one of the routers in the San Jose data center.

9:49pm: Website accessible by its IP address.

9:51pm: No longer accessible by IP.

10:05pm: Twitter search for “softlayer outage” shows other people being affected.

10:05pm: API tier (api.rollbar.com) appears to be working. Sergei verifies that it’s hitting lb3 (in Singapore).

You might notice that we said before that lb3 wasn’t supposed to be in service yet. What appeared to have happened DNS had automatically failed over to lb3 (since lb1 was down because of the Softlayer outage). We had set something like this up before when testing out Dyn, but it wasn’t supposed to be active yet. Fortunately, lb3 was ready to go and handled all of our API load just fine.

10:22pm: Sergei tries fiddling with the Dyn configuration to see if anything helps.

10:35pm: Sergei starts trying to get ahold Dyn

10:58pm: Softlayer posts that “13 out of 14 rows of servers are online”. We must be in the 14th, because we’re still unreachable at this point. Brian tries hard-rebooting the ‘dev’ server to see if it helps. It doesn’t.

11:15pm: Sergei gets a call from Dyn, who tells him that the problem was a “stale Real-Time Traffic Manager configuration” and they’re looking into it.

11:54pm: @SoftlayerNotify posts that “all servers are online however some intermittent problems remain”

11:55pm: Sergei notices that the A record for rollbar.com in the Dyn interface appears to have been deleted, and he can’t add it back.

12:00am: Brian sees that rollbar.com is working again. Cory notices that API calls are hitting lb2, causing them to hit the old, non-optimized API handling code on our web tier, overloading them and causing the website to hang. Frequent process restarts minimize the impact.

12:19am: Sergei gets an email back from Dyn saying that they’re still looking into the problem.

12:28am: Dyn calls to say they were able to fix everything. Sergei confirms. lb3 and lb4 are now fully utilized.

12:42am: Brian tweets that all systems are stable.

2:58am: Softlayer tweets that they’re about to run some code upgrades on the troubled router, which will cause some public network disruption.

4:00am:- A customer reports connectivity issues to rollbar.com

4:10am: Softlayer tweets that the troubled router is finally stable.

So what happened here?

Softlayer experienced a network outage, causing our servers in San Jose to be intermittently, then fully, unreachable
This triggered a DNS failover controlled by a stale Dyn configuration, which cascaded into a broken set of DNS records
After about 3 hours, our San Jose servers came back online, and about 30 minutes after that, the DNS issue was resolved.

What went right

We were notified of the problem by our backup monitoring service, Pingdom. (We’re using Nagios as our primary, but it runs inside of San Jose.)
Dyn’s DNS failover did work, even though wasn’t really supposed to be turned on. Our logs don’t show any large gaps in customer data being received.
A single machine (lb3) was able to handle all of our API traffic during the outage.
The API tier was able to handle a master-offline situation.
When San Jose came back online, data processing quickly caught up, notifications were sent, and the system was stable.
Our team came together, stayed mostly calm, and did everything we reasonably could to restore service as quickly as possible.

As a bonus, our Singapore and Amsterdam servers are now in service.

What went wrong

Parts of our service were unusable for a long period of time
- Notifications for new errors, etc. weren’t sent
- The web app didn’t load, and there was no maintenance page.
- status.rollbar.com didn’t show useful information
Even though the Softlayer private network was at least partially accessible, we couldn’t access it because we only had one way in (‘dev’, in San Jose).
The web tier got crushed trying to handle the API load with its old code.

Action items

In the short term (most of this will get done today):

1b. Set up a web server in a separate datacenter to serve a maintenance page.

1c. Add meta-level checks to status.rollbar.com. It currently gets data pushed from Nagios, but this isn’t helpful when San Jose is entirely unreachable.

2. Add another ‘dev’-like machine that we can use to administer servers, deploy code, etc. if San Jose is unreachable

3. Remove that old code, and make it an error if any API traffic hits the web tier.

And longer term:

1a. Add a host master standby in another datacenter for fast failover. If an episode like last night’s happens again, this will let us get notifications back online in a few minutes instead of a few hours.

1b. Set up a read-only web tier in another datacenter

Conclusion

We hope this was, if nothing else, an interesting look into our infrastructure, and to the journey of building a highly-available we service.

If you have any questions about the outage or otherwise, let us know in the comments or email us at support@rollbar.com

Happy Halloween

Wed, 31 Oct 2012 00:00:00 GMT

Y U No Use Rollbar?

asdf

– Happy Halloween from Rollbar.com!

If you don’t have an account yet, sign up here for early access.

Real-time Search for Exceptions and Errors

Wed, 24 Oct 2012 00:00:00 GMT

We’re happy today to announce the release of real-time search. You can now search your exceptions, errors, and log messages by title:

For exceptions, the title contains the exception class and message. For errors and log messages, it contains the entire message. It’s a full-text search that works best on whole words; we also do a few tricks with camelCase and underscore_separated terms.

The search index is kept up-to-date in real-time as new items are added to the system (that’s the “real-time” part). Typically the delay is ~2 seconds from receiving the input at our API to being in the index and searchable.

Current customers can try it out now; let us know if you run into any issues. What else would you like to see indexed?

If you don’t have an account yet, sign up here for early access.

Under the hood

We’re using the new Sphinx realtime features for indexing and querying. It’s currently running on a single dedicated machine (1 core, 2GB ram, 100GB local disk).

New items are indexed by a long-running script that indexes new items as they are inserted. (It keeps track of its location in the table and polls every second for new rows.) The index includes two full-text fields, title and environment, and two scalar attributes, status and level.

Title and environment don’t change, so we don’t need to update them. But status (active/resolved) and level (critical/error/warning/info/debug) do. We keep these in sync by simply writing to the search server whenever we update the primary database and whenever we modify our tokenizing algorithm.

Queries are routed through our API server, which returns the paged list of matching item ids that we can then use to filter with on our primary database, (in case the search results are out of date) and fetch the other data necessary for the results page (last occurrence, etc.)

Although our setup is straightforward, there were a few gotchas and lessons learned.

Infix queries

Sphinx’s realtime index does not currently support infix queries. That means that if you’re searching for “Error” then exceptions with titles like “ReferenceError” or “notfounderror” or even “(Error)” would not be found. To get around this, we index both the original title as well as another set of tokens that we’ve determined are useful for the lookup.

e.g. “#462 UnicodeEncodeError: ‘latin-1’ codec can’t encode character u’\u0441’ in position 71: ordinal not in range(256)”

gets tokenized and becomes

“#462 UnicodeEncodeError: ‘latin-1’ codec can’t encode character u’\u0441’ in position 71: ordinal not in range(256) can’t u0441 71 256 Unicode Encode Error latin-1’”

By tacking on these extra tokens, we are able to support most of the relevant infix searches our users are likely to make.

Sphinx + MySQL

Sphinx search comes with a super-handy feature that lets you connect, add and query the search index using a vanilla MySQL protocol. This is great for debugging and testing but comes with some caveats.

There are a lot of operations that SphinxQL does not yet support. One of the major ones is the lack of support for “OR” where_conditions and another is lack of a “COUNT(*)” method.

Since our API server is written in node, we were able to use the node-mysql library from Felix Geisendörfer. After plugging in the library, we noticed that the Sphinx server drops client connections fairly rigorously so we implemented a layer on top of the node-mysql library to handle reconnects, disconnects, etc… This has been great since it lets us perform maintenance on the Sphinx server without taking down our API server.

REPLACE

Lastly, we made sure that we were able to re-index our entire database into our Sphinx server by only using the REPLACE command when inserting new items. The docs mention that this can cause memory issues but since it’s so infrequent for our use-case, we haven’t run into any trouble and the benefit of re-indexing whenever we want more than makes up for it.

Using a Request Factory in Pyramid to write a little less code

Fri, 07 Sep 2012 00:00:00 GMT

At Rollbar.com, we’ve been using Pyramid as our web framework and have been pretty happy with it. It’s lightweight and mostly stays out of our way.

Pyramid doesn’t have a global request object that you can just import[1], so it makes you pass around request wherever you need it. That results in a lot of library code that looks like this:

# lib/helpers.py
def flash_success(request, body, title=''):
    request.session.flash({'body': body, 'title': title'})

and a lot of view code that looks like this:

# views/auth.py
@view_config(route_name='auth/login')
def login(request):
    # (do the login...)
    helpers.flash_success(request, "You're now logged in.")
    # (redirect...)

That is, there ends up being a lot of function calls that pass request as their first argument. Wouldn’t it be nicer if we could attach these functions as methods on request itself? That would save a few characters every time we call them, and let us stop thinking about whether request is the first or last argument. Pyramid facilitates this by letting us provide our own Request Factory:

from pyramid.request import Request

class MyRequest(Request):
    def hello(self):
        print "hello!"

def main(global_config, **settings):
    config = Configurator(settings=settings, request_factory=MyRequest)
    # ...

Now the request passed to our view methods, and everywhere else in our app, has our hello method.

So, what can we do with this that’s actually useful? In our codebase, we have a few convenience methods to get data about the logged-in user, flash messages, and check if features are enabled.

Here it is, unedited, in its entirety:

class MoxRequest(pyramid.request.Request):
    # logged-in-user access
    @util.CachedAttribute
    def user_id(self):
        from pyramid.security import authenticated_userid
        user_id = authenticated_userid(self)
        log.debug('authenticated user id: %r', user_id)
        return user_id

    @util.CachedAttribute
    def user(self):
        user_id = self.user_id
        if user_id:
            return model.User.get(user_id)
        return None

    @util.CachedAttribute
    def username(self):
        if self.user:
            return self.user.username
        else:
            return None

    def gater_check(self, feature_name):
        return self.registry.settings.get('gater.%s' % feature_name) == 'on'

    # flash methods
    def flash_success(self, body, title=''):
        self._flash_message(body, title=title, queue='success')

    def flash_info(self, body, title=''):
        self._flash_message(body, title=title, queue='info')

    def flash_warning(self, body, title=''):
        self._flash_message(body, title=title, queue='warning')

    def flash_error(self, body, title=''):
        self._flash_message(body, title=title, queue='error')

    def _flash_message(self, body, title='', queue=''):
        self.session.flash({'title': title, 'body': body}, queue=queue)

This just sits in our top-level __init__.py, along with the main() entry point.

Notes: @util.CachedAttribute contains this recipe. “Mox” is an easy-to-type codename, named after these mountains.

[1] I’m still not sold on this, but I’m getting by. It arguably causes problems with testing and such, but it is pretty nice to magically from flask import request.

Writing a simple deploy script with Fabric and @roles

Thu, 16 Aug 2012 00:00:00 GMT

I first heard about Fabric a couple years ago while at Lolapps and liked the idea of:

writing deployment and sysadmin scripts in a language other than Bash
that language being Python, which we used everywhere else

but we already had a huge swath of shell scripts that worked well (and truth be told, Bash isn’t really that bad). But now that we have at clean slate for Rollbar, Fabric it is.

I wanted a simple deployment script that would do the following:

check to make sure it’s running as the user “deploy” (since that’s the user that has ssh keys set up and owns the code on the remote machines)
for each webserver:
git pull
pip install -r requirements.txt
in series, restart each web process
make an HTTP POST to our deploys api to record that the deploy completed successfully

Here’s my first attempt:

import sys

from fabric.api import run, local, cd, env, roles, execute
import requests

env.hosts = ['web1', 'web2']


def deploy():
    # pre-roll checks
    check_user()

    # do the roll.
    update_and_restart()

    # post-roll tasks
    rollbar_record_deploy()


def update_and_restart():
    code_dir = '/home/deploy/www/mox'
    with cd(code_dir):
        run("git pull")
        run("pip install -r requirements.txt")
        run("supervisorctl restart web1")
        run("supervisorctl restart web2")


def check_user():
    if local('whoami', capture=True) != 'deploy':
        print "This command should be run as deploy. Run like: sudo -u deploy fab deploy"
        sys.exit(1)


def rollbar_record_deploy():
    # read access_token from production.ini
    access_token = local("grep 'rollbar.access_token' production.ini | sed 's/^.* = //g'",
        capture=True)

    environment = 'production'
    local_username = local('whoami', capture=True)
    revision = local('git log -n 1 --pretty=format:"%H"', capture=True)

    resp = requests.post('https://api.rollbar.com/api/1/deploy/', {
        'access_token': access_token,
        'environment': environment,
        'local_username': local_username,
        'revision': revision
    }, timeout=3)

    if resp.status_code == 200:
        print "Deploy recorded successfully"
    else:
        print "Error recording deploy:", resp.text

Looks close-ish, right? It knows which hosts to deploy to, checks that it’s running as deploy, updates and restarts each host, and records the deploy. Here’s the output:

$ sudo -u deploy fab deploy
(env-mox)[brian@dev mox]$ sudo -u deploy fab deploy
[sudo] password for brian:
[web1] Executing task 'deploy'
[localhost] local: whoami
[web1] run: git pull
[web1] out: remote: Counting objects: 8, done.
[web1] out: remote: Compressing objects: 100% (4/4), done.
[web1] out: remote: Total 6 (delta 4), reused 4 (delta 2)
[web1] out: Unpacking objects: 100% (6/6), done.
[web1] out: From github.com:brianr/mox
[web1] out:    c731b57..1d365e0  master     -> origin/master
[web1] out: Updating c731b57..1d365e0
[web1] out: Fast-forward
[web1] out:  fabfile.py |    8 ++++----
[web1] out:  1 file changed, 4 insertions(+), 4 deletions(-)

[web1] run: pip install -r requirements.txt
[web1] out: Requirement already satisfied (use --upgrade to upgrade): Beaker==1.6.3 in /home/deploy/env-mox/lib/python2.7/site-packages (from -r requirements.txt (line 1))

[web1] out: Cleaning up...

[web1] run: supervisorctl restart web1
[web1] out: web1: stopped
[web1] out: web1: started

[web1] run: supervisorctl restart web2
[web1] out: web2: stopped
[web1] out: web2: started

[localhost] local: grep 'rollbar.access_token' production.ini | sed 's/^.* = //g'
[localhost] local: whoami
[localhost] local: git log -n 1 --pretty=format:"%H"
Deploy recorded successfully. Deploy id: 307
[web2] Executing task 'deploy'
[localhost] local: whoami
[web2] run: git pull
[web2] out: remote: Counting objects: 8, done.
[web2] out: remote: Compressing objects: 100% (4/4), done.
[web2] out: remote: Total 6 (delta 4), reused 4 (delta 2)
[web2] out: Unpacking objects: 100% (6/6), done.
[web2] out: From github.com:brianr/mox
[web2] out:    c731b57..1d365e0  master     -> origin/master
[web2] out: Updating c731b57..1d365e0
[web2] out: Fast-forward
[web2] out:  fabfile.py |    8 ++++----
[web2] out:  1 file changed, 4 insertions(+), 4 deletions(-)

[web2] run: pip install -r requirements.txt
[web2] out: Requirement already satisfied (use --upgrade to upgrade): Beaker==1.6.3 in /home/deploy/env-mox/lib/python2.7/site-packages (from -r requirements.txt (line 1))

[web2] out: Cleaning up...

[web2] run: supervisorctl restart web1
[web2] out: web1: stopped
[web2] out: web1: started

[web2] run: supervisorctl restart web2
[web2] out: web2: stopped
[web2] out: web2: started

[localhost] local: grep 'rollbar.access_token' production.ini | sed 's/^.* = //g'
[localhost] local: whoami
[localhost] local: git log -n 1 --pretty=format:"%H"
Deploy recorded successfully. Deploy id: 308

Done.
Disconnecting from web2... done.
Disconnecting from web1... done.

Lots of good things happening. But it’s doing the whole process - check_user, update_and_restart, rollbar_record_deploy - twice, once for each host. The duplicate check_user just slows things down, but the duplicate rollbar_record_deploy is going to mess with our deploy history, and it’s only going to get worse as we add more servers.

Fabric’s solution to this, described in their docs, is “roles”. We can map hosts to roles, then decorate tasks with which roles they apply to. Here we replace the env.hosts declaration with env.roledefs, decorate update_and_restart with @roles, and call update_and_restart with execute so that the @roles decorator is honored:

import sys

from fabric.api import run, local, cd, env, roles, execute
import requests

env.roledefs = {
    'web': ['web1', 'web2']
}


def deploy():
    # pre-roll checks
    check_user()

    # do the roll.
    # execute() will call the passed-in function, honoring host/role decorators.
    execute(update_and_restart)

    # post-roll tasks
    rollbar_record_deploy()


@roles('web')
def update_and_restart():
    code_dir = '/home/deploy/www/mox'
    with cd(code_dir):
        run("git pull")
        run("pip install -r requirements.txt")
        run("supervisorctl restart web1")
        run("supervisorctl restart web2")


def check_user():
    if local('whoami', capture=True) != 'deploy':
        print "This command should be run as deploy. Run like: sudo -u deploy fab deploy"
        sys.exit(1)


def rollbar_record_deploy():
    # read access_token from production.ini
    access_token = local("grep 'rollbar.access_token' production.ini | sed 's/^.* = //g'",
        capture=True)

    environment = 'production'
    local_username = local('whoami', capture=True)
    revision = local('git log -n 1 --pretty=format:"%H"', capture=True)

    resp = requests.post('https://api.rollbar.com/api/1/deploy/', {
        'access_token': access_token,
        'environment': environment,
        'local_username': local_username,
        'revision': revision
    }, timeout=3)

    if resp.status_code == 200:
        print "Deploy recorded successfully"
    else:
        print "Error recording deploy:", resp.text

Here’s the output:

(env-mox)[brian@dev mox]$ sudo -u deploy fab deploy
[sudo] password for brian:
[localhost] local: whoami
[web1] Executing task 'update_and_restart'
[web1] run: git pull
[web1] out: Already up-to-date.

[web1] run: pip install -r requirements.txt
[web1] out: Requirement already satisfied (use --upgrade to upgrade): Beaker==1.6.3 in /home/deploy/env-mox/lib/python2.7/site-packages (from -r requirements.txt (line 1))

[web1] out: Cleaning up...

[web1] run: supervisorctl restart web1
[web1] out: web1: stopped
[web1] out: web1: started

[web1] run: supervisorctl restart web2
[web1] out: web2: stopped
[web1] out: web2: started

[web2] Executing task 'update_and_restart'
[web2] run: git pull
[web2] out: Already up-to-date.

[web2] run: pip install -r requirements.txt
[web2] out: Requirement already satisfied (use --upgrade to upgrade): Beaker==1.6.3 in /home/deploy/env-mox/lib/python2.7/site-packages (from -r requirements.txt (line 1))

[web2] out: Cleaning up...

[web2] run: supervisorctl restart web1
[web2] out: web1: stopped
[web2] out: web1: started

[web2] run: supervisorctl restart web2
[web2] out: web2: stopped
[web2] out: web2: started

[localhost] local: grep 'rollbar.access_token' production.ini | sed 's/^.* = //g'
[localhost] local: whoami
[localhost] local: git log -n 1 --pretty=format:"%H"
Deploy recorded successfully. Deploy id: 309

Done.
Disconnecting from web2... done.
Disconnecting from web1... done.

That’s more like it. Since env.hosts is not set, the undecorated tasks just run locally (and only once), and the @roles('web')-decorated task runs for each web host.