Papertrail

The very first thing that I do when creating a new Heroku product is to provision the free Papertrail add-on.

What’s the problem?

We have many applications running in the cloud at any given time, most of which are provisioned with several smaller parallel servers, each of which log.  Most logs we only need for research after the fact, some of them we want to be monitored and to alert us when exceptional conditions occur.

Why Papertrail?

There may be better products out there, but we’ve been using Papertrail for many years successfully and have no particular reason to change.  If you know of something superior, please let me know!  Having a single, standardized log destination makes life simpler, and simple is good. Its also not an area that I want to innovate in; we don’t typically process millions of log lines per hour, so a basic solution works well.

If you host your backend with Heroku, you can add a Papertrail for free (up to 10MB of logs per day) with a simple click in the add-ons section.  That’s a good place to start.

Filters and Alerts

Once an application is up and running, I like to remove normal, boring responses from the logs.  This has the effect of increasing the log size available for the account, as well as making the remainder easier to read.  Under settings > settings, I generally have the following filters set.  First a plain string:

status=200 bytes=

And second a regex:

API Success 200.*OPTIONS

Between those, I still have visibility into all of the API calls being served without reams of unneeded chaff.

On the Alerts tab, we use the standard Slack integration. For our Java backends, we set up an exception alert:

exception -("javax.ws.rs.ForbiddenException" OR "javax.ws.rs.NotAcceptableException" OR newrelic)

And a more standard Heroku alert:

"error code=H" OR "Error R" OR "Error L" OR "warning code=H27"

Between the two of them we catch most everything that would otherwise surprise us.  I'll talk more in another post about how we structure the messages that we write out for easy reference.

Is there a better way?

To be honest, I’ve been meaning for a couple of years now to sign up for an external Papertrail account and use it to link all of our projects together.  This would make adding a new system simpler, in theory, since we could configure the alerts once.  It should also mean that we could keep two windows open to the same Papertrail account but have different applications in each, something that the current approach doesn’t work with.

In reality, the fact that you can only tail one application’s logs at a time hasn’t been a huge issue, and the one-click addition (plus the natural SSO use of Papertrail for external clients who have application access though Heroku) has outweighed any theoretical benefits of a single account.  Until that’s no longer the case, we’ll just keep one-click adding.