Airbrake Status

Problems processing AMEX cards. [RESOLVED]

Tue, 11 Oct 2011 15:14:00 -0400

UPDATE: 13:34 PST. Amex payments are back online.

We’re working with the payment provider to get the service back online as soon as possible. If you’re having problems making a payment today, please try again tomorrow. If you have any urgent queries please contact support@airbrakeapp.com

500 Errors. [RESOLVED]

Thu, 15 Sep 2011 09:10:00 -0400

We have been seeing an abundance of 500 errors on our web app. We have issolated this to a bug with our cookie-based session storage. We are in the process of moving to a memcache-based system.

This error hasn’t affected error collection, and if you see a 500 error, hit refresh and you should get the page you were looking for.

We will update this post once we now the issue is resolved.

UPDATE. We have updated to memcache for cookie storage, this has fixed many of the 500 errors, but if you see any please report them to @airbrakeapp on twitter.

RESOLVED. We’ve resolved the 500 issue; we’ve reviewed our process for handling errors and will be launching a status page for realtime updates.

Email notifications are down.

Fri, 02 Sep 2011 11:40:00 -0400

We are encountering some unexpected downtime, after an update to our SendGrid account to our new domain. We have updated our SMTP settings and are working on sending out all e-mails from the cue. I will keep you posted on developments.

Apologies for the downtime, you can still see exceptions from the AirBrake online app.

[UPDATE: We are Back. Friday 18:50 GMT. Please check your account as some email notifications will have been lost. ]

Service is operating normally.

Thu, 19 May 2011 17:04:33 -0400

Working to restore access

Thu, 19 May 2011 14:36:00 -0400

We’ve been notified of an application error, and are working to get it resolved asap. We will update here as things progress.

UPDATE: All access restored.

Service is operating normally.

Thu, 12 May 2011 13:05:01 -0400

Database Rebuilding

Thu, 12 May 2011 09:04:00 -0400

We’ve completed the work on our main database from last night’s troubles, and we’re going to be cutting back over to the main database this morning. All of the error data that was accepted since about 5pm Eastern last night is in a temporary database, which we are going to feed into the main database in a few minutes.

We are going to allow the main database to take over, which means some of the very newest error details will be unavailable until our work is complete. Don’t worry, though, it’s all still there and will be restored ASAP.

UPDATE: We’ve readded the data, so everything should look fine. As always, if you notice problems, please let us know at http://help.hoptoadapp.com.

Troubles at the datacenter

Wed, 11 May 2011 15:38:00 -0400

We are currently troubleshooting an issue with database connectivity. We will post back here when we’ve been able to repair and solve the situation.

UPDATE: One of our database servers is currently being repaired after a hard crash. To restore POSTs to our API, we’ve provisioned a new database server and temporarily pointed the app at it. Some data may not be viewable from the UI until we are able to restore the data.

Service is operating normally.

Thu, 28 Apr 2011 13:41:49 -0400

Unintended outage

Thu, 28 Apr 2011 13:01:00 -0400

We’ve been notified of an outage on the web front end of the site. We are currently looking at the issue and identifying the cause. We will get the site back online ASAP, and post back here when we know more.

UPDATE: We’re back. There was an outage in our mongodb replica set arbiter. We are putting better monitoring and recovery in place. Thanks for your patience

Service is operating normally.

Thu, 21 Apr 2011 07:54:13 -0400

Troubles at the datacenter

Thu, 21 Apr 2011 07:39:00 -0400

We have one database server that went down overnight. We are working to get it back up and should be back to normal very shortly.

This only affects the web interface. We are still collecting errors, and still sending you notifications.

We’ll update here when everything is back to normal.

[UPDATE] Everything is back to normal

Service is operating normally.

Mon, 18 Apr 2011 12:46:45 -0400

Database maintenance

Mon, 18 Apr 2011 12:14:00 -0400

We will be performing database maintenance at 12:30EST. We expect this to take between 1 and 15 minutes. We’ll post back with any updates.

12:32PM EST: Maintenance page is up. Restarted database.
12:33PM EST: Enabled app, application back online.

Service is operating normally.

Tue, 12 Apr 2011 18:09:41 -0400

Uprades in the data center

Tue, 12 Apr 2011 17:44:00 -0400

We’re upgrading the ruby running on the app, so we will be back shortly.
Sorry for the inconvenience.

UPDATE: We’re back, upgraded to latest REE and passenger versions.

Service is operating normally.

Thu, 07 Apr 2011 16:35:33 -0400

Quick Downtime for Maintenance

Thu, 07 Apr 2011 15:36:00 -0400

We are currently doing maintenance on our MongoDB cluster. We’ve taken precautions to keep downtime to a minimum, and should be back online very shortly.

[UPDATE] - we’re back. Things took a bit longer than we expected. Thanks for your patience!

Issues with exception details

Wed, 06 Apr 2011 15:47:00 -0400

We’re currently dealing with unavailability of error details, and will post back as soon as we find the fix. We’re sorry for the trouble. We will post back shortly.

[UPDATE] We have encountered the issue. Your data is safe and available! However, if it still does not show up for you, please ping us at http://help.hoptoadapp.com and we will speed up the process for your account. We are currently processinig all accounts to resolve this issue.

Service is operating normally.

Tue, 05 Apr 2011 16:55:03 -0400

After a few fires that we had to deal with, the data center migration is complete and the app is running strong. Our response time is faster than ever, and traffic keeps rising!

Thanks for your patience during this time. We look forward to continue to improve the reliability and usefulness of the service!