UPDATE: 13:34 PST. Amex payments are back online.
We’re working with the payment provider to get the service back online as soon as possible. If you’re having problems making a payment today, please try again tomorrow. If you have any urgent queries please contact support@airbrakeapp.com
We have been seeing an abundance of 500 errors on our web app. We have issolated this to a bug with our cookie-based session storage. We are in the process of moving to a memcache-based system.
This error hasn’t affected error collection, and if you see a 500 error, hit refresh and you should get the page you were looking for.
We will update this post once we now the issue is resolved.
UPDATE. We have updated to memcache for cookie storage, this has fixed many of the 500 errors, but if you see any please report them to @airbrakeapp on twitter.
RESOLVED. We’ve resolved the 500 issue; we’ve reviewed our process for handling errors and will be launching a status page for realtime updates.
We are encountering some unexpected downtime, after an update to our SendGrid account to our new domain. We have updated our SMTP settings and are working on sending out all e-mails from the cue. I will keep you posted on developments.
Apologies for the downtime, you can still see exceptions from the AirBrake online app.
[UPDATE: We are Back. Friday 18:50 GMT. Please check your account as some email notifications will have been lost. ]
We’ve been notified of an application error, and are working to get it resolved asap. We will update here as things progress.
UPDATE: All access restored.
We’ve completed the work on our main database from last night’s troubles, and we’re going to be cutting back over to the main database this morning. All of the error data that was accepted since about 5pm Eastern last night is in a temporary database, which we are going to feed into the main database in a few minutes.
We are going to allow the main database to take over, which means some of the very newest error details will be unavailable until our work is complete. Don’t worry, though, it’s all still there and will be restored ASAP.
UPDATE: We’ve readded the data, so everything should look fine. As always, if you notice problems, please let us know at http://help.hoptoadapp.com.
We are currently troubleshooting an issue with database connectivity. We will post back here when we’ve been able to repair and solve the situation.
UPDATE: One of our database servers is currently being repaired after a hard crash. To restore POSTs to our API, we’ve provisioned a new database server and temporarily pointed the app at it. Some data may not be viewable from the UI until we are able to restore the data.
We’ve been notified of an outage on the web front end of the site. We are currently looking at the issue and identifying the cause. We will get the site back online ASAP, and post back here when we know more.
UPDATE: We’re back. There was an outage in our mongodb replica set arbiter. We are putting better monitoring and recovery in place. Thanks for your patience
