Disqus experienced a database error that resulted in a number of people unable to access the service. This lasted for about 45 minutes until we were able to get everything consistent again. While we don’t have a cute aquatic mammal for this situation, we do have this comparably mundane (and brief while hopefully thorough) post describing what’s up.

Why it happened

Some of the heaviest hits on the database come from the widgets — particularly the “Popular Threads” widget. We use memcached for this relatively expensive query, but an expired cache combined with a spike in traffic will cause a dog-pile effect.

From highscalability.com:

Data freshness requires a refrigeration truck or an expiry time on your cache entry that causes stats to be periodically recalculated. Now, what happens when your cached data expires and a 1000 requests simultaneously try to recalculate the expensive to calculate data? Database load spikes and the world nearly ends.

The world didn’t end, but it doesn’t make for a happy Friday morning.

Why it won’t happen again

Ok, that’s a bit presumptuous. Let me restate that.

What we’re doing to prevent it from happening again

Scaling is not a problem we’re only beginning to address. It has always been a priority ahead of everything else. We’re building out a service that you can depend on, not just a product with fancy features. Fortunately, we know how to address the issues.

We are implementing MintCache, which will alleviate some of the woes around the dog-pile effect. This past week was spent scrutinizing and improving the core architecture. And by next week, we hope to finish scaling out the database servers.

With the new help of Andrew and Devin, we’ll be building out Disqus faster than ever. I’m very proud of what the team has been up to lately and I hope you all will be able to directly experience the results.

Communication

You rely on us to provide a vital component of your blog or website. We don’t ever forget that and we realize we let many of you down this morning. I apologize. As always, if you don’t feel comfortable with Disqus, you can export your data from us. We are working hard to make sure you feel comfortable with our service, so please voice your concerns.

We’re listening. We monitor tweets, email, and blog posts. So let us know the good and bad. Just like you, we want Disqus to be great.