“It’s a good problem to have.”

“…but not when you’re having it.”

We were offered this advice by many different people when discussing the scaling issues we may run into when creating such a distributed web application.

The idea is that it would be a good problem because it would surely indicate that we were popular, thriving, and thus successful. But it isn’t a good problem for all of you that are users of Disqus. It isn’t a good problem for us at Disqus, as we’re building our service based on trust, performance, and reliability.

We had some downtime this morning, starting at 7am PST, for a little under 30 minutes.
We are sorry. I am personally sorry. Being a popular service is not a valid excuse for being unavailable when other websites depend on you.

So, what are we doing about it? Lots. We have spent a lot of time recently optimizing and tweaking. Disqus is in transition-mode to upgrading our web and database servers (plus a few other misc. infrastructure points).

Bonus apology: This week, email us with the subject “Downtime makes me sad” and we’ll send over some cool Disqus swag. Be sure to indicate your blog/website and share a sentence or two about your experiences with Disqus.

Thanks all — it’s not a good problem to have and we’ll do our best to make sure it doesn’t become a problem you have to deal with.

Daniel on April 23rd 2008 in disqus

  • Glitches and growing pains will be inevitable. So what if my comments are unavailable for 30 mins. That's OK.

    The world won't stop spinning and I won't get innundated with email complaints. Although you might want to move Fred Wilson onto a dedicated server :-)

    At least, you come out with a prompt status update and an apology including the word 'Sorry' (in bold).

    Those guys from Twitter should take note.

    Onwards and upwards !
  • you guys are awesome. keep up the great work and great product.
  • Having problem is one aspect. The other aspect is acknowledging the problem and admitting it. All other problems just get washed off once solving the problem becomes your priority. Great going...We have the faith...Downtime is manageable; Reputation is not...You have managed your reputation...Downtime will automatically be managed. That's how the innovations about scalability and reliability will be taken care of.
  • Free stuff is good. I'm a dev, so I understand downtime, but apologies do mean a lot. So thanks :)
  • Sounds great guys, keep up the good work. No need for an apology, although the cool Disqus swag might not be too shabby =D
  • Thanks for the update and the swag. No big deal about the downtime.

    =]
  • Downtime is almost always harder on you than it is on your customers. Take it seriously, but don't take it too hard; hopefully you're learn something and improve from each one.

    It's going to happen again in some form or another; the best you can do is be prepared (backups, redundancy, etc..) and make sure you're always designing with reliability and scalability in mind.

    Keep up the good work!
  • i just want shwag even ythough I am happy!
  • you guys are awesome. keep up the great work and great product.
  • Thanks for the update and the swag. No big deal about the downtime.

    =]
  • Yeah, Downtime is no real problem, if you inform the users and try to fix it as soon as possible.
  • i just want shwag even ythough I am happy!
  • The one thing I've learned from several way-too-long downtime issues is to have a separately-hosted site for status updates. And make sure everyone knows that site.

    The first major downtime we had at my old old company, everything was hosted in the same facility, including email, and there was ZERO way to get the word out. Except to call people... which is what we had to do.

    I learned that as long as you're really good about communicating... then people give you a break.
  • Yeah, Downtime is no real problem, if you inform the users and try to fix it as soon as possible.
  • When a company runs into technical problems, there are the technical issues, and then there are the business issues. In two cases today (the other one being Aaron Forgue's GroupTweet issue), the principals took immediate and decisive action from both a technical and a business standpoint, letting their customers know that they had a problem, they were very sorry about the problem, and that they were going to make right with their customers.

    Andy is right; there's another problem-plagued company that should be taking note.

    I've written about this at http://mrontemp.blogspot.com/2008/04/theres-sup...
  • Sounds great guys, keep up the good work. No need for an apology, although the cool Disqus swag might not be too shabby =D
  • What? No comments for 30 minutes. Its all good. Everything is recovered and these growing pains are good ones to have!

    Plus, who doesn't like free schwag?
  • Hi
    How we do "dısqus comments" system translation TURKISH language
  • Hi
    How we do "dısqus comments" system translation TURKISH language
  • Aw, sorry I missed the swag offer earlier. I was, indeed, sad! But I know you're working on it and I appreciate your work.
  • Hey, guys u are doing very good.........keep up the good work........
  • Hey, guys u are doing very good.........keep up the good work........
  • Hey, guys u are doing very good.........keep up the good work........
  • It is inevitable for popular service to be experiencing downtime, but not too frequently.
  • It is inevitable for popular service to be experiencing downtime, but not too frequently.
  • SEO
    It is inevitable for popular service to be experiencing downtime, but not too frequently.
  • nice article! nice site. you're in my rss feed now ;-)
    keep it up
  • nice article! nice site. you're in my rss feed now ;-)
    keep it up
  • Sorry is the hardest word to say but then if it is easier said, then its won't be happy problem.
  • Nobody is perfect. Everybody makes mistake or fumble sometimes. Important thing is to go into damage control mode and make sure it does not happen again.
  • Nobody is perfect. Everybody makes mistake or fumble sometimes. Important thing is to go into damage control mode and make sure it does not happen again.
  • Nobody is perfect. Everybody makes mistake or fumble sometimes. Important thing is to go into damage control mode and make sure it does not happen again.
  • SEO
    Everything has some pros and cons and if people are finding any problem with Discus than it doesn’t sound new.. But I guess if they have apologized than nothing is more better than this.. I used it and find it awesome as it gives us a new and easy way to get connected with lots of people, make interaction with them and then easily post blogs in your related domain or interest..
  • Carole J Takeshita
    great article!, grats for u site :)
  • Patricia W Olson
    tu blog es excelente! te mando 454 felicitaciones!
  • Patricia W Olson
    tu blog es excelente! te mando 454 felicitaciones!
  • ezditto
    I'm glad that downtime dont' happen frequently
  • Duple Ang
    Hope that you all did manage this well!
  • replicaloh
    I think nobody will blame you all for the downtime with the bonus apology in return!
  • Jasmine Lim
    the bonus apology seems cool!
blog comments powered by Disqus