Something is not right, and the last 17 hours have been a blogging MESS.
Digg brought my site to its knees yesterday - and somehow today it still can’t get back up and running consistently.
16 days ago, I had another post on the front page of Digg - the site didn’t even hiccup. So I’m working hard with a few people and my hosting company to figure out what the heck broke so we can fix it, because I haven’t made any site changes between my last Digg and this Digg.
Take a look at the following visitor numbers and tell me something. Do you think that yesterday’s traffic would have killed your server?!
Anyway, the irony of all of this happening because of my post The Top 10 Blogging Mistakes I Made in My First Year is rather humbling. 
Yep - we’ll all have a good laugh when I write up my next post once we fix it - something like, “How to Hand the Trolls on Digg the Perfect Fodder to Rip You a New One”.
Sparkplugging Founder Wendy Piersall is dang passionate about helping people start & grow a business while maintaining life balance (somehow).
Dawud Miracle has one focus: to help you get it. The it? How your website, and blog, can change the way you do business. You can find out more at 
Wendy,
My server went down from Stumbleupon traffic the other day. I was ecstatic about the traffic, but it made me realize that I needed more server bandwidth!
I’m about to make an upgrade, but I have no real clue how much more bandwidth I’ll need. I’ll probably go with the next level up from my present one which will be 4 times the space, and see how that does during the Digg/Stumble onslaughts!
Glad to see you back up again! It sure took a while…
I pray that one day I get so many visitors that my (or Typepad’s) servers crash too
Ahhh, glad to see you up and running again. I hope you find out the culprit and report it here. I’m also curious to see why your site went down this time, but not the last time you were dugg.
Funny thing, I just read this quote from Pavlina (at Daily Blog Tips) and had to laugh:
“I appreciate Digg’s traffic swells, but a flood of sarcastic teenagers isn’t a great fit for a blog on personal development.”
Being dugg is overrated anyway. It’s just a bunch of drunken, horny frat boys stampeding en masse to your site, pissing all over your blog, nuking the server and then leaving never to return again, while they throw obnoxious barbs from afar in the Digg comments section. Not really conducive to developing a mature readership if you ask me.
Aaron, it’s not bandwidth you have to concern yourself with, it’s the amount of CPU resources your website uses.
When hosting services offer you terrabytes and terrabytes of bandwidth, they’re being full of it, because it’s virtually impossible to use up all that bandwidth without taxing the server’s CPU so much that your account gets suspended as a result. That’s the limitation of shared hosting unfortunately, and it’s a policy that all hosting services follow to some degree. If you want to find out what the CPU quota is for a hosting provider, it’s usually buried in their Terms of Service agreement. Some are more flexible than others, but even then how well your website can stand a traffic spike can depend on other extenuating circumstances, such as who else is also sharing the server your site is on.
Dedicated hosting on the other hand is really not justified unless you are consistently getting the kind of traffic that would normally cause your shared hosting account to shut down. So it’s a matter of finding workarounds where your site can withstand massive traffic spikes at least temporarily until your 15 minutes of fame are up. Since I’m using Wordpress, I use WP-Cache and Digg Defender as preventative measures to avoid seeing my blog go up in smoke. Unfortunately I’ve never been dugg or slashdotted, so it remains to be seen how effective this will be. :-O
Lincoln,
Wow, that’s some great information - I had no idea. Earlier today when I tried to access this post I got an error message that mentioned the CPU.
Much obliged for the extra info. I’ll definitely look into the TOS of my service provider and see what my options are. I had considered dedicated hosting before, but didn’t want to get it unless it was going to be necessary.
Thanks again for the tip!
Yeah, Aaron - I’ve learned more than I ever wanted to know about hosting this weekend.
I’ll definitely post about the lessons learned (as I always do!!) once I get things straightened out.
Currently, I’m deciding between a virtual private server and a dedicated server - or even buying my own server. The move will be a pain, but I just can’t have this kind of down time ever again!
I’m glad to see you’re back up Wendy.
@Aaron: You’re welcome.
The day I get dedicated rejoicing will also be the day I’ll be running down the highway screaming, “I made the big times! I made the big times!!!!!” simply because it would mean I’m finally getting decent traffic to my site. 
Lincoln - thanks for reminding me of the bright side - I’ve been stressing all weekend when I should be celebrating!
@Lincoln - Looking forward to celebrating that day of you running down the street screaming!
@Wendy - I will look forward to your after action review! In fact, I’ll wait until then to make any final decisions about my own upgrades. Any Digg/Stumbleupon onslaughts in the meantime will just have to deal with my present server capacity!
I wondered what happened with the site I had tried to access it four times yesterday and I got this not so lovely grey screen with some garbeld CPU stuff. Good to know it is back up BTW going to plug myself here and say please check out my main site Monday. That is all I can say it is a surprise from here.
It certainly can be frustrated however I’m sure you will be able to learn something of value from this and bring it to the forefront for everyone else to learn from.
Diggin it!
You know what they say? Misery loves company. I’m so glad that I checked here before upgrading my server. I was having that CPU crapola thing going on too. Lincoln, I cannot thank you enough for the tips. I headed right over and installed those plugins.
It’s nice to know my traffic is picking up, but I sure hated getting that error.
Hi Wendy,
I recently published an article about web hosting, after my own site was down for quite a while.
Who do you host with? I’ve been thinking about switching to a dedicated virtual (dv) server with Media Temple, similar to Nate Whitehill and Mark Boulton.
Hey Guys,
We just moved to a DV server on http://www.HostDime.com for this very reason.
What’s hurting you isn’t bandwidth, that would only get you if you were serving large videos direct from your server (not talking even YouTube framing here).
Your problem is system load on the web server - memory, CPU, etc. And on a shared server, if you hog too much, they’ll take you down fast since they don’t want all the other 97 sites on there going down too.
We just moved to a new host - a Dedicated Virtual (DV) server actually - a very powerful box split into virtual servers for just a few users. That means a flood of traffic MAY cause you to miss a few beats, but they shouldn’t shut you down for it. Oh, and you get a lot more CPU power.
I wouldn’t go with MediaTemple. Not just because of the recent bad press, it’s not about their DV systems, but rather about the brand-spankin’-new shared “grid” plan.
OK - we just researched A LOT from everyone we knew, and found what I think is the BEST deal for a DV system.
We just signed up with HostDime.com 2 wks ago, after a good from who’s been with them for years recommended them highly. He gets on the Digg homepage now and then and sees no issues.
The HostDime guys have been SO awesome and helpful, and migrated all our stuff over AND configured it the way we wanted - all with no setup fee. And it’s the best VPS deal anywhere. Go give them a try.
http://www.hostdime.com/services/vps/
We also learned some very hard optimization lessons (read: how NOT to shoot yourself in the foot and lose $1,000 in a single day with stuff like your OpenAds server). More on that in an article soon on our new Pro Blogging Insider site - BlogSuccessJournal.com.
But for a shared host, we recommend HostGator - just for web hosting, let’s not bundle email here. We host our email on Webmail.us, but that’s a different deal. We’re talking about having a cheap and rock solid WEB/BLOG hosting plan. Check out our review of them on our new site. We’ve had 30,000 hits to our site in a single day on that $10/month shared plan, and no issues. Pretty cool.
Have an awesome day!
Dan
Thanks for the tips on DV hosting. I’ll take a look at HostDime.
I’ve read a fair bit of Media Temple’s bad press for the grid system, but have also seen good reviews for their DV, and it’s cheaper than HostDime. I know you usually get what you pay for though.
@Gayla: You’re welcome!
Funny, I’m also using Hostgator, and the same ten dollar plan too, LOL. Good to see they held up under a massive traffic load, so I feel a lot better now about going with them. I have HostDime bookmarked for the future. Thanks for the tip!
I enjoyed your post and i wish my server could crash. It seems that i bump into good stories all the time but there already in digg. Stumble is a great way to boost some traffic to your personal site but i have to agree with @Lincoln once you had all that traffic there all gone and rarely comes back. The best thing for us bloggers is to just keep posting and readers will come.
Wendy - nothing like a little Friday afternoon stress!! I have been on HostGator for about a year now and have been very happy with their service. I have a reseller plan so I can allocate more resources to certain sites as I need to.
Guys, just to emphasize again - the problem you’re seeing is a limitation of the system RAM and CPU.
You don’t have the option to re-allocate those, even with a hostgator reseller account - that just allows you to re-distribute disk space and bandwidth. I went through all that with HostGator, and even their “semi dedicated” $75/mo plan is susceptible to these issues and getting your accunt turned off for hogging resources. it happened to us.
When you start getting REAL traffic, it’s time to consider a dedicated server (going to be $150/mo minimum) or the much better interim option of a “dedicated virtual” (DV) server.
@ David,
It’s not apples to apples as it appears. As we just moved to our brank-spankin’ new DV server, we looked very closely at both HostDime and Media Temple before going with them.
Media Temple APPEARS to be cheaper - but gives you half the RAM. For $50/mo they give you a DV with 256 MB RAM. But HostDime gives you 512 MB of RAM for $60.
256 won’t be enough for very much, since this is a complete virtual system. This means you’ll be running your own Apache web server, your own MySQL database, and so on - all in your RAM alocation.
And to get 512 MB at Media Temple, you would pay $100/mo.
And watch out - MT DOES give you more disk space and bandwidth, but that’s not what you need more of - it’s the CPU and RAM. Even with our explosive growth the last months, we haven’t come close to reaching the HostGator bandwidth or disk limits.
What you have no way of knowing is exactly how much of the CPU slice you get from that large server… they can’t really quantify that for you.
Anyway, check both out and compare the points. We went with HostDime because it made more financial sense and they came highly recommended. And we recommend them to you guys.
Best of luck.
And feel free to ask us if you have any questions on the DV stuff and how it’s working for us; we’ll be happy to help.
Have an awesome day!
Dan & Jennifer
What’s considered real traffic though? 1000 visits a day? 10,000? 5 billion?
@ Lincoln ~ These days I’m feeling like it’s about 10 at once. *sighs*
Hey Lincoln,
Sorry, poor choice of words on my part.
Jennifer and I approached the server upgrade stuff this way: “if it ain’t broken, don’t fix it”. We kept our $10/mo hostgator account until we outgrew it on one magical day.
Fact is, this’ll be different for everyone, depending on what plugins you use, what traffic sources you have, etc.
But one day - which would have been our highest traffic day EVER with estimated 50,000 visitors and projected $1,000 in revenue - we were down pretty much the whole day. Our shared hostgator account crashed hard in the morning and we were down all day until they got us migrated to a semi-dedicated server (don’t do that btw - go with a real DV anyday - that’s what we did).
So that’s when we decided it was time to move somewhere to a bigger server. But that’s got a lot of complexities of it’s own to deal with.
Ultra-techie-speak warning below this line…
Bloggers are particularly hard hit by performance limitations on servers because we’re all running WordPress. That’s written in PHP, which means that for EACH hit to your site, your server is literally compiling dozens of PHP files. LOTS of overhead. so on our new server, we had a PHP object cache software installed into Apache, so now the compiled PHP code is kept after the 1st execution, and served up. Basically cut our server load by 90%. But that’s the kind of stuff you can only do on your own server, whether dedicated or DV.
Have an awesome day!
Dan
@Gayla: LOL, just remember Rome wasn’t built in a day.
so on our new server, we had a PHP object cache software installed into Apache, so now the compiled PHP code is kept after the 1st execution, and served up.
Say what now?
This is one of the few things that concerns me when looking into a VPS or dedicated solution. I really hope I’m not expected to know this stuff on my own and that the hosting provider will be able to optimize the server for me based on the kind of site I run. Otherwise, as they say in ebonics, I be f—–…
By the way Dan, in case you didn’t know, your email option in the “Share This” plugin doesn’t work (in both your blog journal and the dan and jennifer site). It just results in a blank screen when you try to send it. I learned this when I was troubleshooting my OWN Share This plugin and it took me all day to set it up, tweak it, then notice the email feature was broken, cursed, cursed some more, checked the Wordpress Support forums for help and quickly realized how futile that was, cursed and screamed even more, then finally vaulted the bloody cursed thing and installed the terrific Add-This plugin instead (http://www.addthis.com/). Now I’m happy again.
Wendy - if you’re going to write about the effect of Digg, I hope you’ll mention roughly how many backlinks you get as a result. That is the real reason for wanting to get dugg isn’t it?
Now you make me see the importance of Digg… I will place digg on my website I just wish their images weren’t so ugly :\
Hi! Alright.
Virtual Private servers are occasionally teh way to go but there is a lot more involved in finding a VPS host than most people think of. You have to find out exactly how many people will be on the server you on, and as much details about the server as possible. I just purchased a VPS on the recommendation of my friend who went with them and has a few through them, the server was slow and constantly as I would word bubbling. Where it would work really good then pop and be slow. It was highly annoying. I cannot be expected to run wpmu on a server like that, or allow my clients to be placed on a server like that. Ha I dont think so.
So make sure I know douglass karr has a free years worth of VPS hosting coupon on his site to check that out see if that VPS company is good I was going to mosey on over there myself. But really it depends on what you plan on operating on the site. Which programs you plan on installing will tell you if you should waste your time with a VPS or upgrade to a full server.
CPU can get overloaded with a program as well. I know a few people posted about CPU being overloaded thats what causes a server to go down correct. When you’re on a dedicated server the only person you have to blame for a bad script is yourself. But you don’t know if joe blow next door didn’t upgrade his security issues on whatever scripts he’s running and someone backdoored their way into your server, or if he’s running a bad program and all of a sudden it attacked the CPU. With CGI scripts they can suck the cpu right out of a server. I had many issues with that when I had my own server for my (no sold) webhosting company where so many people were still using GreyMatter and it would kill the cpu so I would have to suspend random sites until the server could breathe again.
Just remember that.
Here’s another vote for HostGator. I’ve had a reseller account with them for about a year and no problems.
Killed? It would have destroyed my server!! Congrats Wendy!
DAMN!!!! that is the only thing I can say :D:D….congrats Wendy…I also dugg that story :D….and I would have definitely vote for you in the Blogger Face-off if I hadn’t been distracted by Jennifer’s awesome SEO posts…:D…next time for sure..:D
I doubt my host could handle it, but it’s a problem that I would sure like to have!
Congrats on the Digg love. The first time we hit front page, the server literally keeled over. Some major retooling later, we hit front page and didn’t break a sweat. You’ll have to look carefully at your server error logs to see what the killer was (memory exhaustion, too many connections, etc.). Fun stuff…;)