11-09-2006, 10:59 PM
Just an FYI copy/paste of an email I just received from BlueHost demonstrating their handling of a problem with their servers:
**********************
Dear Bluehost.com Customer,
I am writing today to give you some important information
regarding your Bluehost.com account. As many of you know
Bluehost.com uses Cpanel, a third party product, as our control
panel to allow customers to manage their sites. While this is
normally a good thing, it also takes much of the control away
from us and sometimes forces us to live with deficiencies that
we would normally not tolerate. Lately Cpanel has become so slow
and cumbersome that despite buying some of the highest end hardware
available some sites are sluggish at best and extremely slow at
worst. This was never acceptable to us and we tried to work with
Cpanel to resolve our issues, but it is clear to us that our issues
will never be at their forefront. So what do we do?
Yesterday was my breaking point. A customer called in to complain
of a slow site. I checked it myself and the server load showed it
was running fine. Clearly it wasn't, and I wanted to know what was
causing his issues. To make a long story short Cpanel was hitting
every server's hard drive array close to 50,000 times per day more
often than it needed to (Based on my rough calculations). Our
server monitoring was not doing its job and our customers bore the
brunt of these issues. There wasn't an easy fix and our servers
were all grinding to a halt. We had to implement a custom fix and
it had to done immediately. The main issue causing these problems
was mail related. About 70% of all users had to have their mail
moved to a new system and converted to a new storage method. I
made the difficult decision to execute these fixes immediately
without notifying all our customers first because of a couple of
reasons. First, to mail all our users with our current system takes
approximately 18 hours (We could do faster, but then many emails
would be denied by large ISPs because too many emails come form a
single source too quickly), and second because it was causing such
a severe problem with all our servers that it simply couldn't be
delayed. It was causing so many sites to run poorly that we acted
as fast and effeciently as we could.
This made many users email temporarily unavailable. No emails should
have been lost in the transition. It simply wouldn't allow you to
log in while mail was transitioned from one platform to another.
Again, we greatly regret having to do this without prior notification,
but the benefits were exactly what we hoped for.
While all server issues aren't immediately resolved, the VAST majority
of problems we were seeing on all servers are virtually gone. You
should now see a SUBSTANTIAL improvement in almost all areas of your
site including mail issues, script execution time, and overall
responsiveness of your hosting experience.
Next week we will provide a link to a user monitoring service that will
allow you to monitor your server (From our perspective) to see basically
what we see, and know if issues you are experiencing originate on our
end or if some other factor is involved. We have more speed improvements
being worked on aggressively right now and in the near future those
details will be made available as well.
I know this email is lengthy, and I congratulate those that made it this
far I just wanted you to know that I care about what happens to
your sites, and not just because you pay me to care. I don't get to
be as technically involved as I used to be because of trying to run the
company, but I realize now that a change is needed. I may not be the
best admin out there, but I care the most and will always be your advocate
to making your hosting experience better.
All users (All 210,00 of you) can ALWAYS email me directly at -
matt@bluehost.com
Thanks for your support,
Matt Heaton / President Bluehost.com
**********************
Dear Bluehost.com Customer,
I am writing today to give you some important information
regarding your Bluehost.com account. As many of you know
Bluehost.com uses Cpanel, a third party product, as our control
panel to allow customers to manage their sites. While this is
normally a good thing, it also takes much of the control away
from us and sometimes forces us to live with deficiencies that
we would normally not tolerate. Lately Cpanel has become so slow
and cumbersome that despite buying some of the highest end hardware
available some sites are sluggish at best and extremely slow at
worst. This was never acceptable to us and we tried to work with
Cpanel to resolve our issues, but it is clear to us that our issues
will never be at their forefront. So what do we do?
Yesterday was my breaking point. A customer called in to complain
of a slow site. I checked it myself and the server load showed it
was running fine. Clearly it wasn't, and I wanted to know what was
causing his issues. To make a long story short Cpanel was hitting
every server's hard drive array close to 50,000 times per day more
often than it needed to (Based on my rough calculations). Our
server monitoring was not doing its job and our customers bore the
brunt of these issues. There wasn't an easy fix and our servers
were all grinding to a halt. We had to implement a custom fix and
it had to done immediately. The main issue causing these problems
was mail related. About 70% of all users had to have their mail
moved to a new system and converted to a new storage method. I
made the difficult decision to execute these fixes immediately
without notifying all our customers first because of a couple of
reasons. First, to mail all our users with our current system takes
approximately 18 hours (We could do faster, but then many emails
would be denied by large ISPs because too many emails come form a
single source too quickly), and second because it was causing such
a severe problem with all our servers that it simply couldn't be
delayed. It was causing so many sites to run poorly that we acted
as fast and effeciently as we could.
This made many users email temporarily unavailable. No emails should
have been lost in the transition. It simply wouldn't allow you to
log in while mail was transitioned from one platform to another.
Again, we greatly regret having to do this without prior notification,
but the benefits were exactly what we hoped for.
While all server issues aren't immediately resolved, the VAST majority
of problems we were seeing on all servers are virtually gone. You
should now see a SUBSTANTIAL improvement in almost all areas of your
site including mail issues, script execution time, and overall
responsiveness of your hosting experience.
Next week we will provide a link to a user monitoring service that will
allow you to monitor your server (From our perspective) to see basically
what we see, and know if issues you are experiencing originate on our
end or if some other factor is involved. We have more speed improvements
being worked on aggressively right now and in the near future those
details will be made available as well.
I know this email is lengthy, and I congratulate those that made it this
far I just wanted you to know that I care about what happens to
your sites, and not just because you pay me to care. I don't get to
be as technically involved as I used to be because of trying to run the
company, but I realize now that a change is needed. I may not be the
best admin out there, but I care the most and will always be your advocate
to making your hosting experience better.
All users (All 210,00 of you) can ALWAYS email me directly at -
matt@bluehost.com
Thanks for your support,
Matt Heaton / President Bluehost.com