Last November, one of its datacenters was brought low, embarrassingly, by a truck crashing into a nearby power transformer. That's precisely the kind of thing datacenters are designed to survive, but Rackspace's did not. Its employees all got bonuses anyway. That's a fanatical disservice to its investors, who now include public shareholders. A more deserving tech company, one hopes, will come to the public markets soon, and erase the memory of Rackspace's broken offering.

Last fall, Rackspace went through a major outage that released many of its customers from their long-term contracts. Then, earlier this year, Tumblr founder David Karp publicly voiced his discontent with Rackspace and the company cut its prices by 80 percent in order to keep his business. Karp didn't take the offer, telling us its cheaper for startups to load up on redundant servers than to depend on Rackspace's 100 percent up-time promise.

Rackspace's promise of 100 percent uptime just doesn't mean that much anymore, Karp explained. He said after recent innovations, it has become far cheaper for startups to protect themselves from downtime by buying twice as many servers as they'll ever need rather than to go with managed hosting. If half of them suddenly fail, so what?

Some services and sites have known this for a while, but it's only now that Rackspace is reeling. Why? One disastrous side effect of the truck crash that caused Rackspace meltdown last fall, is that the downtime voided many clients' contracts. Under the terms of their agreements, they're now free to cancel service and go elsewhere.

(Photo by rp72)

"Dumping them is a pain in the ass," Forman told me by IM. "I would if i had the time, but I don't."

Here's four reasons why he'd switch if he "had the time."

1. Their uptime used to be good. But now it's normal.
   
2. Their support used to be good but again, everyone offers the same shit now.
   
3. They've turned lazy and they are basically like — 'yeah — we forgot to do this shit — it will be 3 more days."
   
4. During that outage in their datacenter I didn't get one fucking phone call. I got a fucking mass mail apology and thats it. Their shit was down for like a full day and restarted like 3 times. We probably aren't a major account, but its $8k/month. It's a lot of money to me.

This morning, Tumblr founder David Karp said he planned to move the site to a new hosting services provider, blaming Rackspace for a "botched" firewall upgrade and decrying the "exorbitant price" his company paid for its services.

It's been a bad last couple months for Rackspace. When Web-based collaboration software provider 37signals went down for a few hours on Friday morning, it too blamed Rackspace. Rackspace retorted that the problem was isolated to 37signals. How reassuring to know that Rackspace's finger-pointing services are available 24/7.

Rackspace also went down in early December. And in November, a careening truck destroyed a power transformer, taking all of Rackspace's customers offline. But hey, before that, Rackspace was called the "most reliable" Web host in September 2007.

We're going to have a long, serious talk with our service provider (Rackspace). They're supposed to be the best in the business, but in this instance they failed us, so we in turn failed you. We'll do everything we can to make sure that something as simple as a load balancer (or firewall or switch or any other network equipment) going bad does not cause two hours of downtime.

Rackspace is falling apart again. To my understanding, the whole infrastructure has failed and they now have emergency generators and chillers for the past two or three weeks in the parking lot. This must be vary bad for the remaining customers to go through this again and again, not to mention that Rackspace is not a true redundancy A-B side as they advertise to the public. This company is a bad investment!

(Photo by kylemac)

The Register is reporting that Rackspace's LonB facility in London's Park Royal area has lost power and one generator failed to start up knocking 10 percent of its U.K. customers offline. A company spokesman said, "Obviously, we will be testing to see what went wrong with this one." Obviously. Rackspace's customers likely feel they're being tested, too.

A pickup driven by a man experiencing a drop in his blood sugar crashed into a transformer that provides power to the datacenter. This disabled one source of power to the building. During a power failure, Rackspace's uninterruptible power supplies normally keep the servers running until the generators can be started. The chiller units that keep the server rooms cool automatically shut down and are restarted on generator power — a process that takes approximately 30 minutes. Once everything was up and running normally on generators, Rackspace decided to switch internal power to its secondary utility power source — going through the 30-minute chiller reset period one more time. During this period, the utility company shut off power completely to the building once again. Generators kicked in and servers were kept up and running.

As a result of the second power failure, the chillers once again had to be reset. Because of this repeated reset period with no cooling at all, temperatures inside the datacenter skyrocketed and servers had to be shut down to prevent heat damage. Once the chillers were brought back online and ambient temperature was reduced to acceptable levels, technicians began bringing affected machines back online.

Want more? Here's the full report:

Incident Report — DFW1 Power Review
November 11th / November 12th 2007

Introduction
This Incident Report has been prepared by Rackspace to provide customers with the information relevant to the service disruptions in our DFW Data Center on November 11th and 12th 2007. This report includes a detailed description of the incident and a timeline of our actions. Rackspace Engineering teams are working on a corrective action plan in response to these events. We will prepare a follow-up report on the action plan.

This report documents a series of sequential events in the DFW Data Center. Please note that customers affected by one event may not have been affected by all of the incidents. Your Account Manager will have specific information regarding your configuration.

In line with our promise of Fanatical Support™ and the messages from Lanham Napier, our CEO, regarding this event, we are focused on supporting our customers and providing the information you need for your business and customers.

Incident Overview & Cause
*All listed times are in US central time
Incident 1 — At approximately 4:30 A.M.* Sunday November 11th, Rackspace experienced a power outage in one section of our DFW data center. In the design of the DFW power infrastructure, power flows from the uninterruptible power supplies (UPSs) through a switch board to multiple power distribution units (PDUs). The power outage was caused by a breaker failure in the switch board and affected all downstream PDUs. The reason that this failure did not cause the entire data center to switch to generator power is because its impact was limited to infrastructure components downstream of the UPS.

Rackspace engineers immediately began troubleshooting. Our first objective was to restore service. After our initial efforts to restore power were unsuccessful, we began to investigate alternate solutions. One option was to switch to generator power to bypass the failed breaker but, with the root cause still undetermined, Rackspace needed to be certain that making this transition would not worsen the situation. After confirming that the proposed solution would be an improvement, Rackspace moved the affected portion of the data center to generator power. According to our data, all services were restored by 6:50 A.M. on November 11th.

Incident 2 — At approximately 6:30 P.M. on the evening of the 11th, a breaker in the generator power grid tripped. The impact of this event was similar to the initial outage but on a much smaller scale. When this breaker failed, one PDU lost power. All customer devices with dual power supplies in this section of the data center remained online and were not affected. While the PDU in question was without power, devices with a single power supply were affected. Data Center technicians immediately acted to minimize the impact on these customers by moving these devices manually to alternate power supplies. Rackspace technicians worked to restore service as quickly as possible and by 7:40 P.M. the affected PDU was back online.

Our data center engineers worked through the night to attempt to identify both the root cause of the problem and a safe return path to utility power. At approximately 5:00 A.M. on November 12th Rackspace began the process to return to utility power. By 5:10 A. M. the transfer of power back to utility power was completed.

Incident 3 — At 5:25 A.M. there was a problem with the utility distribution network. This caused another outage that affected the same section of the data center as the initial outage on the 11th. DC Engineers immediately initiated a switch back to generator power. By 5:40 A.M. the generators were once again providing power to the affected portion of the data center. There are several aspects of the distribution network that are currently being investigated and tested to determine the root cause.

Incident 4 — In an unrelated incident, at approximately 6:30 P.M. Monday, November 12th, a vehicle struck and brought down the transformer feeding power to the DFW data center. During a utility power failure like this, the chillers, a component of the HVAC system responsible for maintaining consistent temperature in the data center, automatically lose power as they are not directly powered from the UPS. Upon power failure, our emergency generators kicked in and provided power to the chillers as intended. At this point the DFW data center was fully operational.
The DFW facility has two separate utility feeds and the engineers decided to start moving from generators to the secondary utility feed. Each time Rackspace alternates between utility and generator power, the chillers require us to follow a shut down and restart procedure. This procedure normally takes approximately 30 minutes to complete. We started the transition to the secondary utility feed and initiated the restart process on the chillers.

Unfortunately, at this point, our utility provider shut down the secondary feed that was powering the data center, without notifying Rackspace. This was an emergency action taken by the utility in order to allow safe removal of the accident vehicle and protect the emergency responders. This unexpected power cut required DFW to switch back to generator power and reinitiate the chiller start up procedure. The repeated cycling of the chillers resulted in increasing temperatures within the data center.

At 07:35 P.M. the ambient temperature reached a critical level within the facility so Rackspace began to manually shut down some servers in the affected areas. This action was taken to protect them from overheating, prevent data loss and hardware failures, and reduce temperatures. By powering down these devices and bringing the chillers back online on generator power, we were able to reduce the ambient temperature to an acceptable level. Lower temperatures allowed us to power up the devices that had been shut down earlier. At 11:37 P.M. we switched from generator power back to utility power for the areas of the data center that lost power due to the incident on Monday night.

The devices in the section of the data center affected by the 1st outage on November 11th remain on generator power. Our engineering team is investigating solution options to resolve the original issue. We will not move to transition off of generator power until we can determine the root cause and test the proposed solution. We will notify affected customers of the maintenance window in advanced before we begin this switchover.

Incident Timeline
4:19 A.M. November 11th — A problem in the internal utility power distribution grid caused a breaker failure resulting in loss of power to a portion of the DFW data center.

6:49 A.M. — The transition to generator power is completed and power is restored to the affected portion of DFW.

6:32 P.M. — A breaker in the generator power grid tripped causing a loss of power to a single PDU, including a smaller subset of the original group of customers.

7:40 P.M. — The situation was resolved and the PDU returned to service. Between 6:32 P.M. and 7:40 P.M. all devices with dual power supplies maintained power and were not affected. Customer devices with single power supplies in this area were affected. Data Center technicians immediately acted to minimize the impact on these customers by moving these devices manually to alternate power supplies.

4:00 A.M. November 12th — The utility distribution grid completed realignment and re-synchronization; all systems reported as ready for operation.

4:30 A.M. — Transfer of power was initiated and affected devices were slowly moved off of generator power and back to internal utility distribution power.

5:10 A.M. — Transfer was complete and all devices in DFW were running on utility power.

5:25 A.M. — The internal distribution grid failed again. Data Center engineering acted immediately to transfer all affected devices back to generator power.

5:40 A.M. — The transfer to generator power was completed in under 15 minutes and all affected devices had access to power.

6:30 P.M. — In a completely unrelated incident, an automobile accident caused damage to the primary utility feed to the DFW Data Center. The Data Center automatically failed over to generator power without any service interruption to customer devices.

6:45 P.M. — The Data Center Engineering team decided to transition from generators to the secondary utility power feed. The chillers were moved to the secondary utility power feed and started powering up.

7:12 P.M. — The local utility company shut the second feed down, without notifying Rackspace. This was an emergency action to protect emergency responders and ensure safe clearing of the accident scene. At this time the DFW facility returned to generator power.
Each time the data center alternated power sources the chiller systems reinitiated their start up sequence preventing the data center from receiving proper cooling.

7:35 P.M. — Due to the rising temperature inside the data center, Rackspace started shutting down infrastructure and servers in order to prevent them from overheating, prevent data loss and hardware failures, and to keep the entire facility from overheating.

7:55 P.M. — The chillers were brought back online and the facility was stabilized on generator power.

8:20 P.M. — The temperature and power systems had stabilized to a point where we were able to begin powering on servers that had been shut down earlier.

11:37 P.M. — The area of the facility affected by the Monday evening traffic accident-induced incident was transferred back to utility power. The transfer was accomplished without any service interruption. The chillers continued operating normally.

Investigation
Throughout this situation our data center management team has been meeting with internal and external experts in an attempt to isolate the root cause of the original outage. Rackspace engineers and contractors are onsite evaluating all the systems and the events that triggered the failure. The
original breaker is being thoroughly tested to rule it out as a possible cause. We also brought a load bank onsite to test the UPS cluster. These tests will allow us to determine when it is safe to return to utility power.

Next Steps
We are waiting on the results of the additional tests before planning to return to utility power. Rackspace will provide additional information as it becomes available and will provide advance notice before returning to utility power.

MVA- a man driving a ford truck ran a stop sign and went into a parking lot hitting a power transformer. Prior to our arrival the Southlake FD was on scene. They had made contact with the PT and found that he had low blood sugar. When we arrived we cleared the seen [sic] of all personell [sic] because of the possibility that the transformers may still have power to them. The PT was still in the truck. He was conscious and talking. Once the power company arrived and informed use [sic] that the power was off. We removed the pt to a back board and then to the cot. he was treated in the Mobile Intensive Care Unit and transported to BRMCG by M561 with consent. Management informed IC that a clean up company was contacted to respond to clean the mineral oil spill from the transformer. The scene was released back to management and electric company on scene.

Dear [redacted],

It has been a long day and hard day here at Rackspace. We know it's been a long day for you. We are deeply sorry for the events that have taken place at our Dallas/Fort Worth data center. Your satisfaction is what every Racker works towards every day, especially today.

To those affected by the outage, I apologize. We all apologize. We understand the frustration and uncertainty you have gone through. We take full responsibility for what happened and we will work with you to reach a remedy that satisfies you.

We have made the latest status update to the myrackspace.com customer portal. Please continue to visit it regularly for our most recent updates.

As always, your Account Manager and Support Team are available to help in any way. Likewise, please feel free to contact me if you have any questions or needs.

Sincerely,

Lanham Napier
CEO
Rackspace

Nov. 12th 11:30PM CST — As of 10:50 p.m. CST, all DFW servers that were proactively powered down earlier this evening, to avoid overheating, have now been powered back up. The Data Center Engineering team has been working to resolve the power issues caused by tonight's traffic incident. The team is preparing to transfer machines affected by tonight's power outage from generator power back to utility power. The servers and devices that were affected by the unrelated event over this past weekend will remain on generator power. We anticipate transferring the machines affected this evening back to utility power within the hour and expect the transfer to be non-disruptive to customer environments. We apologize again for the inconvenience these events have caused and have all hands on deck working fanatically to minimize the impact on your business.

Will be using this situation as both a wake-up call and a learning experience. While our systems are engineered to chug through major failure, this "perfect storm" chain of events beat both our set-up and our data center's sophisticated back-up systems. We will work hard to further diversify our systems in order to make an future downtime event like this even more rare.

Threadless

As you noticed Threadless took a little nap this evening. Apparently a truck jumped a railing and attacked our data center. I think their may have even been an explosion. I hope so.

This is the first time we have had power outage like this in the 9 years that we have been in hosting (8 of which at Rackspace).

*crickets*

Monday Nov. 12th 8:00PM CST — Thank you for your patience today as we work through a root cause analysis of the power issues in the DFW data center. The Data Center Engineering team is continuing to work on the plan to move back to utility power without any further interruption in service. Rackspace will notify you in advance before we move to switch back to utility power.
In the meantime, here is a brief timeline of events:

Sunday, Nov. 11, 2007 4:19 a.m. CST — A problem in the internal utility power distribution grid caused an outage to cabinets in one section of the DFW data center. 6:49 a.m. CST — Power was fully transferred to generator power. Based on building monitoring systems, outage times varied for every customer. DC engineering worked to isolate the internal utility problem and restore the integrity of the internal distribution system. 6:32 p.m. CST — A separate incident occurred when a breaker in the generator power grid tripped, causing one of the Power Distribution Units (PDUs) in the same section of the DFW Datacenter to fail, affecting a much smaller group of the customers in this section. All customer devices with dual power supplies in this section of the datacenter remained online and were not affected. Customer devices with single power supplies in this area were affected. Data Center technicians immediately acted to minimize the impact on these customers by moving these devices manually to alternate power supplies - resulting in just a few minutes of downtime. 7:40 p.m. CST — The breaker problem was diagnosed and resolved, bringing the down PDU back online.

Monday, Nov. 12, 2007
4:00 a.m. CST — The Data Center engineering team had the initial utility distribution grid realigned and resynchronized. All systems reported ready for operation.
4:30 a.m. CST — Transfer of power was initiated and affected devices were slowly moved off of generator power and back to internal utility distribution power.
5:10 a.m. CST — Transfer of power was completed.
5:25 a.m. CST — Unfortunately, the internal distribution grid failed again. Data Center engineering was able to transfer all affected devices back to generator power in under 15 minutes.
5:40 a.m. CST — All affected devices were back on generator power. The Data Center environment is stable and is designed to be able to run indefinitely on generator power. Data Center engineering is continuing to diagnose the problem and engaging all vendors onsite.

We will continue to provide updates via the MyRackspace portal. In addition, you will receive notification of any maintenance windows. We are committed to supporting your business and minimizing any impact on your hosted environment.

DFW Datacenter Power Update

Nov. 12th 8:30PM CST

In a completely unrelated incident to this weekend's power problems in DFW, a traffic accident caused damage to a power transformer which provides utility power to our DFW data center. Here is the current sequence of events:

* At approximately 6:00 p.m. CST utility power was lost to the DFW data center
* Power automatically switched over to backup generators without disrupting service for any customers
* When generator power was established two chillers within the data center failed to start back up
* Utility power was re-established through a secondary utility source
* As a result of temporary data center temperature increases, we proactively shutdown a number of customer servers to protect them from overheating

At this point, the chillers are back up and running and we are operating on generator power throughout the data center. We have contractors on site to repair the damage and will be in contact with all customers who have been affected by this outage. We apologize for any disruption to your business operations and will work diligently to restore your service.