UK heat wave causes Google and Oracle cloud outages

A heat wave in the UK led to Google Cloud and Oracle Cloud outages as cooling systems failed in corporate data centers.

Over the past week, the UK has experienced a record-breaking heatwave, which has resulted in scorching temperatures across the region.

Advertising

Today, however, with temperatures reaching a record 40.2 degrees Celsius (104.4 Fahrenheit), the data center cooling systems used by Google and Oracle to power their cloud infrastructure began to fail.

To prevent permanent damage to hardware components and create extended downtime, both Google and Oracle disable equipment and cause downtime in their cloud services.

Oracle was the first to be affected, with the company reporting a cooling failure at 11:30 a.m. EST today that shut down “non-critical hardware.”

“Due to unseasonal temperatures in the region, a subset of the cooling infrastructure in the UK South (London) data center experienced an issue. This led to the shutdown of a subset of our service infrastructure to prevent uncontrolled hardware failures” Oracle Cloud Status Message seems to have been the first found By The Register.

“This action is taken to limit the potential for long-term impact to our customers.”

However, Oracle says customers in this zone may not be able to access their Oracle Cloud Infrastructure resources even if only critical hardware is disabled.

About two hours later, Google also announced cooling restrictions on Europe-West2—a zone in one of its buildings—for Europe-West2.

“A cooling-related outage occurred in one of our buildings serving the West-Europe2-A zone for the West-Europe2 region. This caused a partial capacity outage in this region, resulting in VM shutdowns and loss of machines for a small number of our customers,” Google Cloud says. Incident report.

See also  Paris wins legal battle in US over France.com

“We are working hard to bring cooling back into service and increase capacity in this area. We do not expect any additional impact in the Europe-west2-a region and running virtual machines should not be affected. A small percentage of replicated fixed disk devices are running in single redundant mode.

“In order to avoid machine damage and long-term downtime, we have activated part of the area and we are working to restore redundancy for all remaining duplicated fixed disk devices affected by the potentially prohibitive GED. »

As with Oracle, this cooling failure disrupted Google Cloud customers, with virtual machines down, machines inaccessible and persistent disk devices operating in single redundancy mode.

Both companies say they don’t expect further damage as they work to get cooling systems back online.

Refrigeration systems were restored

Both Google and Oracle have resolved cooling issues at their data centers, with service restored to Google on Tuesday and Oracle on Wednesday.

At 11:45 pm EST on Tuesday, Google restored the service with the following final update.

“There was a cooling-related failure in one of our buildings, which contains part of the West-Europe2-region of a zone, which has now been resolved. The impacts to EPC, fixed disk and auto-scaling have been resolved. Customers in all regions of Western Europe are virtual Machines can be started2. A small number of HDD-backed fixed disk volumes still exhibit impact and I/O errors. If you continue to experience problems with these services, please contact Google Cloud Technical Support and refer to this message.

It took some time for Oracle to recover its cool, and services were restored at 7:00 a.m. EST on Wednesday.

“Following unusually high temperatures in the south of England (London), two cooling units in the data center failed when they had to operate above their design limits. As a result, temperatures in the data center began to rise, shutting down a subset of the computing infrastructure.

Updated 7/20/22 4:55 PM EST: Added updates on cooling system issues.

See also  Obama's birthday party faces criticism despite respecting health rules

Leave a Reply

Your email address will not be published.