Internet was taken down by one small typo at Amazon's web services, company says

Amazon accidentally took down large swathes of the internet with just one typo.

A worker retrieves goods from shelves at Amazon's warehouse on December 5, 2014 in Hemel Hempstead, England Peter Macdiarmid/Getty Images

A worker retrieves goods from shelves at Amazon's warehouse on December 5, 2014 in Hemel Hempstead, England Peter Macdiarmid/Getty Images

This week, many of the world's biggest websites and services stopped working because of a problem with Amazon Web Services, the platform that the retailer provides to power people's websites. One of its sites went offline because of the issue – and since thousands of websites rely on it, including many of the world's biggest, they immediately went competely offline or stopped working.

Websites like Quora, Trello and some of the world's biggest news sites went offline or stopped working properly when the issue happened. It even emerged that people's houses broke down – internet-enabled ovens, lights and front gates stopped working as a result of the outage.

Now it has emerged that all of those problems were the result of just one small typo on a set of servers.

The team at Amazon's Simple Storage Service were working to remove a small set of servers from its system, according to a technical report into the incident. As it did so, someone entered the wrong command and removed "a larger set of servers was removed than intended".

One of the two servers that were affected was one that manages the "metadata and location information of all S3 objects" in the Virginia data centre, according to the post. That meant that many of the core processes broke down and the server centre could no longer be used.

To fix that problem, Amazon had to fully restart all of the affected systems. But that was a huge operation – Amazon said it hadn't done one for "many years – and it took even longer than expected, meaning the problem couldn't be fixed quickly.

The company says it has now added a range of fixes to stop the issue happening again. It has modified the tools that deal with such problems to cope more efficiently, and is auditing the rest of the systems, it said.

It also announced that it would make changes to the page that shows whether the service is actually online. Even as Amazon Web Services and the websites that rely on it were falling over, its update page said that everything was fine – because that same page relied on S3 and so was affected by its own outage. It will now rely on different data centres so that it wouldn't buckle if just one went down, it said.

Independent

Other News

Vietnam joins Artemis Accords on space exploration

Vietnam joins Artemis Accords on space exploration

Vietnam views space science and technology as a strategic sector that can lift national science-technology capacity and support sustainable growth. The country wants to work with member nations on peaceful, safe, equitable, transparent and sustainable space exploration under international law, with tangible benefits for people and the planet.

Vietnam seeks deeper role in global semiconductor value chain

Vietnam seeks deeper role in global semiconductor value chain

As global semiconductor supply chains are restructured, experts said Vietnam’s opportunity lies not only in attracting projects but also in accumulating knowledge, technology and management expertise to move into higher-value activities.

Vietnam moves to standardize digital economy measurement

Vietnam moves to standardize digital economy measurement

The digital economy is a strategic priority for Vietnam, which aims to raise its share of GDP to at least 30% by 2030, in line with the Politburo’s Resolution No. 57-NQ/TW on breakthroughs in science and technology, innovation and national digital transformation.

AI platform launched to help businesses build brands

AI platform launched to help businesses build brands

Addressing the event, experts and business representatives stressed that harnessing digital technologies requires businesses to adopt a new mindset, use data in decision-making, and maintain brand identity and consumer trust.

Top leader calls for greater contribution from Vietnamese experts in Canada

Top leader calls for greater contribution from Vietnamese experts in Canada

At a meeting with representatives of the Canadian Vietnamese Scientists and Experts Association (CVSEA) on September 25 (local time), the top leader praised the contributions and professional capabilities of Vietnamese experts and intellectuals in Canada, describing them as a valuable intellectual resource and an important bridge connecting Vietnam with Canada and the wider world.

Hanoi approves controlled pilot of two technology projects

Hanoi approves controlled pilot of two technology projects

The two projects involve an intelligent autonomous transportation system developed by Phenikaa-X JSC and a technology for producing mixed rare-earth oxides with a purity of over 95% on a semi-industrial scale developed by the Mining Technology and Construction Consultancy JSC.

Banking industry in AI era: When data becomes competitive advantage

Banking industry in AI era: When data becomes competitive advantage

In just a few years, AI has evolved from an experimental technology into an “assistant” for many banks, supporting everything from customer service, credit approval, and fraud detection to risk management. However, as the technological gap between institutions narrows, a new question is emerging: What will determine banks' competitive advantage in the AI era?

Bac Ninh considers establishing AI centre of excellence

Bac Ninh considers establishing AI centre of excellence

A meeting has been held in Bac Ninh to discuss potential cooperation with Plug and Play in developing the northern province’s innovation ecosystem and establishing an artificial intelligence (AI) centre of excellence, with Standing Vice Chairman of the provincial People’s Committee Mai Son receiving a delegation from the global innovation platform.

Hanoi allows 24-month trial of autonomous vehicles at Hoa Lac

Hanoi allows 24-month trial of autonomous vehicles at Hoa Lac

The project aims to assess the safety, reliability and readiness of autonomous vehicles under real traffic conditions, as well as the effectiveness of autonomous public transport and compatibility with existing systems. Its results will contribute to developing technical standards, operating procedures and management models suited to Hanoi’s traffic conditions.

Singapore's experience suggests solutions to Vietnam's human resources challenge in AI era

Singapore's experience suggests solutions to Vietnam's human resources challenge in AI era

As Vietnam accelerates digital transformation and high-tech development, Singapore’s experience and the engagement of global technology companies demonstrate that human resources preparation must stay ahead of the curve. The ability to filter knowledge, collaborate, innovate and effectively use AI will increasingly become essential to helping workers adapt to a changing labour market and strengthening the competitiveness of Vietnam’s economy in the new era.