Major worldwide IT outages

[GrumpyPenguin]

4 hours ago, kevinlms said:

But no one would be silly enough to unplug the server - would they?

Something like a server should not be supplied via a plug & socket - it should be hard wired & any switches should be key operated (circuit breakers upstream should be suitable marked/locked on.

Absolutely no excuse.

[spamcan61]

15 minutes ago, GrumpyPenguin said:

Something like a server should not be supplied via a plug & socket - it should be hard wired & any switches should be key operated (circuit breakers upstream should be suitable marked/locked on.

Absolutely no excuse.

...and I would expect any server worthy of the name to have at least two PSUs, with separate mains cabling. Things were somewhat different 20-30 years ago though- in non safety critical environments anyway.

[KeithMacdonald]

3 hours ago, kevinlms said:

Indeed it was dead, because the UPS had been unplugged from the power point and the battery had gone flat!

I wonder how long they put up with the beeping from the UPS, before everything stopped?

 

Ah yes, fun with UPS.

 

Another job I had was with a certain High Street bank's insurance department, on one floor of an office block in Bristol. Just to protect the guilty party, let's call it the "Not Willing" bank, because t'management was notoriously tight fisted on equipment. One time, there was a major power cut that affected the whole office block the "Not Willing" bank was sharing. Which set off fire alarms, and the whole building was evacuated. Fortunately t'was a nice sunny day, so we didn't mind standing outside. All the desktop computers had crashed, so someone wondered aloud if the servers would have crashed as well. One of t'management pompously said "Oh no, they've got a UPS". So we waited ... and waited ... and then wondered why we could see multiple fire engines arriving, and fire crews rushing into the building seriously quickly, not like they thought it was a false alarm.  Well, it wasn't a false alarm, there was a real fire, and it was in the server room of the "Not Willing" bank! It turned out t'management had refused the IT Dept request for a UPS for each server. So some bright spark (sic) had wired about ten seperate servers into one poor little UPS. When the mains power went down, the surge load on the one poor little UPS was so great it overheated and burst into flames, setting off the sprinkers and another call to the Fire Brigade.

 

[KeithMacdonald]

Boom!

 

Quote

Elon Musk Deletes CrowdStrike "From All Systems," Warns Global IT Disruption Sparked "Seizure In Auto Supply Chain"

 

https://www.zerohedge.com/technology/largest-it-outage-history-sparks-disruptions-worldwide

[Michael Hodgson]

Well I predict the main outcome of all this is just one thing.

Some very rich lawyers and going to rake in a great deal more money and become much richer still.

[uax6]

When working for BT as an installer for business customers, I and a little, twitchy Cockney mate ended up in the server room of a major design and control company (they, at the time did design work for the automotive and petrochemical industries). 

We were there to install some cables for some new telephone system they were having. I was the 'Youff' so was doing what I was told, and Mr Twitch was doing the easy parts of the job (like pushing the rods where they needed to go, and telling me to cleat things to the walls). In the server room we had to get across the ceiling (which was suspended), so Mr Twitch pushes up the ceiling tiles and thrusts across the rods. I tape on the heavy multi-pair cable and then proceed to feed it up into the ceiling. While this was going on there is a sudden click followed by silence. 

It must have been about 12 seconds before the server room door slams open (against a data rack of all things!) and there the words along the lines of 'have either of you kind gentlemen happened to switch anything off?'.

I looked at Twitchy, and noticed behind him that there was what looked like an MK cooker switch, underneath which was a mark on the wall made by the ends of his rods. I think my face told Twitchy all he needed to know and he turned round, flicked the switch back on, and said 'Thats a bit daft having your server fed on one switch', and then carried on pulling the cable in. 

When the manager turned up, I was told to beat a retreat and go to the exchange.....

When he arrived at the exchange, his twitches were some what larger than normal. We didn't go back to finish up.

 

Andy G

[Moxy]

58 minutes ago, KeithMacdonald said:

 

Ah yes, fun with UPS.

 

Another job I had was with a certain High Street bank's insurance department, on one floor of an office block in Bristol. Just to protect the guilty party, let's call it the "Not Willing" bank, because t'management was notoriously tight fisted on equipment. One time, there was a major power cut that affected the whole office block the "Not Willing" bank was sharing. Which set off fire alarms, and the whole building was evacuated. Fortunately t'was a nice sunny day, so we didn't mind standing outside. All the desktop computers had crashed, so someone wondered aloud if the servers would have crashed as well. One of t'management pompously said "Oh no, they've got a UPS". So we waited ... and waited ... and then wondered why we could see multiple fire engines arriving, and fire crews rushing into the building seriously quickly, not like they thought it was a false alarm.  Well, it wasn't a false alarm, there was a real fire, and it was in the server room of the "Not Willing" bank! It turned out t'management had refused the IT Dept request for a UPS for each server. So some bright spark (sic) had wired about ten seperate servers into one poor little UPS. When the mains power went down, the surge load on the one poor little UPS was so great it overheated and burst into flames, setting off the sprinkers and another call to the Fire Brigade.

 

 

That reminds me of a similar incident when I worked for an insurance company at their newly opened office in Manchester back in the late 1980's.

 

They had rented a couple of floors in an open plan city centre office block, and so when we arrived, all the equipment was still being installed, including the computer servers.  Local management decided to build a server room around the servers.

 

Shortly after that, the servers started shutting down at random causing systems to shut down.  After a few days of this, the IT guys from head office turned up, walked in to the server room and said 'Where's the air conditioning?'  To which the local management said 'What air conditioning?' 

 

They hadn't taken into account that building a non-ventilated room full of computer servers would get very warm very quickly, and it was overheating that was causing the servers to shut down.  Needless to say, air conditioning was installed in the server room pretty sharpish.

[adb968008]

2 hours ago, spamcan61 said:

...and I would expect any server worthy of the name to have at least two PSUs, with separate mains cabling. Things were somewhat different 20-30 years ago though- in non safety critical environments anyway.

And is a requirement of DORA also…

it doesn’t just cover software… power resilience is just as important. 
I’ve heard today a facility with over 500 Linux servers went down because the power monitoring system went down, was based on a Windows server.

[ejstubbs]

I read that CrowdStrike’s president tweeted that the incident had been caused by a “defect found in a single content update for Windows hosts” (which we all know by now) and added: “This is not a security incident".  He's presumably never heard of the "CIA Triad" of Information Security; CIA being an abbreviation for Confidentiality, Integrity and Availability.

Edited July 19 by ejstubbs

[rab]

10 hours ago, Coldgunner said:

Exactly this, some things are simply rushed to get them out the door without rigourous testing and limited rollouts beforehand.

Do you want it now, or do you want it right

[J. S. Bach]

41 minutes ago, adb968008 said:

So my trip to DC went out of the window this morning it took 2 hours to get rebooked to London from Newark.

 

The App keeps crashing, customer service online wasn’t working and the two girls on the premier desk were struggling to process anyone.

 

Airside security isn’t much better..

 

And gate calling is all manual (indeed it’s refreshing not to be hearing the automated announcements) but you are reliant on yourself to be at the right gate..

 

 

early finish now for me without the IAD trip I’ve a day off the schedule.

 

corona.. wasn’t that a virus ?

 

[KeithMacdonald]

M'Lady has taken this as an excellent lesson in the perils of the drive towards a "cashless society", which has just proved how fragile it all can be. She is a great believer in holding onto wodges of cash. Shopping is natural directed towards those still willing to accept legal tender in cash form. Especially the market stalls for meat, veg, and cheese. Where the prices are cheaper anyway.

[Vistisen]

We had a railway related IT problem. My ex employer had a training center where people bought their own laptops and logged onto a public guest WiFi which we had set up. But about every 10 minutes a lot of their PCs lost their connection. We checked everything and could not find the cause. Eventually I was sent over to observe onsite. The location was very convenient as it was close to a local metro station. I had my own laptop on the network and sure enough lots of people lost their connection, but my laptop had no problem at all. Then the penny dropped: everytime a train stopped at the station just outside the window, all their laptops hopped on to its free WiFi. When the trains pulled out, they lost their connection.

[kevinlms]

24 minutes ago, KeithMacdonald said:

M'Lady has taken this as an excellent lesson in the perils of the drive towards a "cashless society", which has just proved how fragile it all can be. She is a great believer in holding onto wodges of cash. Shopping is natural directed towards those still willing to accept legal tender in cash form. Especially the market stalls for meat, veg, and cheese. Where the prices are cheaper anyway.

Not necessarily, if the registers at supermarkets and the like are down, they will close, because the system can't calculate the bill and re-order replacement stock. So easier to close for the duration. If it's only a matter of hours, not enough time to worry about alternative means, as by the time you do that, the problem will be resolved.

[jcm@gwr]

13 hours ago, rab said:

Do you want it now, or do you want it right

You give 3 options on anything you are offering:-

cost (how cheap), speed (how quick), quality (how good)

then tell the customer to pick 2, because you can't have all 3

[The Stationmaster]

20 hours ago, adb968008 said:

With nothing else to do for 8 hours I thought I’d take the shuttle to Newark Airport and have a look.. Departure board seems to have bought it…

the desks themselves are very quiet but that’s because the Police are restricting road traffic coming in .

Few canceled, just delayed.. I suspect they arent able to cancel the flights, as theres really not much chance this lot taking off at the rate things are moving.

 

even the checkin desks are blue screened.. you can only checkin if you I’ve mobile checked in.. but that means little as virtually nowt is flying.

 

 

went to the hotel breakfast and well seems the guests have disappeared too..

 

it’s beginning to feel like Sept 11th again at the airport.. like a shopping mall after midnight not a Friday afternoon at one of the US’s busiest airports

 

 

Maybe the airline should change its name to UNTIED?

[The Stationmaster]

18 hours ago, PMP said:

I suspect very few will need to be positioned. The best option is to bin the days flying program, or start from a      specific sector once the systems are serviceable. As many domestic routes (and regional) are out and back, you ‘kill’ those sectors you know you can’t operate, and then fly those you can. That then puts the aircraft back into the correct location for the following days flying program. The crew positioning will be more complex, but a similar rationale will be applied.

How very sensible - exactly what i used to do with trains when I went into emergency service planning and recovery mode.  

I only wish that GWR would do it that way nowadays.

[Michael Hodgson]

21 hours ago, adb968008 said:

 

the desks themselves are very quiet but that’s because the Police are restricting road traffic coming in .

Few canceled, just delayed.. I suspect they arent able to cancel the flights, as theres really not much chance this lot taking off at the rate things are moving.

 

even the checkin desks are blue screened.. you can only checkin if you I’ve mobile checked in.. but that means little as virtually nowt is flying.

 

According the TV news India seems to have managed better.  Perhaps they are more used to systems going down.

In Delhi as they couldn't print boarding cards they apparently just gave everybody blank boarding cards and told the punters to fill them in themselves.    As the departure boards weren't working they wrote the gate numbers on a white board.  The main problem with that was that people kept touching the white board and accidentally erasing the info so it had be written up again.

[KeithMacdonald]

41 minutes ago, Michael Hodgson said:

According the TV news India seems to have managed better.  Perhaps they are more used to systems going down.

 

Indian colleagues (that I work with daily) say this may be because a lot of IT systems in India are based on Linux servers, not Microsoft. They were indeed more used to systems going down, in the past when Indian mains electricity was less reliable, but that's rapidly changing for the better. They also say Indian IT systems are built to be more resiliant, as they are built with failure in mind.

Edited July 20 by KeithMacdonald
Typo fixes

[MyRule1]

4 hours ago, Vistisen said:

We had a railway related IT problem. My ex employer had a training center where people bought their own laptops and logged onto a public guest WiFi which we had set up. But about every 10 minutes a lot of their PCs lost their connection. We checked everything and could not find the cause. Eventually I was sent over to observe onsite. The location was very convenient as it was close to a local metro station. I had my own laptop on the network and sure enough lots of people lost their connection, but my laptop had no problem at all. Then the penny dropped: everytime a train stopped at the station just outside the window, all their laptops hopped on to its free WiFi. When the trains pulled out, they lost their connection.

A similar issue can arise on buses in the UK. in our area most First Bus buses have wifi. If you log on at a bus station there's a very high probability you loose you connection when the bus departs, as you will have logged ontothe system on another bus.

[RFS]

I understand the root cause of the problem was that the updated definition file automatically sent out to all systems was just full of zeroes, and the software couldn't cope with that.  Looks like someone messed up big time!

 

 

 

[Michael Hodgson]

28 minutes ago, RFS said:

I understand the root cause of the problem was that the updated definition file automatically sent out to all systems was just full of zeroes, and the software couldn't cope with that.  Looks like someone messed up big time!

 

 

 

Yes, it's the old story of a big fuss was all about a lot of nothing.

[adb968008]

4 hours ago, The Stationmaster said:

Maybe the airline should change its name to UNTIED?

https://en.wikipedia.org/wiki/Untied.com
 

site existed for many years.

 

This made it famous.. when a bands guitar was trashed, united didn't care so he wrote a song that went viral, called “United Breaks Guitars” and got 25mn hits, became an itunes no1 hit, whilst even dipping the share price at one point..

you can see his broken guitar in some scenes

 

https://en.wikipedia.org/wiki/United_Breaks_Guitars

 

 

Edited July 20 by adb968008

[AndrueC]

6 hours ago, KeithMacdonald said:

M'Lady has taken this as an excellent lesson in the perils of the drive towards a "cashless society", which has just proved how fragile it all can be. She is a great believer in holding onto wodges of cash. Shopping is natural directed towards those still willing to accept legal tender in cash form. Especially the market stalls for meat, veg, and cheese. Where the prices are cheaper anyway.

I had no problems paying with plastic at Tesco and Amazon was happy as well. Tbh apart from the headlines I saw no evidence of any problems. Business as usual.

[adb968008]

2 hours ago, RFS said:

I understand the root cause of the problem was that the updated definition file automatically sent out to all systems was just full of zeroes, and the software couldn't cope with that.  Looks like someone messed up big time!

 

 

 

Probably a test file that wasnt replaced with the payload file after testing.