Major worldwide IT outages

[RFS]

I understand the root cause of the problem was that the updated definition file automatically sent out to all systems was just full of zeroes, and the software couldn't cope with that.  Looks like someone messed up big time!

 

 

 

[Michael Hodgson]

28 minutes ago, RFS said:

I understand the root cause of the problem was that the updated definition file automatically sent out to all systems was just full of zeroes, and the software couldn't cope with that.  Looks like someone messed up big time!

 

 

 

Yes, it's the old story of a big fuss was all about a lot of nothing.

[adb968008]

4 hours ago, The Stationmaster said:

Maybe the airline should change its name to UNTIED?

https://en.wikipedia.org/wiki/Untied.com
 

site existed for many years.

 

This made it famous.. when a bands guitar was trashed, united didn't care so he wrote a song that went viral, called “United Breaks Guitars” and got 25mn hits, became an itunes no1 hit, whilst even dipping the share price at one point..

you can see his broken guitar in some scenes

 

https://en.wikipedia.org/wiki/United_Breaks_Guitars

 

 

Edited July 20 by adb968008

[AndrueC]

6 hours ago, KeithMacdonald said:

M'Lady has taken this as an excellent lesson in the perils of the drive towards a "cashless society", which has just proved how fragile it all can be. She is a great believer in holding onto wodges of cash. Shopping is natural directed towards those still willing to accept legal tender in cash form. Especially the market stalls for meat, veg, and cheese. Where the prices are cheaper anyway.

I had no problems paying with plastic at Tesco and Amazon was happy as well. Tbh apart from the headlines I saw no evidence of any problems. Business as usual.

[adb968008]

2 hours ago, RFS said:

I understand the root cause of the problem was that the updated definition file automatically sent out to all systems was just full of zeroes, and the software couldn't cope with that.  Looks like someone messed up big time!

 

 

 

Probably a test file that wasnt replaced with the payload file after testing.

[melmerby]

On 19/07/2024 at 10:57, bmb5dnp1 said:

Hello,

    Actually I think there is an over-reliance on Microsoft (particularly the excrable Windows), that's the root of the problem.

 

                        Dave

Rubbish. There's nothing wrong with Windows if it is correctly installed and maintained.

Windows is used because it does everything the (business) customer wants.

 

This problem is no different to Kaspersky putting out an update for home PCs that that turns out to be a system killer.

Although the result would be less catastrophic.

 

 

[Hroth]

3 hours ago, RFS said:

I understand the root cause of the problem was that the updated definition file automatically sent out to all systems was just full of zeroes, and the software couldn't cope with that.  Looks like someone messed up big time!

 

 

Someone will be for the chop over that, though the software development team might get slapped wrists for not sanitising updates when received.

 

End of year bonuses may be 00.00 too....

 

 

[melmerby]

12 minutes ago, Hroth said:

the software development team might get slapped wrists for not sanitising updates when received.

Yes

It should've been run through the anti-virus software first,...........Oh, wait a minute........☹️

[adb968008]

2 hours ago, melmerby said:

Rubbish. There's nothing wrong with Windows if it is correctly installed and maintained.

Windows is used because it does everything the (business) customer wants.

 

This problem is no different to Kaspersky putting out an update for home PCs that that turns out to be a system killer.

Although the result would be less catastrophic.

 

 

Yes but its not the dominant force it was..  Cloud services, Linux has eaten much of the business market, IOS, Android the domestic consumer.

Schools here seem quite set on Chrome.

 

This is a good example of why that is necessary too.

 

There is still a lot of old VMS systems out there too. Indeed Airlines themselves have a lot of very old tech in use, which in part why they were up and running so fast (ever noticed those dot matrix printers around airports). I noticed my app on ios was crashing yesterday, my guess was it was relating to authentication (maybe AD) in some way.

 

 

Edited July 20 by adb968008

[melmerby]

IOS penetration of the mainstream computer market hasn't changed dramatically over the years contrary to Apple fans opinions.

It's even diminished significantly on mobile devices from a near monopoly to about 25% of the market.

 

IMHO Without the US market where it dominates, iOS could well be dead by now, apart from the diehard users.

Linux has long been the dominant cloud OS

 

Windows is still the main PC OS (about 70%), Linux Distros (ignoring Chrome OS) are nowhere, with less than 2%.

 

 

[rab]

10 hours ago, jcm@gwr said:

You give 3 options on anything you are offering:-

cost (how cheap), speed (how quick), quality (how good)

then tell the customer to pick 2, because you can't have all 3

And options one and two tend to work against option three.

 

During my working life I used to notice the repeating cycles:

Must keep costs down/ must improve on delivery times.

Six months later the quality standards had dropped/customer returns increased.

So the emphasis switched to improving quality, then part /assembly costs rose

and delivery dates were missed, so back to square one again

[jcm@gwr]

1 hour ago, rab said:

And options one and two tend to work against option three.

 

During my working life I used to notice the repeating cycles:

Must keep costs down/ must improve on delivery times.

Six months later the quality standards had dropped/customer returns increased.

So the emphasis switched to improving quality, then part /assembly costs rose

and delivery dates were missed, so back to square one again

 

Which is why you can only have 2, any combination of 2 negates the third!

[MyRule1]

Just had my first e-mail from "Microsoft" about the outage, containing loads of dodgy links.

[adb968008]

1 hour ago, melmerby said:

IOS penetration of the mainstream computer market hasn't changed dramatically over the years contrary to Apple fans opinions.

It's even diminished significantly on mobile devices from a near monopoly to about 25% of the market.

 

IMHO Without the US market where it dominates, iOS could well be dead by now, apart from the diehard users.

Linux has long been the dominant cloud OS

 

Windows is still the main PC OS (about 70%), Linux Distros (ignoring Chrome OS) are nowhere, with less than 2%.

 

 

No offence but i’ll go with Gartners structured figures (mobile devices, desktops, servers)  to make it a little more factual..

 

Desktops.. windows has it

Mobile.. Android has it

Server.. Linux has it.

 

https://gs.statcounter.com/os-market-share

 

https://www.statista.com/statistics/272698/global-market-share-held-by-mobile-operating-systems-since-2009/

 

and structured content here..

https://en.wikipedia.org/wiki/Usage_share_of_operating_systems

 

 

 

 

[jjb1970]

Software conformity verification is a nightmare for industry. People tend to forget that industrial machinery (engines, boilers, production line plant, switchgear, generators) is now full of devices with embedded software, much of which is networked for performance monitoring and optimization. That's in addition to platform and process management systems. Modern engines use full authority digital engine control systems (and have done for many years) which relies on a lot of those sensors and instruments.

 

That wasn't so concerning for me when key safety functions still had electro-mechanical protection which would operate if software based systems failed. It was more concerning when manufacturers of big engines, boilers etc wanted to go to all software based protection and struggled to answer questions. I was in a meeting where one gas turbine manufacturer was doing the 'trust us' thing, we use the same system on our jet engines for airliners so it is obviously tip top. Which backfired when the MoD technical authority asked if that was the same engine that had recently came close to causing a total loss when an engine protection system failed to function. Have another go.

[adb968008]

2 hours ago, jjb1970 said:

Software conformity verification is a nightmare for industry. People tend to forget that industrial machinery (engines, boilers, production line plant, switchgear, generators) is now full of devices with embedded software, much of which is networked for performance monitoring and optimization. That's in addition to platform and process management systems. Modern engines use full authority digital engine control systems (and have done for many years) which relies on a lot of those sensors and instruments.

 

known as OT, this is still a wild west as often its manufacturer bespoke software and using closed if not proprietary rather than standard industry protocols. It makes customers beholden to the hardware vendor throughout the life of the asset and is basically vendor lock in for support too. There are established protocols for accessing data points from data collectors (modbus, opc, bacnet etc etc) and better hardware vendors will offer open access software or integration points (api etc).

 

2 hours ago, jjb1970 said:

 

That wasn't so concerning for me when key safety functions still had electro-mechanical protection which would operate if software based systems failed. It was more concerning when manufacturers of big engines, boilers etc wanted to go to all software based protection and struggled to answer questions. I was in a meeting where one gas turbine manufacturer was doing the 'trust us' thing, we use the same system on our jet engines for airliners so it is obviously tip top. Which backfired when the MoD technical authority asked if that was the same engine that had recently came close to causing a total loss when an engine protection system failed to function. Have another go.

Whats scary is the industry is much like the rest of us… they buy new IT but instead of throwing the old away much of it is repurposed for less “front line” duties.

 

At this point the documentation starts becoming more opaque as the write down has happened but the assets still burning kw to creating heat for purposes known only to a few.

 

Inevitably the house clean up event arrives (shut down a facility, move to cloud, tech refresh etc) and no one knows what does what… at which point no one wants to take the risk of shutting it down.

 

one thing that may come of this Crowdstrike event is the final shut down of some random IT..if no ones screaming next month and its still blue screened then it maybe time to explore potential savings by disposal.

ive so many horror stories here.. a major oil company in your region I was at once, wanted to shut down a bunch of servers in a relocation and were marked EOL, out of service and ready to dispose.. once scanned revealed they were actually live and managing the pumping wells !

 

Another UK insurance company I was at, revealed they had open connectivity and internal server access on quite a wide scale to/from UK to Sri Lanka for a call center / support business they ceased contracting with 5 years earlier.

 

 

Edited July 21 by adb968008

[melmerby]

13 hours ago, adb968008 said:

No offence but i’ll go with Gartners* structured figures (mobile devices, desktops, servers)  to make it a little more factual..

*That's where I got the figures from (amongst others) and your precis corroborates my statement.

 

I went back over the years to see the trends.

I looked at several different sites and the overall trends are the same, even though the end figures vary from one site to the other.

 

Edited July 21 by melmerby

[adb968008]

2 hours ago, melmerby said:

*That's where I got the figures from (amongst others) and your precis corroborates my statement.

 

I went back over the years to see the trends.

I looked at several different sites and the overall trends are the same, even though the end figures vary from one site to the other.

 

There are products in the market which are designed to discover, map and analyse this data for responsible companies that realise the need to have such data, tools and processes in place to access when these types of events (planned or unplanned) happen.

 

Gartner, Forrester, 451 etc liase with these software vendors for both ranking purposes in sector reports for such software, to  sell these reports to prospective purchasers of such software, and to produce industry trends from the data these tools collect. Some have stronger preferences and relationships who they work with for that data, and the size, quality of data sets from differing tools, which is a reason for such variation.

If you get data from a tool that specialises in AS400/AIX etc your not going to get as good wintel info. Similarly a domestic US company serving every possible OS isnt going to get you a global view. A tool for Cloud sizing isnt going to reflect onprem hardware etc.

 

So careful with stats… as often they extrapolate data based on that sample and multiply it to predict trends.. Anyone notice China isnt blue screened to death.. yet their O/S stats dont appear in any numbers… and theres a lot of IT in China.

 

 

 

 

Edited July 21 by adb968008

[bmb5dnp1]

On 20/07/2024 at 17:35, melmerby said:

Rubbish. There's nothing wrong with Windows if it is correctly installed and maintained.

Windows is used because it does everything the (business) customer wants.

 

This problem is no different to Kaspersky putting out an update for home PCs that that turns out to be a system killer.

Although the result would be less catastrophic.

 

 

Hello,

  At the risk of turning this thread into 80s style computer wars, it's certainly not rubbish to suggest that Windows is the least secure, least reliable OS to have gained such widespread use. I suppose that's the way it came into being (Microsoft thought that the internet was a passing fad for instance) and then wanting to maintain backwards compatibility which meant that lots of bad design and implementation have been kept alive all these years. For instance, the only way to make Windows secure is to constantly annoy users with messages about installations etc. In contrast, I have Linux servers that have literally been running critical business/academic functions for years without reboots with no issues with security, reliability etc. 

 

Dave

[spamcan61]

Another one from the "testing properly is way too expensive" file, this one actually life critical

 

https://www.bbc.co.uk/news/articles/cjr4zkq1kkwo

[craneman]

With increasing (unnecessary) automation of vehicles, and at some point the nightmare of fully autonomous vehicles to look forward to, this sort of thing will get much, much worse.

 

CrowdStrike managed to achieve this level of chaos with nothing more sophisticated than shocking incompetence. Imagine a team of clever, fully-resourced state actors hell-bent on disruption.

 

The future of civilisation - such as it is - looks fairly fragile to me.   

[Coldgunner]

A piece of software is only as smart as the person that programmed it afaik. I know AI has come on leaps and bounds, but still struggles with some of the human qualities like perception, anticipation etc.

 

Also when it comes to cars, I have a theory that the more assistances added, the lazier and potentially more dangerous the driver becomes. I mean, introduce full autopilot into a car and what's stopping the driver having a snooze or reading a book instead of paying attention. Bad enough people already mess about with phones and do their makeup behind the wheel.

[AndrueC]

14 hours ago, bmb5dnp1 said:

I have Linux servers that have literally been running critical business/academic functions for years without reboots with no issues with security, reliability etc. 

I have been running my own mail server for nearly twenty years. Over that period it's been rebuilt three times all incarnations were Windows and software called VPOP3. It is under constant attack from scrotes trying to get in either via the Web UI or SMTP (I've disabled POP3). The only times it's ever failed have been due to hardware and that's why it got rebuilt.

 

Servers generally lead closeted lives. Some of them are exposed to the outside world but usually only in limited and easy to control ways. They are also usually looked after by knowledgeable and sensible people who ensure they are looked after.

 

I do agree that Linux is a better designed and implemented Operating System but until/unless it becomes a significant presence on the desktop I don't think we can conclude that Windows is insecure. Most Windows machines are poorly maintained and (ab)used by people who know little to nothing about the technical aspects of their device. Indeed that's always been the barrier to Linux on the desktop - it requires too much technical knowledge.

 

Consumers just want to use their device and don't have the time or the inclination to get technical. Windows is primarily designed to target that market. The fact it still exists and has market domination at the cutting edge of IT demonstrates that it is supremely well suited to that environment. The fact that market will tolerate all kinds of *censored* is beside the point.. 😉

[Michael Hodgson]

2 hours ago, AndrueC said:

Consumers just want to use their device and don't have the time or the inclination to get technical. Windows is primarily designed to target that market. The fact it still exists and has market domination at the cutting edge of IT demonstrates that it is supremely well suited to that environment. The fact that market will tolerate all kinds of *censored* is beside the point.. 😉

 

It wasn't a Microsoft mistake.  The software that got cocked up wasn't something consumers make use of themselves.  Nobody outside the industry had even heard of Crowdstrike.

 

What caused the severity of the outage was a combination of at least three facts

• that a lot of major businesses use it (because they've got to use something to protect against malware, and this particular package has been widely deemed better than alternatives),
• that many of them happen to use the Windows operating system (it could probably have happened similarly to a different version used under some other OS) and
• that this particular intended upgrade did not allow them easily to reverse its impact and recover in the usual manner.

In that sense, it's rather like a lot of railway accidents - it takes several things happening together rather than one single mistake.

[PeterStiles]

5 minutes ago, Michael Hodgson said:

Nobody outside the industry had even heard of Crowdstrike

Except everyone watching Lewis Hamilton's car go round and round in circles of a weekend. Crowdstrike are one of their sponsors and plastered all over the place...