The BA IT collapse

[Asterix, the former Gaul.]

http://www.bbc.com/news/uk-40069865

British Airways has cancelled all flights from Heathrow and Gatwick because of global computer problems.

It apologised for the "global system outage" and said there was no evidence of a cyber attack.

Mick Rix, GMB's national officer for aviation said: "This could have all been avoided.
"BA in 2016 made hundreds of dedicated and loyal IT staff redundant and outsourced the work to India... many viewed the company's actions as just plain greedy."

My employer took the same course of outsourcing in 2005.  It was not successful because the companies doing the work were also working for other clients and did not give priority to one client.  I understand that much of the work has returned to be done by UK employees.

Whatever the cause of this particular incident, I'd suggest that such critical operations should be maintained by dedicated staff working for the company.  Thus a high degree of urgency could be assured.

[Kim]

Cynically, it seems like a good excuse to ground all your planes in a hurry without causing a mass panic...

[Basil]

Cynically, it seems like a good excuse to ground all your planes in a hurry without causing a mass panic...

Blimey.  I hadn't thought of that.

[Asterix, the former Gaul.]

Cynically, it seems like a good excuse to ground all your planes in a hurry without causing a mass panic...

That's the scary version

If so, when will it be safe to explain?

[ElyDave]

But for what reason?

If there was some kind of generalised threat, why only BA?

[Polar Bear]

Call to radio or tv station:

Hello, this is <codename>

There is a device on a British Airways plane.

Terrorists use code names and code words to get across their warnings.   I recall this from the days when the IRA was active.

[DaveReading]

Call to radio or tv station:

Hello, this is <codename>

There is a device on a British Airways plane.

If that was really the case then (a) passengers would not have been kept on board flights at the terminal, as has happened to many people this afternoon, and (b) inbound BA flights would not be landing at Heathrow but would be diverted, probably to Stansted.

There seems little doubt it is an IT issue, as reported.

[Veloman]

Call to radio or tv station:

Hello, this is <codename>

There is a device on a British Airways plane.

Terrorists use code names and code words to get across their warnings.   I recall this from the days when the IRA was active.

And those days have gone as the aim these days is to kill and get maximum casualty rate, unlike the IRA who were keen to disrupt infrastructure and cause general damage to buildings.  Followers of IS do not use code words and the asymmetric warfare they pursue would not benefit from code words and warnings.

That said, a vicarious phone call stating a threat could cause the closure or withdrawal of facilities. Big decision on behalf of someone to ignore such a threat.

[Veloman]

My cross-post with DaveR and totally agree with his comments.

[PaulF]

Call to radio or tv station:

Hello, this is <codename>

There is a device on a British Airways plane.

Terrorists use code names and code words to get across their warnings.   I recall this from the days when the IRA was active.

Think that was just the IRA, the current crop of terrorists don't seem to be doing the same. Don't think it's a good idea to generalise about terrorists. Or any other group for that matter.

[Kim]

...Or it could just be an infrastructure clusterfuck.  It is a bank holiday weekend, after all.

[Hot Flatus]

You should lay off the weed, Kim.

[Polar Bear]

Call to radio or tv station:

Hello, this is <codename>

There is a device on a British Airways plane.

Terrorists use code names and code words to get across their warnings.   I recall this from the days when the IRA was active.

Think that was just the IRA, the current crop of terrorists don't seem to be doing the same. Don't think it's a good idea to generalise about terrorists. Or any other group for that matter.

I'm not generalising, just offering a possible explanation about behaviour.   We don't know and we're all hypothesising.   Probably more likely imo that BA has been hacked to be honest.

[Asterix, the former Gaul.]

We believe the root cause was a power supply issue."

My employer had back up generators in the basement.  Power supplies issues just made the lights flicker.

What is more they had agreements with other major computer 'owners' that in the event of a crash there would be a switch-over to allow continuity. Resources were not isolated but pooled through people like IBM.

My last job was change management i.e. to ensure that no changes were made unless it had been tested through three stages with a proper test plan approved by users.  Once our IT department 'bought into it' it worked very well.  When they decided to outsource I volunteered for redundancy, quite happily, as did many others.  I understand that once outsourced the system did not work well and the IT director was replaced.

The simple explanation for BA's plight is that a similar thing has happened - they failed to understand the downside of outsourcing.

[Ham]

Put simply, it mostly comes down to poor risk management.  Or, good risk management if the overall cost of this episode is less than it would cost to prevent, which I doubt. When you have an attitude to risk based around fallacious data and empirical assessment of isolated risks combined with the innate obduracy of inanimate objects, this sort of event is inevitable. The only question is, which organisation does it happen to, this time it was BA. Every major organisation that I've had dealings with for IT systems (and that's a metric fuckload) is using cost as one of the most significant factors in their decision making process, it was not always this way.

TL;DR: Shit happens.

[Polar Bear]

I was absolutely gobsmacked when I arrived at a large retailer's HQ to discover that they would be out of business in three days if their computers failed.  They had no backups, no disaster recovery plans, no failover sites, nothing.  When I left 3 1/2 years later, I left as a result of the IT director* rejecting the proposals to implement the basics.   

When I say left, I was made redundant. 

*  He is a Chelski fan.  I wonder if I still have his number ...

[Thor]

The current explanation - that a power failure can knock out one or more critical, global systems for hours - defies credibility. That such systems have no redundancy, failover options, Disaster Recovery procedures, is not credible.

Something's up - and the cover story isn't very convincing.

[Kim]

Maybe someone opted to save costs my installing the redundant systems in adjacent racks?

[Vince]

Odd that BA have a system that is specific to flights in and out of Heathrow and Gatwick.

[Ham]

The current explanation - that a power failure can knock out one or more critical, global systems for hours - defies credibility. That such systems have no effective redundancy, failover options, Disaster Recovery procedures, is not credible.

Afraid to tell you, you are wrong with that assumption. I've done a little FTFY to be a bit more specific.

My job is for %Megacorp, one of the main providers of services to global organisations, my role is intimately involved with understanding any issues and proposing the solution so it would be unethical and inappropriate for me to comment directly; especially given the speculation of where and how the failure occurred. But really, it's no surprise.

One little anecdote about another major org that put in their own dual site HA for some critical systems. Turns out that when one site goes out, it takes out the other. It's been that way for three years and going to be for another one, at least.

Very few organisations understand the difference between HA and DR, even at the highest levels. It's all down to money at the start and end of the day.

[pcolbeck]

The current explanation - that a power failure can knock out one or more critical, global systems for hours - defies credibility. That such systems have no redundancy, failover options, Disaster Recovery procedures, is not credible.

Something's up - and the cover story isn't very convincing.

Yup. I am currently designing the network for an airport and you have two data centres with critical services having a backup server in the second data centre and even within a data centre you have redundant switches in separate racks with everything dual homes to two switches. Each switch has two PSUs (at least) as well and you would normally feed them from separate supplies. The sore and distribution switches for the network that goes out to terminals and connects the baggage handling, check in wireless etc etc would also be redundant. Only the access switches would be a single point of failure and losing one would only take down the stuff directly attached to that.

Mind you I have seen an airport network that was badly designed and a L2 issue like a broadcast storm could have taken the whole thing down.

[Ham]

Yup. I am currently designing the network .....

And there you have, at least part, of it.

Systems are not just the sum of its components. You can have a great system but, if they aren't designed, installed or maintained properly you have a pile of useless. FTR, my role is the integrator of the various stuffs, like Network, DR, Server, Service, etc etc

Go on, another anecdote (dating back about 15 years). Another major corp with highly redundant highly secure systems lost complete contact with their, erm, contact centre. I happened to be there and got involved. Turned out that the dual redundant inbound circuit terminations when installed had not only been plugged in to the same dual socket, but a wall socket instead of a cabinet socket had been chosen. And someone had plugged in a kettle. That was all it needed.

[Mr Larrington]

The current explanation - that a power failure can knock out one or more critical, global systems for hours - defies credibility. That such systems have no redundancy, failover options, Disaster Recovery procedures, is not credible.

Something's up - and the cover story isn't very convincing.

I used to work for an arm of a BigCo in which we had a UPS to keep things going until the backup gennies kicked in.  One day a lightning strike took out half the town's electricity, ours included.  The UPS did its job admirably, the gennies started, all happy.  Then the gennies stopped and the UPS batteries went flat.  Someone had failed to ensure that the gennies had fuel tanks containing diesel rather than fumes...

[Pickled Onion]

Go on, another anecdote (dating back about 15 years).

If we're doing anecdotes, I was at a major financial when 9/11 happened. That was when they found out it was a bad idea to have their secondary systems in Manhattan as a backup to the primary, in Manhattan  Learning from this, they moved the secondary to New Jersey. A few years later Hurricane Sandy hit and took them both down.

[Ham]

Was that the corp with split systems across the two towers, or a different one?