Police delete 150,000 records

[toontra]

How is this even vaguely, conceivably possible?  I would consider myself to be an utter twat if I were to delete files in error without having, at best, at least 1 backup, or at worst, some way of rescuing those files from hard drives.  My files aren't of national security concern. 

Those that have gone missing may very possibly result in serious crimes going un-detected or unsolved.  Cold case reviews are often based on collating seemingly un-related evidence collected in previous arrests, even if no charges were brought at the time, particularly DNA, fingerprints and other forensic stuff.

[TheLurker]

How is this even vaguely, conceivably possible? 

In no particular order..

- Shit UI.  Confusing confirmation prompts.
- Shit UI.  No confirmation prompts at all.
- Shit UI.  An overabundance of (confusing) confirmation prompts lead to learnt "click through" behaviour.
- Shit UI.  Difficult to set up and use record filtering.
- Poorly trained staff, because outsourcing to the cheapest bidder is always a good idea, not setting up weeding params correctly
- Knackered staff,  because outsourced staff being flogged into the ground to keep costs under control, not setting up weeding params correctly
- Coding errors.  Filter to weed out candidate records is set up (apparently) correctly, but code using the search params is shagged.

And, of course, a combination of one or more of the above.

Oh, and one for the tinfoil hat brigade.  It was deliberate.  Someone wanted a particular record disappeared and an "accidental" deletion of many tens of thousands of records is a good way to hide it.  IIRC this is the underlying idea in Agatha Christie's, "The ABC Murders".  The real target's killing is hidden and the identity of the murderer by swamping and misleading the investigators with a series of killings.

ETA.
And even if there were sensible, easy to understand, confirmation prompts there's no sensible and quick way that I know of for reviewing a batch of 150,000 records to make sure you've selected the *right* 150,000 records,  You have to trust that the the process for selection is correct.

[Lightning Phil]

Someone in IT service delivery asked to do deletion on behalf of a senior user , and makes a mistake under pressure. Senior user denies all accountability.  Seen it happen in a large company I used to work for. It took 36 hours to fully recover the systems, as they were operational, and could not be just switched off, whilst a static backup was applied. If a user can directly delete 150,000 separate records directly, that’s a bit worrying.

After above incident in company I worked for, any such requests from users had to go through the change request approval process. It mostly cut out 99% of such requests or forced development to deliver the user functionality they’d dropped previously.

[Davef]

I had read it was a coding error. I am sure they will have back ups but they will need to work out how to merge the backed up data into the live system that has since moved on. It is part of the cunning design of SQL that when you expect to be deleting one record you can delete 150,000.

In my first job I deleted 3 months of work for which no back up existed.

[toontra]

In my first job I deleted 3 months of work for which no back up existed.

 

[Davef]

In my first job I deleted 3 months of work for which no back up existed.

 

I discovered you can do 3 months work in about 3 weeks when it is being done for the second time and under a certain amount of pressure.

[ian]

> cd priti.patel
priti.patel > rm *.*
priti.patel > No files found. The directory is empty.

[Feanor]

In my first job I deleted 3 months of work for which no back up existed.

 

I discovered you can do 3 months work in about 3 weeks when it is being done for the second time and under a certain amount of pressure.

At my former work, my boss had been fiddling about coding some script stuff on one of the Unix boxes.
Rather than storing it somewhere sensible, it was in a /tmp directory, which I cleared out one day.

You can guess how well backed up it was...

[Davef]

> cd priti.patel
priti.patel > rm *.*
priti.patel > No files found. The directory is empty.

That was very similar to my catastrophic data loss (I still have the odd flashback 36 years on)

What I meant to type was

cd testdata
rm -r *

What I actually typed was

cd testdate
rm -r *

Don’t type ahead.
The cd failed and testdata’s siblings src,build etc went.

[andrewc]

How is this even vaguely, conceivably possible? 

In no particular order..

- Shit UI.  Confusing confirmation prompts.
- Shit UI.  No confirmation prompts at all.

Ho Yuss!  A colleague once went to delete an individual entry in a file, which would have prompted a warning message.  Instead he accidentally deleted the master file, which gave no warning.  This didn't have an effect until a midnight file synch, after which all hell broke loose.  Thankfully there was a backup, but it took ages for people to work out what had actually happened & re-install it.
He'd previously warned that this was a possibility & management thought he'd done it deliberately to make a point.  After some 3 months of unpleasantness he was allowed back into the office, but doing a different job.

[rogerzilla]

> cd priti.patel
priti.patel > rm *.*
priti.patel > No files found. The directory is empty.

I actually did that on the LAN at Birmingham University, and they never found out it was me 

[Kim]

I had read it was a coding error. I am sure they will have back ups but they will need to work out how to merge the backed up data into the live system that has since moved on. It is part of the cunning design of SQL that when you expect to be deleting one record you can delete 150,000.

Let's not overlook the other popular option: A backup strategy that diligently duplicates the accidental deletion before anyone works out what's happened.

[Pingu]

I keep telling people to Ctrl-a, Shift-Delete 

[Ham]

As long as it included  De Do Do Do and Wanking on the Moon, why does anybody mind?

[Pickled Onion]

I had read it was a coding error. I am sure they will have back ups but they will need to work out how to merge the backed up data into the live system that has since moved on. It is part of the cunning design of SQL that when you expect to be deleting one record you can delete 150,000.

The way it's being reported there aren't backups. And if there were, it wouldn't be a story. Which as toontra said "How is this even vaguely, conceivably possible?".

As for "work out how to merge the backed up data... " that should be part of any backup process. It doesn't consist of throwing stuff on a tape and then working out how to read it off when you've lost the original. We test our DR (disaster recovery) failover of the whole system every two weeks and we test restoring from backups eight times every single day.

In my first job I deleted 3 months of work for which no back up existed.

Ha, yes. Back in the days when developers used to have access to all systems and we turned up to work in spurs and stetsons, I dropped a table from the live production database thinking it was the dev one. No problem, someone got in a taxi with the tape and an hour later it was back to normal and we replayed that morning's transactions. No-one broke a sweat.

[Pickled Onion]

Those that have gone missing may very possibly result in serious crimes going un-detected or unsolved.  Cold case reviews are often based on collating seemingly un-related evidence collected in previous arrests, even if no charges were brought at the time, particularly DNA, fingerprints and other forensic stuff.

They should have no right to keep DNA and fingerprints from people on the off-chance they might match it up to a crime some day. Lots of people are wrongly arrested, or asked to give samples to eliminate themselves from an enquiry.

https://justice.org.uk/dna-retention-police/

[andyoxon]

Could be 400 000 records...

https://www.theguardian.com/politics/2021/jan/15/priti-patel-under-fire-as-150000-police-records-accidentally-lost

[Davef]

Could be 400 000 records...

https://www.theguardian.com/politics/2021/jan/15/priti-patel-under-fire-as-150000-police-records-accidentally-lost

Unfortunately I could not read that article because it was blocked by my browsers exaggeration filter after detecting 24 uses of the word “potential” or “potentially” but I agree it sounds potentially worse than I thought originally.

[Davef]

I had read it was a coding error. I am sure they will have back ups but they will need to work out how to merge the backed up data into the live system that has since moved on. It is part of the cunning design of SQL that when you expect to be deleting one record you can delete 150,000.

Let's not overlook the other popular option: A backup strategy that diligently duplicates the accidental deletion before anyone works out what's happened.

They are saying they hope to recover the data soon, but that could of course mean running around old crime scenes with swabs.

[Davef]

The way it's being reported there aren't backups.

Saying they are going to recover the data might of course mean men with swabs revisiting old crime scenes. More likely busy negotiating a price with the outsourced IT suppliers.

It doesn't consist of throwing stuff on a tape and then working out how to read it off when you've lost the original.

After several hours playing poker we found a card had fallen on the floor an hour into the game.

After several hours playing poker we found one of the players bodies and been taken over by an alien an hour into the game.

Restoring data in even the reasonably predictable situation is far more complex than backing it up.

[hatler]

Given that some data from these records is required to be permanently deleted, that would explain the lack of a backup. And in that case, all the more reason for significantly stronger checks and controls before the data is permanently deleted.

[Kim]

Given that some data from these records is required to be permanently deleted, that would explain the lack of a backup.

I've wondered about the intersection of data protection and similar regulations on otherwise-sensible backup strategies.  If you've got, say, 3 months worth of backups, what happens when you hold data that's required to be deleted permanently after some shorter period?  Presumably someone has to go into the backups to delete it^[1], which means you're now a single human error away from losing something important...


_{[1] Or there's some automated expiry-date system which provides a friendly interface to doing the same thing.}

[Davef]

Given that some data from these records is required to be permanently deleted, that would explain the lack of a backup.

I've wondered about the intersection of data protection and similar regulations on otherwise-sensible backup strategies.  If you've got, say, 3 months worth of backups, what happens when you hold data that's required to be deleted permanently after some shorter period?  Presumably someone has to go into the backups to delete it^[1], which means you're now a single human error away from losing something important...


_{[1] Or there's some automated expiry-date system which provides a friendly interface to doing the same thing.}

I am not sure the “right to be forgotten” extends to evidence held by the police, but in general gdpr and backups is complicated. It is generally considered unreasonable to have to go and edit all your backups but you will need to put systems in place so that the forgotten data can’t be restored.

[Mr Larrington]

Given that some data from these records is required to be permanently deleted, that would explain the lack of a backup.

I've wondered about the intersection of data protection and similar regulations on otherwise-sensible backup strategies.  If you've got, say, 3 months worth of backups, what happens when you hold data that's required to be deleted permanently after some shorter period?  Presumably someone has to go into the backups to delete it^[1], which means you're now a single human error away from losing something important...


_{[1] Or there's some automated expiry-date system which provides a friendly interface to doing the same thing.}

At my last job most backup tapes got overwritten eventually, but the month-end ones were retained.  The Auditors wanted us to remove credit card data from the older ones.  We politely told them to [“Partake of sex and travel” – The Invigilator], but they then insisted that the keep-for-all-eternity backups could no longer be kept offsite because it wasn’t sufficiently secure, completely ignoring the fact that the biggest threat to the security of said data was a bent BOFH

[ian]

Given that some data from these records is required to be permanently deleted, that would explain the lack of a backup.

I've wondered about the intersection of data protection and similar regulations on otherwise-sensible backup strategies.  If you've got, say, 3 months worth of backups, what happens when you hold data that's required to be deleted permanently after some shorter period?  Presumably someone has to go into the backups to delete it^[1], which means you're now a single human error away from losing something important...


_{[1] Or there's some automated expiry-date system which provides a friendly interface to doing the same thing.}

I am not sure the “right to be forgotten” extends to evidence held by the police, but in general gdpr and backups is complicated. It is generally considered unreasonable to have to go and edit all your backups but you will need to put systems in place so that the forgotten data can’t be restored.

This is how we do it on those systems where it needs to happen – in very basic terms there's a file of records that shouldn't be restored. They're still on the backups but it seems a sufficient approach to satisfy the lawyers.