Author Topic: Photo file corruption  (Read 2671 times)

Ben T

Photo file corruption
« on: 04 September, 2017, 11:33:24 pm »
I've got some photo files backed up onto a USB stick, and today I tried to access them and they look 'corrupted'.
When opened up in a photo viewer they look the right size, but all except a thin strip at the top is just solid grey. The thin strip at the top is that portion of what the photo should look like. Weirdly though, when first browsing to them in "large icons" mode in Windows Explorer, it momentarily displays the icon as the full photo, before reverting to displaying the mostly blank grey version, which sort of suggests it has got the information somewhere...? Or is it simply caching thumbnails in some way?

Furthermore how can it happen in the first place? They were copied onto a san disk 64GB memory stick using robocopy from Windows which didn't report errors. Each photo seems to have the same amount chopped off, about 80 or 90%.
Any recommendations for reliable backup software that verifies the backup?

Jaded

  • The Codfather
  • Formerly known as Jaded
Re: Photo file corruption
« Reply #1 on: 05 September, 2017, 06:46:26 am »
Hmm, I've had similar on OS X, with some pretty important photos - my Olympics and Paralympics photos. It appeared to be some kind of conflict between the RAW files ,the disk they were on and the program I was using, Aperture. Other photo programs didn't do it and other sets of photos taken before and after didn't either. I never got to the bottom of why this was, I think it may have been something to do with how I had copied them from the cards to the library in the first place.

I had a backup and I put these onto a media that was unwriteable, DVD, so that the original files couldn't  be damaged. That worked.

It was scary.
It is simpler than it looks.

T42

  • Apprentice geezer
Re: Photo file corruption
« Reply #2 on: 05 September, 2017, 08:07:13 am »
JPEGs [may]* contain a thumbnail, which is probably where Windoze gets them from.

My problem isn't corrupted files, it's the fact that all my UAF Audax pics from 2009/10 have effed off into the void, on my three backups as well.  Directory structure AFU.

* can't remember if it's always there; it's a long time since I played with JPEG guts.
I've dusted off all those old bottles and set them up straight

Jaded

  • The Codfather
  • Formerly known as Jaded
Re: Photo file corruption
« Reply #3 on: 05 September, 2017, 08:09:58 am »
My experience was as Ben's in that the photo looked ok then suddenly blocks went haywire. I'm pretty sure it wasn't a thumbnail first.

Ben try copying them to a CD/DVD and opening them with a completely different program, or better still, on a different computer.
It is simpler than it looks.

ian

Re: Photo file corruption
« Reply #4 on: 05 September, 2017, 08:35:37 am »
It's something I've seen a few times over the years, the upper blocks of image go AWOL. They look fine on the camera, but it's gone during transfer. Someone cleverer than me put it down a corrupted file index, so the data is there, software just doesn't know how to display it. Presumably, such files would have a different checksum.

Some software is more tolerant than other and will attempt to rebuild the index, but it's a case of trying several different applications.

I've no idea why, it seems to happen in the act of transfer. I've seen similar stuff with video, again mis-indexing, also during transfer. In my experience it's always involved cables, but that's anecdotal.

Ben T

Re: Photo file corruption
« Reply #5 on: 05 September, 2017, 06:49:09 pm »
My experience was as Ben's in that the photo looked ok then suddenly blocks went haywire. I'm pretty sure it wasn't a thumbnail first.

Ben try copying them to a CD/DVD and opening them with a completely different program, or better still, on a different computer.

Hmm, thanks - tried that but still the same.

Fortunately I've got them on flickr. Which is the only place some of them currently are.
But still annoying. Does anybody know, if you "download original" on flickr, is the file the exact same one you uploaded, or has it been processed by flickr in any way - is it the same quality?
It's the same original pixel size, so hopefully it is, but just wondered all the same, if any lossy compression has been applied.


Kim

  • Timelord
    • Fediverse
Re: Photo file corruption
« Reply #6 on: 05 September, 2017, 06:57:54 pm »
Does anybody know, if you "download original" on flickr, is the file the exact same one you uploaded, or has it been processed by flickr in any way - is it the same quality?
It's the same original pixel size, so hopefully it is, but just wondered all the same, if any lossy compression has been applied.

This is what md5sums are for.  I don't have a flicker account to test, thoguh.

Ben T

Re: Photo file corruption
« Reply #7 on: 05 September, 2017, 07:06:30 pm »
I'll report back  :) as I do have some of them as originals.

I blame dropbox - I deleted them off my hard drive, because I originally tried sharing them with others who only had free dropbox accounts before snagging the rule that dropbox shared folders use up space in all sharer's accounts!  >:(

Re: Photo file corruption
« Reply #8 on: 05 September, 2017, 07:48:14 pm »
If you are still stuck try a search / thread on here https://www.talkphotography.co.uk/forums/

It's a common problem and there are software tools but I've never had to use one.

ian

Re: Photo file corruption
« Reply #9 on: 06 September, 2017, 07:59:32 am »
I'll report back  :) as I do have some of them as originals.

I blame dropbox - I deleted them off my hard drive, because I originally tried sharing them with others who only had free dropbox accounts before snagging the rule that dropbox shared folders use up space in all sharer's accounts!  >:(

Dropbox does bit-by-bit synching via whatever the AWS service is. If it gets interrupted there can be problems.

If they ever successfully synched there may be copies on Dropbox itself, as it keeps previous versions.

frankly frankie

  • I kid you not
    • Fuchsiaphile
Re: Photo file corruption
« Reply #10 on: 06 September, 2017, 09:27:28 am »
JPEGs [may]* contain a thumbnail, which is probably where Windoze gets them from.

And independently of that, Windows Explorer may generate its own thumbnails database.
when you're dead you're done, so let the good times roll

T42

  • Apprentice geezer
Re: Photo file corruption
« Reply #11 on: 06 September, 2017, 11:22:20 am »
Yes, all those irritating thumbs.db files.  Back in the 90s I wrote an image-management system that used a similar system. Eventually trashed it in favour of a central DB - people grumbled about all those extra files.
I've dusted off all those old bottles and set them up straight

Afasoas

Re: Photo file corruption
« Reply #12 on: 09 September, 2017, 12:18:50 am »
Photos are stored as binary data - a stream of bytes written to a file.
A byte is a series of bits with a value of 0 or 1.

A bit flipped from a 0 to a 1 or vice versa in a critical place will result in a corruption and the file or part of a file no longer making sense. All sorts of things can cause a bit to flip - some file formats and file systems/protocols will be more resilient to it than others. Flash memory, particularly at consumer level where TLC and QLC are more common, is allegedly more prone to 'silent' bit rot as it relies on charge not dissipating from transformers within the memories 'cells'. TLC and QLC flash storage generally needs some form of error correction to account for flipping bits.

https://en.wikipedia.org/wiki/Multi-level_cell

I wouldn't use USB flash storage as a backup.
And this is one of the reasons I use ZFS for both my 'nas' and 'backup nas', in raidz (mirrored) and raidz2 (raid6) configurations. ZFS stores data with checksums so a regular scrub can identify irregularities and correct them. I haven't had any on current hardware, but I have on previous hardware. If I had not been using ZFS I'd have not been aware of the issue and resolving it might have been a bit more involved, if indeed that was possible.

Of course, storage might not be the issue. The files could have been corrupted by a power surge, high levels of EM interference, faulty RAM, chipset or CPU.

vorsprung

  • Opposites Attract
    • Audaxing
Re: Photo file corruption
« Reply #13 on: 09 September, 2017, 10:12:04 am »
Although SAN is a good brand, normal USB sticks aren't a great way to store stuff you really really want to keep

If you really really want to keep something then the correct solution is to copy it and keep copying.  Hopefully one of the copies will survive

I did have some special "archive" USB sticks once, dunno if they are generally available

Ben T

Re: Photo file corruption
« Reply #14 on: 09 September, 2017, 02:56:05 pm »
ok cheers, so what's a good external backup drive then - is this a better format than a standard flash drive?
https://www.amazon.co.uk/Seagate-Backup-Portable-External-Drive/dp/B00FP50JAG/ref=sr_1_4?s=computers&ie=UTF8&qid=1504965033&sr=1-4&keywords=usb+backup+drive
anything cheaper but just as good? (not that that one's unreasonable)

Afasoas

Re: Photo file corruption
« Reply #15 on: 09 September, 2017, 05:12:36 pm »
How much data are you looking to back up?
Microsoft One Drive, Google Drive or Dropbox is probably the cheapest easily accessible solution - I think they all do 'version' control so you can always retrieve older versions of corrupted files.
If you have too much data for these services, then a good internet connection maybe one of the fuller onliney-cloudy-backup solutions is an option.
Failing that, a 2 bay Synology or QNAP NAS.

I wouldn't use SSD (flash) for archival storage.

Ben T

Re: Photo file corruption
« Reply #16 on: 09 September, 2017, 05:48:58 pm »
No, they're fine, and I do use them - but would like something additional, and non internet based as well.

Ben T

Re: Photo file corruption
« Reply #17 on: 09 September, 2017, 07:00:23 pm »
I'd rather the smallest size they do, because once it gets full I'd rather just get another one, I'd rather 2 x 1tb than 1x2tb.

Re: Photo file corruption
« Reply #18 on: 09 September, 2017, 08:07:06 pm »
I use a combination of an external drive and Windows file history for my backups.
This means all new or changed files are automatically backed up without further effort. Good but with the
drive always plugged in its vulnerable to ransomware. So I've got a second drive for manual backup only one of which is online
at one time.

Secure backup rapidly gets complicated!
 

Afasoas

Re: Photo file corruption
« Reply #19 on: 09 September, 2017, 10:37:14 pm »
Honestly wouldn't rely in a single external HDD anyway. I'd get a few of adequate capacity and rotate them. They are just too fragile.

Mr Larrington

  • A bit ov a lyv wyr by slof standirds
  • Custard Wallah
    • Mr Larrington's Automatic Diary
Re: Photo file corruption
« Reply #20 on: 10 September, 2017, 05:55:11 am »
ok cheers, so what's a good external backup drive then - is this a better format than a standard flash drive?
https://www.amazon.co.uk/Seagate-Backup-Portable-External-Drive/dp/B00FP50JAG/ref=sr_1_4?s=computers&ie=UTF8&qid=1504965033&sr=1-4&keywords=usb+backup+drive
anything cheaper but just as good? (not that that one's unreasonable)

I've got four of them in various size around the place; they are there and they work.
External Transparent Wall Inspection Operative & Mayor of Mortagne-au-Perche
Satisfying the Bloodlust of the Masses in Peacetime

Ben T

Re: Photo file corruption
« Reply #21 on: 10 September, 2017, 03:40:26 pm »


Does anybody know, if you "download original" on flickr, is the file the exact same one you uploaded, or has it been processed by flickr in any way - is it the same quality?
It's the same original pixel size, so hopefully it is, but just wondered all the same, if any lossy compression has been applied.

This is what md5sums are for.  I don't have a flicker account to test, thoguh.

I'll report back  :) as I do have some of them as originals.

I blame dropbox - I deleted them off my hard drive, because I originally tried sharing them with others who only had free dropbox accounts before snagging the rule that dropbox shared folders use up space in all sharer's accounts!  >:(

Yes - they are the exact same files.  :)

frankly frankie

  • I kid you not
    • Fuchsiaphile
Re: Photo file corruption
« Reply #22 on: 10 September, 2017, 06:39:13 pm »
Just to mention that Google Drive is set to change in some way, later this year.
https://www.dpreview.com/news/3622374461/google-drive-and-photos-support-to-end-in-december
when you're dead you're done, so let the good times roll