Author Topic: spanning physical drives into one volume  (Read 2805 times)

Wombat

  • Is it supposed to hurt this much?
spanning physical drives into one volume
« on: 24 April, 2017, 10:53:02 am »
My PC (i7 4770k, 32GB RAM) has 3 1TB SSDs in it, one for OS and assorted general garbage and documents, one for my railway information, and one for photos.  The photos one is going to fill up before very long, despite aggressive editing on my part recently.  *I really DO need all of my photos available in one place, as I often get asked "have you got a photo of the back end of a particular wagon taken in 1972?" or "have you got the ones you took of us at the ball in 2015?" or something like that, so I need to be able to scroll through them to find it, and anyway I just like to be able to see them all easily.

Option one is to buy a new 2TB SSD, which is a moderately scary price at the moment (£559 is the best I've found), but I gather there is this "spanning" business where I could get another 1TB drive, and combine them into one volume.  Its a bit of a pain as I'd have to back it all up first separately as it involves deleting the volumes before you combine them, but is it safe or reliable enough?

I probably could stretch to the 2TB drive, but I'm reluctant on the grounds that I'm about to move house, which we all know is a VERY expensive time... Currently there is room for 2200 or so additional uncompressed RAWs on that drive.  A couple of events could blow that.
Wombat

Morat

  • I tried to HTFU but something went ping :(
Re: spanning physical drives into one volume
« Reply #1 on: 24 April, 2017, 10:56:53 am »
This sounds horribly like you're contemplating a Raid0 setup which is entirely possible but leaves you vulnerable to failures. If one drive fails in the Raid, the whole volume (ie 2Tb) is lost.
Everyone's favourite windbreak

Re: spanning physical drives into one volume
« Reply #2 on: 24 April, 2017, 11:42:02 am »
Do you REALLY need all images on the SSD? Long term storage would be much more economic on standard magnetic, and you could get 2 x 4Tb with change for for the SSD 1Tb.....

iddu

  • Are we there yet?
Re: spanning physical drives into one volume
« Reply #3 on: 24 April, 2017, 11:48:44 am »
I'd offer you some moral support - but I have questionable morals.

ian

Re: spanning physical drives into one volume
« Reply #4 on: 24 April, 2017, 11:54:49 am »
I'm also not clear why you would need SSD for this – it won't offer much in terms of benefits for this type of archive and retrieve use. Use spinny media and don't forget to back-up. You'll do the whole thing for far fewer pounds than a capacious terabyte-plus SSD.

Wombat

  • Is it supposed to hurt this much?
Re: spanning physical drives into one volume
« Reply #5 on: 24 April, 2017, 12:40:06 pm »
My Capture One Pro catalogue will reference this storage, and I've found that the speed difference between SSD and spinny disc is considerable, especially considering each of the recent images (in the last year) are 85MB uncompressed RAWs.  In practice I find it difficult to separate current photos from "archive" ones, as sometimes suddenly a series of photos from several years ago becomes a live project, such as right now, when I have to produce a selection of re-optimised images of older photos for use on display and marketing material for the charitable organisation that I'm a trustee of.  As its "just growed" rather than been created with a formal strategy right from the start, I think separating and recataloguing it is a very major project that would take me half of my retirement, and I just don't have time right now.

I do appreciate that my use case may not be typical!  As well as general photography for my own enjoyment I also do event photography for Hampshire Regency Dancers and the Napoleonic Association, and hold the photo archive for the Friends of Sierra Leone National Railway Museum, and also hold the originals for part of the British Library endangered archives project (the Sierra Leone railway related stuff) as sadly they can't afford to store it at its full quality, so the publicly available online resource is at reduced quality, which is probably not a major issue for most people, but we often need to study old photographs in ridiculous detail, looking for clues to historical issues.

Yes, it does get backed up very regularly, and there are offsite copies of the critical stuff!
Wombat

Re: spanning physical drives into one volume
« Reply #6 on: 24 April, 2017, 01:28:21 pm »
Here's a suggestion - create a directory on your SSD called "Working" or some such, have more than one if you want, whatever. Copy the chunk of stuff that you're working on from magnetic to SSD. Keep the catalogue DB on SSD. That will give you performance for very limited additional time.

Or if money  is no object, just use SSD.  :P

Afasoas

Re: spanning physical drives into one volume
« Reply #7 on: 24 April, 2017, 02:00:21 pm »
If you are running Windows 10, you could use Storage Spaces. You could add an additional 1 TB SSD. Use one for your OS, and span the other 3 into a 2TB RAID5 volume which will at least cope with one disk failing.

OR: Obtain a smaller SSD (256GB will do for OS, unless you are installing lots of programs/software). Clone your existing OS disk (or install fresh) to the new 256GB drive and then use your existing 1TB SSDs in the storage pool.

https://support.microsoft.com/en-us/help/12438/windows-10-storage-spaces

Kim

  • Timelord
    • Fediverse
Re: spanning physical drives into one volume
« Reply #8 on: 24 April, 2017, 03:52:51 pm »
This sounds horribly like you're contemplating a Raid0 setup which is entirely possible but leaves you vulnerable to failures. If one drive fails in the Raid, the whole volume (ie 2Tb) is lost.

Indeed.  Even if you don't use RAID0, the usual LVM-style spanning methods would lose the entire logical volume if a single disk fails, even if the others are intact.  There are ways to do spanning that avoid that problem (unionfs springs to mind), but they tend to be a bit subtle.

I've no idea what tools Windows has to do this sort of thing.  I find that life is generally made easier by not storing data on Windows machines.

On the other hand, I can't think of many applications where you actually care about this failure mode.  If it's real data you care about like photos, then it's backed up somewhere else, so you might as well use RAID0 and get the performance boost (or use RAID5 or RAID6 and save a few headaches).  It would be a reasonable way to store rips and recordings of broadcasts on a media server, so that if you lose a disk you only have to re-rip or seek out an alternate source for the stuff you've lost and still actually care about.  (MythTV handles this at an application level - you can just point it at multiple directories and it will start each recording on whatever has the most free space.)

Re: spanning physical drives into one volume
« Reply #9 on: 24 April, 2017, 04:07:52 pm »
....MythTV handles this at an application level - you can just point it at multiple directories and it will start each recording on whatever has the most free space.

Don't tell me, it's an application for TV shows you mythed first time around?

Kim

  • Timelord
    • Fediverse
Re: spanning physical drives into one volume
« Reply #10 on: 24 April, 2017, 04:14:05 pm »
....MythTV handles this at an application level - you can just point it at multiple directories and it will start each recording on whatever has the most free space.

Don't tell me, it's an application for TV shows you mythed first time around?

It's supposed to be the software for a Mythical TV Convergence Box (back when the idea was new), except it actually works more like an electric monk - watching TV programmes so you don't have to.  Since you can keep throwing more disk at it, you never have to actually get round to watching things.

Back before we gave up on telly, I had a MythTV server that operated as a retirement home for old hard disks.

fuaran

  • rothair gasta
Re: spanning physical drives into one volume
« Reply #11 on: 24 April, 2017, 04:49:34 pm »
For photos, I would just split them in some way, ie by date. Then put them in separate folders on each drive.
Then you should be able to add all of the folders to whatever photo management software you are using, and treat it all as a single library. You can do this in Windows Photo Gallery, so you should be able to do with any proper photo management software.

Or you could use symbolic links. ie one main folder, then create symlinks within that, which actually point to folders on another drive.

David Martin

  • Thats Dr Oi You thankyouverymuch
Re: spanning physical drives into one volume
« Reply #12 on: 24 April, 2017, 09:39:43 pm »
....MythTV handles this at an application level - you can just point it at multiple directories and it will start each recording on whatever has the most free space.

Don't tell me, it's an application for TV shows you mythed first time around?

is it written in Lithp?
"By creating we think. By living we learn" - Patrick Geddes

frankly frankie

  • I kid you not
    • Fuchsiaphile
Re: spanning physical drives into one volume
« Reply #13 on: 25 April, 2017, 09:42:48 am »
I would (I do) just stick everything on a 2-drive Synology NAS, which will routinely handle space expansion and immunity to drive failure without you having to think about it, and give you access from any machine or, if you want, even remote locations. You can use cheap spinny 4Tb disks and set it to only spin up on demand.   Have a small local working SSD if you want, and sync it to the NAS periodically.
when you're dead you're done, so let the good times roll

Morat

  • I tried to HTFU but something went ping :(
Re: spanning physical drives into one volume
« Reply #14 on: 25 April, 2017, 01:18:08 pm »
mmm to be picky nothing is _immune_ to disk failure. It's quite possible for a second drive to fail during a rebuild. I've even had it happen to me on a RAID5 storage box that I now know should really have been a RAID6. That was a really bad time to find out that the replication partner had silently failed due to a power cut.

Synology are excellent choice, though. I like their boxes.
Everyone's favourite windbreak