Viewing 9 posts - 1 through 9 (of 9 total)
  • FAO Anyone involved in data storage/IT/Photography please ;)
  • 0303062650
    Free Member

    Hey,

    A photographer friend of mine currently has plenty of external HD's for which he uses to backup his data, to which is notes what customer info is on what drive in a little paper diary.

    Not good i said, it's 2010 and there is a better way of doing this.

    Other than build a pretty hefty NAS box with approx 20+TB's of storage, is anyone aware of any on-line solutions that may be able to offer him what he wants?

    Storing the data is done over both mac and pc but primarily mac. (PC is used as a printer interface, and ideally, a basis for a storage solution (using pc-based server tech, I am not too afraid of looking at linux-esq OS either)

    Does anyone have any thoughts on how to progress?

    I know the benefits of onsite storage are faster access, but less reliable.

    any thoughts / comments would be greatly appreciated.
    regards,
    Jonathan

    freddyg
    Free Member

    Does he really need 20+TBs of storage?

    You can put together an off the shelf NAS with 8 to 10TB for under a grand. You can do it for even less if you build your own fileserver.

    cranberry
    Free Member

    20 Tb is a lot of storage – what volume of data is he producing each month, and how far back does his collection go? How frequently does he need to access the back catalogue?

    If he's creating large amounts of data he's going to run into issues with bandwidth if storing things online.

    Could he write the back catalogue to DVD/blue ray/tape ( 2 copies ) and store them in 2 different locations then put current data on a NAS ?

    Fresh Goods Friday 696: The Middling Edition

    Fresh Goods Friday 696: The Middlin...
    Latest Singletrack Videos
    0303062650
    Free Member

    Hi there,

    Yeah, he shoots with a hassleblad 50mp camera and regularly 10's of gigabytes per shoot, and busy everyday!

    It seems a (custom built) NAS box would work out the right sort of price, and custom 20tb isn't that much more than custom built 10tb.

    We use carbonite so i'm waiting to hear back from them to see what they have to say.

    jt

    simonfbarnes
    Free Member

    I know the benefits of onsite storage are faster access, but less reliable.

    with that volume of data ordinary broadband would never keep up! One thing rarely mentioned is that hard disks need a periodic scan as the magnetisation on high density drives can lessen with time. A full surface scan with chkdsk will do it. If all the sectors are read, the drive electronics will rewrite any sectors with a reduced signal.

    molgrips
    Free Member

    If you rented a managed server, you could have it with as much storage as you wanted. Although as mentioned, data transfer would be prohibitive.

    +1 for optical storage stored off-site.

    brassneck
    Full Member

    That is a lot of storage to manage offsite , not mention issues with data integrity.. getting Gbytes of data for a single shot over even a T1 would get boring quick I would think.

    Remember that the worst part is managing the data – it's easy to buy something with a capacity of 20Tb, but you've got to back that up, re orgnise when required, expand the space… some sort of hierarchal storage would be best if he needs to retrieve shots (Fast disk pool ,slow disk pool, tape/optical). Dedup would also be nice to wring as much out of the space as possible, though I don't know how good it would be with graphic files.

    Best would be NetApp filers to cover any eventuality but don't expect to get any change out of any money you have.
    Cheapest would probably be a commodity server box with a lot of SATA DAS in RAID5 and a tape drive capable of backing it all up. Openfiler/FreeNAS operating systems let you provison this disk how you need it (iSCSI target, NFS, SMB, whatever) and offer a degree of flexibility, but come at a setup cost (technical knowledge required). W2K8 has some interesting options for storage systems too, this would probably be the most accessible way to provide it on commodity hardware. Don't think I've seen a SoHo NAS box capable of scaling to 20Tb.

    20Tb is going to cost one way or another – cash, or manageability. Just remember to include backup, no disk array is infallible. It might be rare for 3 disks to fail in an array, but I've seen it happen twice. If it's not replicated, you're reaching for the tapes and waiting for a few hours.

    simon_g
    Full Member

    The Drobo boxes are good for this sort of thing – they top out at 14.5TB usable (8 disks, current max size 2TB, some goes to redundancy in case a disk fails) but a couple would meet his needs OK.

    Looking at just over a grand for the base DroboPro box, 2TB SATA disks are just dropping under the £100 mark. So a little over £5k for a usable 29TB online, in quite a small, quiet package. Still have the problem though of what happens in case of theft, flood, fire. Unfortunately tapes haven't been keeping up with disk sizes so 800GB/tape is about as good as it gets.

    Carbonite only cover internal drives on their "home"-type unlimited plans and the business ones you pay for what you use, which will get expensive, fast. Amazon will do you as much storage as you like, but keeping 20TB on their servers will cost over $3000 a month, and there are data transfer fees on top of that to get it there and for updates.

    What does he retain, and why? Does he often have to go back to previous shoots, and does he need full (RAW or equivalent) copies of that? What would be the cost of *not* going back to them, and how does that compare to the cost of storage/backup?

    I think he could do with sorting out his workflow a bit – the guys I know who shoot sports events, weddings, etc generally won't keep RAWs beyond the initial period where people will order prints. Their archive will just be JPEGs of the usable shots (ie. no duplicates, no out-of-focus, no mis-framed shots) and as well as keeping a copy on disk they'll tend to upload those to flickr (a pro account lets you upload as much you like, and you can keep them private) or another cloud service as offsite backup.

    tonyd
    Full Member

    Lots of good advice here already, I'd add that if this guy really needs to store 20+ TB of data and his livelihood depends on it then a budget above £1000 would be a very good idea. I'd go for one of three options:

    1) Hosted solution. Rent a managed server with a lot of storage. Could most likely go for a choice of operating systems. As already mentioned if you're going to need to push lots of GB across to it every day this is going to be a pain and probably not worth pursuing.

    2) Home based file server with attached disk and tape storage. Could be something as simple as a windows/linux box providing a local network share. My preference would be an x86 based server running OpenSolaris and ZFS, but I work for Sun (Oracle) so will say no more 🙂

    3) Buy a high capacity RAID array and attach it to the Mac (assuming it's not a laptop), sounds like this is the machine used for most of the processing? Makes sense to keep the data local to that.

    At the very least whatever disk based storage you provide you should use some form of RAID for redundancy and back it up to tape for offsite storage (mates house?!)

    /tuppence

    Edit to add: Cloud based solution is a good idea for offsite backup option if you don't need to keep *everything*

Viewing 9 posts - 1 through 9 (of 9 total)

The topic ‘FAO Anyone involved in data storage/IT/Photography please ;)’ is closed to new replies.