Cheap Home Cloud ~ Raspberry Pi + USB Drives
Friday random semi-serious question; Like many people I have too much data some work, some create some family, some other...etc. I have some on some backup USB drives that I have never gone back to and also a few unplugged HDD's in the tower (I can't remember even why). I don't really want to go and buy something (or pay for data of cloud storage) but I thought I might do something with the stuff I have lying around. Does anyone know if this is possible: - Set up a Raspberry Pi server which exposes a single file system - Link it to the USB HDD's of different sizes that I have lying around - I save my data to it for backups - The data is placed redundantly over the drives so it can recover from one [or more] dying Bonus point: - I can remove one of the drives and access the data on it directly (so it is FAT32 not Linux RAID) Any ideas? GFS? Cheers P
On 29 Jul 2016 7:03 pm, "Piers Rowan via luv-main" <luv-main@luv.asn.au> wrote:
Friday random semi-serious question;
Like many people I have too much data some work, some create some family,
some other...etc. I have some on some backup USB drives that I have never gone back to and also a few unplugged HDD's in the tower (I can't remember even why).
I don't really want to go and buy something (or pay for data of cloud
storage) but I thought I might do something with the stuff I have lying around. Does anyone know if this is possible:
- Set up a Raspberry Pi server which exposes a single file system - Link it to the USB HDD's of different sizes that I have lying around - I save my data to it for backups - The data is placed redundantly over the drives so it can recover from
one [or more] dying
Bonus point: - I can remove one of the drives and access the data on it directly (so
it is FAT32 not Linux RAID)
Any ideas? GFS?
Not to distract from Piers's enquiry but isn't home cloud an oxymoron?
Not to distract from Piers's enquiry but isn't home cloud an oxymoron?
You have no idea of where I live or how big my house is you insensitive clod!!! :-) I was on Newegg today and they have a product group called home cloud - its basically wireless NAS. My usage was to imply I don't want to have to care what happens on the other side of the storage node - like cloud - but also built from the 100 random IT purchases over the last 10 years......
On Fri, 29 Jul 2016 07:03:30 PM Piers Rowan via luv-main wrote:
- Set up a Raspberry Pi server which exposes a single file system - Link it to the USB HDD's of different sizes that I have lying around - I save my data to it for backups - The data is placed redundantly over the drives so it can recover from one [or more] dying
Bonus point: - I can remove one of the drives and access the data on it directly (so it is FAT32 not Linux RAID)
Any ideas? GFS?
GFS (GFS2 being the supported version) is about multiple nodes using the same shared block devices, pretty much the opposite of what you need. If you want multiple block devices of different sizes in one array then BTRFS is the only option. But RAID-1 is the only level of RAID that should be considered safe with BTRFS at this time. Also this files the "remove one drive" criteria. To satisfy the "remove one drive" criteria you want to have some user-space system for synchronising files by copying them around. Whatever you choose make sure you have checksums, USB devices aren't the most reliable ways of storing data. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Any ideas? GFS?
GFS (GFS2 being the supported version) is about multiple nodes using the same shared block devices, pretty much the opposite of what you need.
Good! I've never managed to get it to work anyway!
To satisfy the "remove one drive" criteria you want to have some user-space system for synchronising files by copying them around.
Something like having the largest drive mounted as the root file system’s /home and then using a cron job ever hour to rsync the data (or portions of it) to the smaller drives? rsync -avh --include '*/' --include '*.php' --prune-empty-dirs /home/ /mnt/code_drive rsync -avh --include '*/' --include '*.jpg' --prune-empty-dirs /home/ /mnt/images_drive rsync -avh --include '*/' --include '*.sql' --prune-empty-dirs /home/ /mnt/data_drive I like this idea as it combines two of my favourite things, rsync and being a cheapskate! Any pitfalls? Cheers P
On 30.07.16 07:40, Piers Rowan via luv-main wrote:
rsync -avh --include '*/' --include '*.php' --prune-empty-dirs /home/ /mnt/code_drive rsync -avh --include '*/' --include '*.jpg' --prune-empty-dirs /home/ /mnt/images_drive rsync -avh --include '*/' --include '*.sql' --prune-empty-dirs /home/ /mnt/data_drive
I like this idea as it combines two of my favourite things, rsync and being a cheapskate!
Any pitfalls?
A missing rsync argument: -c, --checksum even though: With this option, rsync compares a 128-bit checksum for each file that has a matching size. That can slow things down significantly. because: By default, rsync finds files that need to be transferred using a "quick check" algorithm which only looks for files that have changed in size or in last-modified time. and I've had USB drives spontaneously corrupt several % of the bytes in several % of files, and there's no file size change in that. (OK, that was flash, but once bitten, ever sensitive.) My only other concern is: What's the disaster recovery option? (Off-site) Mine includes backup to a laptop which comes with whenever I'm away for days, plus a USB stick which goes in my pocket with the car keys. (So even if I'm only nipping to the shops.) Data which exists only at one location exists only in your imagination, eventually, I figure. Erik
On 01/08/16 19:38, Erik Christiansen via luv-main wrote:
A missing rsync argument: -c, --checksum
Good point.
(So even if I'm only nipping to the shops.) Data which exists only at one location exists only in your imagination, eventually, I figure.
I agree. I have a storage server in Germany that I can back it up to but it has too much space. This leaves lazy people just to back up everything and I end up with spending $$$ on stuff that is too vast and disorganised to ever appreciate the value of (like the 1 in 20 photo of your kids where you actually got the shot *perfect* that renders the other 19 uninteresting and imperfect by comparison). Thanks PIers
On Tue, 2 Aug 2016 06:32:06 PM Piers Rowan via luv-main wrote:
On 01/08/16 19:38, Erik Christiansen via luv-main wrote:
A missing rsync argument: -c, --checksum
Good point.
My preferred solution to such problems is to use BTRFS and put the checksum at the filesystem layer. This means that you can check the validity of the data when the source isn't present. But the downside is that Windows can't read it. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On Fri, Jul 29, 2016 at 07:03:30PM +1000, Piers Rowan wrote:
Friday random semi-serious question;
Like many people I have too much data some work, some create some family, some other...etc. I have some on some backup USB drives that I have never gone back to and also a few unplugged HDD's in the tower (I can't remember even why).
I don't really want to go and buy something (or pay for data of cloud storage) but I thought I might do something with the stuff I have lying around. Does anyone know if this is possible:
unfortunately, doing this with USB drives defeats the purpose. USB drives just aren't reliable enough for a "cloud"-like always-on backup solution. IMO if you're going to do it at all, do it right - and with an eye towards the future. that means spending some money. you'll need a case, a motherboard, RAM and a powersupply. The Silverstone DS380 is very nice (8 x 3.5/2.5" hot-swap SAS/SATA bays + 4 internal 2.5" bays), but costs nearly $200 ($185 from CPL is cheapest i could find). And that doesn't include a power supply - it takes an SFX small-form-factor PSU...that's another $50-$100. http://www.silverstonetek.com/product.php?pid=452 http://www.shopbot.com.au/pp-silverstone-ds380b-price-481778.html the silverstonetek.com site seems to be down at the moment, so here's a review: http://www.silentpcreview.com/SilverStone_DS380 even if you have no intention of ever filling all 8 hot-swap bays, the extras are useful for occasionally inserting a drive, running a backup (e.g with rsync or btrfs send or zfs send) and then removing it, AND taking it off-site (e.g. to work, to a family member, or anywhere else) and having hot-swap bays that can hold 3.5" AND 2.5" drives is ideal. SSDs are rapidly becoming bigger and cheaper. for any business or org with a good budget they're already a better choice than HDDs for storage servers (e.g. samsung 2TB SSDs are under $1000 in au, and they just released a 4TB SSD for $1500 USD), and it won't be more than a few years before 1-4TB drives are cheap enough to be a no-brainer for home use too. There's also the Norco ITX-S8 which looks better made, but is $330 from techbuy (who have some very good stuff but usually at inflated prices). I can't find any other australian suppliers. http://www.norcotek.com/product/itx-s8/ http://www.techbuy.com.au/p/403563/CASINGS_ITX_CASES/Norco/ITX-S8.asp (nice, i just noticed it has 2 x SFF-8087 connectors. probably no use to you, but very convenient if you're using an LSI 9211 or similar SAS/SATA card like I am) I wish these cases had been around when i built my fileserver a few years ago. lack of decent mini-itx cases with hot-swap bays was the main reason i chose a full-tower case. For a motherboard, use the cheapest / lowest-power / quietest mini-ITX board you can find. Needs to have enough sata ports for your drives (or use a PCI-e slot for a cheap LSI card). anything with built-in AMD radeon or Intel GPU and a built-in GB NIC and/or wifi will be fine. Put as much RAM in it as you can afford. more RAM == more disk caching Use btrfs or zfs for the filesystem. snapshots, COW, error-detection & correction, etc etc. if you don't like them for some reason, you could use LVM and ext4 or xfs, i suppose (but i don't see why you'd want to) For the OS, ubuntu or FreeNAS (or FreeBSD) have good it-just-works out-of-the-box ZFS support. All linux distros have good btrfs support. Also think about what other services aside from samba and/or NFS that the box could provide to your network - internet gateway, firewall, dns cache, dhcp server, wireless AP, ntp time server, radicale or other caldav server (so you don't have to use google or apple calendars to have your calendar synced between desktop, phone, and tablet), squid proxy, web server, etc. debian and ubuntu are good - pretty nearly anything you need is available pre-packaged so can be installed easily. fedora has a lot too. You can run all your own cloud-buzzword-enabled software on your own machine! With a decent CPU and enough RAM, it could even run VMs for you - either kvm or lxc (or docker), e.g. for experimenting with stuff or just to keep the various services isolated from each other. Ok, all this adds up to quite a bit. $185 (case) + $50-$100 (PSU) + $100-$250 (motherboard) + $60-$500 for a CPU) + ~$50 (8GB RAM) totals over $400 if you bought everything brand new. but it'll last for years, and should just sit quietly in the corner doing its job without requiring any attention or fiddling with unless a drive dies. Scrounging and ebay-ing for 2nd hand stuff could be considerably cheaper (except for the case and psu - you probably won't find them 2nd hand). There was a thread on luv-main last March that ended up with lots of good info about building a good home NAS (or whatever) for as little as possible. The miser-NAS stuff starts here: https://lists.luv.asn.au/pipermail/luv-main/2015-March/007601.html The QC5000 mb+cpu combo recommended by Rick Moen in the thread seems to be no longer available. a shame, that was a very nice board at around $145. The Asrock Q2900M (using an Intel BayTrail-D J2900 CPU rather than an AMD A4-5000) seems like a reasonable alternative (actually, better in most respects except it only has two sata ports rather than 4, so an extra sata or sas card will be required. intel does it again) http://www.asrock.com/mb/Intel/Q2900M/ Seems hard to find in .au, could only find it on ebay from a shop in sydney for $177. OTOH, you can still get an AMD A4-7300 for $56 from MSY and a ASRock A68M-HD+ (newer than the ASRock A58M-HD+ in my post from last march) for $73. or get one of the newer A6/A8/A8 CPUs (sorted by price): $ msygrep fm2 cpu -v cooler -p 56 AMD AD7300OKHLBOX x2 A4 7300 Dual Core 3.8Ghz/1M/65FM2 CPU Integrated Radeon HD8470D 75 AMD AD740KYBJABOX x2 A6 7400K Dual Core 3.5Ghz/1M/65FM2 CPU Integrated Radeon R5 Series 128 AMD AD765KXBJABOX x4 A8 7650K Quad Core 3.3Ghz/4M/95FM2 CPU Integrated Radeon R7 Series 129 AMD x4 A10 7700K Quad Core 3.4Ghz (3.8Ghz MAX) /4M FM2 CPU AD770KXBJABOX 155 AMD AD786KYBJCSBX x4 A10 7860K Quad Core 3.6Ghz /4M FM2 CPU Integrated Radeon R7 Series 181 AMD AD787KXDJCBO x4 A10 7870K Quad Core 3.9Ghz/4M/95FM2 CPU Integrated Radeon R7 Series $ msygrep -p fm2 motherboard 68 Gigabyte GA-F2A68HM-DS2 A68 FM2 2xDDR3/1xPCI-Ex16/DVI/D-SUB/SATA3/USB3.0/MicroATX Motherboard 73 AsRock FM2A68M-HD FM2 A88X 2xDDR3/1xPCIe/SATA3/FRONT USB3.0/HDMI/DVI/D-SUB/mATX Motherboard 85 ASUS A68HM-PLUS FM2 A68H 2xDDR3/1xPCI-Ex16/SATA3/USB3.0/DVI/D-Sub/HDMI/RAID/MicroATX Motherboard BTW, some relevant links: http://blog.brianmoses.net/2016/02/diy-nas-2016-edition.html https://www.reddit.com/r/buildapc/comments/3minso/build_complete_smallest_12... and this site specialises in articles and reviews and discussions about DIY enterprisy stuff on a home or soho budget: http://www.servethehome.com/
- Set up a Raspberry Pi server which exposes a single file system
rpi only has a few USB ports, no sata.
Bonus point: - I can remove one of the drives and access the data on it directly (so it is FAT32 not Linux RAID)
this defeats the purpose of having a good, reliable backup server. and fat32 is just horrible. craig -- craig sanders <cas@taz.net.au>
On Wed, 3 Aug 2016 12:01:30 AM Craig Sanders via luv-main wrote:
IMO if you're going to do it at all, do it right - and with an eye towards the future. that means spending some money.
you'll need a case, a motherboard, RAM and a powersupply.
The Silverstone DS380 is very nice (8 x 3.5/2.5" hot-swap SAS/SATA bays + 4 internal 2.5" bays), but costs nearly $200 ($185 from CPL is cheapest i could find). And that doesn't include a power supply - it takes an SFX small-form-factor PSU...that's another $50-$100.
It's nice that you can get a case that can handle 8 disks for $200. But as 6TB disks are affordable and 8TB disks are available hardly anyone needs more than 4 disks for a home server.
even if you have no intention of ever filling all 8 hot-swap bays, the extras are useful for occasionally inserting a drive, running a backup (e.g with rsync or btrfs send or zfs send) and then removing it, AND taking it off-site (e.g. to work, to a family member, or anywhere else)
The other option is to just use a USB caddy for extra drives. Even at USB 2.0 speeds I haven't had any problems doing backups. But eSATA is a lot faster as is USB 3.
and having hot-swap bays that can hold 3.5" AND 2.5" drives is ideal. SSDs are rapidly becoming bigger and cheaper. for any business or org with a good budget they're already a better choice than HDDs for storage servers (e.g. samsung 2TB SSDs are under $1000 in au, and they just released a 4TB SSD for $1500 USD), and it won't be more than a few years before 1-4TB drives are cheap enough to be a no-brainer for home use too.
If you want a server for serious performance then a couple of large SATA disks for main storage and a couple of NVMe devices in PCIe cards for ZIL and L2ARC would be the way to go. In the long term I think that 2.5" disks won't be a useful option for many people. Anyone who is designing a laptop now should design for M.2 except for the corner-case of >15" laptops. Anyone who is designing a motherboard now should include 2*M.2 sockets on it. For all storage nowadays you either want the speed of SATAe (which is most easily realised with M.2) or the capacity and price of 3.5" disks. If you want a serious file server (which Piers doesn't seem to want) then ECC RAM is very important. Filesystems like ZFS and BTRFS (which you should use if you want lots of reliable storage) don't handle the type of corruption that bad RAM gives very well. Finally instead of buying systems for such use getting them for free is a better option for most people. I recently gave away a Dell PowerEdge T110 that has 8G of ECC RAM and 4 SATA bays. I have made arrangements to give away a HP micro server with 4 SATA bays (only 3 have caddies at this time). I have given away many dual-core AMD64 systems and a few quad-core systems. I don't think Piers is near enough to take advantage of such free offers, but I'm sure there are free PCs where he is. Linux just needs less hardware than Windows. Apart from laptops and tower servers I stopped buying PCs for myself before AMD64 became commonly used. Windows users just discard better stuff than I need. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
- Set up a Raspberry Pi server which exposes a single file system - Link it to the USB HDD's of different sizes that I have lying around
Just for the excitement factor I set up a 1.8TB USB drive off a Raspberry Pi. Currently we are @ ~ 500GB of data from a rsync of my desktop's /home partition. .....its been 2.5 days! This is definitely a project that proves the difference between using the "right" tools for the job vs playing with what you have lying around! Cheers Piers
On 19 August 2016 at 09:49, Piers Rowan via luv-main <luv-main@luv.asn.au> wrote:
- Set up a Raspberry Pi server which exposes a single file system - Link it to the USB HDD's of different sizes that I have lying around
Just for the excitement factor I set up a 1.8TB USB drive off a Raspberry Pi.
Currently we are @ ~ 500GB of data from a rsync of my desktop's /home partition.
.....its been 2.5 days!
This is definitely a project that proves the difference between using the "right" tools for the job vs playing with what you have lying around!
Cheers
Piers
That's a tick over 8Gb/hour, what are you complaining about? :^) Is that mainly due to the USB2 interface on the PI? And what file system are you running on the USB drive? -- Colin Fee tfeccles@gmail.com
On Friday, 19 August 2016 4:57:15 PM AEST Piers Rowan via luv-main wrote:
That's a tick over 8Gb/hour, what are you complaining about? :^)
Is that mainly due to the USB2 interface on the PI? And what file system are you running on the USB drive?
$ echo 8000/3600|bc -l 2.22222222222222222222 8Gig an hour is 2.2MB/s. USB2 typically gives about 32MB/s nowadays, strangely in the days when a P3 was a new system it gave about 35MB/s. The USB bus wouldn't be the bottleneck here.
Its an encrypted ext3 drive, which I am sure only adds to the overhead
I just did a quick Bonni++ run on my Thinkpad T420 with a i5-2520M and a 500G SSD that's encrypted. It was giving 417MB/s sequential output and 550MB/s sequential output with 19% CPU use. If the SSD was capable of it then my Thinkpad should have been capable of around 2500MB/s encryption speed! The i5-2520M CPU is a lot faster than what's in a RaspberryPi, but is it 1000* faster? It might be worth doing a Bonnie++ run. Also try running zcav (part of the Bonnie++ suite) to test sequential IO speed without encryption. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Recruit Online
The i5-2520M CPU is a lot faster than what's in a RaspberryPi, but is it 1000* faster? It might be worth doing a Bonnie++ run. Also try running zcav (part of the Bonnie++ suite) to test sequential IO speed without encryption.
I should probably look into it......but I hate to interrupt it @ 4.5 days so far!
participants (5)
-
Colin Fee -
Craig Sanders -
Erik Christiansen -
Piers Rowan -
Russell Coker