Hi all, My desktop computer hard drive is now full. I have offloaded a couple of hundred Gb to an external expansion drive to give me breathing space. So I am looking at keeping sda and if possible adding sdb and sdc as a RAID arrangement, from my reading I am leaning toward a software RAID 1 (mirrored drive) using SATA drives. I am progressing further along my intended path of returning to professional photography so data security is now imperative. Can I keep sda as my boot disk and just run sdb and sdc for data storage? sda is 1Tb and sdb and sdc are identical Seagate SATA drives of 2Tb each. I run OpenSuse Tumbleweed. I was intending to get an SSD of 500Gb for my post production work with output being stored only on RAID. Am I on the right track here or do I have incorrect expectations? Many thanks Andrew Greig
On 05/03/18 22:15, Andrew Greig via luv-main wrote:
Can I keep sda as my boot disk and just run sdb and sdc for data storage?
Yes, I've been doing that for years. I started with a HDD for the boot drive and 2x 2TB drives for my data, but eventually upgraded to a 1TB SSD for my boot drive and 2x 4TB for the data. I have two partitions on the 4TB RAID 1 volume, a 3TB XFS filesystem for larger files like videos and a 1TB ext4 filesystem for everything else. I keep my Steam games on the SSD for speed and because they can always be redownloaded so I don't need the reliability of the mirror.
Am I on the right track here or do I have incorrect expectations?
It's been working fine for me. Cheers, Andrew
Thanks Andrew, Do I need to rebuild my installation or can I add a software RAID at any time by downloading the correct packages? Are there any documents that are particularly helpful on set up and partitioning? I imagined a single partition and call it /data. Cheers Andrew On 03/05/2018 10:23 PM, Andrew Pam wrote:
On 05/03/18 22:15, Andrew Greig via luv-main wrote:
Can I keep sda as my boot disk and just run sdb and sdc for data storage? Yes, I've been doing that for years. I started with a HDD for the boot drive and 2x 2TB drives for my data, but eventually upgraded to a 1TB SSD for my boot drive and 2x 4TB for the data. I have two partitions on the 4TB RAID 1 volume, a 3TB XFS filesystem for larger files like videos and a 1TB ext4 filesystem for everything else.
I keep my Steam games on the SSD for speed and because they can always be redownloaded so I don't need the reliability of the mirror.
Am I on the right track here or do I have incorrect expectations? It's been working fine for me.
Cheers, Andrew
On 05/03/18 22:46, Andrew Greig wrote:
Do I need to rebuild my installation or can I add a software RAID at any time by downloading the correct packages? Are there any documents that are particularly helpful on set up and partitioning? I imagined a single partition and call it /data.
My ext4 partition is also called /data and I then softlink other directories onto that partition as required. Yes, you can add it at any time - you only need the "mdadm" package, although it's also possible to set up mirroring using LVM, btrfs or ZFS if you prefer. The way I did it is to physically install the new drives, partition them, use mdadm to set up a RAID-1 mirror, and then finally create a filesystem on the mirror volume. In the old days I used to use the device names for the RAID component partitions and the mirror volume, but now I recommend using UUIDs for both to ensure everything will work correctly regardless of any changes to the device names. You can find the instructions with "man mdadm", and that points you to https://raid.wiki.kernel.org/ if you need even more information. Hope that helps! Cheers, Andrew
Thanks Andrew, I will read and learn Andrew On 03/05/2018 10:54 PM, Andrew Pam wrote:
On 05/03/18 22:46, Andrew Greig wrote:
Do I need to rebuild my installation or can I add a software RAID at any time by downloading the correct packages? Are there any documents that are particularly helpful on set up and partitioning? I imagined a single partition and call it /data. My ext4 partition is also called /data and I then softlink other directories onto that partition as required. Yes, you can add it at any time - you only need the "mdadm" package, although it's also possible to set up mirroring using LVM, btrfs or ZFS if you prefer.
The way I did it is to physically install the new drives, partition them, use mdadm to set up a RAID-1 mirror, and then finally create a filesystem on the mirror volume. In the old days I used to use the device names for the RAID component partitions and the mirror volume, but now I recommend using UUIDs for both to ensure everything will work correctly regardless of any changes to the device names.
You can find the instructions with "man mdadm", and that points you to https://raid.wiki.kernel.org/ if you need even more information.
Hope that helps!
Cheers, Andrew
On Monday, 5 March 2018 10:15:35 PM AEDT Andrew Greig via luv-main wrote:
My desktop computer hard drive is now full. I have offloaded a couple of hundred Gb to an external expansion drive to give me breathing space.
So I am looking at keeping sda and if possible adding sdb and sdc as a RAID arrangement, from my reading I am leaning toward a software RAID 1 (mirrored drive) using SATA drives. I am progressing further along my intended path of returning to professional photography so data security is now imperative.
Can I keep sda as my boot disk and just run sdb and sdc for data storage?
Sure. But it's easier to have RAID for everything.
sda is 1Tb and sdb and sdc are identical Seagate SATA drives of 2Tb each.
If you had memory limits then you could use one disk for swap and the others for data access. But memory limits aren't much of an issue nowadays, systems with 8G of RAM are often thrown out as rubbish and systems that can be expanded to 16G also appear in the rubbish. My main server in Melbourne has 16G of RAM and came from the InfoXchange rubbish pile. My workstation also came from the InfoXchange rubbish pile and has 8G in 2*4G DIMMs, I have systems from rubbish that would take 4*4G DIMMs but they have slower CPUs. When I attend the LUV meetings at InfoXchange I hunt through the rubbish pile for nice systems and offer them to anyone there. There seems to be a correlation between not having nice hardware at home and not being able to recognise nice hardware in a pile of rubbish. ;) More drives means more heat produced and more potential problems in summer. It means more parts that can fail and cause you annoyance. If you are about to buy new disks then getting 2*3TB seems like a better option than 1TB+2*2TB. If you are worried about data loss then latent sector errors is what you should worry about. I've replaced lots of disks in production due to them returning bat data and claiming it to be good. I can't remember the last time I replaced a disk in production due to it giving read errors. The vast majority of disk problems I've had in the last 10 years were only discovered because of using ZFS or BTRFS and would have given silent data corruption on any other filesystem. BTRFS allows you to use RAID-1 across any combination of disks. So if you put in 1TB+2*2TB in a BTRFS RAID-1 you would get 2.5TB of RAID-1 storage. BTRFS also allows you to dynamically resize arrays. You can start with a single disk, add another to make it RAID-1 (with a long operation of copying all the data) and then keep adding disks when you run out of space.
I run OpenSuse Tumbleweed. I was intending to get an SSD of 500Gb for my post production work with output being stored only on RAID.
Try to avoid getting one of the rubbish SSDs. My workstation has a Sandisk SSD, a Samsung, and an Intel all in a BTRFS RAID-1. The Sandisk SSD slows the entire system down, it can't handle 30MB/s of continuous writes so when I copy data from a USB-2 attached device the bottleneck is the Sandisk not USB 2.0. Intel and Samsung devices can be relied on to be reasonably good. Read reviews if you are thinking of buying another brand. 2*250G SSD costs about the same as 1*500G. Would a 250G RAID-1 of SSD be more useful than 500G of unmirrored storage? Even if you have only a single disk use BTRFS or ZFS. They both support multiple copies of metadata on a single disk and they both do checksums on everything so you know if anything gets corrupted.
Am I on the right track here or do I have incorrect expectations?
Mostly right. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On 6 March 2018 at 00:48, Russell Coker via luv-main <luv-main@luv.asn.au> wrote:
If you are worried about data loss then latent sector errors is what you should worry about. I've replaced lots of disks in production due to them returning bat data and claiming it to be good. I can't remember the last time I replaced a disk in production due to it giving read errors. The vast majority of disk problems I've had in the last 10 years were only discovered because of using ZFS or BTRFS and would have given silent data corruption on any other filesystem.
I've had experience of this with my media server at home. I was running 2 x mirrored pairs with soft raid (mdadm) and it was happily mirroring corrupt data across a couple of TV series. I switched to ZFS to manages the mirrored pairs and haven't had any issues since. When it came time to increase the disk pool size (both the physical disk size and pool size) ZFS made the job simple. And luckily I was able to recover the corrupt data from the source DVDs. -- Colin Fee tfeccles@gmail.com
On Tuesday, 6 March 2018 8:12:39 AM AEDT Colin Fee via luv-main wrote:
I switched to ZFS to manages the mirrored pairs and haven't had any issues since. When it came time to increase the disk pool size (both the physical disk size and pool size) ZFS made the job simple.
ZFS doesn't allow changing RAID levels, removing RAID sets, or reducing the size without a backup/format/restore cycle. The support of such things is a major feature for BTRFS on smaller installations. For enterprise use this isn't an issue as storage only increases and all disks will be the same size etc. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
participants (4)
-
Andrew Greig -
Andrew Pam -
Colin Fee -
Russell Coker