I have just received a free HP DL-385 system, it's quite a nice server with 16G of RAM, 3*SATA and 5*SAS disks. I want to set it up to run Linux software RAID-1 with Ext4 on a pair of SATA disks for booting and ZFS RAID-Z2 on the 5*SAS disks for data storage. The problem I have is that the boot disks were wiped, the DVD drive is broken and ILO isn't licensed for remote disk access. This combined with the fact that USB booting apparently doesn't work (it doesn't boot with my usual USB boot device) means that I have to make the SATA disks bootable before connecting them. The fact that the HP system takes ages to boot and is really noisy means that I don't want to make dozens of attempts to get it by trial and error. Does anyone know what the CCISS disk format is for a single disk RAID-0 (IE a JBOD)? Presumably there is some offset where I can just put an image of a regular bootable SATA disk and have it work. Thanks. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On Mon, Jun 24, 2013 at 06:49:34PM +1000, Russell Coker wrote:
I have just received a free HP DL-385 system, it's quite a nice server with 16G of RAM, 3*SATA and 5*SAS disks.
I want to set it up to run Linux software RAID-1 with Ext4 on a pair of SATA disks for booting and ZFS RAID-Z2 on the 5*SAS disks for data storage.
if you can, use the third SATA port for the RAID-Z array as well - RAID-Z gets better performance when the number of data disks in an array is a power of 2 (e.g. 4 data disks + 1 parity for RAID-Z1, or 4 data disks + 2 parity for RAID-Z2). the performance difference isn't crippling or even huge, but it is noticable. OTOH, if performance were the most important criteria, then a pool made up of mirrored pairs of drives would be beter (similar to RAID-10) hmmm...depending on your I/O workload, you might be better off putting an SSD in the third SATA port, partitioned to give you a smallish ZIL (4 or 8GB would be heaps) to smooth out sync. writes, and the remainder as L2ARC cache.
The problem I have is that the boot disks were wiped, the DVD drive is broken and ILO isn't licensed for remote disk access. This combined with the fact that USB booting apparently doesn't work (it doesn't boot with my usual USB boot device) means that I have to make the SATA disks bootable before connecting them.
can you get the server to pxe boot? if so, you could set up a debian installer on a local tftp server. all you need is dhcpd and a tftpd (strongly recommend ipxe as a 2nd stage so you can load the debian installer image with http rather tftp) if you need pointers on how to do this, i can send you relevant snippets of my dhcpd.conf and tftp setup.
Does anyone know what the CCISS disk format is for a single disk RAID-0 (IE a JBOD)? Presumably there is some offset where I can just put an image of a regular bootable SATA disk and have it work.
have you tried just putting a regular bootable disk in? it's been a while since i last used a hp smart array so i can't remember if you have to go into the smart-array bios setup and configure it for JBOD (or, as you say, single-disk-raid-0 because HP's smart-array isn't as smart as the name implies). craig -- craig sanders <cas@taz.net.au>
On Mon, 24 Jun 2013, Craig Sanders <cas@taz.net.au> wrote:
On Mon, Jun 24, 2013 at 06:49:34PM +1000, Russell Coker wrote:
I have just received a free HP DL-385 system, it's quite a nice server with 16G of RAM, 3*SATA and 5*SAS disks.
I want to set it up to run Linux software RAID-1 with Ext4 on a pair of SATA disks for booting and ZFS RAID-Z2 on the 5*SAS disks for data storage.
if you can, use the third SATA port for the RAID-Z array as well - RAID-Z gets better performance when the number of data disks in an array is a power of 2 (e.g. 4 data disks + 1 parity for RAID-Z1, or 4 data disks + 2 parity for RAID-Z2).
The SATA disks will just be for booting. They won't have ZFS because root on ZFS was way too much pain to even consider last time I looked into it. Also I think we are a long way from having ZFS root be reliable enough that I would even consider using it on a remote system with no other forms of boot available. Performance doesn't matter. I'm looking at replacing a system with a pair of 80G IDE disks in a software RAID-1 which is giving more than enough performance.
hmmm...depending on your I/O workload, you might be better off putting an SSD in the third SATA port, partitioned to give you a smallish ZIL (4 or 8GB would be heaps) to smooth out sync. writes, and the remainder as L2ARC cache.
If performance mattered then I'd consider that. However none of my ZFS installations have needed such performance so far, not even the ones where thousands of dollars were spent on storage hardware. Let alone the system that's free.
The problem I have is that the boot disks were wiped, the DVD drive is broken and ILO isn't licensed for remote disk access. This combined with the fact that USB booting apparently doesn't work (it doesn't boot with my usual USB boot device) means that I have to make the SATA disks bootable before connecting them.
can you get the server to pxe boot?
Thanks for the suggestion, I'll give that a go. I had it working a few years ago so I'm sure I can do it again.
Does anyone know what the CCISS disk format is for a single disk RAID-0 (IE a JBOD)? Presumably there is some offset where I can just put an image of a regular bootable SATA disk and have it work.
have you tried just putting a regular bootable disk in?
Yes, then the CCISS BIOS tells me that it found no RAID array and will default to RAID-5 if I don't say anything. Afterwards I configured it for a single disk RAID-0 and one of those options (auto RAID-5 or manual RAID-0) wiped the MBR along the way. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On Mon, Jun 24, 2013 at 07:37:56PM +1000, Russell Coker wrote:
On Mon, 24 Jun 2013, Craig Sanders <cas@taz.net.au> wrote:
if you can, use the third SATA port for the RAID-Z array as well - RAID-Z gets better performance when the number of data disks in an array is a power of 2 (e.g. 4 data disks + 1 parity for RAID-Z1, or 4 data disks + 2 parity for RAID-Z2).
The SATA disks will just be for booting. They won't have ZFS because root on ZFS was way too much pain to even consider last time I looked into it. Also I think we are a long way from having ZFS root be reliable enough that I would even consider using it on a remote system with no other forms of boot available.
i made no mention of having root on ZFS. that's a completely unrelated topic. I was pointing out that there is a small but noticable performance benefit with ZFS RAID-Z arrays if you have a power-of-two number of *data* disks - e.g. 2, 4, or 8 data disks. with 5 SAS ports, you can have 4 data disk (yes, a power of 2) and 1 parity for RAID-Z1. Or you can have three data disks (NOT a power of 2) and 2 parity disks for RAID-Z3. hence, if you want RAIDZ2 with four data disks, then 4 SAS data + 1 SAS parity + 1 SATA parity would achieve that. my own raidz1 pools have only 4 drives each, 3 data + 1 parity - but I made them before I knew about the power-of-two thing, and I'm using a pair of SATA 6Gbps OCZ Vector SSDs for my boot drive, with partitions for mdadm RAID-1 rootfs (xfs) and /boot (ext2), mirrored 4G ZIL, and 2 x 50G L2ARC.
Performance doesn't matter. I'm looking at replacing a system with a pair of 80G IDE disks in a software RAID-1 which is giving more than enough performance.
RAID-1 typically performs much better than RAID-5/6 or RAID-Z, especially on writes, and switching to RAID5/6/Z can result in a surprisingly unpleasant drop in performance. (OTOH, ancient IDE drives probably aren't very fast....but i'd still expect a RAID-1 of them to get better write speed than even modern non-SSD disks in RAID5/RAIDZ) BTW, a secondary reason to use an SSD as an L2ARC cache is that L2ARC is additional to the RAM-based ARC - exactly what you need if you want to enable de-duping, which can require quite large amounts of ARC/L2ARC. de-duplication is very useful if you have a large number of VM zvols (or zfs filesystems for container VMs) that are mostly similar. e.g. all running the same OS, or all created by cloning from a snapshot of a "base" VM
can you get the server to pxe boot?
Thanks for the suggestion, I'll give that a go. I had it working a few years ago so I'm sure I can do it again.
i see in your later reply that you got this working - cool, good.
have you tried just putting a regular bootable disk in?
Yes, then the CCISS BIOS tells me that it found no RAID array and will default to RAID-5 if I don't say anything. Afterwards I configured it for a single disk RAID-0 and one of those options (auto RAID-5 or manual RAID-0) wiped the MBR along the way.
nasty. avoiding stupid crap like that is a big part of the reason why why i prefer dumb HBA cards like my IBM M1015 or LSI 9211-8i SAS cards. they don't do things like that, and they're a lot cheaper (it's hard to beat the M1015, an IBM card using the LSI SAS2008 controller - 8 SAS 6Gbps ports for under $100 new on ebay). smart raid cards are pointless if you intend to run software-raid like mdadm, LVM, ZFS or btrfs anyway - you want just dumb ports or JBOD for those, not hardware raid. useful info on lots of different models and brands here: http://forums.servethehome.com/raid-controllers-host-bus-adapters/599-lsi-ra... craig -- craig sanders <cas@taz.net.au>
On Tue, 25 Jun 2013, Craig Sanders <cas@taz.net.au> wrote:
I was pointing out that there is a small but noticable performance benefit with ZFS RAID-Z arrays if you have a power-of-two number of *data* disks - e.g. 2, 4, or 8 data disks.
with 5 SAS ports, you can have 4 data disk (yes, a power of 2) and 1 parity for RAID-Z1. Or you can have three data disks (NOT a power of 2) and 2 parity disks for RAID-Z3.
hence, if you want RAIDZ2 with four data disks, then 4 SAS data + 1 SAS parity + 1 SATA parity would achieve that.
Does RAID-Z allow you to designate parity disks? I thought it did more of a RAID-5 thing than a RAID-4 thing. Anyway I wouldn't recommend that on a HP server as SATA disks are really slow. It seems as if HP was deliberately making them slow to drive sales of expensive SAS disks. So in this case I'm looking at a RAID-Z2 with 4 disks and one spare disk.
Performance doesn't matter. I'm looking at replacing a system with a pair of 80G IDE disks in a software RAID-1 which is giving more than enough performance.
RAID-1 typically performs much better than RAID-5/6 or RAID-Z, especially on writes, and switching to RAID5/6/Z can result in a surprisingly unpleasant drop in performance.
(OTOH, ancient IDE drives probably aren't very fast....but i'd still expect a RAID-1 of them to get better write speed than even modern non-SSD disks in RAID5/RAIDZ)
I expect that an array of 4*10,000rpm SAS disks running RAID-Z2 with 12G of RAM for cache can handily beat a system with 2*7200rpm (or maybe 5400rpm) IDE disks and hardly any spare RAM for cache.
they don't do things like that, and they're a lot cheaper (it's hard to beat the M1015, an IBM card using the LSI SAS2008 controller - 8 SAS 6Gbps ports for under $100 new on ebay).
smart raid cards are pointless if you intend to run software-raid like mdadm, LVM, ZFS or btrfs anyway - you want just dumb ports or JBOD for those, not hardware raid.
Yes. However the server in question was bought some years before ZFS on Linux was available and it worked quite well as a hardware RAID-6 device with battery backed write-back cache. I wouldn't recommend that anyone buy a server of similar specs today, but for some years it worked well for it's intended purpose. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On Tue, Jun 25, 2013 at 12:14:29PM +1000, Russell Coker wrote:
On Tue, 25 Jun 2013, Craig Sanders <cas@taz.net.au> wrote:
hence, if you want RAIDZ2 with four data disks, then 4 SAS data + 1 SAS parity + 1 SATA parity would achieve that.
Does RAID-Z allow you to designate parity disks?
no, in fact it stripes parity/checksum-metadata across all drives same as it does for actual data. my example was more notional, to illustrate the point, than a literal description of how it works in practice.
I thought it did more of a RAID-5 thing than a RAID-4 thing. Anyway I wouldn't recommend that on a HP server as SATA disks are really slow. It seems as if HP was deliberately making them slow to drive sales of expensive SAS disks.
ok, that's a really good reason not to use the SATA port with the zpool. with decent speed SATA ports, though, it would be worthwhile. craig -- craig sanders <cas@taz.net.au> BOFH excuse #360: Your parity check is overdrawn and you're out of cache.
On Mon, Jun 24, 2013 at 07:37:56PM +1000, Russell Coker wrote:
On Mon, 24 Jun 2013, Craig Sanders <cas@taz.net.au> wrote:
if you can, use the third SATA port for the RAID-Z array as well - RAID-Z gets better performance when the number of data disks in an array is a power of 2 (e.g. 4 data disks + 1 parity for RAID-Z1, or 4 data disks + 2 parity for RAID-Z2).
The SATA disks will just be for booting. They won't have ZFS because root on ZFS was way too much pain to even consider last time I looked into it. Also I think we are a long way from having ZFS root be reliable enough that I would even consider using it on a remote system with no other forms of boot available.
i made no mention of having root on ZFS. that's a completely unrelated topic.
I was pointing out that there is a small but noticable performance benefit with ZFS RAID-Z arrays if you have a power-of-two number of *data* disks - e.g. 2, 4, or 8 data disks.
with 5 SAS ports, you can have 4 data disk (yes, a power of 2) and 1 parity for RAID-Z1. Or you can have three data disks (NOT a power of 2) and 2 parity disks for RAID-Z3.
With old hardware like that I would nominate the 5th disk as a hot spare so on failure you have minimum "degraded" time. This assumes that ZFS can handle a hot spare drive. James
On Tue, 25 Jun 2013, James Harper <james.harper@bendigoit.com.au> wrote:
with 5 SAS ports, you can have 4 data disk (yes, a power of 2) and 1 parity for RAID-Z1. Or you can have three data disks (NOT a power of 2) and 2 parity disks for RAID-Z3.
With old hardware like that I would nominate the 5th disk as a hot spare so on failure you have minimum "degraded" time. This assumes that ZFS can handle a hot spare drive.
http://tinyurl.com/p2wckx5 Page 4 of the above Google Research document shows that years 2 and 3 have annualised failure rates just over 8% while years 4 and 5 have rates of about 6% and 7%. So it seems that there isn't a great increase in risk as the disk gets older. If I have 6 disks in service then each disk has a .93% probability of surviving which gives a 0.93^6 == 65% probability of having no failures in a year. With filesystems such as ZFS some conditions which would be regarded as "failures" under lesser filesystems become correctable errors. So I don't think I've got a great need to be worried here. With RAID-Z2 if a disk dies then the system can cope with a second disk dying or giving read errors on reconstruction. This is even better than RAID-6 (which only copes with hard read errors and disk failures not corruption). -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On Tue, 25 Jun 2013, James Harper <james.harper@bendigoit.com.au> wrote:
with 5 SAS ports, you can have 4 data disk (yes, a power of 2) and 1 parity for RAID-Z1. Or you can have three data disks (NOT a power of 2) and 2 parity disks for RAID-Z3.
With old hardware like that I would nominate the 5th disk as a hot spare so on failure you have minimum "degraded" time. This assumes that ZFS can handle a hot spare drive.
Page 4 of the above Google Research document shows that years 2 and 3 have annualised failure rates just over 8% while years 4 and 5 have rates of about 6% and 7%. So it seems that there isn't a great increase in risk as the disk gets older.
I don't see anywhere that you answered my question about the generation of your server, but I assumed it's a HP DL385 (and not a HP DL385 G2 or newer), which is 7-8 years old, and I can't see how you could extrapolate google's results out that far. In any case, my concern was more related to the replacement cost of a failed disk. If you plan on using the server for 3 years then with a hot spare you can take a single disk failure with no loss of redundancy or performance. Or maybe ZFS raid can handle a failure with no loss of performance? James
On Mon, 24 Jun 2013, Craig Sanders <cas@taz.net.au> wrote:
can you get the server to pxe boot?
http://tinyurl.com/btbsqmf I followed the instructions at the above Debian Wiki page and got a PXE server working. It's all installed now. Also thanks for the suggestions James, I'll keep that in mind for next time. Finally while the DL385 is a nice server (16G of RAM, 8 cores of AMD Opteron 275, and 8 disks) it's a bit excessive to my needs and rather noisy. If anyone would like to offer any sort of workstation with ECC RAM in exchange then I'd be happy to do that. All I really need for this case is 8G of RAM and the ability to support 2 desktop SATA disks. Something like an old Dell T1xx server, Dell Precision workstation (one of the ones with ECC RAM), or a HP WX workstation with something like an AMD1212 CPU would do the job well. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On Mon, Jun 24, 2013 at 11:20 PM, Russell Coker <russell@coker.com.au> wrote:
On Mon, 24 Jun 2013, Craig Sanders <cas@taz.net.au> wrote:
can you get the server to pxe boot?
I followed the instructions at the above Debian Wiki page and got a PXE server working. It's all installed now.
Out of interest, could you now find the offset you were originally looking for? It's commonly 512 or 1024 bytes for JBODs, but I'd be curious to know what it is for this particular system. Regards, Marcus. -- Marcus Furlong
I have just received a free HP DL-385 system, it's quite a nice server with 16G of RAM, 3*SATA and 5*SAS disks.
I want to set it up to run Linux software RAID-1 with Ext4 on a pair of SATA disks for booting and ZFS RAID-Z2 on the 5*SAS disks for data storage.
The problem I have is that the boot disks were wiped, the DVD drive is broken and ILO isn't licensed for remote disk access. This combined with the fact that USB booting apparently doesn't work (it doesn't boot with my usual USB boot device) means that I have to make the SATA disks bootable before connecting them.
What vintage? I have a DL385 G2 in production use. It's a long time ago but I seem to remember that USB booting worked just fine. For some odd reason the default setting in the BIOS is that all USB ports are USB1.1 not USB2.
The fact that the HP system takes ages to boot and is really noisy means that I don't want to make dozens of attempts to get it by trial and error.
Does anyone know what the CCISS disk format is for a single disk RAID-0 (IE a JBOD)? Presumably there is some offset where I can just put an image of a regular bootable SATA disk and have it work.
I'd try this, in this order: 1. Check BIOS version and settings then try USB booting again. The boot order stuff can be tricky on those HP's. I think F9 or F12 allows an override of the boot order. 2. Get a trial key for iLO advanced. Should work and give you full features of iLO (including remote disk) for 30 days [1] 3. Plug a DVD drive into one of the sata ports, even if it had to hang over the side Failing that maybe initialise a single disk as JBOD (F8 in BIOS boot I think) then plug it into another machine and have a look at what the format looks like. Google tells me that the metadata might be 1088 sectors long, so you could try copying your data starting at sector 1088 and see what happens. I'm guessing but there may be another copy of the metadata at the end too so don't write too much. btw, ilo ssh works great for access to serial console on my dl385g2. James [1] This page has some info about it http://h30499.www3.hp.com/t5/Integrity-Servers/ILO-2-trial-license-key/td-p/... and even suggests a trial key that might work. Won't work if your server predates iLO2, and probably won't work if a trial has already been used on it. Otherwise follow the other links and register for a trial.
Russell Coker <russell@coker.com.au> writes:
I want to set it up to run Linux software RAID-1 with Ext4 on a pair of SATA disks for booting and ZFS RAID-Z2 on the 5*SAS disks for data storage. [...] The boot disks were wiped, the DVD drive is broken [...]
Can't you just put in a new optical drive, long enough to do an install?
On Tue, 25 Jun 2013, "Trent W. Buck" <trentbuck@gmail.com> wrote:
I want to set it up to run Linux software RAID-1 with Ext4 on a pair of SATA disks for booting and ZFS RAID-Z2 on the 5*SAS disks for data storage. [...] The boot disks were wiped, the DVD drive is broken [...]
Can't you just put in a new optical drive, long enough to do an install?
One of the down-sides of such name-brand equipment is that their parts are all different from other systems. When you only have one of a particular type of system the options for getting spare parts aren't good. Anyway I've solved the problem with PXE which seems to be the best way to do it. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Russell Coker <russell@coker.com.au> wrote:
Anyway I've solved the problem with PXE which seems to be the best way to do it.
That's how I installed Debian on my current laptop, which does not have an optical drive. I was amazed by how well it worked: I brought the new laptop home, made sure the BIOS configuration was set appropriately, connected it to the network with dhcp and tftp configured, and it all just worked without any difficulties.
On Tue, Jun 25, 2013 at 11:15:33AM +1000, Russell Coker wrote:
Anyway I've solved the problem with PXE which seems to be the best way to do it.
IMO, it's always the best way to do it - pretty nearly any machine can boot off PXE, so you don't have to stuff around with CDs, DVDs, USB sticks, etc. it's my primary method of running installers, rescue disks, freedos based firmware updates(*), clonezilla and whatever else - i really only use USB sticks if i'm working on a network that i haven't installed a tftp server on. (*) there's quite a lot of hardware that works on Linux but doesn't have linux tools to update the firmware - many of them have MS-DOS based tools. most of them work just fine on a freedos floppy/hd image. e.g. one of my pxe-bootable freedos images is for re-flashing my LSI and IBM M1015 cards to IT (JBOD) firmware rather than the stock IR (raid) firmware. craig -- craig sanders <cas@taz.net.au>
participants (6)
-
Craig Sanders -
James Harper -
Jason White -
Marcus Furlong -
Russell Coker -
trentbuck＠gmail.com