On Wed, 3 Aug 2016 06:39:07 PM Craig Sanders via luv-main wrote:

...

But as 6TB disks are affordable and 8TB disks are available hardly anyone needs more than 4 disks for a home server.

I don't know if i trust the 6 or 8TB drives yet. maybe in a year or so.

6tb disks have been around for over a year, I don't think that 8TB has been around for so long.

and size isn't everything. multiple smaller disks in RAID-1/10 will give much better performance. 4x2TB RAID-10 will cost 10-20% more than 2x4TB RAID-1 but will be much faster.

You usually don't need a lot of performance from disks. In most cases saturating GigE is enough. Get SSD for filesystems that need speed and maybe for ZIL/L2ARC and you don't need lots of spindles.

...

In the long term I think that 2.5" disks won't be a useful option for many people. Anyone who is designing a laptop now should design for M.2 except for the corner-case of >15" laptops. Anyone who is designing a motherboard now should include 2*M.2 sockets on it.

i've got nothing against NVMe (in fact, it's one of the technologies I'm keenly waiting to get better and cheaper), but they have different use cases than 2.5" or 3.5" drives.

2.5" drives will be useful for years to come. NVMe (esp. in raid-1) is great for an OS drive and for bcache/l2arc/zil/etc but capacity is limited due to the small size and, worse, pci-e lanes are limited (esp. in Intel motherboards) so very few motherboards will have more than 1 or 2 M.2 slots.

You can get 512G M.2 NVMe cards for $425 and they will continue to get bigger and cheaper. 2*M.2 slots on the motherboard and the option of PCIe cards for more means that very few people will have a lack of fast storage.

i haven't yet seen any NVMe hot-swap capability either, which means taking the system down to replace a drive.

True. But the number of installations that depend on a single server (as opposed to a HA cluster) and need 24*7 operation is quite small. Certainly not anyone's home server!

SATA3 SSDs get up to around 550MB/s. 4 of them in RAID-1 or RAID-10 will get nearly 2GB/second read speed, and over 1TB/s write. and the more mirrored pairs you add, the faster it gets. That's fast enough that it doesn't need an NVMe for caching.

Currently if you want large SSDs the biggest that's in the affordable range seems to be 1TB M.2 SATA for $459 (about $500 with a SATA adapter). If you want a RAID array of fast 1TB storage devices then that would do well. But if you need more then it won't. My home server has a RAID-1 array of 2*4TB disks. A RAID-Z of 5*1TB SSDs would cost $2500, those 2 disks cost about $300 each. My home server has a RAID-1array of SATA SSDs for the things that need speed (root, /home, and swap). But if I was going to do it all again from scratch I'd consider a pair of NVMe devices for root and for L2ARC and a pair of 6TB disks for main storage.

...

For all storage nowadays you either want the speed of SATAe (which is most easily realised with M.2) or the capacity and price of 3.5" disks.

part of my point was that SSDs are getting cheaper all the time. In the not too distant future, they will be cheap enough for home users to build reasonably large arrays of 1-4TB SSDs, with much greater performance, especially on random seeks or programs that open files with O_SYNC or just fsync a lot.

True. The size of storage is increasing much faster than most people use storage. NVMe is currently available in affordable sizes to replace the hard drives in the vast majority of systems I run (note not the majority of hard drives, but the majority of systems having hard drives). A lot of systems that I'm responsible for don't need the performance of NVMe, home PCs are a good example. But the cost difference is dropping so eventually it'll just make sense to do it. Relatives who currently have 120G SATA SSDs might get NVMe when they wear out. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/