On Sat, Dec 6, 2014 at 10:36 PM, Brett Pemberton <brett.pemberton@gmail.com> wrote:

My method was to decide on my fully populated capacity: 9 drives, and
then just source old drives to make that happen.

So my initial raidz array was 3 x raidz1 vdevs, each with 3 drives. In
my initial state these were 500gb drives I had lying around, or
sourced for very cheap. 4 of them were even PATA. I was then able to
replace them with larger drives, 3 at a time.

As long as you can source some cheap old (but still relatively
reliable) drives, you can still use ZFS in the same way as mdadm.

This is a scenario I hadn't considered. My box has a max capacity of 8 properly mounted drives, I have 4 x 3tb for a storage pool and 256gb ssd for the os and for some regularly used but not so precious data. I also have a bunch of older drives of random sizes laying about which I could use. I don't think any of these are the same size.

If I was to follow your example, I could create 2 raidz1 vdevs of 3 drives each, 1 of these using the 3x3tb drives and 1 making use the other 3tb drive and my 2 biggest capacity old drives. This vdev would obviously be limited to the capacity by the smallest of these drives. I could then grow my storage pool when needed by upgrading the two older drives to match the larger 3tb. One downside would be that, for now, I wouldn't be making use of a big chunk of one of the 3tb drives. On the other hand, I don't yet have the data to fill it.

The other obvious path would be to simply use the 4x3tb drives in a raid5 pool managed by mdadm and forgo zfs filesystem features, though I did want to make use of snapshots.

Seems I should have posted here for ideas prior to purchasing my storage. If any of you had my hardware, how would you construct your storage layout?

Vote NO in referenda.