11 Oct
2016
11 Oct
'16
11:13 p.m.
On Tuesday, 11 October 2016 10:30:01 PM AEDT Craig Sanders via luv-main wrote:
I was rebooting anyway in order to replace a failed
SSD on one machine and
convert both of them to root on ZFS. It booted up OK on both, so I made it
the default. If it refrains from sucking badly enough to really piss me off
for a decent length of time, i'll leave it as the default.
That's a bold move. While ZFS has been totally reliable in preserving data I
have had ongoing problems with filesystems not mounting when they should. I
don't trust ZFS to be reliable as a root filesystem, I want my ZFS systems to
allow me to login and run "zfs mount -a" if necessary.
(this is, of courtse, one of the reasons I dislike
journald. logs should be
plain text, so you can access them without specialised tools. and rsyslogd
wasn't running in the semi-broken recovery mode that systemd dumped me in
after making me wait 1m30s until it finished doing mysterious things
(AFAICT, it was doing nothing except twiddling some stars on screen).
I agree that those things need to be improved. There should be a way to get
to a root login while the 90 second wait is happening. There should be an
easy and obvious way to display those binary logs from the system when it's
not running systemd or from another system (IE logs copied from another
system).
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
12 Oct
12 Oct
1:31 a.m.
New subject: How to make systemd more reliable
On Tue, Oct 11, 2016 at 11:13:34PM +1100, russell(a)coker.com.au wrote:
On Tuesday, 11 October 2016 10:30:01 PM AEDT Craig
Sanders via luv-main wrote:
switching my main system to systemd? yes, i know. very bold. very risky.
i'll probably regret it at some point.
:)
I was rebooting anyway in order to replace a
failed SSD on one
machine and convert both of them to root on ZFS. It booted up OK on
both, so I made it the default. If it refrains from sucking badly
enough to really piss me off for a decent length of time, i'll leave
it as the default.
That's a bold move. While ZFS has been totally reliable in preserving data
I have had
ongoing problems with filesystems not mounting when they should.
i know other people have reported problems like that, but it's never
happened on any of my zfs machines...and i've got most of my pools
plugged in to LSI cards (IBM M1015 reflashed to IT mode) using the
mp2sas driver - which is supposed to exacerbate the problem due to the
staggered drive spinup it does.
the only time i've ever seen something similar was my own stupid fault,
i rebooted and just pulled out the old SSD forgetting that I had ZIL
and L2ARC for the pools on that SSD. I had to plug the old SSD back in
before I could import the pool, so i could remove them from the pool
(and add partitions from my shiny new SSDs to replace them).
I don't trust ZFS to be reliable as a root
filesystem, I want my ZFS
systems to allow me to login and run "zfs mount -a" if necessary.
not so bold these days, it works quite well and reliably. and i really
want to be able to snapshot my rootfs and backup with zfs send rather
than rsync.
anyway, i've left myself an ext4 mdadm raid-1 /boot partition (with
memdisk and a rescue ISO) in case of emergency.
the zfs root on my main system is two mirrored pairs (raid-10) of
crucial mx300 275G SSDs(*). slightly more expensive than a pair of
500-ish GB but much better performance....read speeds roughly 4 x SATA
SSD read (approximating pci-e SSD speeds), write speeds about 2 x SATA
SSD.
i haven't run bonnie++ on it yet. it's on my todo list.
http://blog.taz.net.au/2016/10/09/converting-to-a-zfs-rootfs/
the other machine got a pair of the same SSDs, so raid-1 rather than
raid-10. still quite fast (although i'm having weirdly slow scrub
performance on that machine. haven't figured out why yet. peformance
during actual usage is good, noticably better than the single aging SSD
I replaced).
(*) 275 marketing GB. SI units. 256 GiB in real terms.
they're good value for money anyway....i got mine for $108 each. I've
since seen them for $97 (itspot.com.au). MSY doesn't stock them for
some reason (maybe they want to clear their stock of MX200 models
first).
we're just on the leading edge of some massive drops in price/GB. a bit
earlier than I was predicting, i though we'd start seeing it next year.
wont be long before 2 or 4TB SSDs are affordable for home users (you can
get 2TB SSDs for around $800 now). and then I can replace some of my HDD
pools.
I agree that those things need to be improved. There
should be a way
to get to a root login while the 90 second wait is happening.
so there really is no way to do that? i was hoping it was just some
trivially-obvious-in-hindsight thing that i didn't know.
it's really annoying to have to wait and watch those damn stars when you
just wnat to get a shell and start investigating & fixing whatever's
gone wrong.
There should be an easy and obvious way to display
those binary logs
from the system when it's not running systemd or from another system
(IE logs copied from another system).
yep. can you even access journald logs if you're booted up with a rescue
disk? (genuine question, i don't know the answer but figure it's one of
the things i need to know)
craig
--
craig sanders <cas(a)taz.net.au>
9:26 a.m.
New subject: How to make systemd more reliable
There should
be an easy and obvious way to display those binary logs
from the system when it's not running systemd or from another system
(IE logs copied from another system).
yep. can you even access journald logs if you're
booted up with a rescue
disk? (genuine question, i don't know the answer but figure it's one of
the things i need to know)
If the journal is being kept on disk, then yes, journalctl --directory
/var/journal/ will work.
I'm not terribly happy with the current implementation of journal. Having a
structured, fast, searchable, all in one journal is a lovely goal however.
--
-- Clinton Roy, Software Engineer with Bloomberg L.P.
10:42 a.m.
New subject: How to make systemd more reliable
On 12/10/16 01:31, Craig Sanders via luv-main wrote:
...
it's really annoying to have to wait and watch
those damn stars when you
just wnat to get a shell and start investigating & fixing whatever's
gone wrong.
> There should be an easy and obvious way to display those binary logs
> from the system when it's not running systemd or from another system
> (IE logs copied from another system).
On my todo list. It happens when I boot after failing to alter fstab to
match the actual disks connected.
yep. can you even access journald logs if you're
booted up with a rescue
disk? (genuine question, i don't know the answer but figure it's one of
the things i need to know)
Good question - went and looked:
systemctl -b -1
will pull up the log from the boot before the rescue boot.
--list-boots will give you the history in the database.
10:55 a.m.
New subject: How to make systemd more reliable
On 12/10/16 10:42, Allan Duncan via luv-main wrote:
On 12/10/16 01:31, Craig Sanders via luv-main wrote:
...
Look at
man systemd-udevd
The default 180 sec would work, but seems to be retriggered in practise.
There are some kernel params to tweak also, but they seem to be intended
to prolong not shorten.
it's really annoying to have to wait and
watch those damn stars when you
just wnat to get a shell and start investigating & fixing whatever's
gone wrong.
> There should be an easy and obvious way to display those binary logs
> from the system when it's not running systemd or from another system
> (IE logs copied from another system).
On my todo list. It happens when I boot after failing to alter fstab to
match the actual disks connected.
1:48 p.m.
New subject: How to make systemd more reliable
On Wed, Oct 12, 2016 at 10:42:03AM +1100, Allan Duncan wrote:
neither of those work. that would be because they're journalctl options,
not systemctl.
also, the default for the Storage setting in /etc/systemd/journald.conf
(at least in debian) is "auto", so no persistent storage of journal
unless you manually create /var/log/journal (which, of course, you
wouldn't do until *after* you realise you need to and only after you've
figured out what needs to be done. bad default)
craig
--
craig sanders <cas(a)taz.net.au>
On my todo list. It happens when I boot after failing
to alter fstab
to match the actual disks connected.
it seems to happen at the slightest excuse, whether the machine ends up
booting or not. just what everyone needs, one or more 90 second delays
during the boot process.
maximising downtime by putting long delays in the way of whoever's
trying to fix the problem is not a good idea.
is this delay configurable to something more reasonable, like 30 or 15
or even 5 seconds? can it be disabled?
can i at least have a --stop-wasting-my-fucking-time option?
(that would be more useful than their deprecated --kill-my-kernel-please,
aka --debug)
yep. can you
even access journald logs if you're booted up with a
rescue disk?
Good question - went and looked: systemctl -b -1 will pull up the log
from the boot before the rescue boot.
--list-boots will give you the history in the database.
5:08 p.m.
New subject: How to make systemd more reliable
On 12/10/16 13:48, Craig Sanders via luv-main wrote:
On Wed, Oct 12, 2016 at 10:42:03AM +1100, Allan Duncan
wrote:
neither of those work. that would be because they're journalctl options,
not systemctl.
Indeed - I am always mixing the two.
On my todo list. It happens when I boot after
failing to alter fstab
to match the actual disks connected.
it seems to happen at the slightest excuse, whether the machine ends up
booting or not. just what everyone needs, one or more 90 second delays
during the boot process.
maximising downtime by putting long delays in the way of whoever's
trying to fix the problem is not a good idea.
is this delay configurable to something more reasonable, like 30 or 15
or even 5 seconds? can it be disabled?
can i at least have a --stop-wasting-my-fucking-time option?
(that would be more useful than their deprecated --kill-my-kernel-please,
aka --debug)
yep. can
you even access journald logs if you're booted up with a
rescue disk?
Good question - went and looked: systemctl -b -1 will pull up the log
from the boot before the rescue boot.
--list-boots will give you the history in the database. also, the default for the Storage setting in
/etc/systemd/journald.conf
(at least in debian) is "auto", so no persistent storage of journal
unless you manually create /var/log/journal (which, of course, you
wouldn't do until *after* you realise you need to and only after you've
figured out what needs to be done. bad default)
Fedora has it commented out, so I guess that means the default is
"persistent" 'cause I have them waaay back.
After some online searching I have found the fstab options
nofail
and
x-systemd.device-timeout=
given in the systemd.mount man page.
Another tidbit:
https://gryzli.info/2016/06/18/systemd-systemctl-list-unit-files-timeouts/
[refers to https://github.com/systemd/systemd/issues/1961 ]
11:18 a.m.
New subject: How to make systemd more reliable
On Wednesday, 12 October 2016 1:31:33 AM AEDT Craig Sanders via luv-main
wrote:
https://wiki.debian.org/systemd#Debugging
According to the above Wiki page you can give a kernel command-line parameter
for a root shell. However that uses tty9 (which isn't available on serial
consoles), has no password (you can't have it enabled all the time with
hostile console access), and has to be switched on in advance.
I don't know if there is a way to do what you and I want. But there is a way
to do something that will get us by in most situations.
the only time i've ever seen something similar was
my own stupid fault,
i rebooted and just pulled out the old SSD forgetting that I had ZIL
and L2ARC for the pools on that SSD. I had to plug the old SSD back in
before I could import the pool, so i could remove them from the pool
(and add partitions from my shiny new SSDs to replace them).
Did you have to run "zfs import" on it or was it recognised automatically? If
the former how did you do it? Is the initramfs configured to be able to run
zfs import?
not so bold these days, it works quite well and
reliably. and i really
want to be able to snapshot my rootfs and backup with zfs send rather
than rsync.
BTRFS snapshots are working well on the root filesystems of many systems I
run. The only systems I run without BTRFS as root are systems where getting
console access in the event of problems is too difficult.
the zfs root on my main system is two mirrored pairs
(raid-10) of
crucial mx300 275G SSDs(*). slightly more expensive than a pair of
500-ish GB but much better performance....read speeds roughly 4 x SATA
SSD read (approximating pci-e SSD speeds), write speeds about 2 x SATA
SSD.
i haven't run bonnie++ on it yet. it's on my todo list.
If you had 2*NVMe devices it would probably give better performance than
4*SATA and might be cheaper. That would also leave more SATA slots free.
we're just on the leading edge of some massive
drops in price/GB. a bit
earlier than I was predicting, i though we'd start seeing it next year.
wont be long before 2 or 4TB SSDs are affordable for home users (you can
get 2TB SSDs for around $800 now). and then I can replace some of my HDD
pools.
It's really changing things. For most users 2TB is more than enough storage
even for torrenting movies. I think that spinning media is going to be mostly
obsolete for home use soon.
I agree that
those things need to be improved. There should be a way
to get to a root login while the 90 second wait is happening.
so there really is no way to do that? i was hoping it was just some
trivially-obvious-in-hindsight thing that i didn't know. it's really annoying to have to wait and watch
those damn stars when you
just wnat to get a shell and start investigating & fixing whatever's
gone wrong.
Absolutely.
There should
be an easy and obvious way to display those binary logs
from the system when it's not running systemd or from another system
(IE logs copied from another system).
yep. can you even access journald logs if you're booted up with a rescue
disk? (genuine question, i don't know the answer but figure it's one of
the things i need to know) From a quick read of the man page it appears that the
-D option to journalctl
might do what we want. It appears that Debian has moved to
not having the
binary journals so I don't have a conveniant source of test data.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
2:46 p.m.
New subject: How to make systemd more reliable
On Wed, Oct 12, 2016 at 11:18:40AM +1100, russell(a)coker.com.au wrote:
yes, that would certainly be a LOT faster. can't see any way it could
be cheaper. i'd have to get a more expensive brand of ssd plus i'd
need an nvme pci-e card or two.
However, I have SATA ports in abundance. On the motherboard, I have 6 x
SATA III (4 used for the new SSDs, two previously used for the old SSDs
but now spare) plus another 2 x 1.5Gbs SATA, and some e-sata which i've
never used. In PCI-e slots, I have 16 x SAS/SATA3 on two IBM 1015 LSI
cards (8 ports in use, 4 spare and connected to hot-swap bays, 4 spare
and unconnected).
PCI-e slots are in very short supply. and my m/b doesn't have any nvme
sockets.
If I could find a reasonably priced PCI-e 8x NVMe card that actuAlly
supported two PCI-e NVMe drives (instead of 1 x pci-e nvme + 1 x sata
nvme), i'd probably have swapped out the spare/unused M1015 cards for
it. i don't have any spare 4x slots.
so i did what I could to maximise performance with the hardware I have.
everything I do on the machine is noticably faster, including compiles
and docker builds etc.
but yeah, eventually I'll move to PCI-e NVME drives. sometime after my
next motherboard & cpu upgrade.
I'm waiting to see real-world reviews and benchmarks on the upcoming AMD
Zen CPU.
Intel has some very nice (and expensive) high-end CPUs, but their
low-end and mid-range CPUs are more expensive than old AMD CPUs without
offering much improvement....might make sense for a new system, but not
as an upgrade. Every time I look into switching to Intel, it turns out
I'll have to spend around $1000 to get roughly similar performance to
what I have now with a 6 year old AMD CPU. I'm not going to spend that
kind of money without a really significant benefit.
I could get an AMD FX-8320 or FX-8350 CPU for under $250 but I'd rather
wait for Zen and get a new motherboard with PCI-e 3.0 and other new
stuff too. Just going on past history, I'm quite confident that will
be significantly cheaper and better than switching to Intel...i expect
around $400-$500 rather than $800-$1000.
btw, for torrenting on ZFS, you need to create a separate dataset with
recordsize=16K (instead of the default 128K) to avoid COW fragmentation.
configure deluge or whatever to download to that and then move the
finished torrent to another filesystem.
probably same or similar for btrfs.
On Wednesday, 12 October 2016 1:31:33 AM AEDT Craig
Sanders via luv-main
wrote:
after plugging the old SSD back in? can't remember for sure, but i think
so....it wasn't imported before i rebooted again so wouldn't have been
automatically imported after reboot.
I probably did something like:
zpool import -d /dev/disk/by-id/ <poolname>
the only time i've ever seen something
similar was my own stupid
fault, i rebooted and just pulled out the old SSD forgetting that I
had ZIL and L2ARC for the pools on that SSD. I had to plug the old
SSD back in before I could import the pool, so i could remove them
from the pool (and add partitions from my shiny new SSDs to replace
them).
Did you have to run "zfs import" on it or was it recognised
automatically? If the former how did you do it? Is the initramfs configured to be able to run zfs
import?
yes, i have zfs-initramfs installed.
BTRFS snapshots are working well on the root
filesystems of many
systems I run. The only systems I run without BTRFS as root are
systems where getting console access in the event of problems is too
difficult.
yes, but you can't pipe `btrfs send` to `zfs recv` and expect to get
anything useful. my backup pool is zfs.
and so far, i've had 100% success rate (2/2) with zfs rootfs.
Disclaimer: not a statistically significant sample size. contents
may settle during transport. void where prohibited by law. serving
suggestion only. batteries not included.
crucial mx300
275G SSDs(*). slightly more expensive than a pair of
500-ish GB but much better performance....read speeds roughly 4 x
SATA SSD read (approximating pci-e SSD speeds), write speeds about 2
x SATA SSD.
i haven't run bonnie++ on it yet. it's on my todo list.
If you had 2*NVMe devices it would probably give better performance
than 4*SATA and might be cheaper. That would also leave more SATA
slots free. we're just
on the leading edge of some massive drops in price/GB.
a bit earlier than I was predicting, i though we'd start seeing it
next year. wont be long before 2 or 4TB SSDs are affordable for home
users (you can get 2TB SSDs for around $800 now). and then I can
replace some of my HDD pools.
It's really changing things. For most users 2TB is more than enough
storage even for torrenting movies. I think that spinning media is going to be mostly
obsolete for home
use soon.
yep. and good riddance.
i'd still want to buy them in pairs, for RAID-1/RAID-10 (actually, ZFS
mirrored pairs)
i'll read through that (and the fedora one that it links to) before
rebooting my myth box with systemd again.
it was an unintentional reboot anyway. i'd used grub-set-default
intending to reboot to systemd next time, but the thunderstorm caused
a few second power-outage and the UPS for that machine died ages ago
(haven't replaced it yet). was busy with other stuff and didn't even
notice it was down for a few hours.
From a quick read of the man page it appears that the
-D option to
journalctl might do what we want. It appears that Debian has moved to
not having the binary journals so I don't have a conveniant source of
test data.
looks like Storage=auto in /etc/systemd/journald.conf, but
/var/log/journal isn't created by default. so no persistent journal by
default. bad default.
debian configuring it to use rsyslogd by default is a good thing, but
doesn't help when the system won't boot far enough to get rsyslogd
running. should be on by default. maybe even automatically turn off
journald's persistence as soon as rsyslogd (or whatever external logger)
successfully starts up.
craig
--
craig sanders <cas(a)taz.net.au>
3:35 p.m.
New subject: How to make systemd more reliable
On Wednesday, 12 October 2016 2:46:01 PM AEDT Craig Sanders via luv-main
wrote:
Last time I was buying there wasn't much price difference between SATA and
NVMe devices. Usually buying 2 medium size devices is cheaper than 4 small
devices.
The early versions of BTRFS used a 4K allocation block size. Recent versions
(the version in Debian/Jessie and maybe before) use a 16K allocation block
size. There is no way for BTRFS to use the kind of large block allocation
that ZFS does in that regard. This sucks for many use cases but means you
don't lose anything for torrenting in this example.
yes, but you can't pipe `btrfs send` to `zfs recv`
and expect to get
anything useful. my backup pool is zfs.
In the early days the plan was to have btrfs receive not rely on BTRFS, so you
could send a snapshot to a non-BTRFS filesystem. I don't know if this is a
feature they continued with.
If you had
2*NVMe devices it would probably give better performance
than 4*SATA and might be cheaper. That would also leave more SATA
slots free.
yes, that would certainly be a LOT faster. can't see any way it could
be cheaper. i'd have to get a more expensive brand of ssd plus i'd
need an nvme pci-e card or two. PCI-e slots are in very short supply. and my m/b
doesn't have any nvme
sockets.
That's a problem for you then. Also most motherboards don't support booting
from NVMe at the moment, there are many ways of working around this (chaining
boot from CD, USB, etc) but it's an annoyance.
Intel has some very nice (and expensive) high-end
CPUs, but their
low-end and mid-range CPUs are more expensive than old AMD CPUs without
offering much improvement....might make sense for a new system, but not
as an upgrade. Every time I look into switching to Intel, it turns out
I'll have to spend around $1000 to get roughly similar performance to
what I have now with a 6 year old AMD CPU. I'm not going to spend that
kind of money without a really significant benefit.
I keep getting such great systems from e-waste. Mostly Intel CPUs because
there seems to be a correlation between people who buy Intel CPUs and people
who dispose of systems after a few years. I'll try and get some more for LUV
members.
It's
really changing things. For most users 2TB is more than enough
storage even for torrenting movies.
btw, for torrenting on ZFS, you need to create a separate dataset with
recordsize=16K (instead of the default 128K) to avoid COW fragmentation.
configure deluge or whatever to download to that and then move the
finished torrent to another filesystem.
probably same or similar for btrfs. i'd still want to buy them in pairs, for
RAID-1/RAID-10 (actually, ZFS
mirrored pairs)
The failure modes of SSD are quite different to the failure modes of spinning
media. I expect it will be some years before there is adequate research into
how SSDs fail and some more years before filesystems develop to work around
them. ZFS and WAFL do some interesting things to work around known failure
modes of spinning media, they won't be as reliable on SSD as they might be
because of the spinning media optimisation.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
4:03 p.m.
New subject: How to make systemd more reliable
On Wed, Oct 12, 2016 at 03:35:34PM +1100, russell(a)coker.com.au wrote:
well, yes, of course it is. we're talking about my system here, and why
I choose 4 cheap SATA SSDs rather than two pci-e SSDs.
I'd still use some kind of raid-1/mirroring anyway, no matter what kind
of drives I had. raid isn't a substitute for backups, but it does reduce
the risk that you'll need to restore from backup (and the downtime and
PITA-factor that goes along with restoring)
also, there's no way for ZFS to correct any detected errors if there's
no redundancy.
i don't mind paying double for storage. it's a bit painful at purchase
time, but that's quickly forgotten. and a lot less painful than the time
and hassle required to restore from backup, and losing everything new or
modified since the previous backup (nightly, but that's still up to a
full day's worth of stuff that could be lost. Now that i've got rootfs
on ZFS, I can snapshot frequently and backup more often with zfs send)
craig
--
craig sanders <cas(a)taz.net.au>
On Wednesday, 12 October 2016 2:46:01 PM AEDT Craig
Sanders via luv-main
wrote:
nope. from what i've read, they originally intended to make it tar
compatible but tar couldn't do what they needed, so they dropped that
idea.
yes, but you can't pipe `btrfs send` to `zfs
recv` and expect to get
anything useful. my backup pool is zfs.
In the early days the plan was to have btrfs receive not rely on
BTRFS, so you could send a snapshot to a non-BTRFS filesystem. I
don't know if this is a feature they continued with. Last time I was buying there wasn't much price
difference between SATA
and NVMe devices. Usually buying 2 medium size devices is cheaper
than 4 small devices.
right, but there's a difference between the price of Crucial SSDs and
Samsung or Intel. There's also a price difference between having to buy
a pci-e nvme card and not having to buy one.
PCI-e slots
are in very short supply. and my m/b doesn't have any
nvme sockets.
That's a problem for you then. i'd still
want to buy them in pairs, for RAID-1/RAID-10 (actually,
ZFS mirrored pairs)
The failure modes of SSD are quite different to the failure modes
of spinning media. I expect it will be some years before there is
adequate research into how SSDs fail and some more years before
filesystems develop to work around them. ZFS and WAFL do some
interesting things to work around known failure modes of spinning
media, they won't be as reliable on SSD as they might be because of
the spinning media optimisation.
4:14 p.m.
New subject: How to make systemd more reliable
On Wednesday, 12 October 2016 4:03:27 PM AEDT Craig Sanders via luv-main
wrote:
The ZFS copies= feature is designed to operate in single-disk mode. The ZFS
metadata has 1 more copy than the data, so even without using copies=2 you
will have multiple copies of metadata on a single disk. But whether that does
any good on SSD is anyone's guess at the moment.
The failure
modes of SSD are quite different to the failure modes
of spinning media. I expect it will be some years before there is
adequate research into how SSDs fail and some more years before
filesystems develop to work around them. ZFS and WAFL do some
interesting things to work around known failure modes of spinning
media, they won't be as reliable on SSD as they might be because of
the spinning media optimisation.
I'd still use some kind of raid-1/mirroring anyway, no matter what kind
of drives I had. raid isn't a substitute for backups, but it does reduce
the risk that you'll need to restore from backup (and the downtime and
PITA-factor that goes along with restoring)
also, there's no way for ZFS to correct any detected errors if there's
no redundancy. i don't mind paying double for storage. it's a
bit painful at purchase
time, but that's quickly forgotten. and a lot less painful than the time
and hassle required to restore from backup, and losing everything new or
modified since the previous backup (nightly, but that's still up to a
full day's worth of stuff that could be lost. Now that i've got rootfs
on ZFS, I can snapshot frequently and backup more often with zfs send)
It's a pity that the options for RAID on laptops are so poor. Thinkpads used
to have the option of buying a disk bay to replace the CD/DVD drive but I
don't know if that is still available and it was always quite expensive.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
2361
days inactive
2362
days old
11 comments
4 participants
participants (4)
-
Allan Duncan -
Clinton Roy -
Craig Sanders -
Russell Coker