Hey all, I have a few cases where I'd like to encrypt without taking the system down for extended periods, ie, servers. In the windows/apple world truecrypt / bitlocker / filevault will all let you encrypt the root partition as a background process, throttled to a low IO load. Usually this requires a reboot to get started, then runs in the background. Does anyone know how to achieve this in the Linux world? (preferably with luks) -Noah
Hi Noah, You might be able to achieve this with ecryptfs, from Ubuntu. It can mount an encrypted drive with unencrypted-passthrough; you can then run around rewriting all the files to convert them. ie. mount -t ecryptfs encrypted crypted find -type f crypted -exec rewrite_file \{\} \; where rewrite_file does something like cp $FILE tmp_file rm $FILE mv tmp_file $FILE -Toby On Thu, 23 Apr 2015 at 08:36 Noah O'Donoghue <noah.odonoghue@gmail.com> wrote:
Hey all,
I have a few cases where I'd like to encrypt without taking the system down for extended periods, ie, servers.
In the windows/apple world truecrypt / bitlocker / filevault will all let you encrypt the root partition as a background process, throttled to a low IO load. Usually this requires a reboot to get started, then runs in the background.
Does anyone know how to achieve this in the Linux world? (preferably with luks)
-Noah _______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
Hmm... Seems like it might be my only option but it's really precarious. What about: 1. In use files 2. Indexers, backup software, media servers, that will see each file change and will try and back it up 3. File modification dates.... Metadata... etc etc. On 23 April 2015 at 14:59, Toby Corkindale <toby@dryft.net> wrote:
Hi Noah, You might be able to achieve this with ecryptfs, from Ubuntu. It can mount an encrypted drive with unencrypted-passthrough; you can then run around rewriting all the files to convert them. ie. mount -t ecryptfs encrypted crypted find -type f crypted -exec rewrite_file \{\} \; where rewrite_file does something like cp $FILE tmp_file rm $FILE mv tmp_file $FILE
-Toby
On Thu, 23 Apr 2015 at 08:36 Noah O'Donoghue <noah.odonoghue@gmail.com> wrote:
Hey all,
I have a few cases where I'd like to encrypt without taking the system down for extended periods, ie, servers.
In the windows/apple world truecrypt / bitlocker / filevault will all let you encrypt the root partition as a background process, throttled to a low IO load. Usually this requires a reboot to get started, then runs in the background.
Does anyone know how to achieve this in the Linux world? (preferably with luks)
-Noah _______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
Yeah, I don't think it's a good idea, but if you really HAVE to do it, that's a way. For the case of an external hard drive, where you don't want to have to reformat it nor buy a second drive, it'd be a valid way of doing things that I'd trust. But who keeps all their data on just one drive? Just reformat the backup drive to be encrypted.. back everything up to it.. test the backup.. then reformat the main drive and restore to it from the now-encrypted backup. On Thu, 23 Apr 2015 at 16:53 Noah O'Donoghue <noah.odonoghue@gmail.com> wrote:
Hmm... Seems like it might be my only option but it's really precarious.
What about:
1. In use files 2. Indexers, backup software, media servers, that will see each file change and will try and back it up 3. File modification dates.... Metadata...
etc etc.
On 23 April 2015 at 14:59, Toby Corkindale <toby@dryft.net> wrote:
Hi Noah, You might be able to achieve this with ecryptfs, from Ubuntu. It can mount an encrypted drive with unencrypted-passthrough; you can then run around rewriting all the files to convert them. ie. mount -t ecryptfs encrypted crypted find -type f crypted -exec rewrite_file \{\} \; where rewrite_file does something like cp $FILE tmp_file rm $FILE mv tmp_file $FILE
-Toby
On Thu, 23 Apr 2015 at 08:36 Noah O'Donoghue <noah.odonoghue@gmail.com> wrote:
Hey all,
I have a few cases where I'd like to encrypt without taking the system down for extended periods, ie, servers.
In the windows/apple world truecrypt / bitlocker / filevault will all let you encrypt the root partition as a background process, throttled to a low IO load. Usually this requires a reboot to get started, then runs in the background.
Does anyone know how to achieve this in the Linux world? (preferably with luks)
-Noah _______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
Hi Noah, Not sure what you are trying to achieve. But if it is protecting system files from unwanted/malicious modification, then encryption is probably not the way to go. Imagine if a persistent attacker actually managed to change the encrypted data: It would be a much worse result, as you could be unable to de-crypt and access everything!! Even if it did not, it may well be that a change may not be detected until it is too late. So to handle this I would suggest using proper intrusion detection software, design for the purpose, eg: Tripwire. Its better that you know that an attack has taken place and you can react accordingly, than you believing that you are secure (falsely) and an attacker somehow finding its way in and you not knowing about it. For securing sensitive data from unwarranted access, then using a file based encryption system eg:ecryptfs might do it. Cheers Daniel. On 23/04/15 16:52, Noah O'Donoghue wrote:
Hmm... Seems like it might be my only option but it's really precarious.
What about:
1. In use files 2. Indexers, backup software, media servers, that will see each file change and will try and back it up 3. File modification dates.... Metadata...
etc etc.
On 23 April 2015 at 14:59, Toby Corkindale <toby@dryft.net <mailto:toby@dryft.net>> wrote:
Hi Noah, You might be able to achieve this with ecryptfs, from Ubuntu. It can mount an encrypted drive with unencrypted-passthrough; you can then run around rewriting all the files to convert them. ie. mount -t ecryptfs encrypted crypted find -type f crypted -exec rewrite_file \{\} \; where rewrite_file does something like cp $FILE tmp_file rm $FILE mv tmp_file $FILE
-Toby
On Thu, 23 Apr 2015 at 08:36 Noah O'Donoghue <noah.odonoghue@gmail.com <mailto:noah.odonoghue@gmail.com>> wrote:
Hey all,
I have a few cases where I'd like to encrypt without taking the system down for extended periods, ie, servers.
In the windows/apple world truecrypt / bitlocker / filevault will all let you encrypt the root partition as a background process, throttled to a low IO load. Usually this requires a reboot to get started, then runs in the background.
Does anyone know how to achieve this in the Linux world? (preferably with luks)
-Noah _______________________________________________ luv-main mailing list luv-main@luv.asn.au <mailto:luv-main@luv.asn.au> http://lists.luv.asn.au/listinfo/luv-main
_______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
On Thu, Apr 23, 2015 at 04:59:21AM +0000, Toby Corkindale wrote:
find -type f crypted -exec rewrite_file \{\} \; where rewrite_file does something like cp $FILE tmp_file rm $FILE mv tmp_file $FILE
if you're going to do something like this then use 'cp -a' to preserve file ownership, group, permissions, and timestamps. otherwise you'll find everything owned by root.root with permissions set according to your umask, and a timestamp of now. this is precisely what you DO NOT want and something that's extremly difficult and tedious to fix without either a good backup or a reinstall (if the files being copied are system/package files) craig -- craig sanders <cas@taz.net.au>
On 23/04/2015 7:51 PM, Craig Sanders wrote:
On Thu, Apr 23, 2015 at 04:59:21AM +0000, Toby Corkindale wrote:
find -type f crypted -exec rewrite_file \{\} \; where rewrite_file does something like cp $FILE tmp_file rm $FILE mv tmp_file $FILE
if you're going to do something like this then use 'cp -a' to preserve file ownership, group, permissions, and timestamps.
Yes, but rsync would be better. I hate it when I see "cp" without at least "cp -p" ... ;-) Having a 3 way RAID-1 disk set would be the way to go, take one disk out of RAID and LUKS that disk, then add the LUKS volume back in as a RAID component, let it sync; when done do the same for each other mirror disk that you want to keep. Oh and if you are going to LUKS encrypt the root file system, then you'll need to consider other things. I use dropbear to enter pass phrases at boot time via ssh with "cryptsetup luksOpen ...." for each container and then LVM to provide all the required file systems (including root). A.
All solutions that involve copying the data elsewhere or restoring data don't actually solve my problem statement - Which is to encrypt with minimal downtime. Yes, of course - I keep backups. In two different physical locations. But restoring them is actually a worst case scenario which involves days of downtime. It takes time to copy TB's of data. Even if you break a RAID 1 you're still going to have to make the other half read only or you're going to have new data on your original that doesn't make it across. And correct me if I'm wrong with the ecryptfs solution - but doesn't it leave unencrypted copies of your files in free space? Because it's not encrypting in place at a block level, some / most / many files are left unallocated but in clear text on your volume, until you later overwrite them with new data? I'll try to give another example of a use case: Imagine you take over as sysadmin for a company. You run a web based instant messenger solution, hosted off premises in someone else's VPS. eg, an OpenStack provider. The previous sysadmin has been storing logs / message history on the server disk. These are required because your web clients need to be able to query conversion history (think Facebook Messenger). It concerns you that these logs are stored in plain text, because often you have drive failures (and some are RAID 1 for performance) , and drives cannot be cleanly sanitized before they go back to the manufacturer because they are dead, and besides, are the responsibility of your Openstack provider anyway. You decide that to replicate your environment or start fresh would be too costly, and prefer an incremental approach that you test on your dev environment first. It strikes me as strange that we have kernel level APIs for abstracting blocks, we can do all this fantastic crazy stuff with LVM, yet can't do this simple operation that every other OS seems to be able to.. Imagine if Android said "sorry, you must wipe your phone first" when you turned on device encryption... On 23 April 2015 at 22:27, Andrew McGlashan < andrew.mcglashan@affinityvision.com.au> wrote:
On 23/04/2015 7:51 PM, Craig Sanders wrote:
On Thu, Apr 23, 2015 at 04:59:21AM +0000, Toby Corkindale wrote:
find -type f crypted -exec rewrite_file \{\} \; where rewrite_file does something like cp $FILE tmp_file rm $FILE mv tmp_file $FILE
if you're going to do something like this then use 'cp -a' to preserve file ownership, group, permissions, and timestamps.
Yes, but rsync would be better.
I hate it when I see "cp" without at least "cp -p" ... ;-)
Having a 3 way RAID-1 disk set would be the way to go, take one disk out of RAID and LUKS that disk, then add the LUKS volume back in as a RAID component, let it sync; when done do the same for each other mirror disk that you want to keep.
Oh and if you are going to LUKS encrypt the root file system, then you'll need to consider other things. I use dropbear to enter pass phrases at boot time via ssh with "cryptsetup luksOpen ...." for each container and then LVM to provide all the required file systems (including root).
A.
_______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
On Fri, 24 Apr 2015 at 09:22 Noah O'Donoghue <noah.odonoghue@gmail.com> wrote:
Yes, of course - I keep backups. In two different physical locations. But restoring them is actually a worst case scenario which involves days of downtime. It takes time to copy TB's of data. Even if you break a RAID 1 you're still going to have to make the other half read only or you're going to have new data on your original that doesn't make it across.
I think people were suggesting blanking each removed half of the array, prior to making the underlying block device encrypted, then adding and re-mirroring.
And correct me if I'm wrong with the ecryptfs solution - but doesn't it leave unencrypted copies of your files in free space? Because it's not encrypting in place at a block level, some / most / many files are left unallocated but in clear text on your volume, until you later overwrite them with new data?
Yeah, you'd totally want to run "zerofree" over the device after you were done. I was just pointing out that ecryptfs might be able to be made to work for you.
It strikes me as strange that we have kernel level APIs for abstracting blocks, we can do all this fantastic crazy stuff with LVM, yet can't do this simple operation that every other OS seems to be able to.. Imagine if Android said "sorry, you must wipe your phone first" when you turned on device encryption...
Ubuntu can do a (not live) encryption of your home directories.. so it is a thing. (They use ecryptfs, but also require the user to log out while it occurs) I'd say it's about consumer demand. Which exists for things like Windows and Android, and even desktop installs of Ubuntu, but doesn't so much for linux server environment. So there's no incentive for people to write the tools to do it.
On 24 April 2015 at 12:54, Toby Corkindale <toby@dryft.net> wrote:
I think people were suggesting blanking each removed half of the array, prior to making the underlying block device encrypted, then adding and re-mirroring.
Wait, you're suggesting encrypting each drive separately, and then adding the result to a linux RAID device? That's.... crazy, but it might even work. Would be interesting to see what happens to performance considering you have to do a different encryption operation per drive of the RAID array. Unfortunately it looks like it's impossible to convert an existing single drive to a RAID drive.. Or at least the instructions here do the good old copy from one to the other dance. https://wiki.archlinux.org/index.php/Convert_a_single_drive_system_to_RAID I suppose you could in theory do this for systems that are already RAID 1 though.. (as Russell suggests)
On Fri, 24 Apr 2015 02:00:49 PM Noah O'Donoghue wrote:
On 24 April 2015 at 12:54, Toby Corkindale <toby@dryft.net> wrote:
I think people were suggesting blanking each removed half of the array, prior to making the underlying block device encrypted, then adding and re-mirroring.
Wait, you're suggesting encrypting each drive separately, and then adding the result to a linux RAID device?
That's.... crazy, but it might even work.
It will work.
Would be interesting to see what happens to performance considering you have to do a different encryption operation per drive of the RAID array.
For as long as LUKS has been available commonly available CPUs have been able to encrypt/decrypt significantly faster than disks can write/read. CPUs have been increasing in speed at a greater rate than disks, so I really don't think a pair of separately encrypted disks is going to take much CPU time. If you had a pair of SSDs then bulk IO wouldn't be a problem as the commonly available SSDs aren't THAT fast for bulk IO. But if you had lots of random seeks then the latency of encryption might slow things down a bit, but it would probably only be noticable in benchmark results not in real world operations.
Unfortunately it looks like it's impossible to convert an existing single drive to a RAID drive.. Or at least the instructions here do the good old copy from one to the other dance.
https://wiki.archlinux.org/index.php/Convert_a_single_drive_system_to_RAID
I suppose you could in theory do this for systems that are already RAID 1 though.. (as Russell suggests)
You can convert a BTRFS filesystem from a single disk to RAID-1 while it's running. But upgrading to BTRFS is another step that involves downtime. I think that the best thing to do is have some scheduled down-time to convert the system to RAID-1 with encryption. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On Fri, Apr 24, 2015 at 09:33:08PM +1000, Russell Coker wrote:
I think that the best thing to do is have some scheduled down-time to convert the system to RAID-1 with encryption.
yep. note that downtime can be minimised because you can set up the degraded RAID-1 on LUKS disk (a few minutes downtime to powerdown, install disk, and then power up) and then repeatedly rsync everything to it while the system is operating normally....this will minimise the amount of time needed to do the final rsync of the files while the system is offline. when you're ready to do the final conversion, run rsync again. when that's finished, immediately reboot to a rescue CD/USB-stick, mount the old partition(s) and the new degraded RAID-1 partition(s) and then do a final rsync from the old to the new. chroot into the new RAID rootfs, reconfigure grub so that it boots from the RAID rootfs, and re-run grub-install. reboot, then repartition and setup LUKS on the old disk and add it to the RAID-1. total downtime: 5-10 minutes plus however long it takes to run the final rsync (probably only a few minutes). this method is the best way to minimise downtime while transferring a linux filesystem from old storage media to new, whether LUKS or RAID or LVM or whatever is involved or not. craig -- craig sanders <cas@taz.net.au>
On 24 April 2015 at 21:33, Russell Coker <russell@coker.com.au> wrote:
For as long as LUKS has been available commonly available CPUs have been able to encrypt/decrypt significantly faster than disks can write/read. CPUs have been increasing in speed at a greater rate than disks, so I really don't think a pair of separately encrypted disks is going to take much CPU time.
I dispute this assertion, 1. I don't think it's necessarily true.. Benchmarks such as this one indicate average 20-50% performance loss on a 2014 *core i7* ultrabook. ( http://www.phoronix.com/scan.php?page=article&item=ubuntu_1404_encryption&nu... ) 2. I think CPUs / chipsets that make their way into home servers have been increasing in performance per watt, but haven't steadily increased in performance. Especially when you look at the trend for NASes and file servers to feature power efficient Atom & AMD cpus (and ARM..) rather than quad cores, and for some manufacturers preferring dual core i7's now instead of quad core i7's for power / thermal efficiency in laptops.. I think the RAID suggestion is a good one for servers that already have RAID configured, but the rsync and set up crypto (on the LVM or RAID device rather than the disk device) is probably the better long term solution for performance / power efficiency / simplicity. Going to use the rsync solution. Thanks everyone :)
On Mon, 27 Apr 2015 09:11:12 AM Noah O'Donoghue wrote:
On 24 April 2015 at 21:33, Russell Coker <russell@coker.com.au> wrote:
For as long as LUKS has been available commonly available CPUs have been able to encrypt/decrypt significantly faster than disks can write/read. CPUs have been increasing in speed at a greater rate than disks, so I really don't think a pair of separately encrypted disks is going to take much CPU time.
I dispute this assertion,
1. I don't think it's necessarily true.. Benchmarks such as this one indicate average 20-50% performance loss on a 2014 *core i7* ultrabook. ( http://www.phoronix.com/scan.php?page=article&item=ubuntu_1404_encryption&n um=1 )
http://ecryptfs.org/about.html The 50% performance loss was for "home directory encryption" which is using eCryptfs. The above URL describes eCryptfs and it's obvious why performance is reduced.
On Fri, 24 Apr 2015 09:33:08 PM Russell Coker wrote:
If you had a pair of SSDs then bulk IO wouldn't be a problem as the commonly available SSDs aren't THAT fast for bulk IO. But if you had lots of random seeks then the latency of encryption might slow things down a bit, but it would probably only be noticable in benchmark results not in real world operations.
For LUKS the worst case was on Postmark where LUKS delivered 76.9% of unencrypted performance on a SSD. I've pasted back in the above section from my previous message. Anyway unless the server that you are referring to is using SSDs and performing operations that are latency dependent such as running a mail server then you have nothing to worry about.
2. I think CPUs / chipsets that make their way into home servers have been increasing in performance per watt, but haven't steadily increased in performance. Especially when you look at the trend for NASes and file servers to feature power efficient Atom & AMD cpus (and ARM..) rather than quad cores, and for some manufacturers preferring dual core i7's now instead of quad core i7's for power / thermal efficiency in laptops..
While there has been a small trend towards low power systems the general trend is that everything gets faster. Sure the Raspberry Pi is somewhat slow, but it's probably still faster than the Cobalt Qube. Anyway if you run a Raspberry Pi as a server (and some people do) then you are going to be limited by USB speed and reliability. The servers with low power CPUs tend to be the ones that can't contain many disks (Dell apparently now sells low end servers that are designed for a single disk). If you look at affordable servers that are capable of having a few internal SATA disks then you will see options like the Dell PowerEdge T110 which has adequate CPU power.
I think the RAID suggestion is a good one for servers that already have RAID configured, but the rsync and set up crypto (on the LVM or RAID device rather than the disk device) is probably the better long term solution for performance / power efficiency / simplicity.
Going to use the rsync solution. Thanks everyone :)
If it's a server you should still use RAID though. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On Thu, 23 Apr 2015 08:36:32 AM Noah O'Donoghue wrote:
I have a few cases where I'd like to encrypt without taking the system down for extended periods, ie, servers.
In the windows/apple world truecrypt / bitlocker / filevault will all let you encrypt the root partition as a background process, throttled to a low IO load. Usually this requires a reboot to get started, then runs in the background.
Does anyone know how to achieve this in the Linux world? (preferably with luks)
LUKS doesn't permit doing per-file operations because it operates on the block device. However any real server should have RAID-1. So you could break your RAID-1 array, apply LUKS to one half and then re-mirror. Then you break it again apply LUKS to the other half and re-mirror. No down-time as long as you don't have a disk error during the process... A more cautious way of doing it is to replace one of the internal disks with a USB attached external disk so you can replace each disk with a LUKS encrypted block device without ever having less than 2 running disks. USB is slower than SATA but you just wait a while as it all happens in the background. On Thu, 23 Apr 2015 05:18:10 PM Daniel Jitnah wrote:
Not sure what you are trying to achieve. But if it is protecting system files from unwanted/malicious modification, then encryption is probably not the way to go. Imagine if a persistent attacker actually managed to change the encrypted data: It would be a much worse result, as you could be unable to de-crypt and access everything!!
We had a discussion on this list about someone who had their NAS compromised in such an attack. The fact that they hadn't used encryption on the NAS in the past didn't stop the attacker from doing a background encryption of all the files and then asking for a ransom. NB before anyone says "Noah could solve the online encryption problem with the same software that the hackers used" please note that by definition attackers aren't too bothered about the risk of losing data, so the quality of the code is probably much lower than that which you would willingly trust with your data.
Even if it did not, it may well be that a change may not be detected until it is too late. So to handle this I would suggest using proper intrusion detection software, design for the purpose, eg: Tripwire. Its better that you know that an attack has taken place and you can react accordingly, than you believing that you are secure (falsely) and an attacker somehow finding its way in and you not knowing about it.
Tripwire is easy to fool. Sure it can do some good (I've had one of my systems compromised by a trojan that was caught by debsums and would have been caught by tripwire) but I wouldn't rely on it. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
participants (6)
-
Andrew McGlashan -
Craig Sanders -
Daniel Jitnah -
Noah O'Donoghue -
Russell Coker -
Toby Corkindale