Hi All, I'm currently getting our DR system fine tuned at work, and as part of this I would like some software to take images of some of our larger Linux servers (RHCE 5 & 6) similar to how programs like Shadow Protect does for Windoze based systems, in other words while the system is live. Some of these server have up to 6TB of data stored on them and as you could appreciate, downing the server to run something like Clonezilla would take far too long to be practical. I would also like to be able to schedule these snapshots, say once a week. TIA, Chris -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
Hi Chris, Perhaps "fsarchiver" may do the job? It does backup live systems. Cheer Daniel On 28/08/12 11:24, Christopher M. Bailey wrote:
Hi All,
I'm currently getting our DR system fine tuned at work, and as part of this I would like some software to take images of some of our larger Linux servers (RHCE 5 & 6) similar to how programs like Shadow Protect does for Windoze based systems, in other words while the system is live. Some of these server have up to 6TB of data stored on them and as you could appreciate, downing the server to run something like Clonezilla would take far too long to be practical. I would also like to be able to schedule these snapshots, say once a week.
TIA, Chris
-- This message has been scanned for viruses and dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is believed to be clean.
_______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
Daniel Jitnah <djitnah@greenwareit.com.au> wrote:
Perhaps "fsarchiver" may do the job? It does backup live systems.
There are also dump utilities for various file systems, for example xfsdump/xfsrestore for XFS, and probably similar tools for EXT2/3/4. If you're using lvm, you could take an lvm snapshot and then create a backup of that. I'll leave it to the lvm experts to comment in detail.
On Tue, 28 Aug 2012, Daniel Jitnah <djitnah@greenwareit.com.au> wrote:
Perhaps "fsarchiver" may do the job? It does backup live systems.
http://www.fsarchiver.org/Main_Page According to the above web page it merely does file level access to the data. So if you have a live database or something similar then you WILL get corrupt data. On Tue, 28 Aug 2012, Jason White <jason@jasonjgw.net> wrote:
There are also dump utilities for various file systems, for example xfsdump/xfsrestore for XFS, and probably similar tools for EXT2/3/4.
I believe that they all have the same issues.
If you're using lvm, you could take an lvm snapshot and then create a backup of that. I'll leave it to the lvm experts to comment in detail.
To make a backup of a live system you need to snapshot ALL inter-related filesystems at the same time. If for example you have a database store and it's log files on separate filesystems then it may be theoretically impossible to create snapshots in a way that guarantees no data loss on restore (based on previous discussion on this list about database logging). If there are no cross-filesystem data dependencies then LVM snapshots will work well. As will BTRFS and ZFS snapshots. But for RHEL there is no reasonable option of ZFS for BTRFS Red Hat say: https://access.redhat.com/knowledge/docs/en- US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/ch-btrfs.html # Btrfs is not a production quality file system at this point. With Red Hat # Enterprise Linux 6 it is at a tech preview stage So for a RHEL user who wants live snapshots of databases it seems that any good option would involve using LVM and keeping all correlated data on one filesystem. http://etbe.coker.com.au/2012/08/08/hard-drives-backup/ For someone who's not using a Red Hat supported environment ZFS seems to be a necessary part of an ideal solution at this time, see my above blog post for details. For backing up the boot partition etc (the other part of the original request) I'm not sure what is the best option on Linux and so far none of the answers have addressed this. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On 28/08/12 13:03, Russell Coker wrote:
# Btrfs is not a production quality file system at this point. With Red Hat # Enterprise Linux 6 it is at a tech preview stage
I would argue that is the case for btrfs in any distro as the upstream code in the mainline kernel is still marked experimental. cheers, Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
[oops - missed this bit previously] On 28/08/12 13:03, Russell Coker wrote:
According to the above web page it merely does file level access to the data. So if you have a live database or something similar then you WILL get corrupt data.
I would have thought this would be a risk for any mechanism that doesn't get the database to dump its state in a consistent manner, whether it be file level, filesystem level or disk level snapshots. cheers, Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
On Tue, 28 Aug 2012, Chris Samuel <chris@csamuel.org> wrote:
According to the above web page it merely does file level access to the data. So if you have a live database or something similar then you WILL get corrupt data.
I would have thought this would be a risk for any mechanism that doesn't get the database to dump its state in a consistent manner, whether it be file level, filesystem level or disk level snapshots.
Why wouldn't filesystem level snapshots allow a consistent database backup if a single filesystem is used for all tables and logs? -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Hi All, On 28/08/2012 11:24 AM, Christopher M. Bailey wrote:
I'm currently getting our DR system fine tuned at work, and as part of this I would like some software to take images of some of our larger Linux servers (RHCE 5 & 6) similar to how programs like Shadow Protect does for Windoze based systems, in other words while the system is live. Some of these server have up to 6TB of data stored on them and as you could appreciate, downing the server to run something like Clonezilla would take far too long to be practical. I would also like to be able to schedule these snapshots, say once a week.
Unless I am mistaken, all Shadow Protect does is create snapshot of the filesystem at a point in time. No different to taking an LVM snapshot. If at that point in time there are processes still with open files and data in memory, then you have every risk of corruption and lack of a suitable backup. Non-database files are less likely to be a problem with rsync type backups -- I use rsnapshot for these. But database tables need to be fully consistent and so a short "freeze" will be required to be sure that everything is good. Lots to consider, are you using Xen, KVM or any other virtualization in your setup? Are you using RAID1 and can you shutdown briefly, disconnect a drive from RAID1 (preferably with 3 or more RAID members in normal operation). LVM snapshots are also a good option. I shutdown entire VM's just long enough to snapshot all file systems, then restart the VM and proceed with backing up off the snapshots. How many and what type of disks make up the 6TB ? How are they set up? Using dump tools is likely to cause problems unless the dump is of a consistent snapshot. Linus sees dump of running file systems as "stupid" because you can never be sure that the OS has written everything to disk, so you cannot guarantee the state -- unless of course it is an unmounted snapshot area that you are dumping, you will not get a "proper" backup. Oracle has been claiming production ready BTRFS, but I don't know anyone whom really believes that, I mean *really* believes. Using BTRFS today has it's own risks. With 6TB of data, you really should have RAID1 protection at least and LVM snspshots with perhaps the rsnapshot tool and/or dumps (again of non-running file systems). I would say that you need a standby server to rsync to as well, a DR server, best at a separate DR site if possible. Cheers -- Kind Regards AndrewM Andrew McGlashan Broadband Solutions now including VoIP Current Land Line No: 03 9012 2102 Mobile: 04 2574 1827 Fax: 03 9012 2178 National No: 1300 85 3804 Affinity Vision Australia Pty Ltd http://affinityvision.com.au http://securemywireless.com.au http://adsl2choice.net.au In Case of Emergency -- http://affinityvision.com.au/ice.html
Andrew McGlashan wrote:
Non-database files are less likely to be a problem with rsync type backups -- I use rsnapshot for these. But database tables need to be fully consistent and so a short "freeze" will be required to be sure that everything is good.
The other way you can do it, is tell rsync to exclude the "real" database files, and don't worry if they're incoherent, because you have a daily cron job that tells the db to make a coherent dump of itself to a local file, which rsync will back up.
On 29/08/2012 4:24 PM, Trent W. Buck wrote:
Andrew McGlashan wrote:
Non-database files are less likely to be a problem with rsync type backups -- I use rsnapshot for these. But database tables need to be fully consistent and so a short "freeze" will be required to be sure that everything is good.
The other way you can do it, is tell rsync to exclude the "real" database files, and don't worry if they're incoherent, because you have a daily cron job that tells the db to make a coherent dump of itself to a local file, which rsync will back up.
Yes I use .rsync-filter files for that purpose, exclusion. Cheers A.
On Wed, 29 Aug 2012, Andrew McGlashan <andrew.mcglashan@affinityvision.com.au> wrote:
Unless I am mistaken, all Shadow Protect does is create snapshot of the filesystem at a point in time. No different to taking an LVM snapshot.
AFAIK the only ways of making a snapshot on Linux are LVM (which is supported everywhere - including RHEL), BTRFS (experimental everywhere except Oracle and "tech preview" in RHEL), and ZFS (reliable but not supported in RHEL). Unless Shadow Protect uses LVM snapshots I don't think it can do what you claim.
If at that point in time there are processes still with open files and data in memory, then you have every risk of corruption and lack of a suitable backup.
No, if you have all MySQL data on a single filesystem then you should be safe with an LVM snapshot.
Lots to consider, are you using Xen, KVM or any other virtualization in your setup? Are you using RAID1 and can you shutdown briefly, disconnect a drive from RAID1 (preferably with 3 or more RAID members in normal operation). LVM snapshots are also a good option. I shutdown entire VM's just long enough to snapshot all file systems, then restart the VM and proceed with backing up off the snapshots.
How many and what type of disks make up the 6TB ? How are they set up?
6TB wouldn't be a RAID-1. On Wed, 29 Aug 2012, "Trent W. Buck" <trentbuck@gmail.com> wrote:
The other way you can do it, is tell rsync to exclude the "real" database files, and don't worry if they're incoherent, because you have a daily cron job that tells the db to make a coherent dump of itself to a local file, which rsync will back up.
That is very good for a small database. For my blog I have a cron job which dumps the database and another cron job on a workstation to copy the daily dump file. Then the directory full of daily dump files is part of the workstation backup. This is for a dump file that's 5.6M compressed. When you have a big database (or even a database that's a few gigs in size) you don't necessarily want to wait for a dump to be loaded before going live. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Russell Coker wrote:
AFAIK the only ways of making a snapshot on Linux are LVM (which is supported everywhere - including RHEL), BTRFS (experimental everywhere except Oracle and "tech preview" in RHEL), and ZFS (reliable but not supported in RHEL).
Nitpick: doesn't linux support an alternative to LVM (EVMS?) that nobody uses? That probably has a snapshot facility, too.
On Wed, 29 Aug 2012, "Trent W. Buck" <trentbuck@gmail.com> wrote:
Nitpick: doesn't linux support an alternative to LVM (EVMS?) that nobody uses? That probably has a snapshot facility, too.
http://en.wikipedia.org/wiki/Enterprise_Volume_Management_System EVMS is long dead and the management tools were migrated to using LVM2 kernel code. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On Wed, 29 Aug 2012, Russell Coker wrote:
When you have a big database (or even a database that's a few gigs in size) you don't necessarily want to wait for a dump to be loaded before going live.
In these cases a cold standby and replication may do the trick for you. With MySQL it is not that hard to set up. Regards Peter
Hi All,
On 28/08/2012 11:24 AM, Christopher M. Bailey wrote:
I'm currently getting our DR system fine tuned at work, and as part of this I would like some software to take images of some of our larger Linux servers (RHCE 5 & 6) similar to how programs like Shadow Protect does for Windoze based systems, in other words while the system is live. Some of these server have up to 6TB of data stored on them and as you could appreciate, downing the server to run something like Clonezilla would take far too long to be practical. I would also like to be able to schedule these snapshots, say once a week.
Unless I am mistaken, all Shadow Protect does is create snapshot of the filesystem at a point in time. No different to taking an LVM snapshot. If at that point in time there are processes still with open files and data in memory, then you have every risk of corruption and lack of a suitable backup.
Unlikely. Windows VSS is _very_ good. Firstly, it allows snapshot sets to be taken atomically across multiple volumes (think /var/db_data and /var/db_logs) ensuring true crash-consistent snapshots. If /var/db_data and /var/db_logs were on separate LVM volumes under Linux, all bets are off. Secondly, VSS is tightly integrated with the OS. Any decent database or other application will provide a VSS writer which tells the backup application exactly how to back up the data for the database. To give an example, for Windows 2003 and older, when a VSS snapshot is taken the registry files are dumped to a different location and instructions are provided to the backup application to back the files up from there as if they were really in C:\Windows\System32\config. For an MSSQL differential backup, the backup application is told exactly which file fragments (eg offset 1234 length 456) have changed since the full backup was taken, so that only those need to be backed up. For Exchange, log truncation etc is handled automatically. Even without fancy instructions required, the VSS snapshot process gives applications a chance to ensure application level consistency. Sorry to sound like a Microsoft shill, but I've contributed code to the Bacula project to do all the above stuff so I can appreciate the differences. James
On 29/08/2012 9:57 PM, James Harper wrote:
Unlikely. Windows VSS is _very_ good. Firstly, it allows snapshot sets to be taken atomically across multiple volumes (think /var/db_data and /var/db_logs) ensuring true crash-consistent snapshots. If /var/db_data and /var/db_logs were on separate LVM volumes under Linux, all bets are off.
Secondly, VSS is tightly integrated with the OS. Any decent database or other application will provide a VSS writer which tells the backup application exactly how to back up the data for the database. To give an example, for Windows 2003 and older, when a VSS snapshot is taken the registry files are dumped to a different location and instructions are provided to the backup application to back the files up from there as if they were really in C:\Windows\System32\config. For an MSSQL differential backup, the backup application is told exactly which file fragments (eg offset 1234 length 456) have changed since the full backup was taken, so that only those need to be backed up. For Exchange, log truncation etc is handled automatically.
Even without fancy instructions required, the VSS snapshot process gives applications a chance to ensure application level consistency.
Sorry to sound like a Microsoft shill, but I've contributed code to the Bacula project to do all the above stuff so I can appreciate the differences.
Yes, I have to agree that VSS is seriously good, but you still need application aware programs to be sure that the backup is fully consistent. MSSQL might be fine without special consideration, but just because it is a Windows box, doesn't mean it needs to have a native database. Oh and to Russell... 6TB could easily be multiple RAID1 volumes, who knows, it could be anything -- hence why I asked and didn't ass-u-me ... ;) There is always more than one way to skin a cat, so to speak and usually quite a number of ways. Cheers A.
participants (9)
-
Andrew McGlashan -
Chris Samuel -
Christopher M. Bailey -
Daniel Jitnah -
James Harper -
Jason White -
Peter Ross -
Russell Coker -
Trent W. Buck