Hi Bill, I see you mention that you have tried the find | xargs rsync option without a lot of luck so i will just put this here in case it is different to what you have tried but we are using this quite sucessfully cd <source>; find . -maxdepth 1 -mindepth 1 ! -path './.*' -print0 | xargs -0 -n1 -P<no_of_threads> -I% rsync -irlt % <destination>/. I am however keen to look at the parsync linked by Nic. Thanks for the link Nick Evans On 16 February 2017 at 16:01, Nic Baxter via luv-main <luv-main@luv.asn.au> wrote:
On 16/02/17 12:12, Bill Yang via luv-main wrote:
Hi there
I need to transfer 200+ TB data from one storage server (Red Hat Linux based) to another (FreeBSD). I am planning to use rsync with multiple threads in a script. There are a number of suggestions on the Internet (find + xargs + rsync), but none of them worked well so far. I also need a reliable way to check whether all files/directories from the source server have been copied to the destination server. Any suggestions/help would be appreciated.
This may help. http://moo.nac.uci.edu/~hjm/parsync/ scripts running on top of rsync with some crude load balancing and throttling.
_______________________________________________ luv-main mailing list luv-main@luv.asn.au https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main