Hi Bill,

I see you mention that you have tried the find | xargs rsync option without a lot of luck so i will just put this here in case it is different to what you have tried but we are using this quite sucessfully

cd <source>; find . -maxdepth 1 -mindepth 1 ! -path './.*' -print0 | xargs -0 -n1 -P<no_of_threads> -I% rsync -irlt  % <destination>/.



I am however keen to look at the parsync linked by Nic. Thanks for the link


Nick Evans

On 16 February 2017 at 16:01, Nic Baxter via luv-main <luv-main@luv.asn.au> wrote:



On 16/02/17 12:12, Bill Yang via luv-main wrote:

Hi there


I need to transfer 200+ TB data from one storage server (Red Hat Linux based) to another (FreeBSD). I am planning to use rsync with multiple threads in a script. There are a number of suggestions on the Internet (find + xargs + rsync), but none of them worked well so far. I also need a reliable way to check whether all files/directories from the source server have been copied to the destination server. Any suggestions/help would be appreciated.


 This may help. http://moo.nac.uci.edu/~hjm/parsync/
scripts running on top of rsync with some crude load balancing and throttling.

_______________________________________________
luv-main mailing list
luv-main@luv.asn.au
https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main