[ale] Parallel data transfers (NFS).

JD jdp at algoloma.com
Thu Nov 6 08:29:02 EST 2014


On 11/06/2014 07:47 AM, Raj Wurttemberg wrote:
>> The slowdown is likely to be the storage on the dev server or the network.
>> You have to look at each point in the chain to determine if the current
>> solution is already flooding any part therein.
>>
>> A simple way to cause XX concurrent copies is to spawn them based on first
>> character in the filename.
>>
>> I suspect you were looking for a -j10 switch? Something like GNU
> "parallel"
>> perhaps? https://www.gnu.org/software/parallel/
>>
>> https://superuser.com/questions/327659/whats-the-best-way-to-perform-
>> a-parallel-copy-on-unix
>> has more discussions on this.
> 
> 
> Great information on those links. Thank you. Yes, I just discovered the
> parallel command late last night. Looks interesting. I think that rsync
> might be the best option.
> 


And just to be clear - I didn't know about parallel before yesterday either.
There's always something new to learn.

I'd love to hear from people using it with a tiny description of the use.


More information about the Ale mailing list