[ale] Speed un-tar?

Robert L. Harris robert.l.harris at gmail.com
Wed Jul 30 10:08:05 EDT 2014


Maybe if I used smaller chunk loads but not sure what that would get me.


On Tue, Jul 29, 2014 at 10:33 PM, Jeff Hubbs <jhubbslist at att.net> wrote:

> Do you have enough RAM to read from disk and write to a ramdisk or vice
> versa, whichever helps?
>
> On 7/29/14, 6:44 PM, Jim Kinney wrote:
>
>> Ugh. Sounds like you'll need to do it stages. Coarse grain search written
>> to new files and a fine grained search on those new files.
>> On Jul 29, 2014 6:08 PM, "Robert L. Harris" <robert.l.harris at gmail.com>
>> wrote:
>>
>>  Unfortunately I can't touch the VM's configuration or the hardware
>>> underneath it.  Supposedly I'm spread across a minimum of 6 "fast" disks
>>> already.  I can't really go less than 10 files though as I am concerned
>>> with information being spread across multiple files.  I was hoping
>>> someone
>>> knew a tool/util which would rip through the data faster I had not found
>>> yet.
>>>
>>> Robert
>>>
>>>
>>>
>>> On Tue, Jul 29, 2014 at 4:00 PM, Jim Kinney <jim.kinney at gmail.com>
>>> wrote:
>>>
>>>  unless you can spread that read/write load out over many, many spindles,
>>>> you're stuck. Now add in the VMmust access through the virtual drive
>>>> process and you've got another performance hit.
>>>>
>>>> You _could_ add extra drives to the VM that are hosted on a decent array
>>>> (fiber channel or LA network iSCSI), copy the files to the new home in a
>>>> batch and hit the 4G RAM limit.
>>>>
>>>> If possible, can you add more RAM to that VM?
>>>>
>>>>
>>>> On Tue, Jul 29, 2014 at 5:10 PM, Robert L. Harris <
>>>> robert.l.harris at gmail.com
>>>>
>>>>> wrote:
>>>>> I'm working on a tool to parse through a lot of data for processing.
>>>>>
>>>>   Right
>>>>
>>>>> now it's taking longer than I wish it would so I'm trying to find ways
>>>>>
>>>> to
>>>
>>>> improve the performance.  Right now it appears the biggest bottleneck
>>>>>
>>>> is
>>>
>>>> IO.  I'm looking at about 2000 directories which contain between 1 and
>>>>>
>>>> 200
>>>>
>>>>> files in tar.gz format on a VM with 4 Gigs of RAM.  I need to load the
>>>>>
>>>> data
>>>>
>>>>> into an array to do some pre-processing cleanup so I am currently
>>>>>
>>>> chopping
>>>>
>>>>> the files in each of the directories into an array of groups of 10
>>>>>
>>>> files
>>>
>>>> at
>>>>
>>>>> a time ( seems to be the sweet spot to prevent swap ) and then a
>>>>>
>>>> straight
>>>
>>>> forward loop of which each iteration executes:
>>>>>
>>>>>    tar xzOf $Loop |
>>>>>
>>>>> and then pushes it into my array for processing.
>>>>>
>>>>> I have tried:
>>>>>
>>>>>   gzcat $Loop | tar xO |
>>>>>
>>>>> which is actually slower.  Yes, I'm at the point of trying to squeeze
>>>>> seconds of time out of a group.  Any thoughts of a method which might
>>>>>
>>>> be
>>>
>>>> quicker?
>>>>>
>>>>> Robert
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> :wq!
>>>>>
>>>>>  ------------------------------------------------------------
>>> ---------------
>>>
>>>> Robert L. Harris
>>>>>
>>>>> DISCLAIMER:
>>>>>        These are MY OPINIONS             With Dreams To Be A King,
>>>>>         ALONE.  I speak for                      First One Should Be A
>>>>>
>>>> Man
>>>
>>>>         no-one else.                                     - Manowar
>>>>> -------------- next part --------------
>>>>> An HTML attachment was scrubbed...
>>>>> URL: <
>>>>>
>>>>>  http://mail.ale.org/pipermail/ale/attachments/20140729/
>>> 38cb3da3/attachment.html
>>>
>>>> _______________________________________________
>>>>> Ale mailing list
>>>>> Ale at ale.org
>>>>> http://mail.ale.org/mailman/listinfo/ale
>>>>> See JOBS, ANNOUNCE and SCHOOLS lists at
>>>>> http://mail.ale.org/mailman/listinfo
>>>>>
>>>>>
>>>>
>>>> --
>>>> --
>>>> James P. Kinney III
>>>>
>>>> Every time you stop a school, you will have to build a jail. What you
>>>>
>>> gain
>>>
>>>> at one end you lose at the other. It's like feeding a dog on his own
>>>>
>>> tail.
>>>
>>>> It won't fatten the dog.
>>>> - Speech 11/23/1900 Mark Twain
>>>>
>>>>
>>>> *http://heretothereideas.blogspot.com/
>>>> <http://heretothereideas.blogspot.com/>*
>>>> -------------- next part --------------
>>>> An HTML attachment was scrubbed...
>>>> URL: <
>>>>
>>>>  http://mail.ale.org/pipermail/ale/attachments/20140729/
>>> 385b6337/attachment.html
>>>
>>>> _______________________________________________
>>>> Ale mailing list
>>>> Ale at ale.org
>>>> http://mail.ale.org/mailman/listinfo/ale
>>>> See JOBS, ANNOUNCE and SCHOOLS lists at
>>>> http://mail.ale.org/mailman/listinfo
>>>>
>>>>
>>>
>>> --
>>> :wq!
>>> ------------------------------------------------------------
>>> ---------------
>>> Robert L. Harris
>>>
>>> DISCLAIMER:
>>>        These are MY OPINIONS             With Dreams To Be A King,
>>>         ALONE.  I speak for                      First One Should Be A
>>> Man
>>>         no-one else.                                     - Manowar
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL: <
>>> http://mail.ale.org/pipermail/ale/attachments/20140729/
>>> e382a9b2/attachment.html
>>> _______________________________________________
>>> Ale mailing list
>>> Ale at ale.org
>>> http://mail.ale.org/mailman/listinfo/ale
>>> See JOBS, ANNOUNCE and SCHOOLS lists at
>>> http://mail.ale.org/mailman/listinfo
>>>
>>>  -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <http://mail.ale.org/pipermail/ale/attachments/
>> 20140729/4b9bfb79/attachment.html>
>> _______________________________________________
>> Ale mailing list
>> Ale at ale.org
>> http://mail.ale.org/mailman/listinfo/ale
>> See JOBS, ANNOUNCE and SCHOOLS lists at
>> http://mail.ale.org/mailman/listinfo
>>
>>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>



-- 
:wq!
---------------------------------------------------------------------------
Robert L. Harris

DISCLAIMER:
      These are MY OPINIONS             With Dreams To Be A King,
       ALONE.  I speak for                      First One Should Be A Man
       no-one else.                                     - Manowar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ale.org/pipermail/ale/attachments/20140730/fcdf40c9/attachment.html>


More information about the Ale mailing list