[ale] mysql and high load levels

[djinn at djinnspace.com]

More folks on the mysql list are demanding answers...seems like its happening to a lot
of mysql users and nobody is speaking up.  Well, you get what you pay for, right, and
it's served me really, really well for a long time...it's not like the mysql
developers are deliberately doing wierd stuff.

So first on my list is to drop back to a known good version of mysql (the one I was
running on the old machine before I got the bright idea of upgrading everything at
once).  If that doesn't fix it, I will go back to a 2.2 kernel.

Joe, to answer your question, I initially wanted to use 2.4 for iptables, but have
since disabled packet inspection behind my second firewall to try to speed up delivery
of web content.  I figure, by the time it reaches my web server, it's gone thru 3
inspections and is being portsentry'd anyway, I'll take my chances. ;)  So there is no
reason at all for me to be running a 2.4 kernel any more.

Thanks for the excellent info about memory, buffers, caching and CPU...the *only*
bright side of this nightmare is that I'm learning a lot about how to diagnose my
systems.  :)

Jenn

Jeff Hubbs wrote:

> Wow, how about that!  I'm sad to see that this isn't fixed as of 2.4.9,
> which, according to kernel.org, is current.  Anyone here park on the
> kernel devel mailing list and know what the story is?
>
> - Jeff
>
> djinn at djinnspace.com wrote:
> >
> > This showed up on the mysql list...and it caught my attention given what you guys
> > are saying about 2.4 and its wierd memory handling....
> > This post (below) describes my situation almost exactly...he's using more
> > 'spensive hardware than I am, but scale it down a bit and it's *exactly* what I'm
> > seeing.  I'm also seeing 98% RAM usage and even swapping on my web
> > server....which has 1GB of RAM and should NEVER need to swap.  Ever.  Even with
> > lots of PHP usage...the old server with 256MB ram never swapped.
> >
> > So...given this and ya'll's advice...should I drop back to 2.2.19 and forget 2.4
> > altogether, or should I disable swap, compile himem into my 2.4.7 kernel, and see
> > if it fixes it?
> >
> > jenn
> >
> > <originally posted on lists.mysql.com by  Jeremy Zawodny >
> > -------------------------------------------------
> > We had a couple of Linux 2.2.19 servers with 2 CPUs (P3-850) and 1GB
> > of RAM.  They were solid and they flew.  The never swapped.  They had
> > well tuned my.cnf files.  But we needed large files and ReiserFS, so
> > we upgraded to 2.4.9.
> >
> > After the upgrade, IO performance on the master server began to suck
> > at backup/snapshot time.  In fact, Linux claimed that it was running
> > out of RAM.  So we added RAM (a 2nd gigabyte).  The problem didn't go
> > away.
> >
> > So I poked around and watched it really closely for a couple days.
> > What I found is that the server would run pretty well until I
> > initiated some other type of intensive IO in addition to MySQL.  For
> > example, I'd copy a 3GB data file from the main RAID-5 data volume to
> > /tmp.  As it copied, Linux decided to swap out most of mysqld (which
> > is in the neighborhood of 400MB) to use it for a disk buffer.  This
> > not only killed MySQL performance, because it had to swap those pages
> > back into physical memory as soon as they were needed, it also made
> > the copy take much longer than it should have.
> >
> > I could go on, but the pattern was clear.  Linux tried to use as much
> > memory as it could for disk buffering--even if it meant "stealing" RAM
> > from mysqld.
> >
> > The solution?  I disabled swap and it's been humming along just fine
> > ever since.  After all, with 2GB of RAM and a good understanding of
> > what our RAM requirements really are, the system should never swap.
> > -----------------------------------------------------
> >
> > Joseph Andrew Knapka wrote:
> >
> > > Jeff Hubbs wrote:
> > > >
> > > > Jenn -
> > > >
> > > > If it were me - and please understand that I have very, very little
> > > > experience with MySQL - but I'd try going back to a 2.2 kernel first.
> > >
> > > Seconded (including the disclaimer :-). There seem to be a number of
> > > ways the 2.4 kernels can snafu WRT memory management. When your load
> > > goes up, what process(es) are using all that CPU? I'll place a small
> > > wager on kswapd.
> > >
> > > Cheers,
> > >
> > > --
> > > # Joe Knapka
> > > # "You know how many remote castles there are along the
> > > #  gorges? You can't MOVE for remote castles!" - Lu Tze re. Uberwald
> > > # Linux MM docs:
> > > http://home.earthlink.net/~jknapka/linux-mm/vmoutline.html
> > > --
> > > To unsubscribe: mail majordomo at ale.org with "unsubscribe ale" in message body.

--
To unsubscribe: mail majordomo at ale.org with "unsubscribe ale" in message body.