[ale] GATech mirror problems - from the admin

[Neil Bright]

Hi folks,

As the admin of the GT linux mirror, I'd like to comment on some of  
the issues people have been raising.  Hopefully, I can at least  
provide some explanation as to the behaviors people have been seeing.

General FTP problems -
I'm using vsftpd with PASV enabled (the default behavior).  Please  
ensure that you use ftp clients that understand this portion of the  
protocol.

DNS issues -
These reports seem to stem from the use of the ftp- 
linux.cc.gatech.edu name.  GTLib is no longer housed within the  
College of Computing, but rather from our central campus IT  
organization - the Office of Information Technology.  The CoC  
maintains their own DNS infrastructure and I can't vouch for it's  
reliability.  The *.gtlib.gatech.edu names use our central campus DNS  
infrastructure.  Another portion of my responsibilities at GT is  
campus hostmaster, so I'm a bit more comfortable making guarantees  
about these names.  If you are having problems with them, please let  
me know.  Output from tools like dig and the contents of resolv.conf  
will be especially useful in these situations.  ;)

General performance problems -
We've been getting pretty hammered lately...  We're on the Fedora YUM  
lists, we host debian and ubuntu, we're part of rsync.us.gentoo.org,  
we host Suse, Mandrake and Mozilla.  The There's an architecture  
diagram here [1] for those interested.  A couple of the switches have  
been rearranged since I last updated the graphic, but the important  
parts are correct.  Of course, this means that the MRTG graphs are  
incorrect now....  *sigh*  However, there is also a Ganglia instance  
at [2].  In addition to the usual stuff, I also have additional  
gmetrics tracking the number of active apache, vsftpd and rsync  
connections.  There are also some bits in there related to NFS as well.

For the last while, the default 256 connections on all three of the  
apache front end machines have been in use.  Likely, the performance  
issues people have been seeing lately are latency issues while  
waiting to establish a connection.  (At least, this is my current  
theory, feel free to shoot holes.  :)  Many of these connections have  
been coming from the dreaded "download accelerators", often times  
with 20+ connections coming from the same IP address.  I understand  
that NAT on the other end of the pipe will look like this, but I am  
unaware of a programatic way to distinguish between NAT and abuse.

For now I've increased the MaxClients setting, but my current long  
term thinking is to employ some sort of per-address connection  
limiting.  If anybody has suggestions on how to do this I would love  
to hear from you.  I have RHEL4 with the provided apache, vsftpd and  
rsyncd on the hosts.  All four hosts are attached to the same Cisco  
2970 with a Cisco 6509 (Sup2 & SFM) router running IOS 12.2 and Cisco  
FWSM running 2.3(4) firewall context.  The hosts could be moved to a  
4948 if that would make a difference.  (Due to topology changes, this  
will likely happen anyway.)  Donations of memory for a SunFire v20z  
would always be appreciated as well.  *grin*





[1] - http://www.gtlib.gatech.edu/gtlib.pdf
[2] - http://www.gtlib.gatech.edu/ganglia

+======================================================================= 
=+
Neil Bright (neil.bright at oit.gatech.edu)                    (404)  
385-6954
OIT - Academic and Research Technologies / Georgia Institute of  
Technology
258 Fourth Street, Rich Bldg, Rm 266 / Atlanta, GA  30332-0700