[ale] Security breach on kernel.org
Michael H. Warfield
mhw at WittsEnd.com
Thu Sep 1 15:41:40 EDT 2011
On Thu, 2011-09-01 at 14:13 -0400, Bob Toxen wrote:
> Mike,
> Why DO the developers need to change their private SSH keys? I hope
> they were not stored on the compromised systems.
I would say it's highly unlikely that many, if any, private keys of
developers have to be changed, but possible. There might be some keys
used in mirroring that might. People shouldn't have been using those
servers as their home servers, merely connecting to them remotely for
git. Some people might have had private keys on one machine to
communicate with other machines in the cluster (which could account for
the spread of the compromise). For that, people really REALLY should be
using auth forwarding, instead of private keys on a remote server, but
it is a too common practice, unfortunately. Only other concern would be
if an attack were propagated back to one of their servers by a trojaned
file. Again, unlikely.
A more likely scenario was described in another message. The attackers
could have added additional authorized_keys file, necessitating the
removal and rebuilding of those files. That's a very common attack
scenario that's consistent with the statements that have been made. I
don't know the exact functionality of Hera or how other servers were
subsequently compromised through Hera (or even if they were compromised
through Hera).
Another concern is that the suspicion is high that it was compromised
user credentials that were used in the initial break-in against Hera.
Here, it could have been a developer's machine that was initially
compromised. Obviously, they would have to change their keys and they
would have had to be positively identified as the source of the attack.
That may not be clear cut and they may have a set of potential sources
which are having to update their keys. If it was compromised user
credentials, then the escalation could also have been through sudo or a
loop-through to root on local host. They're just saying the escalation
was unknown at this time.
> In my book I discuss special procedures to cover very high security
> situations which kernel.org certainly qualifies as. One part is to keep
> the private ssh keys (and other crypto keys) OFFline to avoid being
> compromised from the Internet.
That could be a little difficult with ssh private keys, since their used
for remote access, unless you mean by "off-line" being on USB keys and
smart cards. Having certain critical communications keys on isolated
machines which can only be logged into locally but then can still access
the Internet and using auth forwarding out of that for server-to-server
access makes more sense in that regard. If you have git access to the
cluster through ssh using hardened RSA or DSA keys, you'd have to have
the private keys on-line on your development any time doing a clone or
pull or update or a host of other git activities.
Certain "data at rest" type keys and things like CA private keys make a
lot of sense to be kept off-line and yet we've got yet another CA
compromise that's making a much of things... Sigh...
Regards,
Mike
> Bob Toxen
> http://www.verysecurelinux.com [Network&Linux security consulting]
> http://www.realworldlinuxsecurity.com [My book:"Real World Linux Security 2/e"]
> Quality Linux & UNIX security and SysAdmin & software consulting since 1990.
> Quality spam and virus filters.
>
> On Thu, Sep 01, 2011 at 09:21:06AM -0400, Michael H. Warfield wrote:
> > On Thu, 2011-09-01 at 08:42 -0400, Jim Kinney wrote:
> > > Major bad news. They host loads of code.
> >
> > Read the articles. Several machines were compromised but not all.
> > Compromised machines have been taken off line for diagnostics and
> > reinstallation. A number of developers (close to 500) are having to
> > change their ssh keys, which sucks.
> >
> > Bad but highly unlikely to have any impact on the source code thanks to
> > the nature of git and the highly distributed development model along
> > with cryptographically secure hashes and history on every single file.
> > They'd need a time machine to go back and poke changes into past sources
> > and change sets and they're need a transporter to get to all the
> > thousands of machines hosting git repos at developer sites for the
> > development their development. They're validating the the change sets
> > and hashes but it's unlikely to contain anything and it's unlikely the
> > sources have been contaminated. Unexpected changes should show up
> > rapidly to the subsystem maintainers as unexpected conflicts or
> > validation checks or unapproved changes sets.
> >
> > http://www.linux.com/news/featured-blogs/171-jonathan-corbet/491001-the-cracking-of-kernelorg
> >
> > He points out that the sources are distributed from kernel.org but are
> > developed on and hosted all over the world.
> >
> > Regards,
> > Mike
> >
> > > On Sep 1, 2011 8:14 AM, "Watson, Keith" <krwatson at cc.gatech.edu> wrote:
> > > > Security breach on kernel.org
> > > > https://www.kernel.org/
> > > >
> > > > Earlier this month, a number of servers in the kernel.org infrastructure
> > > were compromised. We discovered this August 28th. While we currently believe
> > > that the source code repositories were unaffected, we are in the process of
> > > verifying this and taking steps to enhance security across the
> > > kernel.orginfrastructure.
> > > >
> > > >
> > > > There is more information on their home page.
> > > >
> > > > keith
> > > >
> > > > --
> > > >
> > > > Keith R. Watson Georgia Institute of Technology
> > > > IT Support professional Lead College of Computing
> > > > keith.watson at cc.gatech.edu 801 Atlantic Drive NW
> > > > (404) 385-7401 Atlanta, GA 30332-0280
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Ale mailing list
> > > > Ale at ale.org
> > > > http://mail.ale.org/mailman/listinfo/ale
> > > > See JOBS, ANNOUNCE and SCHOOLS lists at
> > > > http://mail.ale.org/mailman/listinfo
> > >
> > > _______________________________________________
> > > Ale mailing list
> > > Ale at ale.org
> > > http://mail.ale.org/mailman/listinfo/ale
> > > See JOBS, ANNOUNCE and SCHOOLS lists at
> > > http://mail.ale.org/mailman/listinfo
> >
> > --
> > Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
> > /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
> > NIC whois: MHW9 | An optimist believes we live in the best of all
> > PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
>
>
>
> > _______________________________________________
> > Ale mailing list
> > Ale at ale.org
> > http://mail.ale.org/mailman/listinfo/ale
> > See JOBS, ANNOUNCE and SCHOOLS lists at
> > http://mail.ale.org/mailman/listinfo
>
>
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
Url : http://mail.ale.org/pipermail/ale/attachments/20110901/521657c9/attachment-0001.bin
More information about the Ale
mailing list