[ale] Little OT: Bad Linux Sysadmin Practices

Fri Oct 12 11:13:58 EDT 2012

So we should really be blaming the bean-counter wannabe posers and not the
_actual_ bean-counters.

got it.

Insert a little bit of "Ops-Man will be held personally responsible for
performance of subordinate team and systems" and it's amazing the resources
that can be found to do the job right. Pay bonus for team based on time
with no un-planned outages or loss of service, etc.

On Fri, Oct 12, 2012 at 10:27 AM, Lightner, Jeff <JLightner at water.com>wrote:

> After the "missing" backup issue they fully embraced the original plan
> they'd previously nixed.
>
> SLAs weren't really in the vernacular until some time later and I'll have
> to say that once the company started focusing on SLAs (even between
> departments) more decisions were made based on SLA requirements than pure
> dollar requirements.
>
> By the way in a former life I WAS a bean counter for a long time and had
> worked my way up to Financial Controller.   You are wrong to blame the bean
> counters for poor decisions like this - it is almost always operational
> managers trying to look good on paper that do this.   More than once in my
> former career I was the one pushing operations to make "smart" money
> decisions rather than "instant savings".    Many decisions made by
> operational types are made to meet this month's budget to the cost of this
> quarter's or this quarter's to the cost of this year's or this year's to
> long term stability.
>
> It is even worse in publically traded companies where everything is judged
> on today's stock price rather than underlying company value.    There's a
> reason so many lost money in the Enron debacle despite the fact that what
> they were publishing in their required financial reports (especially the
> notes) would have let any person with a little "bean counter" training know
> how badly they were setup.   The problem is nobody wants to believe you
> when you tell them things are overvalued.  I remember how badly Warren
> Buffet was chastised by financial media pundits because he refused to by
> into the tech boom of the 90s.   The fact he had become a billionaire
> making smart decisions seemed to escape them.   When the bust finally hit
> and all the dot coms went belly up Berkshire Hathaway quietly went on about
> their business of making money based on value.   If I'd been Warren I'd
> have been using some of my billions to take out full page ads saying "I
> told you so!".  :-)
>
> For years co-workers derided me for buying company stock at a place I
> worked but when the tech bust finally came that company stock turned out to
> be my second best investment because it did have intrinsic value.   (In
> fact when they sold out the company I was annoyed because I had to take the
> capital gains tax hit as they went private on the sell so I got cash rather
> than stock as in most mergers.)
>
>
>
>
>
> -----Original Message-----
> From: ale-bounces at ale.org [mailto:ale-bounces at ale.org] On Behalf Of Jay
> Lozier
> Sent: Friday, October 12, 2012 10:03 AM
> To: ale at ale.org
> Subject: Re: [ale] Little OT: Bad Linux Sysadmin Practices
>
> On 10/12/2012 09:30 AM, Lightner, Jeff wrote:
> > I worked for a company that had a large data center in Little Rock which
> had both the telecom systems and the banking/mortgage systems (mainframe
> and open systems) for multiple telecom, banking and finance companies (at
> the time 20% of mortgages in the US went through these systems).
> >
> > On two separate occasions while doing supposedly "planned" maintenance
> they took down the whole data center accidentally.   In one of these events
> they had poorly wired one of the breaker cabinets and caused a short which
> made it fail.  No problem because they had brilliantly set up the power so
> that failures from one cabinet would shunt the load to the next one.  Oops
> - there's still a short so now we've fried that next cabinet.  No problem
> because it fails over to another one etc...  - OH WAIT....!
> >
> > It is truly gratifying when there is a major production outage that you
> as a sysadmin could NOT have prevented and do NOT have to be involved in
> resolving (at least until it comes time to power everything back up).
> >
> > At that same company a co-worker put together a very good backup policy
> on our first implementation of NetBackup but when they saw how much it was
> going to cost for all the tapes required to do the various retention levels
> management balked.   They nixed his plan and said they'd never need a
> backup more than 6 months old and "saved money" by not buying as many tapes
> and adjusting the retention policies.    8 months later when they asked for
> a backup from the first month due to a critical issue they asked why we
> didn't have it.  When we told them (and even showed to them in writing) how
> they had said they would never need more than 6 months did they accept
> responsibility for poor decision?  They did not - They instead said that
> the admins were not forceful enough in trying to convince them of the need
> for the original plan.    Sometimes you just can't win.
> >
> Did management have pointy hair? It sounds like they let the bean counters
> determine all policy whether there was overriding technical/commercial
> issue involved. I wonder what the contractual obligations were to the end
> users? The contracts probably specified minimum backups required and that
> at a minimum should have driven the backup policy. From your comments it
> sounds like PHB's ignored the contractual obligations (lawsuit for breach
> of contract)
>
> <snip>
>
>
> --
> Jay Lozier
> jslozier at gmail.com
>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>
>
>
>
> Athena(r), Created for the Cause(tm)
> Making a Difference in the Fight Against Breast Cancer
>
> ---------------------------------
> CONFIDENTIALITY NOTICE: This e-mail may contain privileged or confidential
> information and is for the sole use of the intended recipient(s). If you
> are not the intended recipient, any disclosure, copying, distribution, or
> use of the contents of this information is prohibited and may be unlawful.
> If you have received this electronic transmission in error, please reply
> immediately to the sender that you have received the message in error, and
> delete it. Thank you.
> ----------------------------------
>
>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>

-- 
-- 
James P. Kinney III
*
*Every time you stop a school, you will have to build a jail. What you gain
at one end you lose at the other. It's like feeding a dog on his own tail.
It won't fatten the dog.
- Speech 11/23/1900 Mark Twain
*
http://electjimkinney.org
http://heretothereideas.blogspot.com/
*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.ale.org/pipermail/ale/attachments/20121012/e79b6365/attachment.html 