Thursday, October 15, 2009

the dangers of outsourcing

i've been in my new job for 3.5 months. Been to much data centres, and find that the ones that give somewhat bigger problems are managed by internal outsourced teams. 2 cases that i see:

- a well known household brand locally until it was absorbed into another financial institution in 1998. A not-so critical clustered logical partition failed with data corruption. The joke - sysadmin specialists requesting software support to recover a crashed OS, when they support their own software requests. Next joke - the backup was undated, and there were multiple ones. Next joke - the boot disk started to fail 4 months ago, and was only partially mirrored.

oh yes, the clustered system failed to work. High availability... indeed.

- a tyre brand. Server is in a particular off-site location, principally managed by a Blue team in China, co-managed by a local team. The local member asked for technical support insisting we go on site to look at a server that is down. Yes, down (lightning strikes). So we went on site but nobody was at the location. He was sitting nicely at Changi, happily acting as a middleman between me and the Chinese engineer using his phone. He was clueless as to what happened, saying "i'm not sure, i don't know" and kept asking his Chinese counterpart to call me all the time. The Chinese guy thought i'm part of his Singapore local team when i'm from maintenance, asking me to do things that i'm not authorised to do.

why the hell these sysadmin are employed for? What are the data protection policies for? If they are doing a lousy job for too long, it's time for the managers to take action and tell them to go.

on a more positive note, i have praises for...

- a well known electronics company based in Japan, you'd always pay a premium for their products. The server rooms are well furnished and managed. The staff are always seen using only products from the company, such as their monitors and laptops. When there's a disk failure, the sysadmin will do the necessary to move out all the data from the disk without any intervention on our part. The boot disks are always properly mirrored with another.

No comments: