Change “the Change”?

I’ve made up a little story.

The story is fictious. Characters in the story are fictious. Any resemblance to any existing persons are purely in the reader’s mind. Any resemblance with your company’s processes are mere coincidence.

Change?

Is it the process or is it communication?

Characters:

  • George: a very helpful infrastructure project manager located somewhere in Europe
  • Francis: an application manager located somewhere else in Europe
  • Olaf: an even more helpful datacenter operator geographically close to George
  • Dakshi: a very talented cloud operations engineer in/from India
  • Hans: a knowledgable senior cloud technician, European
  • Bob: the senior architect for infrastructure engineering (US)
  • Eveyln: the infrastructure engineering project manager (US, close to Bob)
  • cloud-team.operations@company.com: a support mailbox

On Feb, 4th Francis writes:

Hi Hans, Dakshi,

I currently have an issue you’ll probably able to solve easily.

Our company has signed a deal with a European government’s ministry to host an instance of our social enterprise application. For obvious reasons, that instance needs to be hosted in the customer’s country (not in our cloud)

As this project has a very tight schedule, and to be sure we can deliver in time, I need a dump of a few virtual machines from our existing V2 implementation so we can import them on our local VMWare infrastructure.

Machines needed are:

  • webserver01
  • dbserver01
  • accesssrv01
  • worker01
  • viewer01

I’m not sure what would be the best and fastest way to transfer them. A network transfer may be too long so maybe we can use a hard drive sent to us by UPS as we did for earlier transfers across the ocean.

I hope we can do this very quickly as the schedule is very tight, and I’m a bit late due to the time spent trying to fix the current V1 issues.

Best regards,

Francis


Let us briefly elaborate on the geographical circumstances of this request:

  • The datacenter running the V1 and V2 versions of the application under question is located in Europe
  • The customer is located in a different European country from the datacenter
  • The DC operations team is located in India

Dakshi is a very talented and quick operations guy, who picks up the request just 5 minutes later and responds willingly to export the requested VMs but being in need of a contact geographically close to the cloud DC to x-plug (and deliver) a USB harddisk . For this he redirects to Hans. Hans is known to know virtually everything which in certain organizations sometime means that Hans would be the one also doing everything – not so here: Hans redirects to George. George is a project manager located close to and responsible for activities in the DC in question, hence supposed to be best choice to coordinate the HDD x-plug and delivery process.

* FULLSTOP *


On Feb, 13th Francis writes:

Hi George,

I’m Francis, in charge of our social enterprise application. We’ve been working with our cloud team on the company’s “DriveSocial” project (V1 and V2). I’m writing to you on behalf of Hans. Our company has signed a deal with a local government’s ministry to host an instance of our social enterprise application. For obvious reasons, that instance needs to be hosted in the customer’s country.

As this project has a very tight schedule, and to be sure we can deliver in time, I need a dump of a few virtual machines from our application’s V2 implementation so we can import them on our local VMWare infrastructure.

Machines are:

  • webserver01
  • dbserver01
  • accesssrv01
  • worker01
  • viewer01

Can you please have them dumped and sent on a hard drive to:

DC1GER – Markus Verdinger, Sackstrasse 240, 99999 Praiss, Germany

The latest snapshot of the virtual machines, even though from last week would be perfect.

Thank you very much in advance,

Francis


A few minutes later the same day, Hans (in a supportive manner) shares a diagram of the V2 implementation with everybody; included is a detailed directive how to discover the right VMs within the “complicated jungle” of virtual organizations and virtual appliances [2(!) ORGs; 5(!) vApps].

George, the project manager and always eager to exactly specify the right activities, now kicks off a conversation about which server to plug the harddisk into – which ultimately involves Bob, the infrastructure architect.

On Feb, 13th Bob writes clarifyingly:

George,

They are looking for physical jump server in the EU1DC rack 0815.  On the rack elevation diagram, you are looking for CWRWTs001 in RU99.

Looks like these UCSs have the old US names on them.  We need to review the names for the UCSs in Rack 0815 and make the corrections.

@Evelyn, please set up meeting to talk about and correct this issue.  We should double check EU2DC too.

Thanks. Bob


Evelyn confirms instantly. Case closed for today.

  • Total #mail: 11
  • Total #mail today: 8

The next day passes by with Georges proactive attempts to get the right HDD into the right server. He’s supported by Olaf who gives regular live reports from the actual situation in the DC (i.e. precisely explaning which HDD is plugged into which server and asking for admittance to x-plug).

On Feb, 15th Geroge writes:

Hans,

Can you let us know who will be baring the costs for sending this HDD over to Praiss, Germany?

cheers G


Hans in all honesty responds not to know this and directs back to Francis “coram publico” assuming, that the cost question will for sure be no issue here as it all is about a very important customer and a very urgent request.

For the first time in our story things become “a bit complicated” here, as George has to ask admittance to book costs for x-plugging a HDD, putting it into an envelope and filing it with UPS. The representative of the respective delivery organization for this customer kicks in and asks whether sales has a budget (and PO) for this effort. George suggests to use a cloud development PO for the sake of simplicity, Hans again suggests – for the sake of even more simplicity – to invoice the department responsible for the social enterprise application directly (as this will be the benefitting party in this whole story anyway).

On Feb 15th, late in the evening (after having patiently watched the emails on the matter so far), Francis writes:

Jesus Christ…

Please invoice our department

In any case this is still our comapny’s money – so what!

Best regards

Francis

… and George kicks off the task of x-plugging and exporting with Dakshi by — — — asking Hans to open a Change Ticket in the Change Management system for this activity!

… which now – for the first time – leaves Hans standing in complete awe and totally leaking to understand his involveness in the case (especially as Hans is a future oriented agile minded technician who disbelieves in the flexibility of traditional change processes based on ITIL; ITIL was great some 10 years ago – Hans believes, that this is the cloud era which asks for more rapid process definitions and especially executions – but that’s a different story …)


This is the last we hear from our story’s heros before the weekend begins …

  • Time elapsed: 11 days
  • Total #mail: 34
  • Total #mail today: 17

On the following Monday, Hans and George spend some time (writtenly and verbously) to clarify how to rightly kick off such an activity and George (who asked for a change ticket just a few days ago) suggests, that the right way would be to engage with the cloud operations team in India directly. … Wait. … Directly with India. … I need to scroll up to the beginning here … Wasn’t that what Francis … did …

On Feb, 18th late in the evening, George writes:

Hi Dakshi,

could you please give me a feedback of the status regarding the transfer of the desired files?

Thanks and Best Regards

George

Dakshi now asks his colleague Abhu to act on the request (giving information on which VMs needed), Abhu asks Prahti to start the export. And Prahti re-queries the right ORG and vApps. Wait. … I need to scroll up again … Didn’t Hans … provide this very same information … directly to … Dakshi; well – it’s only copy-paste anyway and Hans is known to have an everlasting rapidly-searchable email-archive. Info delivered. Now to Prahti.

It’s Monday, Feb 18th.

  • Time elapsed: 14 days
  • Total #mail: 41
  • Total #mail today: 7

* FULLSTOP *


On Feb, 25th, Francis writes:

Hi Hans,

Sorry to disturb you but do you know if the virtual machines hard drive has been sent to our DC? I just had a call with them and apparently they had nothing delivered yet.

Best regards,

Francis

Hans in all honesty responds not to have any new status and Francis redirects his question to George. George in turn asks Dakshi. Dakshi confirms to have started the export but reports, that he had issues with some of the VMs (their export causing high load on the servers, which is why he did (does) not want to continue in order not to disturb productive environments). This leads Francis to ask the blunt question why it wouldn’t be possible to just use latest backups of the very same VMs.

cloud-team.operations@company.com thereafter confirms that backups can be restored to a seperate location and the export can then be started. Wait. … ? … Let’s contemplate briefly on why a restore … … …

On Feb, 26th, Prahti writes:

Hi Francis,

We have exported following machines to the attached hard drive:

  • webserver01
  • dbserver01
  • accesssrv01
  • worker01
  • viewer01
  • loadbalancer01

Prahti


  • Time elapsed: 22 days
  • Total #mail: 49
  • Total #mail today: 2

* FULLSTOP *


On Mar, 15th, Francis writes:

Hello Geroge and Hans,

I just had our DC people on the phone, and they’re still waiting for the VMs. It’s been one month now. Do you have a status on this please?

Best regards,

Francis

George, some hours later, replies:

Hi Hans, Hi Francis,

sorry for the delay, but there was no change ticket in place.

But in this case the colleagues will do an unconventional approach. However, the arrival of the disk at the agreed shipping address is expected during next week only.

Best Regards

Geroge

* FULLSTOP *

On Mar, 27th, Francis writes:

Hi George,

I’ve been contacted by the DC, and they’re still waiting for your shipment to be delivered. Do you have any update on this please?

Best regards,


After this we lose track and the emails trickle away.

  • Time elapsed: 51 days
  • Total #mail: 53
  • Total #mail today: 1

* FULLSTOP *



On June, 6th, Francis writes:

Good afternoon,

Our DC has mounted the machines and it appears you did not provided the right machines. Actually, you sent us virtual machines that belongs to another client and that are not even running WindowsServer.

Beside the fact that it is unacceptable to get the data from another client, the delay introduced by this makes our company at risk with that very important client (government ministry for employment).

For the record, virtual machines needed to be cloned, copied onto a USB drive and sent to our DC are:

  • webserver01
  • dbserver01
  • accesssrv01
  • worker01
  • viewer01
  • loadbalancer01

The client’s DC address is: DC1GER – Markus Verdinger, Sackstrasse 240, 99999 Praiss, Germany

Please, fix ASAP.

Regards,

Rewind to start …


* FULLSTOP *

Questions:

  • Where is the leak of communication?
  • Where is the process leaking clarity?
  • What could have been done by whom to improve the result of this operation/request?
  • Who should consider their job attitude?
  • How would ITIL support such a case?
  • Would this be possible to happen in your company? Why not?

Leave a Reply

Your email address will not be published. Required fields are marked *