Agile cultural shift proves critical in ANZ’s automation project

‘Automation’ and ‘agile’ are two words at the start of most CIOs’ dictionary, and many would argue that you can’t have the first without the second. But how do you implement both with the same team at the same time, just after they’ve deployed a major transformation project?

That was the problem facing Anthony Watson, general manager, technology operations for New Zealand at ANZ Bank. His group has around 400 staff in New Zealand and offshore and its responsibilities include running the data centres, all compute, storage, networks, and 24/7 operations, as well as infrastructure programme and project delivery.

“Our network is incredibly large and complex, and it’s critical because if we get it wrong people end up leaving trolleys in supermarkets and can’t pay for their fuel,” Watson says. “If you look at our investments from an automation perspective, we’ve made some great progress in parts of our organisation. But there was a component that was harder to shift: the network side of things. We’re not yet a fully virtualised network architecture; there were some impediments there and we felt that automation was really a key asset or capability that we needed to achieve. A lot of the organisation had moved to agile delivery, but the network team hadn’t. They were still very much ‘Log a ticket and we’re going to get to it.’”

As a result, the networking function had become a bottleneck in the organisation, interrupting the networking flow, with other teams ending up in a ‘wait state’ which weren’t automated.

“Wait state refers to when a program or project team are unable to make progress on their delivery goals because they are in a line waiting for a service to be delivered, or infrastructure to be provisioned to enable them to keep working. An example is if your project needs a firewall rule change — you submit the change to the team that manages this  and they have an SLA of “X” to provision this, so you must wait. Our goal through automation is to remove as many ‘wait states’ for service/infrastructure provision as we can for these project teams,” Watson says.

The network team had just spent two years working on a major upgrade involving a move to Spark’s Optical Transport Network, which Watson describes as “telco-grade”. It services the two data centres that the ANZ Bank has in New Zealand. These handle most of the local traffic, but the bank has also adopted a multi-cloud strategy, working with Amazon Web Services and Google as required.

Watson says the network team’s goal was to automate 30 per cent of network services, using Red Hat’s Ansible Automation Platform. But rather than get the vendor, or an integrator, to implement the system and hand it over to them, Watson says ANZ looked for a way to build the technical and cultural skills in house.

ANZ uses a residency lab programme to bring the new skills in house

After hearing a bunch of recommendations from banks, airlines and major car manufacturers at Red Hat’s Summit in Boston in 2019, Watson opted to put the team through a residency at Red Hat’s Innovation Lab. “We invested money in professional services for the labs engagement, over and above the basic licenses and subscription that we paid for the technology. The key thing is we were going to implement the technology anyway. It was part of our roadmap. It had ticked all the right technical boxes,” he says.

Red Hat head of innovation labs and architecture practice Arash Dadras says ANZ Bank is its first customer in New Zealand for the residency, which took the form of working intensely with a cross-functional team of eight people located in Auckland and Wellington. During the first six weeks they learned and practiced the technical application of Red Hat’s Automation technology, software engineering practices as well as how to work according to agile methodology and lean methodology.

Cultural alignment within the team was critical to it being able to successfully achieve its automation goal. At the start of the lab, a few grievances were shared as members identified what was frustrating about the old processes but from which a culture had been created. They adopted the name RoboOp and had T-shirts made with ANZ logos to express team pride.

Dadras refers to the process of coming together as a cross-functional team and adapting to a new way of working as ‘storming, warming and performing’.

Moving from the labs back to the data centre

What began as weekly sprints in the lab has transformed into fortnightly sprints, with the team breaking down tasks into manageable sized chunks of work. While improving team capability and creating a network automation squad were the key goals, the team also automated the DNS provisioning process with over 90% efficiency gain.

“The great thing about the way the team works now is that they are self-directed,” says ANZ’s Watson. “30 per cent automation is the goal, but that’s not a cap. Really, they are defining it as they go. They engage the consumers of the services as well, by asking what’s important to them.”

The benefits of automation and agile on display during the pandemic

Both the two-year network refresh and the focus on automation have been beneficial during the COVID-19 pandemic. Over the pandemic lockdown, 90 per cent of the 900-strong contact centre staff worked from home and the network was able to scale up and down to meet spikes in customer demand. The spikes tended to follow the prime minister’s daily 1pm press conference, when policies relating to mortgages and business loans might be announced.

Post-lockdown, Watson’s entire team, in common with most IT departments in New Zealand, is adapting to the new hybrid working environment with people opting for a mix of locations — at home or at the office.

Watson says the next conversation is how to expand the process to incorporate more services. Automation has been achieved around server builds, clients builds and a number of other areas, but there is more to do. “We want to get that point where we say: ‘How do we do this more cross functionally?’.”