By David Spark
“Networks are expensive, get a bad rap for reliability (even when they are meeting their technical specifications), and at the end of the day few businesses could operate without them,” said Matthew H. Podowitz, (@mpodowitz), Senior Director, Operations and Technology Advisory Services, for Pine Hill Group.
Even when networks are so critical for business operations, the money that’s spent on them is always seen as a sunk cost. To get some advice on ways to reduce the operational costs of networking, we reached out to over one hundred industry experts and asked, “What don’t people understand—but should—about the costs of operating a network?"
Here’s a collection of our favorite and most useful advice. We invite you to include your tips and opinions in the comments at the end of this article.
1: It costs less to ask questions first
“The biggest mistake I constantly see is a business making hasty, uninformed decisions about investments and network infrastructure that business owners think they need,” said Jacob Lang (@backbaynetworks), an IT Consultant for Back Bay Networks IT Consulting.
There’s an easy way to avoid this problem, recommends Lang, and that’s to get advice.
“Free advice from a professional is the absolute best way to ensure you don't overspend. IT consulting companies are neutral, meaning no matter what you choose we get paid,” said Lang. “We want you to have all the facts and understand the costs associated with each potential decision.”
Asking questions also holds true when you’re in the middle of a business critical situation, explained Susan Tran, blogger and Senior Principal Product Marketing Manager, Infrastructure Management Solutions for CA Technologies.
During this period of high-stress, precious time is wasted trying to disprove a claim that the network is the cause for poor performance.
“This is a costly approach to network management,” said Tran. “It’s a reactive and emotionally-driven, as opposed to data-driven.”
Tran advises to ask questions that get to the root of the problem, such as:
- Which applications and hosts are consuming the most bandwidth?
- Are critical applications prioritized over less important applications?
- When do these applications consume the most bandwidth?
2: The applications, stupid
In the early 1990s, James Carville, Bill Clinton’s Presidential strategist, hung up a sign at campaign headquarters that said, “The economy, stupid,” to remind everyone to focus and not be distracted by other issues.
A variation of that sign should be hung up in every IT department.
“The network is there to support applications servicing users,” said Ian Rae (@ianrae), CEO for CloudOps. “Building in application networking operations tooling and practices has the fastest ROI and makes sure that the network can meet the needs of the business.”
“Workers that have time sensitive activities to complete are at the mercy of the network powering their applications,” said Everett Dolgner (@nerdsped), blogger and Director of Storage and Replication Product Management for Silver Peak.
You’ll be setting yourself up for failure if there isn’t a direct correlation between network spending and applications. Improvements that aren’t connected to application performance will be viewed as being outside the company’s goals, generating additional costs, and a cause for delay, noted Razvan Ungureanu, CTO Caribbean and Central America at Digicel Group.
3: Senior employees should spend less time firefighting, and more time planning
“When the L3 engineer gets regularly pulled in to do routine firefighting such as fixing network issues related to outages, they lose a lot of time to work on strategic projects that drive business innovation. This opportunity cost, along with the inordinate amounts of time they spend with telcos to chase down circuit issues, must also be factored into the costs of operating a network,” Raju Chekuri (@Netenrich), CEO of Netenrich.
“It’s easy to lose focus on the things your IT department should be doing to add more value, things like innovation and transformation. You want your people to spend more time making the business better and less time simply ensuring the business runs as usual,” said Stefaan Hinderyckx (@DiDataInsights), Network Integration and Security Director at Dimension Data.
4: As number of support contracts goes up, so does cost of management
“Most IT environments today have infrastructures containing multiple technologies from multiple vendors, supported by multiple services contracts. Managing such a complex environment can get ‘messy’ when too much time and money is wasted on simple, run-of-the-mill tasks,” said Dimension Data’s Hinderyckx.
“Contract management introduces new costs such as calls to vendors and help desks re-explaining your specific environment,” continued Hinderyckx. “Your engineers should be focusing their time on business growth. This problem can be easily solved with a single ICT provider managing all vendors and support contracts on your behalf.”
5: Beware of the cost of vendor lock-in
“Choosing closed technologies can lead to huge operational expenses which can eat up capital expenses very quickly,” explained ADARA Networks’ Subrmaniam.
“With the increased use of mobile devices, cloud computing and big data, enterprises need to move towards software-defined networks that are built on open-standards,” argued Kash Shaikh (@KashShaikh), Global Marketing Leader, <HP Networking. “This means that organizations are not locked into a single vendor when it comes to purchasing networking equipment, which can be a tremendous cost-saver, up to 35 percent.”
6: Plan for the inverse cost curve between hardware and management of hardware
“The more complex [networks] become the more they cost to operate and maintain,” said Eric Ingram, Adjunct Instructor at APT College, LLC.
Every year we’re always amazed by how cheap and more capable our networks have become. What’s possible today always greatly surpasses what we could do last year, often for much less. Initial logic causes us to assume that the cost of operating the network should reduce as well. That’s usually not the case.
“The cost of programmers and IT professionals has continued to increase almost directly proportionally with the capabilities of the systems and inversely proportionally to the cost of those systems,” said Ingram.
7: Invest in people
“Organizations often treat their hardware assets better than the technical staff,” noted Michele Chubirka (@MrsYisWhy), security professional and host of the “Healthy Paranoia” and “Packet Pushers” podcasts. “An unhappy, disengaged staff is death to an organization. This dissatisfaction impacts efficiency and the bottom line of an enterprise, but organizations continue to cut professional development spending or fail to compensate appropriately to retain talent.”
Bill Schrier (@BillSchrier), blogger and former CTO for the City of Seattle, notes that while employees are to some extent motivated by money, “they also want recognition for their work and they want to know that their work is important, that it makes a difference to the world at large.”
Two years ago, I attended the Tech Career Expo where I asked attendees, all tech professionals looking for jobs, what their top motivator was. Of the 40+ people I spoke to, only two were motivated by money. The rest, agreed with Schrier, in that they wanted to believe in what they were working on.
8: Hire good negotiators
“There are many business relationships required to develop and maintain an advanced enterprise network. The most effective large-scale network solutions typically feature experts who truly understand how to identify and negotiate the best pricing from manufactures and ISPs. Without the right talent shepherding equipment procurement, a company will find that they are paying much more than market,” said Isaac Conway (@Latisys), Director, Network Engineering for Latisys.
9: Security programs must evolve
“Your network security is only as good as the smartest person that has reviewed its architecture and security standards,” said Don Druckenbrodt (@ZixCorp), VP, Services Policy and Compliance for ZixCorp.
“Companies should move away from the traditional static security setup and work to be more nimble and dynamic,” said Charles Tendell (@CharlesTendell), Cyber Security Expert and Founder of Azorian Cyber Security. “Threats continuously evolve; so should security programs.”
10: Diverse redundancy
Having a redundant network is obvious, but keeping that redundancy diverse with no shared services is not.
When it comes to building network redundancy, Kevin Gilchrist, Product Manager at Comodo advises to “maintain transit diversity with no less than two independent providers at each location, with no shared infrastructure between the providers…plus, verify that one carrier is not of the same fiber you already have.”
To maintain continuous service even with regional outages, Gilchrist also recommends you ensure that each location is geographically fault-tolerant with its neighboring locations.
Once redundancy is in place, Laurent Vaills, Solution Architect for Bonitasoft strongly advises you set up and test some crash recovery scenarios. While this can take quite a few man-hours, monitoring the results will allow you to define the metrics that will trigger the alerts.
11: Monitor your network
“Even with the best network, where traffic is rerouted when a link goes down, companies are vulnerable if no one knows that a link is down and gets it repaired,” said Abdul Jaludi, CEO of TAG-MC. “Without monitoring, companies that spend more for redundant feeds with automated switching will be just as vulnerable as those who have single links without any redundancy.”
“The key is proactively monitoring a network’s machine data to identify anomalous activity in real-time,” said Joan Pepin, Director of Security for Sumo Logic. “It can mean the difference between a breach and a mega-breach.”
“Firewall and antivirus are like locks on the door of your house,” said Michael Fimin, (@netwrix), CEO of Netwrix. “If the locks are broken, you have to understand what to do next and fight the insiders once they are inside.”
12: Avoid solution creep
You probably read the last tip and thought to yourself, “Duh, I know that.” The question is, how much monitoring are you doing and is it spinning out of control?
According to a recent study by Enterprise Management Associations (EMA), network engineers and managers use an enormous suite of disparate tools for monitoring and troubleshooting. The overwhelming majority use 4-10 solutions, with some organizations relying on as many as 25 different solutions.
No organization plans on using 10 different solutions from the start, noted Jay Botelho (@JayBotelho), Director of Product Management at WildPackets. It’s a situation of “solution creep,” and it can be costly in terms of purchasing, learning, maintaining, and integrating.
To save money Botelho recommends that “the network operations team develop the requirements for network visibility and troubleshooting at the same time that major upgrades or new networks are being specified.”
One customer of CA Technologies leveraged a unified monitoring platform and reduced time managing multiple tools by more than 25%, said Umair Khan (@UmairMoheet), Product Marketing Manager, Infrastructure Management Solutions at CA Technologies.
13: Focus on how you handle downtime
“While IT network budgets usually include the equipment and personnel to manage infrastructure, they often fail to consider the cost if the network doesn’t work,” said Daryle DeBalski (@FlukeNetVisual), Vice President and General Manager of Fluke Networks visual business unit.
“The reason companies pay so much for networks is not to ensure they are up 99.8% of the time, but to make sure that the 0.2% they are not available doesn’t bring the business to a standstill,” said Pine Hill Group’s Podowitz.
14: Get out of the network business, if it’s not your business
“Many companies are in the network business and don’t even realize it,” said Podowitz who noted that “while virtually every company relies on a network to make the business function, very few companies’ competitive advantage is based on the companies’ networks.”
If that describes your company, why maintain so many physical technical assets and train staff?
“Many companies would benefit from getting out of the network business and instead paying a third party services provider to take care of the network,” said Podowitz.
15: Understand revenue opportunities from the network
“Many consider CAPEX and OPEX as the costs of operating a network but they should understand that there is another critical cost factor; which is the loss or gain of revenue because of the network,” said Karthi Subramaniam (@ADARAnetworks), Chief Software Architect at ADARA Networks. “One should understand that real ROI should be measured in terms of increased transactions, customer satisfaction (faster response), and then cost savings.”
“IT should be treated as a fluid business unit, just as any other function of the company,” agreed Dan Tully, EVP at Conduit Systems.
“Network operations should always be aligned with business strategy, and its focus must remain on improving core business performance,” said Bruno Scap (@MaseratiGTSport), President of Galeas Consulting.
16: Don’t focus on CAPEX. Look at OPEX as well.
“Organizations can sometimes place too much emphasis on the initial capital expenditure and installation cost of setting up a network and not enough value placed on long-term, on-going costs which can affect the overall operating expenses,” said Cal Calamari, Global Solutions Lead for Enterprise Networks and Connections, Motorola Solutions. “A true total cost of ownership analysis must take into account five key operational elements: installation, provisioning, maintenance, change management, and decommissioning.”
17: Prepare for and predict growth
“The biggest costs that get overlooked or that are difficult to manage in running a modern network are in predicting the pace of business and the pace of change,” explained Sean Lutner (@attivio), Director, IT Operations for Attivio.
“If your company grows significantly faster than anticipated, it's very easy for your network costs either in terms of hardware or bandwidth to grow beyond expectation,” said David Mortman (@mortman), Chief Security Architect and Distinguished Engineer for Enstratius/DELL.
“Failing to account for ever-increasing traffic requires much more capacity and equipment, often procured on an emergency basis, which raises costs,” noted Bernard Golden (@BernardGolden), VP, Strategy for ActiveState.
“With proper capacity planning and network optimization, you can accurately plan for future requirements, which ultimately helps reduce CAPEX and OPEX,” said Bill Berutti, President Performance & Availability Business at BMC.
“Working with a vendor to implement a managed network solution allows an organization to remove the costs of increasing physical hardware and software licensing,” said David Morgan, VP of Sales Engineering at Windstream, of one possible option.
18: One petabyte of data may be cheap to buy, not to manage
“Once stored data grows to one million gigabytes (one petabyte) or more, traditional storage approaches become unsustainable because required costs to manage and protect information at that level using traditional methods are unsustainable,” Russ Kennedy (@Cleversafe), Senior Vice President of Product Strategy and Customer Solutions at Cleversafe. “While cost per gigabyte of raw storage is generally becoming cheaper annually, operating expenditures associated with data growth are not.”
Kennedy points to obvious costs such as data center floor space, power, cooling, networking, and additional staff members to manage the data. The greatest cost will probably be maintaining three copies of the data to protect against data loss.
Instead, said Kennedy, “IT leaders need to find an alternative approach that can deliver protection against failures at a lower cost point than making multiple copies.”
19: Standardize don’t customize
“Customization is the enemy of repeatability,” warned Michael Bushong (@mbushong), blogger and VP of Marketing for Plexxi. “[Customizing] forces your networking team to be contextually aware… Slight differences in network architecture, configuration, and behavior can lead to dramatic differences in issue triage and resolution. Forcing your team to do a context check in these often frantic moments costs time and money.”
“Consider preempting unique requests by being more inclusive in creating architectural building blocks,” said Bushong. “Business buy-in at design time can save you lots of money in ongoing maintenance.”
20: Beware of the cost of not upgrading
“IT needs to look not just at the cost of upgrading, but the cost of not upgrading,” said Mykola Konrad (@mykolakonrad), VP and General Manager, Cloud Business Unit and Strategic Alliances at Sonus. “[For example,] new SIP and unified communications technologies are proven to drive improved employee productivity by enhancing collaboration, providing better access, and simplifying the user experience.”
Nestor Rincon (@RinconDynamic), owner of Rincon Dynamic IT Support, has a customer who refuses to upgrade their 10-year-old switches. The net result is that it takes 10 minutes for each of their 25 employees to log onto the network. That’s a lot of wasted time every day.
“Infrastructure requires care, feeding, and periodic review and refresh. Building this into standard operations will save long-term costs and potentially even your job,” said Plexxi’s Bushong.
Conclusion: The cost of making a network run smoothly
“Operating a network is like getting on a train and expecting it to drive you to your destination unimpeded and on time, all the time. We expect immediate and fault-free service, and many of us complain loudly when this doesn't happen,” said Steve Prentice (@steveprentice), Senior Writer for CloudTweaks.
“A network needs a range of diagnostic and preventative tools to watch for potential leaks, hack attacks, and errors. It needs redundancy to ensure a smooth workaround, as well as the skilled staff, both onsite and remote to be there to handle challenges the moment they occur. It also needs constant upgrades, in terms of hardware, software and people skills to keep pace with increasing demand, and with constant changes to the rulebook,” said Prentice. “What people don’t understand about the costs of operating a network but should, is that a significant percentage of operating cost and personnel time should go into preventative ‘insurance’ and education.”
For more IT cost saving tips, please also read “20 Real-World IT Cost Cutting Mistakes You Need to Avoid.”
Please add your thoughts and tips in the comments below. Thanks.