When it comes to network uptime, the devil is in the details. We’re talking about each of these eight best practices potentially preventing one or two outages, thereby delivering thousands or even millions of dollars in savings to your business.
Just so you can get a better idea of what kind of savings we’re talking about with our goal at 99.99% uptime, here’s a more granular look at how improving network uptime per month can benefit your business:
Most companies operate at 99.9% or below. But even the smallest amount of network outages can greatly damper employee productivity. And since productivity is directly tied to your company’s revenue stream, shrinking downtime to be as small as possible should be a priority.
For the sake of quantifying the cost of network downtime, let’s pretend your company is operating at 99.9% uptime and has 500 employees. That’s pretty good, right?
Well, consider that this uptime rate means each employee is losing about 43 minutes of productivity every month, which adds up to 8 hours and 46 minutes annually (times the number of employees). In total, 4,383 employee hours are lost annually due to outages.
If you only operate at 99% uptime, your annual downtime would add up to 3 days, 15 hours and 36 minutes. The revenue lost from non-productive employees would add up to $1,533,000. The lower your uptime percentage is, the more compelling these numbers get.
As this example shows, network downtime can cost companies millions of dollars. If we can erase a few hundred hours of employee productivity downtime, we’ve just generated more revenue for your company.
Following the best practices outlined in this whitepaper will help minimize network downtime and positively contribute more to your company’s revenue — there’s no question about it. The fundamentals below, provided by EnableIP’s networking experts, can help increase your network uptime to at least the 99.99% level.
This may seem obvious, but it’s not so easy to do. If you’re still reading at this point, we’ll assume you understand the importance of your network. It’s the foundation from which you’re going to build upon. Your overall business revenue depends on your network being available day and night, and that’s only becoming more important as your company (and network) grows.
What “highly availability” really means is eradicating as many single points of failure as possible, both with equipment termination and circuits. Two circuits in each location on diverse last mile routes is a must, and the single most important thing a company can do to overcome a majority of all network issues is removing the loss of last mile service.
Building failover and failback between the primary and backup service will reduce 80% of your network issues. This is essential. Rather than looking at failover solutions as an extra expense, consider it as a return on your most important investment: your employees. Your employees are more productive when the network is up and operational, which means the company is generating more revenue.
Below is an illustration of a very typical design for high availability. We have coined this solution “Cloud Assurance.”
For understandable reasons, a great deal of emphasis tends to be placed on a circuit’s bandwidth. But when you’re trying to achieve a high level of network availability, there are other important things to consider. Cable, DSL, shared/dedicated Ethernet, WiFi, dedicated fiber, wavelengths — each of these circuit types have considerably different Mean Time to Repair (MTTR) as offered by the service level agreements (SLA).
MTTR for cable and DSL is the best effort in most cases. Business-class cable can provide for MTTR, but it’s generally the same business day. DSL or Cable service is probably sufficient for a small office/home office (SOHO), but a twenty-five employee office will need a higher level of SLA and MTTR.
The bottom line is to align your primary and backup service with the solution that is most cost effective and will allow for the highest probability of supporting the business. These are the details that must be explored and discussed. In many cases, these can and will be negotiated by the provider.
For example, if a dedicated fiber service is showing and your SLA needs to be requested or changed, the carrier provider won’t provide service initially until you’ve negotiated. In general, dedicated fiber offers guaranteed low latency as well as low levels of jitter and throughput, but occasionally an issue will arise. When it does, and an outage occurs, the last thing you want to hear your carrier say is, “We know you’re down, but we’re already meeting our SLA of X number of hours per month and we don’t guarantee throughput, so we can’t help you.”
We’ve seen this kind of scenario catch more than one IT department totally off-guard. We recommend covering your backside with as much detail about the circuit and what you’re getting so you can make sure it aligns with your needs. Even if the carrier representative says you’re covered by the SLAs, you should still take a second look.
We suggest you seek the counsel of an attorney to help with the negotiation of a telecom carrier agreement and who can provide your company with credibility. We find that contracts are almost always easier to negotiate to the business’s benefit when they work through an attorney. When you’re negotiating an agreement to provide for high availability, an experienced attorney will undoubtedly save you time and money in the long run.
However, it’s worth noting that some attorneys fail to understand that SLAs and termination liability should be considered when building a network for high availability (as mentioned above). You’ll want to be sure to cover yourself and your business by not only making sure the service works as advertised, but also that you have a means to fix the service if it isn’t working as advertised. Include a clause that allows you to walk away from the contract without penalties if a problem doesn’t get fixed in a set amount of time. This agreement should be very cut and dry.
In the case of (x), the cure period is (y), which is a negotiable point, and the consequence is (z), another negotiable point.
An example of x, y and z are as follows:
These type of contract clauses are far more aggressive than what the provider will offer, and to your benefit. And should you ever need to exercise this clause, you’ll sure be glad you added it.
The next two recommendations are not exclusively related to network uptime, but they’re certainly pertinent during all contract negotiations, especially if you have network or support issues of any kind.
Firstly, over the years we’ve found value in identifying all of the areas where you’re legally committing services to the carrier. We have seen up to four different areas where a customer is contractually committed to keeping their network with the carrier. It’s not just about the overall contract commitment anymore. Specifically, make sure you’re only committed to a minimum 1-year term for both ports and loops. Also, make sure that the addition of a new service doesn’t cause the committed term to increase.
Secondly, contract term auto-renewals are evil. Avoid them at all costs and don’t let the carrier commit you to anything more than the initial term. You can accomplish this by simply asking for a month-to-month renewal term and, in some cases, understand and agree that the rates may change. Most of the time, though, a carrier won’t increase your rates because they want to retain your business. Start the dialogue about a month-to-month term at any time, regardless of your intention to renew.
Prior to cutting a network, especially in a converged voice and data environment, it’s imperative that you test configurations prior to the turn-up date as to avoid unwanted surprises. There are many different human touch points involved in the provisioning of a network circuit, and we humans are most certainly fallible. Hardware, software, and application configurations need to be tested and emulated so there are no surprises when making a cut.
Even after making two or more tests of equipment configurations and applications, we suggest hiring a tier 3 or 4 level engineer for the first three location turn-ups. Unexpected problems can and will occur because all that time in the lab still doesn’t compensate for the real world.
We know you and your IT department are very intelligent and can troubleshoot/repair almost any circumstance. We also know that you’re IT group is convinced no one can do a better job. And we completely agree!
The trouble is that IT personnel have a life outside of work, but troublesome repair issues can arise at any time of the day, night, or during holidays and vacations. When this occurs, telecom carriers typically require the customer to report the issue by making a phone call and identifying the problem. The business must also verify that power and equipment is available.
All of this must be done before a carrier will even address a circuit outage.
Unfortunately, if an outage occurs while your IT person is at lunch or out of the office, that could instantly increase your network downtime another 15-30 minutes. If your IT person is on vacation, maybe a couple hours. Or if your IT guy had a big night out on their birthday and an outage occurs, perhaps several hours.
Our point is that placing the burden on IT to open tickets with the underlying provider creates undue stress as well as additional lost productivity and revenue. This potential issue can be circumvented by outsourcing to a service provider that watches your network 24/7/365 and automatically opens tickets on your behalf within five minutes of an outage. This can help take your network up to 99.99% uptime.
We hoped you learned from these eight best practices. You don’t have to implement all of these steps immediately. But if you want to insure maximum network uptime, then there’s no question that this list is a great start to achieving high availability — no matter what size company you’re operating.
How do we know these steps work? Because this exact list has provided our clients with 99.99% uptime time and time again. If you have any questions or would like more information about our SD WAN solutions, feel free to contact us.
At EnableIP, we build highly available WANs so our customers can fully leverage their cloud applications. Let us do what we do best so that you can do what you do best!
Enable IP is a technology solutions provider founded by Wired Networks founder Jeremy Kerth and Wired Networks’ head engineer Steve Roos, after they realized there was a deep market need for helping mid-size businesses establish better uptime rates for their Wide Area Networks. Armed with the best-in-class carriers and partners, Jeremy and Steve set out with a bold plan: Guarantee better uptime rates than the industry standard of only 99.5%.
Their bold plan became a reality: Enable IP ‘s solutions guarantee clients 99.99%, even 99.999%, network uptime. But, they don’t stop there. Many telecom providers promise high availability network solutions but fail to deliver, because they are in the business of providing services, not solutions.
That’s the Enable IP difference: We deliver highly available networks by providing a complete system, called Cloud Assurance, that ensures our 99.99% or above uptime guarantee.
We deliver this bold promise by: