With competition heating up, network testing and monitoring become critical priorities
Historically, executive suites were nearly impervious to any mention of test and measurement. Lately, however, cable companies have come to appreciate that the latest generation of T&M equipment may be as critical for retaining subscribers as DVRs have been in attracting them and gigabit routers have been for serving them.
Why? Because the importance of network maintenance, and of having installation and repair done right the first time, are no longer just a couple more items on the long, long list of MSO “priorities,” but are actual priorities.
At SCTE Cable-Tec Expo in June, Comcast COO Steve Burke said his company is now “laser-focused” on network reliability, concentrating on node health and on video quality.
That’s a clear change in direction in the view of Tektronix Marketing Manager Keith Cobler. “MSOs typically used to throw more people at problems – the brute force approach,” he said. “That’s starting to change. They’re looking at network monitoring. The fixed and wireless guys are a few years ahead on that, but MSOs are aware of the situation and are not that far behind.”
Comcast’s Burke said that until recently, monitoring simply wasn’t practical. “Five years ago, the question was: do we do any status monitoring? It didn’t pay. If things went right 98 percent of the time, everything was okay.”
But now, 98 percent isn’t nearly good enough. At Cable-Tec, Burke ran through the numbers: Comcast has 25 million subscribers, and if it has 2 percent errors in all the services it provides, the numbers compound to the point where the company is taking 200 million phone calls a year and engaging in 10 million truck rolls. That’s simply not sustainable, Burke said.
A truck roll can reportedly cost anywhere from $100 to $600, depending on the skill level of the technician dispatched and the intractability of the problem, and revisits tend to reside at the costlier end of that spectrum. If an MSO were to reduce the number of revisits by 20 percent to reduce its total number of truck rolls, it would be saving real money.
And though no MSO is content having a bad reputation, historically, there wasn’t much cost to being just a little cavalier about the consequences of an unsuccessful installation or repair failure. Sometimes the aggrieved subscriber had nowhere else to go, and even if disaffected customers did go elsewhere, there was always the potential of winning them back with cool new technology or a superior bundle.
But now with FiOS and U-verse, and with DBS packages being more attractive than they’re often given credit for, there’s a real risk that lost customers will never, ever come back.
Win-back, when possible for an MSO, is an incredibly expensive proposition. Incentives to entice customers not to leave in the first place are also becoming costly.
Charter CEO Neil Smit, also at Cable-Tec Expo, said, “People love our products. We can’t give them a reason to leave.” Later, he added, “in a competitive environment, customer service will make a difference.”
Good-bye, best effort. Hello, five nines.
Factors in the customer service equation include making sure the plant works optimally, giving installers and repairman sophisticated equipment, and then making sure that they actually use it.
Proper application of T&M is where the solution starts.
 |
“A few years ago, it was just a land grab,” noted JDSU Senior Manager, marketing and communications Matt Pitchford. “Everyone was just getting services out as fast as possible. They were just trying to get things to work.”
Now, the expense of returning to fix problems and the cost of losing customers is too much to bear.
JDSU testifies it’s possible to reduce repeat truck rolls by 20 to 30 percent.
There are problems in any cable system’s drops, Pitchford said. Frequently, installers will make sure services are working, but because they need to perform as many installs a day as possible, they often ignore marginal plant – or perhaps never take the time to determine if the drop is as good as it could be – so those problems never get fixed the first time out to a new subscriber’s home.
According to JDSU’s experience, 5 to 8 percent of all installs are undermined by bad drops, leading to an expensive return trip.
The first thing you have to do is just get the right tools to the technicians. The second thing is to make sure they use them, Pitchford said.
Several years ago, JDSU introduced software for its test equipment that tracks the number of jobs a tech has been on, and the number of tests the tech performed. The data is downloaded when the tech returns, and reports are generated. MSOs can evaluate which technicians require more guidance or training.
Pitchford said that JDSU customers who use the testers with the test-tracking software are reducing their repeat truck rolls to fix problem installations by 20 to 30 percent.
Once truck rolls are reduced, MSOs can catch their breath and start getting proactive.
Pitchford said that MSOs might perform a sweep on each node perhaps once a year, and maybe as infrequently as once every two or even three years. Now operators can take the same status reports they’re generating from installation tests, and start analyzing the data to determine which nodes have the most problems, and are therefore most likely to need maintenance.
“Once you get the gear to react, there’s not much more of an investment to get proactive. The cost is in the effort to do it,” Pitchford said.
VGI Solutions has a similar approach to attacking ingress and leakage problems. It used to be that MSOs would try to detect leakage at 20 microvolts/meter, said VGI Vice President Magella Bouchard. Now they’re looking to detect leaks at 2 microvolts.
“That was almost impossible before. Now, with better hardware connectors, better cable, new tools for detecting leakage, MSOs can achieve that,” Bouchard said.
The system is mounted on installation and repair vehicles. Detected leaks are mapped using the GPS system. The data is collected and updated every day.
Rogers, Shaw, Videotron and Cogeco in Canada are using the Canadian vendor’s products. Bouchard said VGI is in trials with all the major operators in the U.S.
Trilithic also has a leakage detection system with GPS, with data that can be displayed using Google Maps. Steve Windle, product manager at Trilithic, said the system is attracting customers.
Contract installers are beginning to see the advantage of the system, he said. Typically, if an installation ends up being unsuccessful, MSOs bill back the contractor for the subsequent truck roll. The installers want the automated reports to demonstrate that they’ve performed the installation correctly.
Phone companies have for decades lived under the strictures of government regulation that compel them to provide maximum service reliability. The telcos are steeped in the cult of five 9s – making sure that everything works 99.999 percent of the time.
Even so, competition is forcing the telcos to strive to be even better, said Tek’s Cobler.
“Looking back 10 years ago, the telecom industry’s mentality was, ‘build it and they will come.’ That used to work, but now they’re getting closer to the customer.”
Tek’s customer base includes many fixed line and wireless phone companies. Cobler said the company’s customers are now asking for end-to-end network monitoring, including monitoring of interconnect partners.
As with MSOs, monitoring allows telcos to not only set off alarms when threshold conditions are approached, but they can also comb through network statistics to become proactive about predicting where problems might occur.
Tek addresses the market with its GeoProbe passive network probes, which work with the company’s Unified Assurance Platform. Last year, Tek tweaked its UAP to work with converged networks. Last November, it introduced a GeoProbe version specifically for monitoring IPTV on converged networks.
Tek’s most recent contribution to automated testing was the June debut of Zoey, an Interactive Voice Response (IVR) system that allows anyone to test and diagnose the quality of VoIP, wired, or wireless phone service. Tek is positioning it as a self-install tool, but it can be an effective means for installation technicians to make some baseline determinations about a customer’s phone service.
For the most part, MSOs don’t have any new requirements for test systems, though that doesn’t mean T&M vendors can rest easy.
“The networks aren’t changing,” Trilithic’s Windle said, “but there are more things on the network that require more rigorous testing.” Telephone service requires an absolute level of reliability, especially for business customers, Windle noted.
A while back, Hewlett-Packard discontinued its HP8591C spectrum analyzer. Then edge QAMs upped the ante on what cable technicians needed from a spectrum analyzer. Trilithic responded with a firmware upgrade for its 860 DSPi Field Analyzer.
The 860 now features error vector spectrum, which enables the user to view the spectrum characteristics within a QAM signal band (including deep-interleave signals). This time-saving feature permits viewing ingress or distortion products that can cause tiling of video, slow data transmission, or dropped calls.
The return path for DOCSIS 3.0 has a frequency range up to 85 MHz, and until recently, most testers topped out at 65 MHz, according to Windle, who said that Trilithic has responded by upping the range on its testers to 100 MHz.
