With so much manual testing, test automation becomes an issue

Service providers are beginning to roll out EBIF and tru2way applications, but the pace, by necessity, is deliberate. There’s a whole lot of test activity that has to go on first to make sure that all of these various apps – program guides, interactive advertising, polling – work from the outset.

For those who specialize in test and measurement (T&M), that breaks down into three basic tasks:

1) Testing the network to make sure it’s suitably prepared to handle upstream traffic.

2) Testing the applications in the lab to make sure they work before going live with them.

3) Monitoring the applications after they’re live to make sure they’re behaving the way they’re supposed to.

There’s little that’s particularly difficult about any of this. But if there’s a problem, it’s with task 2, and it’s not a problem inherent in the test technology.

Rather, the potential problem is an outgrowth of having many operators building their own test beds and their own test routines largely independently from each other. When multiple organizations are doing their own testing based on home-grown rigs and procedures, that creates a risk that an EBIF (or tru2way) application that tests OK in one environment might fail in another.

If an EBIF application runs as it’s supposed to on one MSO’s network but not on another’s, that undermines the purpose of even having an EBIF standard. One of the major advantages advertised for EBIF and tru2way is that an application developer can write one application and be assured that if it works on one EBIF-compliant (or tru2way-compliant) network, it will work on all of them. There’s just no way of telling if that’s the case right now.

PREPPING THE NETWORK
EBIF and tru2way have no special network requirements, nor do they represent any special strain on a network. But by their nature, they are two-way technologies, relying in part on the upstream, an element of the network that even today remains lightly used and maintained lessdiligently than the downstream.

That’s been changing in the last few years, but not necessarily because of EBIF and tru2way. The introduction of VoIP has brought some attention to the upstream.

“But you can’t just assume, ‘Well, I rolled out VoIP, my return path has to be good,’” said Ron Shanks, who is in charge of video solutions at JDSU. VoIP is in a different frequency range than EBIF or tru2way traffic would be.

Some operators have cleaned up their upstream paths in preparation for deploying switched digital video.

Others may have done so when deploying DOCSIS 3.0. Many operators believe the vast majority of subscribers will be downloading far, far more than they’ll be uploading anytime soon, however, and so may not be as attentive to the upstream as one might assume.

Most operators haven’t deployed DSG (DOCSIS set-top gateway) signaling, noted Shanks. “They’re using the low frequencies, and they have to make sure that’s clean enough.”

JDSU has a return path monitoring system called ReturnTrak that complements its forward-path monitoring system: PathTrak.

The upstream channel is host to several well-known problems, including susceptibility to noise and ingress. And as activity increases on the upstream, autototal power load on the upstream lasers increases. Upstream lasers can end up being nudged into non-linear (distortion) mode, reminded Jack Webb, a product manager at Sunrise Telecom.

“On the upstream, you want to look for noise, distortion and non-linear performance,” he said. “You want to make sure the frequency response is correct.”

Regular maintenance, including sweeping, will likely take care of that.

Under the best of circumstances, the spectrum from 5 MHz to 20 MHz tends to be a tad more susceptible to ingress and noise, so as a practical matter, many operators choose to work within the 17 MHz between 20 and 37 MHz. In the past, MSOs might use a single frequency in that band, but now to increase bandwidth, they might be dividing that into two, three, or even four frequencies, Webb explained. EBIF and tru2way aren’t going to aggravate that, but they will represent more traffic in that spectrum.

APP PREPARATION
It appears that different operators are doing their own testing based on home-grown rigs and procedures. If everyone is testing supposedly similar things in dissimilar ways, then there’s no telling if the thing being tested will actually work in both test environments, let alone both networks. If an EBIF application runs on one MSO’s network but not on another’s, that undermines the purpose of even having an EBIF standard.

In a paper presented at the Cable-Tec Expo in late 2009, Cox and its tru2way development partner itaas reported: “Standards and specifications allow MSOs to build intelligent, automated and cost-effective test beds, where manual testing can be kept to very minimal levels.”

Automating testing, of course, serves to reduce both costs and the time it takes to roll out new services.

“One important aspect of tru2way is it provides various troubleshooting mechanisms with the physical presence of resources using technologies such as SNMP walk of devices. This again saves time before and after deployment,” the Cox/itaas paper continues. “There are thirdparty test suites on the market that can also provide a level of compliance reporting on the devices. … These tools are often highpriced, but experience has shown that they can potentially save much more in support and customer care costs.”

Available, yes, but perhaps not widely used just yet.

Fanfare Software specializes in automating test suites. Vice president of marketing David Gehringer said: “It all comes back to testing. In order to satisfy the growing demand for interactive digital entertainment, cable providers and manufacturers must implement a strategic test automation process that enables them to collaborate on testing, communicate test results and address bugs in near real time in order to accelerate the testing process.

“Despite all of the logical reasons to implement automation – enhanced productivity, accelerated testing cycle, shorter product and service development lifecycles,” Gehringer continued, “the majority of testing is still conducted manually. Time-intensive manual testing simply cannot keep up with the complex device and system testing that is required in order to provide sophisticated services like tru2way. As a result, companies are falling behind in their attempts to capitalize on this market.”

CableLabs and Canoe Ventures have been holding interoperability events.

Canoe, meanwhile, has been working at developing what it’s calling templates for bound EBIF applications – common approaches to various functions, which if commonly used will contribute to being able to scale up those functions.

That’s well and good for assuring that various system elements work with each other, but operators including Time Warner Cable and Comcast have their own labs for qualifying the applications themselves. The downside is that currently an application developer has to get separate qualifications from each MSO with a lab.

The question is whether anyone might be able to put together a functional test environment, which goes beyond merely simulating a cable network, that developers could go to and make sure their applications will run across the systems of the different MSOs.

While EBIF is often considered a scaled-down version of tru2way, more suitable for deployment to the most basic set-top boxes, in practice it’s being used differently, according to Eric Conley, CEO of Mixed Signals, a monitoring specialist recently acquired by Tektronix (itself a subsidiary of Danaher).

Tru2way is largely being used as a carousel for when applications or data have to be delivered, he said, “but EBIF is a different animal. EBIF is driving interactivity, and that has a timing element to it.

“If something has to happen at a certain time within a certain show – it works like a clock,” Conley continued. “When a tagged piece of data is put in the video stream regularly, we watch that, or if at a particular time you have to make something happen – a banner over an ad or a poll – we watch that.”

That requires decoding the stream. For tru2way, he said, the entire stream has to be decoded: “It’s all or none.” For EBIF, however, “you can look for a specific PID” (short for packet ID distribution, a marker indicating the presence of an EBIF application), although Mixed Signals always does a full decode, he said.

MONITORING DEPLOYMENTS
Once the network is set up for EBIF or tru2way, then the issue is provisioning, Conley said.

“You want to make sure you’re sending what you want to send, and you want to make sure that it’s not slowing other things down. Generally, that’s a pretty straightforward thing to do,” he said.

And in an increasingly complex network, operators need to guard against something else, too: “Sometimes an app is deployed and there are unintended consequences,” Conley noted.

Mixed Signals helped Time Warner Cable when it first began rolling out tru2way applications and is among the first companies with a monitoring solution adapted specifically for EBIF environments, with an expansion of capabilities to its Sentry line.

“Operators will use our products to check that if there’s data or a stream going downstream to a set-top box, they’ll use Sentry to see if one, that it’s there, and two, there’s no adverse impact on anything else,” Conley said.

Andrew Sachs, Volicon’s senior director of marketing for cable/IPTV, said: “You have to hook up a stream analyzer and capture the stream. If you get the PIDs through ingress grooming, it’s good, it’ll show up on the set-top box.”

That’s the downstream. There also should be some assurance that the system responded to a subscriber’s click, but Conley reports that that’s not an issue his customers are bringing up with Mixed Signals, though he allows they might have some internal tools for that.

Same story with Empirix. “Was there a button push? Did it get recorded? That’s right in our wheelhouse,” said Pete Quigley, director of product management. Has anyone called? Not recently.

Gehringer said: “If you’re going to vote for ‘American Idol,’ if you call in, no one knows if that vote is counted. But if you do it through your TV, people are going to want their votes confirmed. Someone will eventually do that as a differentiator.”

Some of that sort of thing is happening, but it’s still not entirely automated. Sachs said: “We’re working with one large cable operator, testing a direct commerce application. Every hour we go through the application all the way to the order screen.”

Beyond the order screen, the company uses a credit card with an $80 limit and orders more than $80 worth of merchandise. “It should get denied. If it doesn’t get denied, that’s another alarm,” Sachs said.