This story appeared on Network World at

http://www.networkworld.com/columnists/2005/072505bradner.html

'Net Insider

Conclusions based on isolated data

By Scott Bradner, Network World, 07/25/05

Scott Bradner

Performance measurement company Keynote Systems earlier this month issued a study of Internet phone service quality and concluded that there is a "need for considerable improvement." I do not know just what led the company to that conclusion, but I do caution anyone reading reports of this study not to conclude that VoIP has no future.

I have no way of judging the quality of the survey, since all that Keynote has made available is a press release . The company doesn't say in the release how much it charges for the whole study, but I'm sure it's more than I will get for writing this column, so I'll get by on the release.

According to the press release, the study seems to explore a reasonable number of the relevant variables, including multiple VoIP and connectivity providers, call location and time of day. Keynote used 10 factors to evaluate the "end-user experience." The company then reduced these factors into two magic numbers representing reliability and audio quality. That seems to me to be rather over-reduced - for example, lumping the quality at 2 a.m. (when no one is using the local link) with that at midday (when the local loop is congested) does not produce the information I'd want to get.

I use VoIP and a lot of people I know do, as well. Maybe it's just the environments that we work and live in, but it is not my experience that VoIP has a "need for considerable improvement in service." In fact, almost all the time the quality of the VoIP call is perceivably better than my office ISDN phone. I have made calls where the quality sucks (to use a technical term), but that happens a few times a year - far less frequently than the poor-quality public switched telephone network connections I keep getting to and from all sorts of locations.

But let's assume that Keynote is correct in its claim that VoIP too often does not "live up to the dial-tone reliability and crystal-clear communication quality" we have come to expect with plain old telephone service, at least in the environments where the company ran its tests. Can we garner anything about the potential of VoIP from that conclusion?

This information, in isolation, is not all that meaningful in the real world. Other factors overwhelm these perceptions of poor "end-user experience." One only has to imagine what the report would have been if some earlier year Keynote had run exactly the same tests on 1990-era cell phones, which were unambiguously and almost universally crappy. Any reader looking only at the results of the testing would have concluded that cell phones had no future at all.

But anyone coming to that conclusion would have to ignore three important factors. The most important is portability - the phone goes where you go instead of you having to go to the phone. A second factor is that technology keeps getting better, and the third is that the per-call cost of cell phones would plummet because of competition to a point where it's far cheaper than land-line phone calls.

Any conclusion that VoIP has a poor future based on the Keynote study would be to ignore the last two of these factors. Ignoring important factors is not a good way to come to useful conclusions.

Disclaimer: Harvard has enough of a past to know that predicting the future is hard but has not expressed a formal opinion on VoIP futures.

Bradner is a consultant with Harvard University's University Information Systems. He can be reached at sob@sobco.com.