From chipsets to network equipment to full-scale networks, benchmarking of communications products sets the bar for performance.
Whether you are a mobile-network operator (MNO), a network-equipment manufacturer (NEM), a handset maker, or an IC maker, you can never have too much information about how your product performs against expectations and compared to competitors. Benchmarking the performance of mobile technologies has never been more important than with 5G.
MNOs cannot afford to deploy a device or piece of equipment unless they understand exactly how that device or equipment will perform on their network. The ability to measure and compare a network’s performance across many key-performance indicators (KPIs) has never been more critical.
Although benchmarking of 5G devices and networks is vital, it can leave you confused because it comes in many forms, each with its own uses, tools, and methodologies. Let’s look at some prevalent types of benchmarking in 5G technology, their uses, and key attributes.
Benchmarking on live networks
MNOs validate the performance of the cellular broadband data applications that drive 5G’s growth. Network benchmarking helps MNOs:
- Reduce customer churn by improving quality of service (QoS) and quality of experience (QoE),
- Detect network coverage gaps,
- Find locations with signal issues,
- Identify edge coverage issues and weak spots,
- Determine which devices perform best on a particular network,
- Shorten the time-to-market for new technologies and services,
- Make informed decisions about where to add cell sites and which KPIs need improvement.
The only reliable way to measure a network’s maximum throughput and other key performance features is through controlled, active bulk-data-transfer tests between a device and a network.
Benchmarking of live mobile networks involves collecting information to measure QoS and QoE. A typical benchmarking campaign involves comparing QoS and QoE across different networks — including other networks maintained by the same MNO as well as competitors’ networks (Figure 1). Such a benchmarking campaign also involves summarizing the results as KPIs and assigning an overall network performance score.
Figure 1. The chart shows QoE and QoS comparison between multiple networks maintained by multiple MNOs.
Different tools and methodologies required to conduct benchmarking of voice and data transmissions can add to the confusion. MNOs typically have about six different voice and data use cases for each network benchmark. Each of these use cases needs testing, both indoors and outdoors, including urban areas, city centers, and highways. The individual environment of each test location determines test equipment needed to conduct benchmarking in each area. This equipment includes different hardware components to capture and measure signals in different locations and environments. Benchmarking also requires software to conduct post-processing and analysis of the results, calculate NPIs and network performance scores, and create useful reports.
Benchmarking in the lab
MNOs and NEMs need to conduct benchmark testing of their networks and equipment in a laboratory environment. Conducting this benchmark testing in a lab (Figure 2) is critical to understanding how devices and network equipment will perform on a live production network, thus ensuring the smooth and timely rollout of new devices and services. Benchmarking devices and base stations in the lab also lets MNOs troubleshoot uncovered issues prior to deploying the equipment on the live network.
Figure 2. A test setup for device benchmarking with a live base station adds confidence that the device will function as designed.
For example, prior to deploying the latest smartphone on its 5G network, an MNO needs to conduct extensive benchmarking in the lab to learn exactly how that smartphone will perform on the network. This type of benchmarking lets the MNO learn how a device will operate as expected on the live 5G production network. Such testing may also uncover any major compatibility issues between the device and the network.
Much like performing network benchmarking on the live production network, benchmarking in the lab enables scoring of the devices’ performance against KPIs. Scoring the results also lets MNOs compare the performance of different devices, equipment from different vendors, and new versions of equipment and operating systems. This enables the MNOs to thoroughly evaluate performance before deploying equipment and devices on the live network.
Device benchmarking
Benchmark testing typically conducted by chipset and device makers lets them evaluate their products against those from competitors compared to their own previous generation builds. In addition, device benchmarking enables these vendors to troubleshoot, identify, and resolve issues, ultimately enhancing product quality.
Device benchmarking also lets device makers ensure that their devices conform to major MNO’s specifications and will be accepted by the MNOs prior to major device launches. This assurance provides device makers with a reasonable certainty of acceptance prior to putting the devices into volume production.
Chipset manufacturers also run device benchmarking tests to verify that their products perform according to the major MNO and NEM requirements. MNOs also conduct device benchmarking before benchmarking them on lab networks. Doing so lets MNOs confirm that the devices comply with their specific device test requirements.
Just as with the other types of benchmarking discussed above, device benchmarking involves scoring devices for overall performance and against specific KPIs.
The good news
Sorting through the various types of benchmark testing commonly employed in the wireless communications industry can get complicated. Fortunately, all of the benchmarking tests discussed above have been well established within the industry and will apply to 5G networks. Each comes within its own well-established track record, history, and accompanying set of industry best practices, and methodologies.
