1. Introduction

Software-Defined Networking (SDN) is a data network paradigm that defines the separation of the data plane and control plane so that applications can change network behavior through software algorithms ^[1-3]. SDN enables innovation and programmability through a centralized structure and controller. The centralized controller maintains a global view of the network. SDN allows a network administrator to dynamically configure, manage, and optimize the network using software, regardless of the vendors ^[3,4].

Many technologies have been deployed to improve several features of SDN management, and one of the basic tasks is network monitoring. Efficient tools for SDN monitoring improve the utilization and optimization of network resources ^[5]. Many academic solutions are focused on network performance and resource utilization reduction, but these works lack experimental evaluation in physical environments. Many of these proposed solutions were implemented and evaluated in simulated/emulated environments, so these solutions are not suitable for production cases. There is a need for evaluating research works in real SDN environments to determine the influence of a Networking Monitoring System (NMS) on SDN. The proposed NMS is based on a previous work that was published in the International Conference on Electronics, Information, and Communication (ICEIC 2022) ^[6]. The present research work determined the influence of the proposed NMS in a real SDN environment.

This document has the following structure. Section II presents a summary of the problem, related works, and the research contribution. Section III explains the proposed NMS adapted to ONOS, a test environment in a research laboratory, and application deployment. Section IV shows statistical analyses of the results. Lastly, Section V presents the conclusions of the research work and guidelines for upcoming research.

2. Related Work

A literature review of related works was done to determine the state-of-the-art SDN monitoring tools. These works, which were found in indexed databases from 2016, evaluated SDN monitoring performance with related well-known metrics. However, these works lack experimental evaluation in a real SDN environment for deploying their applications. Also, many of them worked with non-production SDN switches, so they could not be suitable for real implementations.

AlZoman and Alenazi ^[7] proposed a link failure detection system for smart cities in 2020. Yang et al. ^[8] presented an extensible-network traffic monitoring system for SDN/NFV. The authors said this system could match the performance of traditional networks but with lower costs. In addition, Wang et al. ^[9] implemented an SDN-monitoring tool called SCSCDaylight on the OpenDayLight SDN controller. Usman et al. ^[10] focused on large-scale device monitoring through a tool named SmartX MultiView Visibility Framework, which gathers, validates, and integrates the created data with large storage based on the integration of databases: ElasticSearch, InfluxDB, and MongoDB. These authors showed results by using metrics of storage-saving time and display response time of web interface components.

Lin et al. ^[11] implemented a process for monitoring the status of network elements (NEs) and integrated an NE alarm and health information for service impact analysis and alarm correlation. Also, published work by Kavitha et al. ^[12] presented a NMS diagram. In 2018, a single work by Vela et al. ^[13] presented a software architecture called CASTOR, which allows active monitoring by using a CASTOR agent. This tool works with an ONOS controller. In 2017, Lin et al. ^[14] proposed GolfEngine that was built as a framework for working with the OpenDayLight SDN controller. GolfEngine’s architecture aims to be a plane abstraction for other network applications. This could solve app policy conflict by using a module for managing the network resource.

Wassapon et al. ^[15] published a research tool called Opimon, which monitors and displays an OpenFlow network, including flow tables of each switch and network topology. This tool was assessed in terms of the OpenFlow packet response per second. Gangwal et al. ^[16] presented a set of mechanisms to find per-port and per-flow traffic statistics, network topology information, and the data transfer rate for each link of an SDN-based network. Cai et al. ^[17] proposed a real-time deep packet inspection (DPI) monitoring system that detects network topology and monitors network services such as Facebook or Google.

In 2016, Guimaraes et al. ^[18] introduced a reuse-based approach to enable visualizations in SDN to improve productivity and reduce costs. Noskov et al. ^[19] proposed an active monitoring tool working with a Floodlight SDN controller for issuing notifications based on a subscription to relevant events on the network. Park et al. ^[20] proposed the Route Convergence Visualizer (RCV), which provides a quick and easy tool to understand what exactly has happened and who has been affected by a network event so a network administrator can make a decision over the network.

Kim et al. ^[21] presented OFMon (OpenFlow Monitor) to monitor all OpenFlow messages between ONOS controllers and OpenFlow switches. The authors evaluated the OFMon performance in terms of CPU and memory usage in an emulated environment. Lastly, Caicedo et al. ^[22] implemented a tool named Rich Dynamic Mashments (RDM) and analyzed the performance in terms of time recognition, time consumption, time composition, display response time, and network traffic.

This paper shows the influence of NMS in a real SDN environment. This paper's contributions are the following:

· A proposed NMS ^[6] works with an ONOS SDN controller, which interacts with production SDN switches such as the OF-DPA© ^[23] and Pica8© ^[24]

· An experimental evaluation of the proposed NMS was performed in a real SDN environment

· The aim for the proposed NMS and real SDN environment is to be extensible to future network innovations in cybersecurity

3. Methodology

An NMS adapted to ONOS is proposed for an SDN. Additionally, a real SDN test environment is presented. Lastly, the NMS deployment is explained.

3.1 Network Monitoring System

The proposed NMS was presented in a previous research paper published at the International Conference on Electronics, Information, and Communication (ICEIC 2022) ^[6]. This subsection explains the components that interact to make the NMS work with the ONOS controller. A significant change was made for the present research work to interact with a real-world topology ^[25,26]. This change focuses on the modification of specific files due to the architecture of low coupling and high cohesion offered by the NMS application.

Fig. 1 shows the NMS software architecture, the internal mechanisms, and the interaction with the ONOS (ver. 1.12.0) SDN controller. Fig. 1 shows the main components of the software architecture. The adaptation to the ONOS API is explained below.

The Monitoring Manager retrieves, analyzes, and stores states of network devices and links ^[6]. The Network State Collector component has been modified from its published version ^[6] to call the ONOS API. The Topology Manager component compares network topologies and makes downlink detection alerts. The component saves the handled data in MongoDB. The communication between the Monitoring Manager and Visualization Manager through a common non-relational database permits a GUI application to show network visualizations (e.g., topology view, downlink detection, and alerts).

Fig. 1. NMS software architecture[6].

3.2 Test Environment

Implementation in a real SDN environment was done in the Advanced Networking Laboratory at Facultad de Ingenieria Electrica y Electronica, Universidad Nacional de Ingenieria, Peru. The real test environment consists of 5 physical SDN switches, one legacy switch, and one server. The basic characteristics of these devices are shown in Table 1.

A real-world topology was implemented in the laboratory as shown in Fig. 2. The topology follows a common data center design for ensuring the capacity and availability of network services.

Fig. 2. Real-world SDN switch topology.

Table 1. Real-world network devices.

Qty.	Type	Device	Characteristics
1	Server	QuantaPlex T41S-2U	-02 x Intel Xeon E5-2630 v4 processors, 10 cores 2.2 GHz. -RAM of 128 GB DDR4 (4 modules of 32 GB). -02 x 150 GB SSD disks and 02 x 1.8 TB disks.
1	Legacy switches	QuantaMesh T1048-LB9 [27]	48 x 1GbE and 04 x10GbE ports
1	SDN core switch	QuantaMesh BMS T7032-IX1 [28]	OF-DPA with 32 x 40GbE QSFP28 ports
2	SDN aggregation switches	QuantaMesh BMS T3048-LY8 [29]	OF-DPA with 48 x 10GbE SFP+ and 6 x 40GbE QSFP+ ports
2	SDN access switches	Edge-Core AS4610-54T [24]	PicOS (modified Open vSwitch) with 48 x 1GbE BASE-T, 4 x 10GbE SFP ports

3.3 Deployment

The NMS application was deployed on a physical server in the real SDN environment. A relevant process in this deployment was the dockerization of the NMS to install all application requirements. The NMS was installed on a Ubuntu 20.04 docker, and a MongoDB 4.4 non-relational database image was used. The ONOS controller was a native installation. This deployment is shown in Fig. 3. Fig. 4 shows the NMS web view of monitoring real SDN switches.

Fig. 3. NMS application deployment.

Fig. 4. NMS web visualization of real SDN switches.

4. Results

This section presents results obtained from an experimental evaluation and some discussion from previous works. Table 2 shows the metrics to evaluate SDN applications that were taken from related works to measure the NMS performance. For each metric, we did measurements in three cases. (1) Baseline is the measurement done when the server does not have any SDN app. (2) Measurement with ONOS means the server is executing the ONOS SDN controller. (3) Measurement with ONOS + NMS means the server is executing ONOS plus the docker containerized NMS.

4.1 CPU

CPU usage is a metric of CPU percentage used by the server. For the evaluation, we used a tool developed by the authors using psutil, a python library. This tool records CPU usage every second for a given time in a file. In addition, this tool is used to measure memory usage. Fig. 5 shows the charted data of CPU usage during 180 measurements for the three cases.

Fig. 6 shows a statistical test of normality. The graph is a normal distribution. The X-axis shows CPU usage values, and the Y-axis shows the position in the normal distribution. The divergence of both lines shows that further measurements would maintain the increase in CPU usage of measurements with ONOS + NMS. In Fig. 6, the average CPU usage for measurements with ONOS is 1.47%. However, the average CPU usage for measurements with ONOS + NMS is 1.84%, which is higher than in the evaluation with solely ONOS.

Fig. 6 also shows the p-value for a statistical test of normality. First, it is assumed that the data has a normal distribution. Then, both measurements with ONOS and measurements with ONOS + NMS have a p-value < 0.05 (5% error). Thus, it is concluded that both do not have a normal distribution.

Based on the normality test in Fig. 6, both variables do not have a normal distribution. Therefore, a Wilcoxon test with the 95% CI was used to verify the following hypotheses:

H$_{0}$: NMS does not influence the server performance in terms of CPU usage

H$_{1}$: NMS influences the server performance in terms of CPU usage

Table 3 shows the ranks obtained from the data (180 measurements) comparing measurements with ONOS and measurements with ONOS + NMS. Negative ranks are the number of measurements (N) where CPU usage of measurements with ONOS + NMS is lower than that of measurements with ONOS. Positive ranks are the numbers of measurements where measurements with ONOS + NMS are higher than measurements with ONOS. ``Ties'' are the number of measurements where measurements with ONOS + NMS and measurements with ONOS are equal.

In addition, Table 3 shows the Wilcoxon Test from SPSS© software, which determined the standard normal distribution Z-score and the p-value (Asymp. Sig) < 0.05 (5% error). H$_{0}$ is rejected, and there is enough evidence to conclude that NMS influences the CPU usage, so H$_{1}$ is accepted.

Fig. 5. CPU usage in 180 seconds.

Fig. 6. Normality test on CPU usage data.

Table 3. Wilcoxon test on CPU usage data.

Parameters		Values
Ranks
	N	Mean Rank	Sum of Ranks
Negative Ranks	48	74.96	3598.00
Positive Ranks	123	90.31	11108.00
Ties	9
Total	180
Wilcoxon Signed Ranks Test
Z		-5.796a
Asymp. Sig. (2-tailed)		0.000

a. Based on negative ranks

4.2 Memory

Memory usage is the metric of amount of Memory used by the server. Fig. 7 shows the charted data of memory usage during 180 seconds for the three cases. Fig. 7 shows clearly that the measurements with ONOS + NMS are always higher than measurements with ONOS.

Fig. 8 shows a statistical test of normality. The graph has a normal distribution formed by the X-axis’s memory usage values and the position in the normal distribution on the Y-axis. The divergence of both lines shows that further measurements would maintain the increase in memory usage of measurements with ONOS + NMS. In Fig. 8, the average memory usage of measurements with ONOS is 18.81 GB. However, the average memory usage of measurements with ONOS + NMS is 19.24 GB, which is higher in all measurements than the evaluation of measurements with ONOS.

Fig. 8 also shows the p-value for a statistical test of normality. First, it is assumed that the data has a normal distribution. Then, at least one variable, measurements with ONOS, has a p-value < 0.05 (5% error), which means the data does not have a normal distribution. Thus, it is appropriate to apply a Wilcoxon test.

Based on the normality test in Fig. 8, the data does not have a normal distribution. Therefore, a Wilcoxon test with the 95% CI was used to verify the following hypotheses:

H$_{0}$: NMS does not influence the server performance in terms of memory usage

H$_{1}$: NMS influences the server performance in terms of memory usage

Table 4 shows the ranks obtained from the data comparing measurements with ONOS and measurements with ONOS + NMS. Negative ranks are the numbers of measurements (N) where memory usage of measurements with ONOS + NMS is lower than that of measurements with ONOS. Positive ranks are the numbers of measurements where measurements with ONOS + NMS are higher than those of measurements with ONOS, and ``Ties'' are the numbers of measurements where measurements with ONOS and measurements with ONOS + NMS are equal. In addition, Table 4 shows the Wilcoxon test from SPSS© software, which determined the standard normal distribution Z-score and the p-value (Asymp. Sig) < 0.05 (5% error). H$_{0}$ was rejected, and H$_{1}$ was accepted. It is concluded that there is enough evidence to state that the NMS influences the memory usage.

Fig. 7. Memory usage in 180 seconds.

Fig. 8. Normality test on memory usage data.

Table 4. Wilcoxon test on memory usage data.

Parameters		Values
Ranks
	N	Mean Rank	Sum of Ranks
Negative Ranks	0	0.00	0.00
Positive Ranks	180	90.50	16290.00
Ties	0
Total	180
Wilcoxon Signed Ranks Test
Z		-11.919a
Asymp. Sig. (2-tailed)		0.000

a. Based on negative ranks

5. Conclusion

We determined the influence of an NMS in a real SDN environment. This was due to a lack of works in the literature with experimental tests on physical SDN switches. Thus, the authors' contribution is applying an NMS application in a real SDN environment. The NMS was adapted to the ONOS SDN controller, a real SDN environment description, and a deployed application, which were explained. In conclusion, it was determined that the NMS influences the server performance in a real SDN environment.

Relevant results and an exhaustive statistical test were shown to evaluate the influence of the NMS on the physical server performance in a real SDN environment. The average CPU usage of measurements with ONOS was 1.47%. Nevertheless, the average CPU usage of measurements with ONOS + NMS was higher at 1.84%. Average memory usage of measurements with ONOS was 18.81 GB, but the average memory usage of measurements with ONOS + NMS was higher at 19.24 GB. The focus of NMS is on being a platform that permits future networking specialists to learn and do research in a real SDN environment. Future networking researchers are encouraged to use production controllers such as ONOS, which works with OF-DPA and Pica8 switches, and standardized tools. The proposed NMS is going to be used for future research in cybersecurity.