Amulya Karaadi, Is?Haka Mkwawa, and Lingfen Sun

Internet of Things （IoT） applications such as environmental monitoring， healthcare， surveillance， event recognition and traffic control are amongst the most commonly deployed applications over the Internet. These applications involve multimedia content that has to be collected， processed and delivered appropriately over the Internet for further processing by human or machines. These applications come with their own set of requirements such as quality， computational power and bandwidth. It is， therefore， vital to minimize power consumption and bandwidth usage in IoT devices without compromising the quality of multimedia delivery. Since the delivery of the multimedia can be destined to a machine or human， it is important to distinguish multimedia quality between the two. Quality of Experience （QoE） for video services involves human visual system， but what will involve a machine or process？ To distinguish between the two， this paper defines a new concept of Acceptable Quality of Things （AQoT） which involves IoT devices and their applications. AQoT aims at minimizing bandwidth without compromising quality in IoT devices. Experimental results based on human detection and license number plate detection use cases have demonstrated that the AQoT concept can significantly reduce bandwidth usage.

QoE； QoT； IoT； bandwidth； video streaming

1 Introduction

he global Internet has been expanding at an unprecedented speed. It is now connecting over 3.7 billion people [1] and around 22 billion “smart objects” via the Internet of Things （IoT） [2]. According to the latest forecast from the Cisco Visual Networking Index [3]， IP video traffic will account for about 82 percent of all consumer Internet traffic by 2021， increase threefold from 2016 to 2021. Within this five?year period， the most fast?growing IP video traffic is expected to be Internet video （such as video services provided by YouTube and Netflix） with an estimated growth of fourfold from 2016 to 2021； Internet video surveillance traffic 7?fold； live video 15?fold； gaming traffic nearly tenfold； and virtual reality and augmented reality traffic 20?fold. In addition to the above consumer Internet video traffic， machine?to?machine （M2M） communications and IoT services for multimedia applications further increase the video traffic on the Internet.

The ever?growing Internet video traffic has posed a real challenge to the healthy operation of the Internet. The Internet is feeling the strain， far beyond the imagination of its original developers in 1970s and 1980s. Any technologies or approaches to reducing the traffic for a service to be delivered over the Internet without compromising the users experience for the service would be welcomed by all parties involved. For consumer?based IP video traffic， such as live video streaming， Internet TV and video gaming， keeping the customer happy and reducing the churn rate is key to the success of launching new service or maintaining an existing service for a service provider. In general， increasing video bit rate for a video streaming service will have a positive impact on end?user perceived video quality if there are no constraints on network bandwidth. However， in some cases， increasing video bitrate further does not result in a clear increase in perceived video quality or Quality of Experience （QoE）. In some applications， it would be too costly to always transfer the maximum video bit rate for a multimedia service. In our previous work [4]， we have demonstrated the gain in utilizing “Acceptable QoE” （i.e. Mean Opinion Score （MOS） over 3.5） in LTE downlink resource scheduling for VoIP services to improve the cell capacity. In this paper， we expand the concept to the domain of multimedia IoT applications. We define the term of ‘Quality of Things （QoT） [5] to refer to the quality of fulfilling an IoT task/process with multimedia IoT services and demonstrate how a similar concept， named as ‘Acceptable QoT （AQoT）， could be applied in IoT applications to reduce video traffic without compromising the quality of delivered multimedia services to a machine or a ‘thing over the Internet.

The remaining of the paper is organized as follows. In Section 2， related work based on the quality of multimedia in IoT is discussed. Section 3 provides QoE and QoT definitions together with QoT scheme and management. Experimental setup， results and evaluation are presented in Sections 4 and 5， respectively. Section 6 concludes the paper.

2 Related Work

Multimedia communications on the Internet of Things research has received wide attention in the literature in recent years. However， the growth and popularity of multimedia data pose new challenges to the IoT devices. Multimedia IoT （MIoT） devices consume more bandwidth and require high processing power to transfer the acquired multimedia data. Multimedia applications such as face or object detection， surveillance system and event detection are captured by the MIoT devices and then the video sequences are sent to edge nodes or cloud for further analysis depending on their tasks.

Research in edge computing framework for cooperative video analysis [6] proposed a cooperative framework for delay?sensitive multimedia IoT tasks， where high?quality video streams acquired by the camera node， are sent to the edge node to process sub?tasks e.g.， feature detection and extraction and send the processed results to cloud for further video analysis if necessary. In [7]， an architecture was designed to run in the hostile environment， where captured images by the camera node will be sent to the cloudlet over high?speed bandwidth connection. If the cloudlet lacks the necessary data from the database， it will send some of the tasks to the remote cloud for further processing. A two?stage procedure was implemented in [8]， which included face detection， extraction and face matching. The face detection and extraction tasks are performed in a cloudlet node while the complex face matching is performed on a remote cloud node. One possible limitation of these implementations is such that the bandwidth requirement is still considerably high due to sending high?quality video and images to the edge and cloud nodes for processing.

In [9]， image and video frames were divided into important premium blocks and unimportant regular blocks to save energy on IoT devices and provide high QoE to end users. A dynamic surveillance video stream was processed at fog node [10]； instead of sending a whole video frame， a sub?part of the original video frame was sent to the fog node to meet the real? time processing requirement. However， these approaches would be difficult to be used in surveillance systems since all the video frames need to be sent to the cloud for further investigations， and this requires high network bandwidth.

Authors in [11] introduced a concept of Quality of Contents （QoC） and proposed QoC based video encoding rate allocation scheme in mobile surveillance networks. This scheme allocates different data rate constraints to each camera node based on corresponding information and delivers video tasks to the remote cloud. Although QoC could save some bandwidth， transmitting video sequences directly to the cloud would lead to congestions and delays. Edge nodes could， therefore， be used to ease congestion delays to the cloud.

In [12]， an intelligent surveillance video coding technique was proposed. A background model was used to extract foreground objects and encoded in high quality while background frames were encoded with low quality. Although this approach could save some bandwidth， processing video locally at the camera node would cause computational delay due to limited network bandwidth.

A fuzzy?based approach that considers some internal and external parameters in order to define the sensing， coding and transmission configuration of visual sensors was proposed in [13]. In [14]， authors defined MIoT in 3 scenarios based on the use of multimedia content such as multimedia as IoT input and output， multimedia as IoT input， and multimedia as IoT output. The paper proposed a QoE layered model for MIoT applications， presented a use case related to the remote monitoring driving practices and conducted subjective assessments to measure the QoE. However， current QoE concepts and models might not be applicable in IoT M2M concept since no humans are involved in the cycle.

In this paper， a concept of AQoT with two use cases is proposed. The goal of AQoT is to meet the acceptable quality to fulfil or complete an IoT task “successfully”. For the meaning of “successful” completion of a task， it might be a 100% detection accuracy or might be 95% accuracy （or other values） depending on applications/scenarios. By meeting the acceptable quality， the system will avoid over?provisioning of multimedia quality. Hence， it will use less bandwidth without compromising its quality for other IoT devices and applications. A similar concept was used in [4] and was termed as an acceptable QoE. It aimed at increasing the number of users in a single eNodeB of an LTE cellular network for VoIP applications.

The approach of this paper is to process MIoT tasks such as human detection， face detection and license number plate recognition at the edge nodes. If further analysis is needed， results of these tasks or some tasks will be delivered to the cloud nodes.

3 Quality， QoE and QoT

Multimedia services over the Internet can be generally categorized as human?to?human （e.g. VoIP and video conferencing services）， human?to?machine （e.g. speech recognition and video recording/uploading）， machine?to? machine （e.g. surveillance camera/video to a server）， and machine?to?human （e.g. Internet video streaming） applications （Fig. 1）. If a recipient involves human as depicted in black lines in the figure， the concept of Quality of Experience （QoE） applies. Otherwise， if a recipient is a machine （including devices/things， data processes， as depicted in red lines in Fig. 1）， we will utilize the Quality of Things （QoT） concept instead.

In the QoT framework for acceptable QoT， a MIoT device such as a camera is used to monitor the surveillance area and sends the acceptable quality video streams to the near edge node for further processing depending on the task. If a task is not computational intensive （light task） e.g.， license plate detection or speed detection， the edge node will complete the task and send the detection results to the cloud node for post?processing and/or general management. If the task is complex （heavy task） e.g.， face detection or recognition， the edge node will share or distribute the task to other neighbouring edge nodes. If all neighbouring edge nodes are busy， the task will be offloaded to the cloud node.

These processes and communications are modelled in a layered architecture consisting of things， edge and cloud layers （Fig. 2）. Servers in the distributed cloud can host several IoT applications such as human detection， face recognition and license number plate recognition applications. Edge nodes can be grouped into domains associated with a set cloud nodes with particular applications or close proximity. IoT devices can also be grouped into domains with similar tasks such as temperature sensors and surveillance cameras. IoT nodes can process tasks locally and then transmit them to edge or cloud nodes. Edge nodes can process tasks from IoT nodes or share them with other edge nodes in the same domain. Edge nodes can also forward tasks from IoT devices to cloud nodes for further processing and analysis. Cloud nodes can process tasks from IoT devices or edge nodes and send feedback to IoT devices or edge nodes. Cloud nodes can also send feedback to a human if there is a requirement.

IoT applications will be residing in edge node [ej] and cloud node [ck] for j=1…J and k=1…K. The aim of the QoT is to fulfil the minimum requirements for these applications in order to effectively execute task [τij] or [τik]coming from IoT device i to edge node j or cloud node k without compromising the performance of the IoT system. Assuming that overall resources are the same such as computing and network resources， the optimization of an MIoT scheme will be to maximize the number of IoT devices that can be served， subject to acceptable QoT.

Fig. 3 depicts the scenario of a surveillance IoT system consisting of a surveillance camera as an IoT node， edge node， cloud node and an IoT application which can reside either in edge or cloud nodes.

If [τminij] and [τmaxij] are minimum and maximum requirements of a task from IoT node i to edge node j， respectively， then [aQoTij] is defined as an acceptable Quality of Things if [τminij]is achieved without compromising the performance of an IoT system.

If [τminik] and [τmaxik] are minimum and maximum requirements of a task from IoT node i to cloud node k， respectively， then [aQoTik] is defined as an acceptable Quality of Things if [τminik] is achieved without compromising the performance of an IoT system.

For MIoT， if a task is license number plate recognition or human detection， the requirement will be an image or video quality and the performance will be the recognition accuracy or detection accuracy. The minimum requirement for MIoT is taken as the minimum quality of an image or video parameter that an IoT system will still be able to effectively execute a task without compromising its performance. The minimum quality of a video task [τij] is the minimum bitrate [bminτij] or quantization parameter [qminτij] that an IoT system will still be able to effectively execute a task without compromising its performance.

For the license number plate recognition task， the recognition accuracy [rτij] will either be 1 or 0. 1 denotes that the license number plate is accurately recognized while 0 shows that the license number plate is falsely recognized. The purpose of an acceptable QoT concept for the license number plate recognition task is to achieve [rτij=1] or [rτik=1] at [qminτij] or [qminτik].

For the human detection task， the detection accuracy [dτij] is defined as the ratio of a number of recognized humans to the total number of humans in a frame. If [dτij=1]， this implies that all humans were accurately detected and if [dτij=0]， this implies that all humans were falsely detected， otherwise， [dτij=x]， for [0

4 Experimental Setup

As a proof of concept for QoT， two use cases were considered， human detection and license number plate recognition.

The platform for development and experiment was conducted in Ubuntu 16.04 Xenial. OpenCV 3.3.0 [15] on Python was used as an IoT application for coding human detection algorithm. Histogram Oriented Gradients （HoG） [16] was applied to detect humans in video frames. HoG is a feature descriptor which uses a global feature to describe a person. This approach trained a Support Vector Machine （SVM） for classification to recognize HoG descriptors of people， which is an effective human detection method.

In the human detection use case， one video sequence was used to demonstrate the concept of the acceptable QoT at which the detection task at minimum bitrate could still be able to accurately detect humans. The video sequence information is given in Table 1. A human video sequence was encoded with FFmpeg version 2.8.11 as H.264/MPEG?4 AVC at a bitrate from 800 kbit/s to 5 kbit/s and the human detection algorithm was deployed at each bitrate. There is a varying number of people in each frame as humans enter and leave a scene. The maximum number of humans in some frames is 5 and the minimum is 1. The snapshots for human detection frame and license number plate are illustrated in Figs. 4a and 4b， respectively.

For the license number plate recognition use case， Open Automatic License Plate Recognition （OpenALPR） [17] was used to recognise license number plate. In this use case， single number plate image （Car1） in Joint Photographic Experts Group （JPEG） format was used.

The image information is in Table 2. The Car1 image was taken close to the number plate in bright lighting conditions. The quality compression of Car1 video sequence was ranging from 90% to 1%. ImageMagick 6.8.9?9 was used to compress the JPEG images into different compression levels. The snapshot for Car1 image thumbnails is depicted in Fig. 4b.

5 Results and Discussions

For human detection， the detection accuracy is used as the performance metric to demonstrate the concept of an acceptable QoT. Detection accuracy is a ratio of accurately detected number of humans to the total number of humans in a frame. The detection ratio between 0.9 and 1 is considered accurate [18]. Three human detection video frames are selected for demonstration （Fig. 5）. Frame 1 has two people very close to each other from the camera point of view； frame 44 has two people who are far from each other and frame 102 has three people who are far from each other.

Fig. 6 depicts the human detection accuracy against the bitrate for frames 1， 44 and 102 of the 10 seconds video sequence. It can be observed that for frame number 1， the human detection rate of 1 ranges from 800 kbit/s to 35 kbit/s. If the bitrate is below 35 kbit/s， the detection accuracy is significantly reduced. The detection accuracy drops to 0 at 10 kbit/s.

For frame number 44， it can be observed that the human detection rate of 1 ranges from 800 kbit/s to 45 kbit/s. If the bitrate is below 45 kbit/s， the detection accuracy is significantly reduced. The detection accuracy drops to 0 at 5 kbit/s.

For frame number 102， it can be observed that the human detection rate of 1 ranges from 800 kbit/s to 30 kbit/s. The bitrate below 30 kbit/s， the detection accuracy is significantly reduced. The detection accuracy drops to 0 at 20 kbit/s.

Since detection accuracy of at least 0.9 is considered as accurate， instead of transmitting the original video sequence of 800 kbit/s over the Internet to another edge node or cloud node， bitrates ranging from 50-70 kbit/s of the same video sequence will be used for transmission. This range of bitrate is considered as minimal at which the IoT system could still be able to perform human detection without negatively affecting the detection accuracy. 50-70 kbit/s is what considered as an acceptable QoT. This has resulted in a saving of more than 10 times of the original bandwidth requirement.

The structural similarity （SSIM） index [19] is used to measure the similarity in perceptual quality between the original images and the degraded ones. The SSIM index values are depicted in Fig. 7 for each bitrate for frames 1， 44 and 102. The SSIM index values between 0.99 and 1 are considered to be excellent in terms of QoE. The values between 0.95 and 0.99 are considered as good while those between 0.88 and 0.95 are considered as fair. The values between 0.50 and 0.88 are poor and below 0.50 are bad [20].

50-70 kbit/s is an acceptable QoT for human detection task， however， SSIM index values for this range denote poor quality in terms of QoE. This is what differentiates QoT and QoE. Fig. 8 depicts the quality of frames 1， 44 and 102 at 50 kbit/s.

For the license number plate recognition task， the Car1 picture was taken close to the camera in bright weather conditions. The license plate number format was European based license plate. The recognition accuracy for the license number plate is 1 if the number plate is accurately recognized and is 0 if not. The recognition accuracy of Car1 is shown in Fig. 9 for each compression ratio. It can be observed that the recognition accuracy is still 1 at 5% compression of the original image. The original image quality was at 90%.

Fig. 10 illustrates the Car1 number plate sequence SSIM index values. It can be seen that although the video quality is considered poor （SSIM index value of 0.7） in terms of QoE at 5% quality level， the license number plate is still recognized. The range of 5%-10% quality level of Car1 can be considered as an acceptable QoT in this scenario of the license number plate recognition task.

Fig. 11 depicts the image size at each quality level of Car1. Since the acceptable QoT of Car1 is in the range of 5% and 10%， instead of transmitting the Car1 image in its original quality at 160 KB， the Car1 image of 40 KB at 5% quality level can be transmitted over the Internet with the same recognition accuracy of 1.

Fig. 12 depicts the Car1 number plate at 5% of the original quality. As per a human visual perception， 5% is considered poor quality， but for the QoT it can still be able to accurately recognize the license number plate.

Based on the results obtained in the described uses cases， it can be observed that the QoT is different from QoE because QoE involves human and QoT involves machine/applications. If the direct mapping is considered， the acceptable QoT for human detection and license plate number recognition tasks is less than the acceptable QoE which is 3.5 and outlined by the authors in [4].

The goal of QoT for M2M communications is to meet the minimum quality that an IoT object can meet the minimum requirement of an IoT application. It focuses on the minimum quality of multimedia data captured by the camera node to be processed and delivered by edge and cloud nodes. For M2H applications （e.g.， if a human being is an end user of an IoT application） the visual quality is needed for subjective viewing. Therefore， an acceptable QoE for multimedia data will be processed and delivered by the edge and cloud nodes. The ultimate goal of this study is to design such intelligent system to optimise network resources usage， so both AQoT and AQoE can be achieved depending on the use case scenario.

6 Conclusions

The IoT has been addressed as one of the biggest technological advances in the recent decades. IoT will soon be an inherent part of our daily lives ranging from smart homes， intelligent cars to aeroplanes and virtually everything we will interact with. With all the benefits that come with IoT， multimedia IoT comes with its own set of requirements such as power consumption， bandwidth usage and quality. This paper has defined a new concept， Acceptable QoT， whose experimental results have shown that it could significantly reduce bandwidth usage to fulfil IoT tasks without compromising the performance of the IoT system. Future work will be to develop intelligent IoT systems which can deliver multimedia IoT services to human or machine according to QoE and QoT automatically over edge/cloud integrated networks.