Enabling Rapid Experimentation of Contextual Network Traffic Management using SDN

The non-cooperative and unsupervised resource competition between adaptive media applications (such as Youtube and Netflix) leads to significant detrimental quality fluctuations and an unbalanced share of network resources. Therefore, it is essential for content networks to better understand the application and user-level requirements of different data flows and to manage the traffic intelligently. I am glad to have been part of a team of talented researchers which was one of the first to experiment Software defined networking (SDN)-assisted QoE-aware network management using physical OpenFlow network switches. SDN is a network paradigm that decouples network control from the underlying packet forwarding. Combined with Fog Computing and Network Function Virtualization (NFV), this opens compute locations that are close to the edge to enable intelligent network traffic management services (I also name this cognitive networking).

Following the publications [1,2,3] we made, there have been numerous requests to open-source our experimentation environment (named REF – Rapid Experimentation Framework) from the research community.  REF is an experimentation framework and a guide to building a testbed that together provides a blueprint for an SDN-based contextual network design facility. Contrasting to existing facilities that typically provide very detailed low-level control to just the network infrastructure, our work provides higher level abstractions of both the network and virtualisation infrastructures through orchestration, automating the creation, connection, running, and cleaning of nodes in an experiment. REF also provides abstraction over the network for making the creation of context aware traffic management applications as streamline as possible. Additionally, with the unique configuration using slicing and port multiplexing, REF can create much larger physical networks with limited hardware than its competitors. Finally, the entire REF framework can be used and modified by anyone without any kind of registration or subscription to a federation.

Needless to say, to “open-source” a framework is not a straight-forward task. Our source codes are pretty much meaningless if they are not well connected with well-configured hardware equipment and a comprehensive guideline of do’s and don’ts. We wanted to publish this tutorial-style guideline in an elite outlet (so more people can benefit from it) while keeping the writing style suitable for SDN beginners, and there is nothing more suitable for our work than the IEEE Communications Magazine.  Furthermore, because we are using HPE’s network switches (3800 and later 3810 series) as reference equipment (and we know for sure that the implementation of standards (such as OpenFlow) by the vendors is a deterministic factor), we must work with HPE to make sure our analysis and conclusions are accurate. Fortunately, Bruno Hareng, an SDN and Security Solution Manager at HPE, provided invaluable input to our work.

Framework for rapid SDN experimentatin 

The manuscript is set to describe the framework (shown in Figure above), covering the requirements of the framework then the purpose of each component within the system as well as the abstractions that it provides to the user. Next, the experiment testbed is detailed, providing a guide on how to construct your own virtualisation and network infrastructure for experimentation. After this, both use cases are described and used to show REF in operation, this includes a Quality of Experience (QoE)-aware resource allocation model, and a network-aware dynamic ACL. Finally, the article goes into a discussion on interesting findings that arose during the creation and use of the system. The manuscript is now accepted by IEEE Communications Magazine for publication in a July 2017 issue:

Fawcett, L., Mu, M., Hareng, B., and Race, N., “REF: Enabling Rapid Experimentation of Contextual Network Management using Software Defined Networking”, in IEEE Communications Magazine, 2017


Online video streaming is becoming a key consumer of future networks, generating high-throughput and highly dynamic traffic from large numbers of heterogeneous user devices. This places significant pressure on the underlying networks and can lead to a deterioration in performance, efficiency and fairness. To address this issue, future networks must incorporate contextual network designs that recognise application and user-level requirements. However, designs of new network traffic management components such as resource provisioning models are often tested within simulation environments which lack subtleties in how network equipment behaves in practice. This paper contributes the design and operational guidelines for a Software-Defined Networking (SDN) experimentation framework (REF), which enables rapid evaluation of contextual networking designs using real network infrastructures. Two use case studies of a Quality of Experience (QoE)-aware resource allocation model, and a network-aware dynamic ACL demonstrate the effectiveness of REF in facilitating the design and validation of SDN-assisted networking.


[1] Mu, M., Broadbent, M., Hart, N., Farshad, A., Race, N., Hutchison, D. and Ni, Q., “A Scalable User Fairness Model for Adaptive Video Streaming over SDN-Assisted Future Networks”, in IEEE Journal on Selected Areas in Communications. 34, 2168-2184, 2016. DOI: 10.1109/JSAC.2016.2577318

[2] Fawcett, L., Mu, M., Broadbent, M., Hart, N., and Race, N., SDQ: Enabling Rapid QoE Experimentation using Software Defined Networking, to appear in IFIP/IEEE International Symposium on Integrated Network Management (IEEE IM), Lisbon, Portugal, 05/2017

[3] Mu, M., Simpson. S., Farshad. A., Ni. Q., and Race. N., User-level Fairness Delivered: Network Resource Allocation for Adaptive Video Streaming (BEST PAPER AWARD) in Proceedings of 2015 IEEE/ACM International Symposium on Quality of Service (IWQoS), Portland, USA, 06/2015

Breaking the filter bubble

A collaboration with the data mining group at TU-Berlin and folks at Lancaster and Glasgow has seen a full paper accepted by ACM International Conference on Interactive Experiences for Television and Online Video (TVX 2017), Hilversum, The Netherlands, 06/2017. The acceptance rate is 31%, a competitive year for this conference series.

The paper describes our recent efforts in breaking the filter bubble, a term used to reflect the phenomenon that a recommendation algorithm guesstimates a user’s preference from limited contextual information (such as user clickstream data) and only provides the user with a very small selection of content based on the preference. A side-effect of such an approach is that it often ends up isolating a user from (a large amount) of content that the system does not believe would interest him or her. As a user selects from within the bubble, the bubble may also become smaller and more “specialised”, causing a negative cycle. We believe that the recommender should be smarter than it is and “talk” to its users as their friend. A friend who knows what you like and yet very often surprise you with new and cool things. We studied this contextual bias effect in an online IPTV system (to which I was a project lead for some years), and developed a novel approach to re-balance accuracy and diversity in live TV content recommendation using social media.

Yuan, J., Lorenz, F., Lommatzsch, A., Mu., M, Race, N., Hopfgartner, F., and Albayrak, S., Countering Contextual Bias in TV Watching Behavior: Introducing Social Trend as External Contextual Factor in TV Recommenders, to appear in 2017 ACM International Conference on Interactive Experiences for Television and Online Video (TVX 2017), Hilversum, The Netherlands, 06/2017

Context-awareness has become a critical factor in improving the predictions of user interest in modern online TV recommendation systems. In addition to individual user preferences, existing context-aware approaches such as tensor factorization incorporate system-level contextual bias to increase predicting accuracy. We analyzed a user interaction dataset from a WebTV platform, and identified that such contextual bias creates a skewed selection of recommended programs which ultimately leaves users in a filter bubble. To address this issue, we introduce a Twitter social stream as an external contextual factor to extend the choice with items related to social media events. We apply two trend indicators, Trend Momentum and SigniScore, to the Twitter histories of relevant programs.The evaluation reveals that Trend Momentum outperforms SigniScore and signalizes 96% of all peaks ahead of time regarding the selected candidate program titles.

Next Generation Internet: what’s next?

The EC’s NGI group recently published the final report of their open consultation for next generation Internet. The report identifies seven Technology Areas (TAs), which are believed to have pivotal roles in future Internet. We shouldn’t be surprised to see the TAs being bound by current FP7/H2020 or RCUK programmes as the respondents of the report wish to continue evolving their work in those programmes. The vast majority of the respondents come from research institutions, civil society, and SME while only 47 out of 449 are linked to industry. This is not to say that the conclusions from the report are far from realistic. Many initiatives, old (the Internet) or new (OpenFlow), stemmed from projects at research institutions.

I can see many connections between the seven TAs and my research in software-defined cognitive networking and immersive media. Having said that, is there any researcher in the area of computing and communications, whose work doesn’t cover multiple of these TAs? Is there any ICT research today that doesn’t consider data, network, and people as a whole? It seems that nearly the entire community envisages NGI as a super-intelligent, self-programming, and human-caring thing or things. There are, of course, brave ones who think differently. I vividly recall an ex-colleague of mine once saying that he contributes his success in networking research to “focusing on moving every single [network] package as fast as possible and nothing else”. Not many would think like that today…

TA 1 Discovery and identification tools

One of the premises of the Internet of Things is that devices around us will be partly physical and partly digital, with a vast majority of those devices being “headless”, lacking buttons, screens and other means by which the user interacts with the device. This premise forces us to figure out ways to discover, identify, and interact with the objects, devices and services in our lives in a seamless way, as well as ways to be made aware of the connected devices that surround us at any given moment.

TA 2 New forms of interactions and immersive environments

Increased computing, transmission power and next generation of devices (enabled by micro-nano-bio technology) allows conceptualizing new forms of interactions with machines and immersive environments that will have an impact in our professional and private life. New challenges are raising related to augmented and virtual reality, behaviour, human-computer interactions, haptics, human-human interactions through computers, machine-to-machine, spatial recognition and geographic information systems.

TA 3 Personal data spaces

Personal data is everything that identifies an individual, from a person’s name to telephone number, IP address, date of birth and photographs. The next generation Internet aims to develop technologies to help us achieve greater control of our personal data, knowing what is being shared and with whom.

TA 4 Distributed architectures and decentralized data governance

Distributed open hardware and software ecosystems are capable of supporting decentralised data management (so that each piece of user-generated information remains under the full control of the entity who generated it, and is subject to on-demand aggregation by third parties), leveraging on decentralised algorithms based on blockchains, distributed ledger technology (DLT) or peer-to-peer (P2P) technologies.

TA 5 Software-defined technologies

There is an evolution towards software-defined technologies. These may provide more functionalities and control for the allocation of resources, configuration and deployment, and may open new opportunities to develop the Internet.

TA 6 Networking solutions beyond IP

The current internet has certain limitations derived from its protocols that were developed in the 70’s, like the transmission control protocol/internet protocol (TCP/IP) and its limitations on mobility, IP address management and task limitation. Quality of Service (QoS) is another problem derived from TCP/IP, which is a problem generated by the inherent nature of networking technologies and the focus on pumping data from point A to point B as fast as possible without focusing on how the data is sent. The internet of the future should be able to overcome these limitations.

TA 7 Artificial Intelligence

Artificial intelligence will also change the Internet. Inspired by how the human brain works, mathematical models can learn discrete tasks by analysing enormous amounts of data. So far, machines have learnt to recognize faces in photos, understand spoken commands, and translate text from one language to another. But this is only the beginning. Artificial Intelligence will greatly sharpening the behaviour of any online services and be core technical enabler of the future Internet.

Overton, David., Next Generation Internet Initiative –  Consultation, https://ec.europa.eu/futurium/en/content/final-report-next-generation-internet-consultation-0, 2017

VR. ready, steady…

Transforming online learning experience using virtual reality and gamification


Nothing cheers you up more than a new gadget in the middle of a term. Oculus Rift, controllers and earphones (Thank you, Nick!) The 60+mph wind gusts (Storm Doris) nearly took them off my arms in the car park, but I managed.

We are still expecting a Google Pixel+Daydream and a(n?) FOVE (with eye-tracking capabilities) to arrive.

View original post

Packet probing for media synchronisation

[A piece of work with Hans Stokking from TNO]

Cross-device immersive media has been one of my main research topics for many years. To get the “immersion”, we inevitably need a mechanism to orchestrate media playback on different devices. This sounds easy but very often very hard to do in practice, especially over different devices (iPad, Smart TV, PI…). One challenge is the delay. When we send commands (such as “START PLAYING”) to another device, your application takes some time to code and pack the information into manageable chunks, which then take some time sitting in a queue and waiting to be serialised as packets for network distribution. Will your packets then travel at the speed of light? Not quite, yet still at 177,000 km/s (59% the speed of light) in Ethernet cables. But the Internet is not an empty highway, your packets will “bump into” others at network nodes such as routers or switches, which simply means they’ll probably sit in the queue again and wait for their turn. When the packets finally arrive at the receiver, they must then go through the network interface card, driver, TCP/IP stack, and any application that control the media playback, before your command can be executed… The whole process, after all the waiting, may only take a few hundred milliseconds. Not bad. OK, let’s make things easier for ourselves and put all the device very close to each other on the same network, then the overall delay is probably just under 100 milliseconds (1/10 of a second). Sounds good, right? Surely, it’s ok for one speaker to lag the other for 100 milliseconds???

What did our recent research say about the human perception of latency in an inter-destination audio-visual test?

20-40 milliseconds perceivable.
60-100 milliseconds annoying.


If you are thinking “No worries, let’s actively measure all those delays (serialisation delay, queuing delay, propagation delay, processing delay, reply delay, etc.) which our messages encountered and compensate them when we execute the command.”, then you are with the group of “mad” researchers in the MediaSync community, who’ve spent many years investigating the sources of Internet delays and the feasible ways to measure them.

To give an overview of different packet probing methods to estimate delay and available bandwidth (and to show how such measurements can really make or break a cool media application), Hans and I are now finalising a manuscript for a Springer book chapter.

It looks like we are going over the 25-page limit though…

UoN Waterside campus in VR?

Screen Shot 2017-02-10 at 16.21.37.png
Creative hub, Waterside campus, The University of Northampton  (All rights reserved)

In the past few months, I have been working with a few colleagues, including Dr Anastasios Bakaoukas and Ewan Armstrong, at Computing on initialising a VR project. The idea is simple: build our new Waterside Campus (due open in 2018) in virtual reality using game development engine so we can all (virtually) walk around on our new site before it’s fully completed in the physical world. So why have we volunteered to do this?

  1. Because we can! Our Game Development/Arts/Design programmes are strong and fast growing. We have expertise in modelling, artistic design and artificial intelligence for developing immersive games.
  2. It can potentially help the University to promote our infrastructure/facilities at the new site to prospective students. To this end, we have worked with the marketing team to understand their needs. The tool could also help improving the visitor/student experience once we move to the new campus.
  3. Using the new campus as the case for teaching. Students can drop their designs or game logics directly into this VR environment and test their work in a unique context.
  4. It will be a great platform for media, AI, and traffic analysis research.

Despite an enormous workload on teaching and marking, the team has worked with an external media company and has committed many many hours transform 3D models for the gaming environment (special thanks to Ewan!). We hope to deliver some interest results very soon!

BTW, if you like one of those “cut-in-half” arts, here are some of my takes on the Creative Hub:

Taking a small slice off and you can see some networking space on the left and entrances to blocks of lecture theatres on the right:

Creative Hub, Waterside campus, The University of Northampton (All rights reserved)


Cutting it further:

Creative Hub, Waterside campus, The University of Northampton (All rights reserved)


It looks like there is a standalone building wrapped inside…

Creative Hub, Waterside campus, The University of Northampton (All rights reserved)

And here is the Engine Shed, a Grade II listed building currently being restored and it will be the home of the Student Union.


Screen Shot 2017-02-10 at 20.18.48.png
The Student Union, Waterside campus, The University of Northampton (All rights reserved)