802.11 Channel Utilization

I was recently asked about a product that requires 90Mbps (bits per sec) of throughput. With a Gig link, no problem. If the link is 100Mbps, technically it could work but that is a little beyond my engineering safety factor of 20%, so about the most I would want to see on a link is 80% of the line rate, or 80Mbps. But if we put in all Gig links, it should be fine.

Then they broke the news that what they really want is to be able to go wireless, too. That's a lot of throughput for wireless; even with new devices that claim super speeds:

https://www.linksys.com/ru/p/P-EA9500/ “Wi-Fi speeds up to 5.3 Gbps*“

Trying to find the asterisk note at the bottom of the page (and not singling out Linksys for any particular reason as all the commodity product vendors do similar things):

*The standard transmission rates–1000 Mbps or 2166 Mbps (for 5 GHz), 1000 Mbps (for 2.4 GHz), 54 Mbps, and 11 Mbps–are the physical data rates. Actual data throughput will be lower and may depend on the mix of wireless products used and external factors.

So you don't really get 5.3Gbps… these are devices that have three radios

Tri-Band (5 GHz + 5GHz + 2.4 GHz)

And the speeds listed are datarates, and then they sum the maximum datarate on each radio:

Wi-Fi Speed: AC5400 (N1000 + AC2166 + AC2166)

I ask: what type of WiFi system do you have available? Turns out the product will have an 802.11 abgn 1x1:1 radio, that supports 20MHz & SGI (short guard interval). So now we know some capabilities. From http://mcsindex.com/, we know the maximum datarate this device can support is 72.2Mbps:

The description tells us:

1. 802.11a means device can utilize the 5GHz band. Exactly how many channels that can be used depends on a number of factors: regulatory domain (i.e. US uses FCC, Europe has ETSI, etc), whether DFS channels are supported, etc. So some information here, but not the complete picture. See for more details https://en.wikipedia.org/wiki/List_of_WLAN_channels.
2. 802.11bg means the device works on 2.4GHz. There is less room for interpretation here, but it still does depend on the regulatory domain. For example, FCC provides for channels 1-11, where three are practically usable simultaneously in an infrastructure deployment (the so called three channel plan, channels 1/6/11). But other regions get additional 2.4GHz channels for use.
3. 802.11n means that the device uses enhancements to both 2.4 and 5GHz bands for performance improvement. Note that n is not a band, but a series of performance-enhancing features. For practical reasons, those enhancements do more for 5GHz channels than 2.4GHz, but even 2.4GHz operation benefits from these enhancements. To see all of the specific performance capabilities of n, look at the MCS Index table for the HT (high throughput) Index values. Note that there are many; indeed, there are many options and different levels of support so just saying n support does not provide a lot of detail as to the maximum datarate capability of the device. In this case, the device supports 1 spatial stream, 20MHz channel bandwidth, and SGI=400ns (so short guard interval) and a maximum MCS Index of 7, so maximum datarate is 72.2Mbps. One popular device used in the WiFi diagnostic industry claims

802.11 a/b/g/n packet injection at all rates

yet does not support SGI, so how could it support all rates?

So inevitably, after showing the MCSIndex table, the gap to close is 90 Mbps required Vs. 72.2 Mbps capability. But not so fast… who is getting 72.2Mbps?

“But you just showed us the table and told us we can do 72.2!”  they say.

Not really. There are liars, damn liars, and 802.11 WiFi engineers. No one gets real throughput that matches the datarate. Let's be clear on the choice of words: datarate is the rate at which data can be transferred. Throughput is the actual amount of data of that is transferred; make no mistake - users care about throughput:

“How fast can I download the file?”

“The Internet seems slow today”

“Why is my network game so slow?”

“This Youtube video keeps stopping!”

Of course there is some relation: datarate would be the upper bound of throughput and the confusion stems from this fact; in the wired world, in general, throughput generally equals datarate.

Let's try iperf between two hosts, wired, with Gig links. All switched network:

user@host:~$ iperf3 -c 192.168.20.21 -f m -i 1 -t 10

Connecting to host 192.168.20.21, port 5201

[ 4] local 192.168.20.14 port 41333 connected to 192.168.20.21 port 5201

[ ID] Interval Transfer Bandwidth

[ 4] 0.00-1.00 sec 113 MBytes 944 Mbits/sec

...

[ 4] 9.00-10.00 sec 113 MBytes 949 Mbits/sec

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval Transfer Bandwidth

[ 4] 0.00-10.00 sec 1.10 GBytes 949 Mbits/sec sender

[ 4] 0.00-10.00 sec 1.10 GBytes 948 Mbits/sec receiver

So throughput here is about 950Mbps for a link speed of 1000Mbps, so about 95% of line rate, where line rate is datarate. In this case, throughput is basically the same as datarate. There is still some overhead when using TCP so we would never reach 100% for actual data throughput. The link may be saturated, but due to overhead when using TCP/IP, it's not all available for data. For full size frames with an assumed MTU of 1500bytes (so max frame size is 1514, which is MTU+Ethernet Header),

TCP/IP Frame Component	Size [bytes]
Ethernet	14
IP (no options)	20
TCP (no options)	20
Data (TCP MSS)	1460

We can calculate the efficiency. However, it's not that important for this topic as 95% of line rate is good enough for our needs at this point to assume throughput=datarate.

However, this is not always the case in the wired world. It's very common, but sometimes we can run into products where the bottleneck is upstream of the network interface, so the maximum throughput could be less than the datarate. For example, take an iMX6 maker board. There are several manufacturers, for example,

• https://www.wandboard.org/
• https://www.udoo.org/udoo-dual-and-quad/

The errata (https://www.nxp.com/docs/en/errata/IMX6DQCE.pdf) for the CPU includes a note regarding maximum throughput:

ERR004512

Description: The theoretical maximum performance of 1 Gbps ENET is limited to 470 Mbps (total for Tx and Rx). The actual measured performance in an optimized environment is up to 400 Mbps.

And indeed, using flowcontrol to even attain these speeds on the infrastructure switch, we can get for the forward and reverse directions:

user@host:~$ iperf -c 192.168.20.21 -f m -i 1 -t 10

------------------------------------------------------------

Client connecting to 192.168.20.21, TCP port 5001

TCP window size: 0.02 MByte (default)

------------------------------------------------------------

[ 3] local 192.168.20.224 port 35568 connected with 192.168.20.21 port 5001

[ ID] Interval Transfer Bandwidth

[ 3] 0.0- 1.0 sec 46.8 MBytes 392 Mbits/sec

...

[ 3] 0.0-10.0 sec 501 MBytes 420 Mbits/sec

user@host:~$ iperf -c 192.168.20.21 -f m -i 1 -t 10 -R

------------------------------------------------------------

Client connecting to 192.168.20.21, TCP port 5001

TCP window size: 0.02 MByte (default)

------------------------------------------------------------

[ 3] local 192.168.20.224 port 35569 connected with 192.168.20.21 port 5001

[ ID] Interval Transfer Bandwidth

[ 3] 0.0- 1.0 sec 47.8 MBytes 401 Mbits/sec

...

[ 3] 0.0-10.0 sec 504 MBytes 423 Mbits/sec

So we see in this case that we have a link speed of 1Gig, so datarate is 1Gbps, but maximum throughput is somewhat less. We know we do not have a 100Mbps link, or the max would be much lower than what is observed. This validates the errata from the chip manufacturer, as expected, and we match the advertised performance. However, this is an edge case. It does come up, probably most notably in the consumer space when adding an Ethernet interface via USB. If USB 1.1 or 2.0, these speeds can be much less than 1Gbps, so we would run into the same problem as the data flow limitation will come from the USB bus, not the network interface datarate.

So where does that leave us with WiFi and 802.11? It turns actual throughput will be MUCH less than sticker (i.e. datarate) for three reasons:

1. Sticker performance will always be the maximum possible datarate that is physically possible from the system, and includes best case. Best case in this context means fantastic signal to noise ratio (SNR). So basically if you are in sealed chamber, and are sitting on top of the access point (AP), this will be the datarate. It's not really that bad, but the maximum requires a very healthy SNR and datarate will fall as one moves away from the AP.  5GHz does not travel as far as 2.4GHz signals, so will drop faster for a given distance change than would be observed if using 2.4GHz.
2. Protocol overhead is significant in the 802.11 world to improve robustness;  one of the biggest issues with wireless is the pervasive issue of packet loss. In the wired world, especially on LANs, packet loss is minimal from the network (not always true, but usually on a good network). However, wireless networks are transient as things come and go that affect the RF environment so we have to manage this packet loss as well as provide for other signaling capabilities. For example, in general, every unicast frame is ACKd per the 802.11 protocol which takes time and bandwidth. Also APs send beacons and other control and management frames either per schedule or as needed to maintain the functioning of the network (e.g. beacons, probe responses, RTS/CTS, etc). All of this extra traffic takes bandwidth away from clients trying to send data.
3. Finally, the RF environment used by WiFi on a given channel is shared. It's inherently half duplex: if one station is communicating, all others must defer; only one gets the network at a time. Obviously, the more stations that are trying to communicate, the less time available for any one station to make use of the network to transmit data. This is evaluated as channel utilization but can be a somewhat elusive number; in some ways it can be useful to think of channel quantity as time; there is only so much available time to access the channel and when it is gone, it is gone. So we want to optimize what is done during that time; the more we can transfer in a given period of time, the more we can get out of the limited resource available.

So what can we get out of a given WiFi radio in a particular environment? What types of throughput are possible? What is the limiting resource that we have to manage?

In evaluating the maximum throughput, we can look at our three reasons of why we don't get sticker performance and see if we can fix some of them. Those that are left we can then design some experiments to observe and evaluate the behavior.

For SNR, we can fix the test devices in known locations ensuring a healthy SNR so that communications will typically be at highest possible datarates. For protocol overhead, there is not much to do as the protocol is what it is: it behaves in a particular way, per the standard, and that is it. We will just try not to make design choices that would exacerbate this effect: use typical best practice settings. For instance, beacon intervals are typically at just about100ms; we could manipulate the amount of protocol overhead in a number of ways:

1. Decrease the beacon interval, so more beacons are present for a given time period. This also affects the channel utilization as more frames means higher utilization.
2. Decrease the datarate of the control and management frames: for a fixed size frame, a lower datarate will take more time to transmit. This takes time away from other stations that want access to the shared RF resource.
3. Add additional SSIDs, where each SSID will send a beacon at the given datarate. There are spreadsheets available on the Internet which caclulate the overhead due to beacon datarate and number of SSIDs (http://revolutionwifi.blogspot.com/p/ssid-overhead-calculator.html).

However, if we fix these items to what best practice might be, this can eliminate some factors in our evaluation. So we can plan to:

1. Leave beacon timing at default
2. Use a channel with few other interfering devices
3. Use a datarate for management and control frames that is consistent with a high density design, so we want high datarate here (the physics of it shows that higher datarate frames do not travel as far, and for high density designs, we want small, high performance WiFi cells). On 5GHz, lets choose 24Mbps by setting the lowest mandatory rate to 24. All other 802.11a rates are supported.

So the last factor to evaluate is channel utilization. This is in fact the precious resource that needs to be managed; there is only so much, and we really want to optimize the time that we have when accessing the resource to maximize the throughput.

Thought experiment – in a vacuum, compare the following scenarios:

1. Sending 100Mb at 1Mbps datarate – expect that it might take 100sec to transfer 100Mb (bit, just so numbers are easy to calculate). What would channel utilization be?
2. Sending 100Mb at 100Mbps datarate – expect that it might take 1sec to transfer 100Mb at this datarate. This is obviously better than the 1Mbps datarate case as it goes much faster. So what would channel utilization be in this case?

For case 1, since the station is transmitting for 100sec, the network is blocked the whole time (ignoring other network effects for simplicity). So the channel utilization will be 100% for this 100sec; there is no time left for other devices to do anything.

For case 2, only while station is transmitting will the utilization be counted; so for the 1sec it is 100%, then for the other 99sec, it would be 0%, and we could do a weighted average and show that it is much lower, on average, than 100%.

I often hear for low throughput requirements: “we don't need 802.11n, or 802.11ac – we don't move a lot of traffic”. It's not really true; using the higher datarate modulations available from 802.11n/ac allow for more optimum use of the channel; though throughput requirement may be low, a high datarate allows for channel access, transmission, then release of the channel faster, allowing more time for others to access the channel and do useful work.

A couple of points:

1. It's tough to get 100% channel utilization; maximum usable appears to be about 85% when in heavy operation. Values higher usually indicate some type of problem. Values this high will starve out stations as well, so operating even close to hear will be problematic.
2. It would be nice to trend channel utilization over time on all channels to see what is happening. In the event of an issue, we could look here first to see if heavy utilization correlates to the problem.

For a simple test, we can use the beacons to tell us channel utilization if the QBSS load element is available:

Experiment: how does actual throughput and channel utilization change as we vary datarate?  We can vary the datarate by changing the supported MCS Index values at the AP; since wifi client and AP communication is a sort of negotiation (i.e. both sides announce what they support, and the highest common parameters are usually chosen).  We can use a laptop for test with an Intel 7265 wifi radio; this supports 802.11abgn-ac 2x2:2 adapter and we will work on a 5GHz/20MHz channel.  The chipset will actually do 40 and 80MHz bandwidth, but for testing, we will limit to 20MHz so we can easily see the effect on channel utilization on a single channel.

The MCS Index table for support in this case: 144.4Mbps should be max with 802.11ac capability disabled (Intel 7265). Wireless client is iperf3 test client (so TCP client, and sends data) and report read receiver values (they do differ, slightly - due to packet loss maybe?)

admin@kali:~$ iperf3 -c 192.168.30.21 -f m -i 1 -t 10

Connecting to host 192.168.30.21, port 5201

[ 5] local 192.168.30.157 port 38148 connected to 192.168.30.21 port 5201

[ ID] Interval Transfer Bitrate Retr Cwnd

[ 5] 0.00-1.00 sec 5.63 MBytes 47.2 Mbits/sec 1 269 KBytes

[ 5] 1.00-2.00 sec 13.7 MBytes 115 Mbits/sec 0 570 KBytes

[ 5] 2.00-3.00 sec 13.5 MBytes 114 Mbits/sec 0 706 KBytes

[ 5] 3.00-4.00 sec 13.8 MBytes 115 Mbits/sec 0 706 KBytes

[ 5] 4.00-5.00 sec 12.5 MBytes 105 Mbits/sec 0 748 KBytes

[ 5] 5.00-6.00 sec 12.5 MBytes 105 Mbits/sec 0 783 KBytes

[ 5] 6.00-7.00 sec 5.00 MBytes 41.9 Mbits/sec 41 256 KBytes

[ 5] 7.00-8.00 sec 12.5 MBytes 105 Mbits/sec 0 554 KBytes

[ 5] 8.00-9.00 sec 12.5 MBytes 105 Mbits/sec 0 621 KBytes

[ 5] 9.00-10.00 sec 13.8 MBytes 115 Mbits/sec 0 738 KBytes

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval Transfer Bitrate Retr

[ 5] 0.00-10.00 sec 115 MBytes 96.8 Mbits/sec 42 sender

[ 5] 0.00-10.04 sec 112 MBytes 93.3 Mbits/sec receiver

To get at RSSI we want to look at both sides as we need frames to go back and forth; if the AP power is much higher than the wireless client, then it's entirely possible for the client to continue to hear the AP, but the AP cannot hear the client which causes roaming problems as roaming is typically a client decision.  Buying a more powerful AP for better coverage is not always the answer; actually turning down the AP power to balance with the clients is often a better design decision.  From the client we can get RSSI signal strength from the AP:

admin@kali:~$ iwconfig

wlan0 IEEE 802.11 ESSID:"LWAPV6"

Mode:Managed Frequency:5.825 GHz Access Point: A0:E0:AF:4E:B0:6F

Bit Rate=144.4 Mb/s Tx-Power=22 dBm

Retry short limit:7 RTS thr:off Fragment thr:off

Power Management:on

Link Quality=66/70 Signal level=-44 dBm

Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0

Tx excessive retries:0 Invalid misc:1 Missed beacon:0

The Cisco AP will give us the signal strength of the wireless client from the AP point of view:

It's also common that AP power is greater than wireless client power, so the client RSSI as measured at the AP will likely be the minimum of the two values.  However, there is no guarantee of this being true.

The Intel client will use the highest datarate available, legacy or HT, not necessarily the highest MCS Index rate.  Therefore, to configure the test let's set as follows:

For a typical high density deployment, I would likely choose different values here.

Results

To interpret, we can see that as MCS Index increases, so does the actual throughput (red squares). Indices 8-11 show some abnormality, to be explained. Notice, though, the channel utilization is not very sensitive, at all, to the actual throughput or the datarate in use; it hovers just over 90% for all tests, regardless of the actual datarates in use.

To explain the issue with indices 8-11, we can see that the maximum projected datarate actually falls. Reference the MCS index table; indices 0-7 for HT (i.e. 802.11n) are for single stream; when we move to indices 8-15, these are two spatial streams. But note that Index 8 has a maximum datarate of 14.4 Mbps; this is well below index 7 value of 72.2Mbps. What is observed via an OTA capture is that the client (in this case an Intel 7265) seems to choose the highest datarate available, not necessarily the highest MCS Index. So until the two stream datarate exceeds the single stream, the maximum single stream rate of 72.2Mbps is selected for transmission. Note that this behavior could be very chipset and version dependent. Other systems could behave very different; it appears that the Cisco AP utilizes the maximum MCS Index as part of the rate selection algorithm. Since the bulk data transfer is upstream with wireless as a client to a wired server, the maximum throughput is heavily dependent on the rate selection of the client, in this case the Intel chipset. Perhaps this explain the slight increase in throughput as MCS Index increases from 8 to 12; the client would be fixed in Tx datarate (MCS 7 single stream datarate of 72.2) but the AP increases it's datarate. The last datapoint is with 802.11ac rates enabled; in this case, it would be 2SS (spatial stream) VHT Index 8 for a datarate of 173.3Mbps.

Observe also that actual iperf throughput (in TCP mode) is always below the datarate; due to the extensive amount of control and management traffic, which uses a rate selection of relatively low basic rates so it's difficult to match the actual throughput with datarate. Note, also, that this implies a number of things when trying to capture traffic: it's very unlikely you won't see the client at all; if so, it's likely not due to a modulation mismatch but more likely that the capture is for the wrong channel, or the adapter does not support promiscuous mode, SGI not supported, etc. The ACKs/Block ACKs are at low datarates so would be picked up even if the data frames (usually QoS-Data) might be missed due to too high a modulation, but evidence will still exist for this MAC address in the form of control traffic.

So we can see that we can get more real throughput with higher datarates, but the channel utilization is the same: we consume channel time (the precious resource to conserve) when transmitting so the higher the datarates in use, the more real throughput is available for a given amount of channel utilization.

Some ideas as to go forward:

1. Datarate is usually not artificially limited in this way by configuration; this is a test.  What really happens is the SNR is reduced due to distance or other obstructions in the RF path.  This will cause a reduction in throughput as well.
2. It would be nice to get a rollup of channel utilization, on all channels, as part of a comprehensive data collection initiative that we might get with a typical network manager.  Something like this:

Graphs for all channels would be useful.

Packet Generation at Layer 2

Packet generation / reception

Looking at different ways to generate the three basic types of layer-2 Ethernet traffic on a wired network:

1. Unicast: destination MAC address to a specific NIC
2. Multicast: destination MAC address starting with 01:00:5e
3. Broadcast: destination MAC address that is ff:ff:ff:ff:ff:ff

I found some tools that work well to send/receive various types of traffic.

 

 Background

Working on some course material for Wireshark, being able to create these three traffic types is useful to test various concepts related to

1. Promiscuous mode
2. IGMP snooping
3. Switching/bridging
4. Hubs
5. Broadcast domains
6. How CPU load varies with traffic type
7. OS and driver differences

 

Some tools

I categorize packet creation utilities into two broad categories - those that are raw packet generators, and those that utilize host services.  For instance, some tools will just create a raw frame and put it on the wire, so all fields must be defined.  This is an example of what a raw packet generator might do.  There are no OS defaults, as their really is no interaction with the host OS.  For instance, the typical programmer might decide to send data to another host, and would select that host by either hostname or IP address.  Even if a host name is used, DNS (or hosts file) will be used to lookup the IP address, so in either case, data is sent to an IP address.  However, with the next hop model of Ethernet and TCP/IP, to actually forward this frame along to another host some MAC address is required, so the sending host OS will lookup the destination IP in it's route table and determine the next hop.  To find the MAC address of the next hop, the sending host OS will usually use ARP to determine the MAC for a given IP address.  With the raw packet generation tools, none of this support infrastructure works - there is no DNS lookup, no route tables to consult, no ARP protocol with cache support, etc.

Contrast this with other tools that utilize some of the host OS's services.  Many of these are for packet generation, but will utilize host services to varying degrees to actually send and/or receive frames.  This includes use of ARP and the ARP cache, the host route table and gateways, etc.  

These lists are not exhaustive; they are just some of the tools that I have used with varying degrees of success.  Indeed, a google search will yield MANY options to send test data on a network for testing.

 

Pure Packet Generators

Ostinato - this is nice, and works on Windows and Linux.  Creates custom packets and can define streams to send.  Obviously limited to system resources, so really happy frame rates are difficult to sustain depending on PC hardware.  Concept of interface selection is not that obvious, either.  I have only used with the GUI, not sure if anything else is available.  Supports sending packets from a pcap file, but struggles with a file of any real size.  

packETH - similar to Ostinato, but looks native to Linux (there may be ports, but don't know how fresh they are for other OSs.)  Rather simple to use, and has GUI or CLI capability, and has capability to read pcap files (functionality not tested).

Colasoft Packet Builder - free Windows tool.

Smartbits, IXIA, and other hardware tools - these solutions tend to be the most expensive, but are purposed designed for this type of operation so have great capabilities.  Always nice to have one of these if you can fit it into the budget.

Bit-Twist - multi-OS support and will easily send pcap capture file packets on to the wire; actually this s my preferred tool for capture file replay.  Includes tool to edit capture files - say you want to change all the destination MAC addresses in a pcap trace file to replay them to a different host: this will allow us to easily make these types of changes.  Note that replay of TCP really is not effective without significant support that I have not seen in any of these tools (it may exist, but I have not seen nor verified it for replay).

 

Generators that utilize host OS services

netcat, ncat, nc - tool for sending and receiving either unicast or multicast traffic.  Would not work with broadcast traffic.  All these tools do so much more than my specific needs for this project so a search of the various features based on what you need is recommended.

iperf/jperf - command line (or Java GUI with jperf, which is recommended as it has graphing capability) for testing performance.  While doing performance testing over Ethernet for unicast or multicast, it is sending packets so could be used as a packet generator depending on what is required: if only layer 2 destination generation is important, than it could fit the bill.  If more customization is needed, then it might not work.  This runs on nearly all platforms, including Windows, Linux, RaspberryPi w/ Pidora, Android tablets and phones, Mac OS X...

hping3 - CLI tool to send and receive packets.  Functionally flexible, but has a somewhat unique capability to send unicast, multicast, and broadcast traffic.  Also has listener services, but these have never been tested here.

socat - CLI tool for Linux that makes a great generic server.  Will listen on a port and accept unicast, multicast or broadcast traffic and echo the contents to the shell.  The broadcast reception capability seems to be somewhat unique.

 

Test Setup

To see how other hosts might view the layer-2 test traffic under various configurations, we can attach other devices to the L2 switch and run Wireshark or other packet sniffing tool (tcpdump, OmniPeak, etc.).

On the server side, to be able to receive all three types of layer-2 traffic - unicast, multicast, or broadcast - we use the socat command:

[root]# socat - UDP4-RECV:54321,ip-add-membership=224.10.20.30:0.0.0.0,broadcast

and we can see an IGMP join is generated, for this group:

This command will accept data from UDP port 54321, whether it is sent via layer-2 unicast, multicast or broadcast.  Some diagnostic commands to see how things are going.  First, what multicast groups is the pidora server now listening to:

[root@pidora ~]# netstat -g

IPv6/IPv4 Group Memberships

Interface       RefCnt Group

--------------- ------ ---------------------

lo              1      all-systems.mcast.net

eth0            1      224.10.20.30

eth0            1      all-systems.mcast.net

lo              1      ff02::1

lo              1      ff01::1

eth0            1      ff02::1:ff0f:8b09

eth0            1      ff02::1

eth0            1      ff01::1

What UDP port is the pidora server listening on:

[root@pidora ~]# netstat -lnu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
udp        0      0 0.0.0.0:46359           0.0.0.0:*                          
udp        0      0 0.0.0.0:54321           0.0.0.0:*                          
udp        0      0 127.0.0.1:323           0.0.0.0:*                          
udp        0      0 0.0.0.0:68              0.0.0.0:*                          
udp        0      0 0.0.0.0:123             0.0.0.0:*                          
udp6       0      0 ::1:323                 :::*                               
udp6       0      0 :::3690                 :::*                               
udp6       0      0 :::123                  :::*                           
                          
Now on the client side, we need different commands to create the different traffic streams.  If we use a pure traffic generator, we have to fill in all the information: source and destination MAC and IP addresses, select packet types, etc.  If we use a tool that can utilize host services, we need to provide less information.  As the command will utilize the routing table and ARP services/cache, we just need to worry about layer-3 information and above (destination IP, UDP port, etc.).

To create unicast traffic from the client:

[george@clt]$ hping3 192.168.10.101 -2 -c 100 -i u10000 -p 54321 -s 1025 -k -e 1
[open_sockraw] socket(): Operation not permitted
[main] can't open raw socket

Obviously this command results in an error - we need to be root:

[root@clt]# hping3 192.168.10.101 -2 -c 100 -i u10000 -p 54321 -s 1025 -k -e 1
HPING 192.168.10.101 (p16p1 192.168.10.101): udp mode set, 28 headers + 1 data bytes
[main] memlockall(): Success

Some of the options used:

Destination IP: 192.168.10.101

-2 for UDP mode

-c 100 as packet count, send 100 packets

-i u10000 as packet send delay, in microseconds, so send a packet every 10ms

-p 54321 is UDP destination port 

-s 1025 is UDP source port

-k is to fix the source port - if not used, each packet sent will increment source port by 1

-e 1 is the data to send - so the UDP segment will contain an ASCII value of 1, or 0x31

So this is better - we can now see traffic flow across the network, and the console prints out '1' on the pidora server side.  Note the display of '1's as packets come in - socat will display the UDP data sent with the -e switch as part of the hping3 command:

[root]# socat - UDP4-RECV:54321,ip-add-membership=224.10.20.30:0.0.0.0,broadcast

11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111

Note that the hping3 command gives frequent segmentation faults when running on this platform (Fedora20 in a VirtualBox VM, on a MacBook host with bridged networking to the Apple thunderbolt GigE adapter).

To create multicast traffic:

[root@clt]# hping3 224.10.20.30 -2 -c 100 -i u10000 -p 54321 -s 1025 -k -e 2

HPING 224.10.20.30 (p16p1 224.10.20.30): udp mode set, 28 headers + 1 data bytes

[main] memlockall(): Success

The only difference is we send an ASCII '2' as the data this time, as well as the destination IP address.  Note that since we are using a tool that can use OS services, specifying the IP as multicast will cause the packet to be generated to have the correct multicast MAC, per the RFC.  It does not need to be specified as the Linux kernel on the client knows how to do this, and will do it for us.  Also for broadcast traffic:

[root@clt]# hping3 192.168.10.255 -2 -c 100 -i u10000 -p 54321 -s 1025 -k -e 3

HPING 192.168.10.255 (p16p1 192.168.10.255): udp mode set, 28 headers + 1 data bytes

[main] memlockall(): Success

sending an ASCII '3' with this data set, and choosing a subnet broadcast address.  The host OS will map this subnet local broadcast IP address to a broadcast MAC of ff:ff:ff:ff:ff:ff.  Subnet local broadcast is defined by the IP address and the subnet mask, where the subnet mask here is class C, \24 or 255.255.255.0.  And the results from the server:

[root]# socat - UDP4-RECV:54321,ip-add-membership=224.10.20.30:0.0.0.0,broadcast

111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111122222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222223333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333

Here we can see all three packet type were accepted by the presence of the ASCII data sent.  

The broadcast address used is the subnet broadcast.  The Linux system client has two interfaces, and if the general broadcast address is used, I could not generate traffic, even if I specify the interface. Disabling the interface not used in the test: 

[root@clt]# ifconfig p7p1 down

allows this command to now work:

[root@clt]# hping3 255.255.255.255 -2 -c 100 -i u10000 -p 54321 -s 1025 -k -e 4

HPING 255.255.255.255 (p16p1 255.255.255.255): udp mode set, 28 headers + 1 data bytes

[main] memlockall(): Success

With these results:

[root]# socat - UDP4-RECV:54321,ip-add-membership=224.10.20.30:0.0.0.0,broadcast

1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222233333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333334444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444

Wireless

This proved to be much more difficult.  We need capability for packet injection, and required specific hardware support and the right software tools.  Next time, we will show how to inject wireless packets on Windows and Linux, and show some of the issues present when doing so.  Also up for discussion is the concept of sniffing traffic on 802.11 wireless interfaces, which can be challenging.

TCP Timing and the RTO

There are two general transport layer protocols in use with Industrial Fieldbus Protocols, TCP and UDP.

Both of these are at layer 4 of the 7-layer ISO-OSI network model, and they provide different services to hosts desiring to send data.  In general, layer 2 and layer 3 (datalink layer and IP layer, respectively), send frames on Ethernet as 'best effort'.  There is no guarantee that they arrive at their destination- quality and reliably is generally high in Ethernet networks, so frames sent usually get to there destination, but this is not a guarantee.  At layer 1, it is possible to have some amount of reliability built-in to the protocol.  For example, if the network detects collisions at layer 1 with CSMA-CD, there is a simple retransmission algorithm to send the frames a number of times until it arrives safely.  Also with 802.11 wireless networks, which are CSMA-CA, most frames sent over the wireless link are typically acknowledged, and failing an acknowledgement, data is resent.

UDP does not provide any type of reliability service.  Data is packaged with a simple header and sent down the stack to to the IP layer to forwarded along to it's destination.  It can be thought of as almost using the IP layer directly.  Several examples of use of UDP in ICS protocols is Class 1 EtherNet/IP messaging, also commonly known as I/O or Implicit messaging.  It is designed to be high speed, and their are a series of counters and a timeout mechanism at the application layer (layer 7) which indicate packet loss.  Also Schneider Electric's Global Data service uses UDP, as does Codesys' Network Variable concept, where in both cases, data is published using UDP, often using multicast addressing at layer 3 (IP layer).   Usually a single UDP datagram contains a single message or some related collection of data.  This is not always true, as the application using UDP can spread data over several UDP datagrams, which can end up in several IP packets, and be delivered over multiple Ethernet frames, but still a UDP segment can be used as a type of container where the applications can infer message or data boundary by being contained in a single UDP datagram.  For instance, if we only want to send a short message of a few bytes, but many of them - we could use distinct UDP datagrams and then the receiving application would be able to tell that each datagram (or packet, or frame) was a unique message.  The packet itself provides information that a logical boundary exists.

TCP, on the other hand, provides a reliable byte stream of data, as well as many other services such as flow control, full duplex operation, and in-order delivery.  One point of note in comparison to UDP is the byte stream concept.  TCP guarantees a stream of data, delivered in order, with no gaps.  There is no concept here of a message boundary - all boundary assumptions have to come from the data itself based on the specific protocol and how it is defined at the application layer.  TCP counts each separate byte of data, not segments, packets or frames, and this is fine based on the service provided - the same byte stream needs to arrive at the receiver, but it does not have to be in a single message.

There are several ways TCP provides reliability to the byte stream it is transporting.  In particular, when data is put on the wire (a generalization for network media, in fact it could be a wireless network so also when data is sent out over the airwaves, etc...) a timer is started.  This timer is called the RTO, or round trip timeout.  This is a very important number in TCP when considering reliability because it controls retransmissions, which is how lost data is recovered in the byte stream.

If the RTO expires without the data being explicitly acknowledged it is usually retransmitted (acknowledgment is done through the receiver providing feedback to the sender that it has, in fact, received the data as indicated by acknowledging the sequence number of the sent data).  Lost data will be retransmitted a certain number of times until it is either acknowledged, or the sender gives up and moves on.  In either case, there are a number of parameters, some dynamically calculated, others fixed by configuration, that control the specifics of this process.

Where does this RTO timing come from?  Often when a TCP connection is initiated, little is known about the path between two communication hosts, and little may be known about the receiver in general.  Therefore it is difficult to estimate what the optimum value should be: too small, and we get retransmissions when we don't need them and this just consumes more bandwidth.  Too long, and it can slow down the data communications and provide poor quality to the data link.  In fact there is much research in optimizing this and other TCP timer-based mechanisms to improve data flow.

Most hosts, when initiating a TCP connection, will set a default RTO of some value.  Historically, this was 3 seconds, and still is for some hosts (almost always set by the operating system in use).  So if a host puts some data on the wire at the beginning of the communication stream, and no ACK is received for these bytes in 3sec, the data would be resent.  It is also a dynamic mechanism - what if data is delivered almost immediately with almost no packet loss: then 3 sec is very long to wait in this case.  So the RTO value is adjusted as data is transferred: measurements are taken, called the RTT, or round trip time, and a formula is used to adjust the RTO to optimize the communication channel.  If data is very slow to be ACKed, then we would want to move to a longer RTO.  If data is ACKed very fast, we would want a smaller RTO.  There are many good references on how this is calculated.  Many hosts now have an initial RTO of 1sec, while there are some ICS products designed for high speed use that have initial RTOs in the 25-50ms range.

However, though dynamic, there are limits to what values the RTO can take.  Since ICS protocols are typically 'local' type applications in close proximity and on generally small networks (not always true, but it's common to have these communication protocols running on a single machine, or in a single building, or in a campus).  It's much less common, though possible, to have these protocols running around the world with high latency satellite links.  Therefore, the parameter for us to focus on is the minimum allowed RTO.  There would be a maximum as well, but in my experience the minimum RTO has a bigger impact on communications in the industrial space.

As a test, I used a relatively recent Linux system running on a PC, communicating with ModbusTCP to a Schneider Electric Momentum PLC.  This type of communication that was done simulates typical SCADA type traffic where a SCADA system, usually running on some type of PC, will use an ICS protocol to pull data from a controller for visualization/storage and also send data to the controller to implement specific actions.  Many SCADA systems run on Windows systems, but some work on other OSs, including Linux, Solaris, etc.

For a recent Linux kernel, the initial RTO seems to be around 1sec.  This is difficult to see with on-wire behavior because usually when the connection builds, data starts to send quickly, and the dynamic mechanism of the RTO based on the RTT starts to work.  It is easiest to visualize the steady-state RTO value, which in my test, is hopefully the minimum RTO.

Baseline RTO Test

As a baseline, this is what we get from the Linux client:

[george@linux george]$ uname -a
Linux linux 3.11.2-201.fc19.x86_64 #1 SMP Fri Sep 27 19:20:55 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux 

This is the kernel in use.  Linux actually allows us to adjust the minimum RTO on a per-route basis, so lets have a look at the routing table now before any adjustments are made:

[george@linux george]$ ip route show
default via 10.171.182.1 dev wlan0  proto static
10.171.182.0/23 dev wlan0  proto kernel  scope link  src 10.171.183.155  metric 9
192.168.10.0/24 dev em1  proto kernel  scope link  src 192.168.10.11  metric 1

In this case, the 192.168.10.0/24 network is in use.  With all of this in place, start communications with the Linux PC as the ModbusTCP client and the Momentum PLC as the server.  We are able to look at the current RTO for this particular TCP connection:

[george@linux george]$ watch -n 1 ss -tn -o dst 192.168.10.99

We can see with the ss command, wrapped inside the watch command, every second the current RTO will be displayed.  In this case, it is at 203ms, which is the middle number of the timer field.

Looking at the raw data from Wireshark, we get the following for the ModbusTCP Response time:

In this case, we can see the typical response is on average about 1.5ms, but there is a significant mode of response up around 2.5ms.  Based on the typical RTOs observed with this data set in Linux, I propose the RTO is calculated for this case as minRTO + RTT.  The minimum RTO is 200ms by default (as evidenced by Linux kernel source code):

#define TCP_RTO_MIN ((unsigned)(HZ/5))

from include/net/tcp.h.

For this network, there are only a few devices and everything is in close proximity with high quality cabling in a lab environment. Therefore, we expect really no packet loss so the RTO should migrate to the minimum value very quickly, which is what was observed.

Packet Drop Test at Default RTO

Now it would be useful to see the RTO in action.  However, this presents a minor problem as it is quite difficult to just drop packets, on command, to test retransmission behavior.  Most of the time I see engineers pull the cable from a device and try to see what the TCP stack does once this link is lost.  This is one particular type of failure event, however, I argue it is only one type of event, and is actually not the most common.  What I have observed that is more common is the loss of a single Ethernet frame that then requires TCP to initiate recovery operations (which is usually a retransmission).  Pulling the cable does not represent this; and the single packet drop on the network is usually not done by the network itself.  Though the network can drop a frame if it is too busy, nearly all switches and bridges in use today make use of dedicated switch chips which can forward frames in hardware at full line rate.  In contrast, most Ethernet devices used in industrial control operate at MUCH lower levels than, say, 100MBps Ethernet is capable of doing - most of the times less than 10% of line rate.  Therefore, it's likely not the network dropping a packet, but an end device not being able to handle it within it's TCP/IP stack so drops a frame due to overload or a bug.

So how to drop a single packet?  The best way is with custom hardware designed to do just this - one such machine is a Spirent or Ixia GEM impairment tool.  These have great control over a TCP connection and can be programmed to drop specific frames and the retransmission behavior can then be analyzed experimentally.

But these can cost thousands of dollars - so what if we don't have one?  Aside from pulling the cable, can I just drop a frame now and again to see what retransmission will be, and see how the RTO affects these retransmissions?

A small Latvian company, Mikrotik, makes some very nice, inexpensive, routers and other networking equipment.  Perusing one of their $120 (street price) routers, RB-450G, I ran across a feature in the IP Firewall configuration section that could be of use in this case.  Note the product has three different firewalls - one at the switch chip, one at the bridge level, and the third at the IP layer.  Here is the configuration of the IP Firewall rule - the first is the selection rule for choosing TCP data with a destination port of 502.  This should be familiar to those who know ModbusTCP and TCP/IP as a selector for client traffic, or Modbus queries.  The ModbusTCP server, which accepts queries and issues responses, is usually listening on TCP port 502, while the client will usually use an ephemeral port (some number, perhaps greater than 32K) as it's port.

Selecting the forward chain, as data is passing through the firewall device, and TCP with a destination port of 502 for client traffic to server traffic:

The interesting part for us is the random field:

This will select random packets at this frequency (based on 100%) - so this is to select 1/100 packets, on average, from the traffic selector we have, that of packets (mostly ModbusTCP queries) destined for the server device. 

Finally, we need to disposition our selected packets... in this case we want to drop, and see what the TCP stack does with the RTO setting:

Let's be clear: this is not as good as a dedicated machine purpose-built to drop packets.  With the GEM tool, you have very specific traffic selectors with much better control, and can do a lot more impairment (multiple drops, change any bit within the frame, duplicate packets, reorder, delay, etc.)  However, for the price, the Mikrotik is tough to beat.  Note that if you are faced with a specific issue, it may take some time to actually drop the specific frame you are looking for with this method due to the lack of richness in the selector tools.  However, probability says if we let it run long enough, we will probably see the event we are looking for.

So now we can drop packets and look at the RTO in action.  

Hypothesis: we will use the Mikrotik to drop a ModbusTCP query, and the Linux client will wait about one RTO period and then retransmit.  

Let's have a look with Wireshark, from the viewpoint of the Linux PC client.  Where you sniff traffic is important: we know the ModbusTCP software will produce the query, and it will get transmitted onto the network.  But before it gets to the server, it will be dropped by the firewall.  So if we sniff traffic at the client end, we will see this query, but it will appear from behavior that it was ignored as Wireshark will then show the same frame again, indicating a retransmission.  If we sniffed traffic at the server end, we would never see the original query, so Wireshark would not indicate a retransmission.  Note that the network has no concept of retransmission: it is a state held by a particular TCP/IP stack.  Even with these two devices, they both do not have to agree on what a retransmission is: based on where you sniff here in this test, the client knows it is a retransmission and Wireshark shows it.  But I propose the retransmitted query from the server's point of view is not a retransmission - the first one never arrived so to this device, the second retransmitted query is the original query.  Dropping a query downstream from here we se a delay of 204ms in retransmit, just as expected:

Adjusting the Default minimum RTO

Linux allows us to adjust the min RTO for a given route using the ip suite of tools.  To do this we perform the following.  Display route:

[george@linux george]$ ip route show

default via 10.171.182.1 dev wlan0  proto static

10.171.182.0/23 dev wlan0  proto kernel  scope link  src      10.171.183.155  metric 9

192.168.10.0/24 dev em1  proto kernel  scope link  src    192.168.10.11  metric 1

192.168.10.20 via 192.168.10.11 dev em1

Note in particular the addition of specific host route now - the last entry for 192.168.10.20.  To change the minimum RTO:  

[root@linux george]# route change 192.168.10.20/32 via 192.168.10.11 dev em1 rto_min 50 

The documentation for this command indicates I need to add units to the new RTO value, in this case, 50.  However, the command will error when I add units (such as 50ms, or 50 ms, etc.) but this seems to work.  Verifying the change:

[george@linux george]$ ip route show
default via 10.171.182.1 dev wlan0  proto static
10.171.182.0/23 dev wlan0  proto kernel  scope link  src 10.171.183.155  metric 9192.168.10.0/24 dev em1  proto kernel  scope link  src 192.168.10.11  metric 1
192.168.10.20 via 192.168.10.11 dev em1  rto_min lock 50ms

Notice we need root privileges to make the route change.  The last host specific route now shows the minimum RTO at 50ms.  So time to try it. 

Packet Drop Test at 50ms RTO

Using our ss command, as we did earlier, we can see what Linux has for the current RTO now:

The RTO is now 53ms, inline with before: minRTO+RTT.

Wireshark shows, on packet drop:

As expected, a dropped query resulted in a retransmission at the RTO time, 53ms.  

Summary

We have explored use of TCP with ModbusTCP and how a TCP system retransmits data.  We have also shown how it is possible to use Linux to adjust the minimum RTO on a per route basis, and how inexpensive hardware can be used to drop selective packets in a TCP stream to study the TCP protocol under actual use conditions.