Channel Utilization

Channel Utilization

I want to be able to collect channel utilization statistics for 2.4 and 5GHz channels in my supported wireless network environments.  Some basic requirements that would be nice to collect the utilization parameter:

1. Automatically collect samples
2. Collect for all 2.4 and 5GHz channels we could possibly use - not just the channels in use by our configured networks
3. Store data for short to medium term availability (weeks to months, one to two year data retention would be fine)
4. Graph data on demand
5. Sample at no longer than 5min intervals, preferably faster
6. Inexpensive solutions using commercially available products would be preferred

It turns out it that it is not that easy to get at this number. There are some ways, but the ultimate goal would be to trend, for all channels, over long periods of time. Feeding this data into a network manager would be ideal – they are usually setup to store data over time and display in graphical form. So let's see how to get at this critical parameter in evaluating the Wifi networks around us.

Example target graph, as created with Observium (http://www.observium.com):

This is for five channels.  Notice channel 165 is under very heavy utilization.

Where can we find channel utilization?

Available Tools

Several places contain the number, but unfortunately, none of these provide a simple method of long term datastore and presentation.

Packet capture

The QBSS load element contains channel utilization, as seen by the AP sending this element:

It's sent in multiple frame types (e.g. beacons, etc) but since beacons are typically produced at a 100ms period, it's easy to just look at these for value. Some advantages of this method are that it provides engineers with the utilization that the AP is observing; the AP is self-reporting it's conditions. Some negatives with this are that an OTA (over the air) packet capture is required; so monitor mode on an 802.11 adapter is needed to pick up this type of information. Also there is no ready way to pull out the value and insert into the datastore; plus multiple sources of QBSS can be observed in a single trace as all beacons might contain this IE (information element). Which value would we use? Each beacon is reporting what it's AP sees at that time, and that changes with both time and spatial position. To get all channels, we will have to cycle through all channels to pick up data; what if there are no beacons on that channel? The QBSS element is optional – not all systems provide it; many don't.

The next level in complexity might be the controller or AP. It might self report. Cisco, for instance, reports for each AP the observed utilization (WLC 3504 controller and AP3802, software version 8.8.111):

This is nice, but gives me only the channel this AP is on. What about all the others?

Controller Systems

Cisco has the capability and tools to look at multiple channels if we dedicate an AP to monitor mode:

With the key, highlighted, and Cisco's Spectrum Expert Windows tool, we are starting to get onto something. With the tool, we have to choose 2.4 or 5GHz (not both), and even then, we get to trend only four channels:

So the upside here is a tool that is available with an easy-to-use GUI interface. Downsides are numerous: need a dedicated AP – has to be in monitor mode and so won't serve clients. The key that has to be copied from the controller changes after every reboot of the AP; not convenient for an always-on, automated data collection system. With the GUI Windows tool, there does not seem to be a long term storage and graphing option (i.e. database capability). Also channels are limited as to what can be collected. It looks like two instances of the GUI tool can be started and pointed to a single AP in monitor mode, but that is just inconvenient.  This seems to be better suited for spot troubleshooting.  

Ubiquiti presents channel utilization for a given channel(s) used by the AP in their software controller:

Under tools, we can look at all channels with one (major) caveat: all users will be kicked off the AP:

Even with new top of the line model (https://unifi-shd.ubnt.com/) that advertises a separate security radio, it seems like this RF utilization could be done with this radio instead of the ones in use by actual wireless clients.  Some of the other tools present are certainly a good start.

Build our own with Linux

What about a Linux system? They make pretty good packet capture systems, so is there anything we can do to bring in channel utilization?   Some Internet searching turned up some ideas, like this one:

https://www.tablix.org/~avian/blog/archives/2016/02/ieee_802_11_channel_survey_statistics/

With the iw tool, there is a command that may allow for presentation of channel utilization. With this, we can start to form a process to get this data stored into a network manager for store and graphical presentation. General workflow at this point, as an idea:

1. Use a shell script or other (e.g. python, perl, c, etc)) to cycle through the various 802.11 channels, and run the iw command to extract channel utilization on a compatible adapter
2. Execute the script in cron at a certain interval, and make the data available to collectd tool (https://collectd.org/)
3. Configure Observium to display collectd datasets

Of course, you can use your favorite data collection / storage / presentation system.

Which adapters support this feature, and how does it really work? There are a lot of wireless adapters available for Linux, and they don't all behave the same way. We have PCIe, mini PCIe, M.2, and USB adapters, just to start, with various chipsets and feature sets.

Let's have a look at one that works (Linux kernel 4.18, AR928X with Ath9K driver):

user@host:~$ iw wlan91 survey dump  Survey data from wlan91  frequency: 5825 MHz [in use]  noise: -95 dBm  channel active time: 7496 ms  channel busy time: 6386 ms  channel receive time: 626 ms  channel transmit time: 0 ms

We have channel active time and busy time; seems like the ratio here would be utilization. In this case, we have 6386ms/7496ms = 85.1%.

What devices/chipsets actual output values from this command? Test command:

user@host:~$ iw <device> survey dump | grep -A 5 "in use"

Some adapters to test, though some only support 2.4GHz, some only 5 GHz. Obviously, a dual band adapter would be most useful so we can get all channels with only a single adapter. Adapters to test:

Ndx Iface  Phy Driver       Mode     Channel     Width Adapter
Linux System A

 0  wlan0  phy0 iwlwifi     monitor  11 (2462MHz) 20MHz Intel Corporation Wireless 7265 (rev 59)
 1  wlan1  phy3 rt2800usb   monitor  11 (2462MHz) 20MHz Ralink Technology, Corp. RT3573

 2  wlan2  phy1 carl9170    monitor  11 (2462MHz) 20MHz NetGear, Inc. WNDA3100v1 802.11abgn [AtherosAR9170+AR9104]

 3  wlan3  phy4 rt2800usb   monitor  11 (2462MHz) 20MHz Ralink Technology, Corp. RT5370

 4  wlan4  phy2 ath9k_htc   monitor  11 (2462MHz) 20MHz Atheros Communications, Inc. AR9271 802.11n

 5  wlan5  phy6 carl9170    monitor   6 (2437MHz) 20MHz CACE Technologies AirPcap NX [Atheros AR9001U-(2)NG]

Linux System B

 6  wlan10  phy2 iwlwifi    monitor 165 (5825MHz) 20MHz Intel Corporation Wireless 8265 / 8275 (rev 78)

 7  wlan90  phy1 ath9k      monitor 165 (5825MHz) 20MHz Qualcomm Atheros AR928X Wireless Net Adapt (PCIx)(rev 01)

 8  wlan104 phy0 ath10k_pci monitor  36 (5180MHz) 20MHz Qualcomm Atheros QCA986x/988x 802.11ac Wireless Net Adapt

Configure test traffic so we can provide at least some sort of validation to any result returned. For 2.4GHz, run an iperf test on channel 6 and for 5GHz, use channel 165. Set each adapter to channel 6 (if possible) and look for survey dump results. Compare to beacon QBSS load element values. For channel 6 and 165, use these commands:

iw wlan10 set freq 2437 HT20

iw wlan10 set freq 5825 HT20

Results

2.4GHz channel utilization from a beacon under load - this AP/SSID is handling the current iperf load (77%):

Baseline load, with no iperf stream, is ~17%. iperf stream throughput for 802.11 abgn 1x1:1 / SGI / 20MHz client is about 36Mbps at this channel loading. Note that this is throughput; datarate is 72.2Mbps.

5GHz baseline is essentially 0%. With iperf stream and same client, we get about 47Mbps at 84% channel utilization:

Some observations:

1. The QCA9862 device (wlan104/Ath10k, PCI 168c:003c) gives inconsistent results from the survey dump command. It varies from 0-100%, even in an environment that has relatively constant channel utilization.
2. The Intel devices do not provide any survey information at all.
3. Some of the USB devices can be used to collect data, as well as the mini PCIe/M.2 cards. wlan90 is miniPCIe adapter, while wlan0-5 are USB based wireless adapters.

Integrate into Collectd

Now that we have what each adapter can do, we need to integrate into a data system such that we can get our routine graphs. This is a proof of concept that works but there are some security considerations ignored so would not be considered a preferred deployment strategy. Others with expertise in collectd and Observium may have easier methods of integrating generic data.   

Add to /etc/collectd/collectd.conf:

TypesDB "/usr/share/collectd/types.db" "/etc/collectd/typedb.local"

Then configure /etc/collectd/typedb.local as:

channel_utilization chn_1:GAUGE:0:100, chn_6:GAUGE:0:100, chn_11:GAUGE:0:100, chn_36:GAUGE:0:100, chn_40:GAUGE:0:100, chn_44:GAUGE:0:100, chn_48:GAUGE:0:100, chn_52:GAUGE:0:100, chn_56:GAUGE:0:100, chn_60:GAUGE:0:100, chn_64:GAUGE:0:100, chn_100:GAUGE:0:100, chn_104:GAUGE:0:100, chn_108:GAUGE:0:100, chn_112:GAUGE:0:100, chn_116:GAUGE:0:100, chn_120:GAUGE:0:100, chn_124:GAUGE:0:100, chn_128:GAUGE:0:100, chn_132:GAUGE:0:100, chn_136:GAUGE:0:100, chn_140:GAUGE:0:100, chn_149:GAUGE:0:100, chn_153:GAUGE:0:100, chn_157:GAUGE:0:100, chn_161:GAUGE:0:100, chn_165:GAUGE:0:100

channel_util_24GHz chn_1:GAUGE:0:100, chn_6:GAUGE:0:100, chn_11:GAUGE:0:100

channel_util_UNII1 chn_36:GAUGE:0:100, chn_40:GAUGE:0:100, chn_44:GAUGE:0:100, chn_48:GAUGE:0:100

channel_util_UNII2 chn_52:GAUGE:0:100, chn_56:GAUGE:0:100, chn_60:GAUGE:0:100, chn_64:GAUGE:0:100

channel_util_UNII2e_0 chn_100:GAUGE:0:100, chn_104:GAUGE:0:100, chn_108:GAUGE:0:100, chn_112:GAUGE:0:100

channel_util_UNII2e_1 chn_116:GAUGE:0:100, chn_120:GAUGE:0:100, chn_124:GAUGE:0:100, chn_128:GAUGE:0:100

channel_util_UNII2e_2 chn_132:GAUGE:0:100, chn_136:GAUGE:0:100, chn_140:GAUGE:0:100, chn_144:GAUGE:0:100

channel_util_UNII3 chn_149:GAUGE:0:100, chn_153:GAUGE:0:100, chn_157:GAUGE:0:100, chn_161:GAUGE:0:100, chn_165:GAUGE:0:100

Each one of these lines (spaces left between only to separate the different items) is a graph definition and will produce a datastore and ultimately a graph.  The first items is all FCC channels that are relevant to me (so 2.4GHz channels 1,6, & 11) and then 5GHz channels that are available in my area that I might want to use.  However, this single graph is difficult to read with any precision so separate graphs are made with subsets of channels.  Since the channel utilization is a %, it is gauge type and goes from 0 to 100.  The legend name is included here (in bold):

chn_1:GAUGE:0:100, chn_6:GAUGE:0:100, chn_11:GAUGE:0:100

and the graph name is the type name, such as channel_util_UNII1.  This should produce 8 graph sets.

Other changes to the config file include enabling the exec plugin, and defining where to get data from:

LoadPlugin exec

<Plugin exec>

        Exec "admin" "/usr/bin/cat" "/tmp/chnutil.txt"

</Plugin>

For more detail, see the collectd documentation set.  In this instance, the user admin will cat the temp file and read in the various values at some interval as determined by collectd.

Restart the service when configuration is complete (for this system, others differ):

user@host:~$ sudo systemctl restart collectd.service

Collect Values from Adapter

Create a bash script to cycle through the various channels and then run the requisite iw command to collect the utilization data; perform the math on the data, and then output so that the collectd plugin can read and process the data.  Execute the shell script through 

crontab -e

as root and add:

*/2 * * * * /path/to/channelutil.sh > /tmp/chnutil.dbg 2>&1

and then restart the cron service on the system:

systemctl restart cron.service

Root is required to perform some operations with the radio; it would be better to allow those functions to be executed by a regular user so that this whole script does not need to be run with elevated privilege.  A suitable sudo configuration could be used.  As this is proof of concept, security improvements will be implemented in a production environment by allowing the specific iw commands that require root privilege to be executed by our admin user.

The referenced bash script channelutil.sh must create our /tmp/chnutil.txt file and redirects script output to the .dbg file.  A better approach would be to use something like python with a library to access wireless information and make the script more flexible; it could be called directly from collectd and output the requisite data depending on how it is called.  We could use some overrides to default behavior (channels to use, adapter, etc) and also solve our screen scraping of iw output (not suggested) and security issues with running as root.  Next year 😄

Here is the prototype script - save as channelutil.sh in an appropriate place (as defined in the cron entry) and make read/write/executable for the user only (chmod 700 channelutil.sh) :

#!/bin/bash

#Grab channel utilization stats from a qca-atheros wifi chipset (maybe others)

#Cycle through the channels, running survey dump

#Version        Chipset supported

#0.1            Qualcomm Atheros AR928X Wireless Network Adapter (PCI-Express) (rev 01) / Ath9k / kernel 4.18.0-kali3-amd64

#0.2            Add 148f:3573 Ralink Technology, Corp. RT3573 Wireless Adapter / kernel 4.18.0-kali3-amd64

############################################################################################################################

#Local Settings

#Choose adapter - can't be used for anything else as it will cycle through channels

Adapter='wlan90'

#Channels to cycle through - will automatically add HT20 so we can get 802.11n traffic.  Use frequency equivalent

#FCC 2.4 and 5GHz

Channel=(2412 2437 2462 5180 5200 5220 5240 5260 5280 5300 5320 5500 5520 5540 5560 5580 5600 5620 5640 5660 5680 5700 5745 5765 5785 5805 5825)

#All 2.4 and 5GHz

#Channel=( 2412 2417 2422 2427 2432 2437 2442 2447 2452 2457 2462 5180 5200 5220 5240 5260 5280 5300 5320 5500 5520 5540 5560 5580 5600 5620 5640 5660 5680 5700 5745 5765 5785 5805 5825)

#Subset for testing

#Channel=(2412 2437 2462 5180)

#Commands - when run from cron may not have an environment so need absolute paths

IW=/usr/sbin/iw

IWCONFIG=/usr/sbin/iwconfig

IFCONFIG=/usr/sbin/ifconfig

#Dwell time on channel, in sec

dwelltime=2

#Output files - in /tmp directory

outfile=/tmp/chnutil.txt

#Systemwide variables - for use with collectd

ChUtilAr=()

host="${COLLECTD_HOSTNAME:-localhost}"

pause="${COLLECTD_INTERVAL:-10}"

i=0                     #simple counter

time="$(date +%s)"      #timestamp - not currently used

############################################################################################################################

#Functions

#

############################################################################################################################

#Setup adapter

#Set each NIC - only have 1

echo -e "[1]\tInitial NIC configuration"

#Unblock adapters

        #Display adapter status

        echo -e "[2]\tBlocking status..."

        /usr/sbin/rfkill

        retVal=$?

        if [ $retVal -ne 0 ]; then

                echo -e "ERROR\n \t$cmdresponse"

                exit $retVal

        fi

        echo -en "[3]\tUnblock ALL adapters \t"

        cmdresponse=$(/usr/sbin/rfkill unblock all 2>&1)

        retVal=$?

        if [ $retVal -ne 0 ]; then

                echo -e "ERROR\n \t$cmdresponse"

                exit $retVal

        else

                echo -e "\tSUCCESS"

        fi

#Configure adapter to monitor mode

for NIC in $Adapter

do

        echo -en "[4]\tBring adapter $NIC down "

        cmdresponse=$($IFCONFIG $NIC down 2>&1)

        retVal=$?

        if [ $retVal -ne 0 ]; then

                echo -e "ERROR\n \t$cmdresponse"

                exit $retVal

        else

                echo -e "\tSUCCESS"

        fi

        #Set monitor mode

        echo -en "[5]\tSet MONITOR mode for $NIC "

        cmdresponse=$($IWCONFIG $NIC mode monitor 2>&1)

        retVal=$?

        if [ $retVal -ne 0 ]; then

                echo -e "ERROR\n \t$cmdresponse"

                exit $retVal

        else

                echo -e "\tSUCCESS"

        fi

        echo -en "[6]\tBring adapter $NIC up \t"

        cmdresponse=$($IFCONFIG $NIC up 2>&1)

        retVal=$?

        if [ $retVal -ne 0 ]; then

                echo -e "ERROR\n \t$cmdresponse"

                exit $retVal

        else

                echo -e "\tSUCCESS"

        fi

        #Set configs

        #echo -e "\tSet config for $NIC"

        #$IW dev $NIC set monitor fcsfail control

done

############################################################################################################################

#Manage channel change

#Number of channels to evaluate from setup array - cycle through each one

NoChnToScan=${#Channel[@]}

echo -en "[7]\tChannel dwell time $dwelltime sec\n"

for (( i=0;  i<$NoChnToScan; i++ ))

do

        echo -ne "[${Channel[$i]}]\tSet channel for $Adapter\t"

        #SetChannel=$(echo ${Channel[$i]})

        #SetChannel=$(printf "%s HT20" $SetChannel)

        #echo -ne" Channel set to $SetChannel"

        cmdresponse=$($IW $Adapter set freq ${Channel[$i]} HT20 2>&1)

        retVal=$?

        if [ $retVal -ne 0 ]; then

                echo -e "ERROR\n \t$cmdresponse"

                exit $retVal

        fi

        #Check - did we get our selected channel?  Something changing the channel besides this script?

        echo -ne "\tCheck "

        #Query system

        adapterinfo=$($IW dev $Adapter info)

        retVal=$?

        if [ $retVal -ne 0 ]; then

                echo -e "ERROR\n \t$CurrentChannel"

                exit $retVal

        fi

        #Recover elements of adapterinfo like channel and frequency

        CurrentChannel=$(echo -e "${adapterinfo}" | grep channel | awk -F ',' '{print $1}' 2>&1)

        #Strip off leading whitespace

        CurrentChannel="$(echo -e "${CurrentChannel}" | sed -e 's/^[[:space:]]*//')"

        #Strip off leading channel text...  channel 1 (2412 MHz) --> 1 (2412 MHz)

        CurrentChannel=$(echo -e "${CurrentChannel}" | awk -F 'channel' '{print $2}' 2>&1)

        #echo -en "PV: $CurrentChannel\t"

        printf 'PV: %15s\t' "$CurrentChannel"

        ActualFreq=$(echo -e "${adapterinfo}" | grep channel | awk -F ',' '{print $3}' | awk '{print $2}')

        ActualChannel=$(echo -e "${adapterinfo}" | grep channel | awk  '{print $2}')

        #echo -ne "Set/Actual "; echo -ne $SetChannel; echo -ne "/"; echo -ne $ActualChannel

        if [ ${Channel[$i]} != $ActualFreq ];

        then

                echo -e "\t FAILED"; exit 1

        else

                echo -en "SUCCESS\t"

        fi

        #Manage survey dump command on this channel - dwell for configured time

        sleep $dwelltime

        surveyresult=$($IW dev $Adapter survey dump | grep -A 5 "in use")

        retVal=$?

        if [ $retVal -ne 0 ]; then

                echo -e "ERROR\n \t$surveyresult"

                exit $retVal

        fi

        activetime=$(echo $surveyresult | awk -F "channel active time: " '{print $2}' | awk '{print $1}')

                #echo "Activetime: $activetime"

        #Active time needs to be a positive number.  Assumes integer return with no sign.

        re='^[0-9]+$'

        if ! [[ $activetime =~ $re ]] ; then

                echo "Error: Active time not a number" >&2

                echo "Surver result: $surveyresult"

                exit 1

        fi

        if ! [[ $activetime -gt 0 ]] ; then

                echo "Error: Active time must be greater than zero" >&2

                echo "Survey result: $surveyresult"

                exit 1

        fi

        #Busy time needs to be a positive number or zero.  Assumes integer return with no sign.

        busytime=$(echo $surveyresult | awk -F "channel busy time: " '{print $2}' | awk '{print $1}')

        if ! [[ $busytime =~ $re ]] ; then

                echo "Error: Busy time not a number" >&2

                echo "Surver result: $surveyresult"

                exit 1

        fi

        if ! [[ $busytime -ge 0 ]] ; then

                echo "Error: Busy time must be greater than or equal to zero" >&2

                echo "Surver result: $surveyresult"

                exit 1

        fi

        #Channel busy is the ratio of busy to active... in percent

        ChannelBusy=$(awk "BEGIN {printf \"%.2f\n\", $busytime/$activetime*100}")

        echo -ne "\tUtilization\t"; echo -ne $ChannelBusy; echo "%"

        ChUtilAr[$i]=$ChannelBusy

done

        #CollectdOutput=$(printf "PUTVAL $host/wifi/channel_utilization %s:%s:%s:%s:%s\n" "$time" "${ChUtilAr[0]}" "${ChUtilAr[1]}" "${ChUtilAr[2]}" "${ChUtilAr[3]}")

        #CollectdOutput=$(printf "PUTVAL $host/wifi/channel_utilization N:%s:%s:%s:%s\n" "${ChUtilAr[0]}" "${ChUtilAr[1]}" "${ChUtilAr[2]}" "${ChUtilAr[3]}")

        #Output all channels - for proof of concept.  Can't really read the graph.

        CollectdOutput_all="PUTVAL $host/wifi/channel_utilization N"

        for (( i=0;  i<$NoChnToScan; i++ ))

        do

                printf -v CollectdOutput_all "%s:%s" "$CollectdOutput_all" "${ChUtilAr[$i]}"

        done

        echo $CollectdOutput_all

        #Just 2.4GHzchannels

        channel_util_24GHz=$(printf "PUTVAL $host/wifi/channel_util_24GHz N:%s:%s:%s\n"    "${ChUtilAr[0]}" "${ChUtilAr[1]}" "${ChUtilAr[2]}")

        echo $channel_util_24GHz

        #UNII1 5GHzchannels

        channel_util_UNII1=$(printf "PUTVAL $host/wifi/channel_util_UNII1 N:%s:%s:%s:%s\n" "${ChUtilAr[3]}" "${ChUtilAr[4]}" "${ChUtilAr[5]}" "${ChUtilAr[6]}")

        echo $channel_util_UNII1

        #UNII2 5GHzchannels

        channel_util_UNII2=$(printf "PUTVAL $host/wifi/channel_util_UNII2 N:%s:%s:%s:%s\n" "${ChUtilAr[7]}" "${ChUtilAr[8]}" "${ChUtilAr[9]}" "${ChUtilAr[10]}")

        echo $channel_util_UNII2

        #UNII2e_0 5GHzchannels

        channel_util_UNII2e_0=$(printf "PUTVAL $host/wifi/channel_util_UNII2e_0 N:%s:%s:%s:%s\n" "${ChUtilAr[11]}" "${ChUtilAr[12]}" "${ChUtilAr[13]}" "${ChUtilAr[14]}")

        echo $channel_util_UNII2e_0

        #UNII2e_1 5GHzchannels

        channel_util_UNII2e_1=$(printf "PUTVAL $host/wifi/channel_util_UNII2e_1 N:%s:%s:%s:%s\n" "${ChUtilAr[15]}" "${ChUtilAr[16]}" "${ChUtilAr[17]}" "${ChUtilAr[18]}")

        echo $channel_util_UNII2e_1

        #UNII2e_2 5GHzchannels

        channel_util_UNII2e_2=$(printf "PUTVAL $host/wifi/channel_util_UNII2e_2 N:%s:%s:%s:%s\n" "${ChUtilAr[19]}" "${ChUtilAr[20]}" "${ChUtilAr[21]}" "0")

        echo $channel_util_UNII2e_2

        #UNII3 5GHzchannels

        channel_util_UNII3=$(printf "PUTVAL $host/wifi/channel_util_UNII3 N:%s:%s:%s:%s:%s\n" "${ChUtilAr[22]}" "${ChUtilAr[23]}" "${ChUtilAr[24]}" "${ChUtilAr[25]}" "${ChUtilAr[26]}")

        echo $channel_util_UNII3

        printf "%s\n" "$CollectdOutput_all"     > $outfile

        printf "%s\n" "$channel_util_24GHz"     >> $outfile

        printf "%s\n" "$channel_util_UNII1"     >> $outfile

        printf "%s\n" "$channel_util_UNII2"     >> $outfile

        printf "%s\n" "$channel_util_UNII2e_0"  >> $outfile

        printf "%s\n" "$channel_util_UNII2e_1"  >> $outfile

        printf "%s\n" "$channel_util_UNII2e_2"  >> $outfile

        printf "%s\n" "$channel_util_UNII3"     >> $outfile

Observium

This can be configured to integrate collectd data easily.  See instructions here: https://docs.observium.org/collectd/

With this configuration, Observium will display a collectd tab, and under that, a wifi tab will show which contains our graphs:

 

Review of requirements:

1. Automatically collect samples
2. Collect for all 2.4 and 5GHz channels we could possibly use - not just the channels in use by our configured networks
3. Store data for short to medium term availability (weeks to months, one to two year data retention would be fine)
4. Graph data on demand
5. Sample at no longer than 5min intervals, preferably faster
6. Inexpensive solutions using commercially available products would be preferred

Did we meet the basic goals?

1. We are using cron to automatically generate samples every two minutes
2. Channels are configurable in the script.  Make sure the adapter supports the channels you want to collect.  Regulatory domain issues do creep up from time to time.
3. These tools store data for at least a year in rrd files
4. Accessing the webpage for Observium will display the graphs on demand
5. Observium functions on a 5 min interval, so that appears to be as fast as can be collected/analyzed without changing the default configuration
6. For a system that has a free miniPCIe slot, an Ath9k adapter 928X works fine and might be around $US50.  For USB only systems, I am using the Netgear carl9170 based adapters which are old but might be found used for $US40.  Both appear to give similar results.  I am using what I already had available.

Due to constant changes in the WiFi industry, the availability of these chipsets can change so it may be required to find different products and evaluate them as replacements.