Properties of ICS Protocols

I want to propose a method or philosophy to study ICS protocols but first I need to define some of their properties.  These properties are not unique, or new, only that with the analysis of the network communication, it's best if the terminology is consistent.

There are many ICS protocols.  Many are called fieldbus protocols, and some are based on the pre-Ethernet days, while some are not.  Some popular Ethernet ones are

ModbusTCP
EtherNet/IP 
Profinet
PowerLink
EtherCAT
sercosIII

and many others.  Some non-Ethernet ones are

CANopen
DeviceNet
Modbus
ASI

and many more...

My focus is on Ethernet mostly, in particular ModbusTCP and EtherNet/IP, which is CIP on Ethernet.

Some particular features of ModbusTCP is that it is usually a polling protocol.  A request is made, and then this request is fulfilled in a response.  This applies to CIP explicit messaging as well (CIP is a suite of protocols, and the explicit messaging type behaves this way).  Based on this behavior, there are several factors to note:

1. No response with updated data is usually available until a request is made.  There are exceptions, but the vast majority of these types of communications is pure polling - there is little to no report-by-exception.  The server issues responses.
2. There is typically a request time - how often a request for new data is made.  This is called many things on many different platforms, but the concept is simple: every period of time, make a request for new data.  The client makes the request.
3. The device that acts as the server has a response time.  How long does it take to answer a request that comes in?  Wireshark is a nice tool to evaluate this, if the number of samples is not too large.
4. There is a concept of data freshness - how long does it take to update data in the client?  This is obviously a function of both the request period and the response time of the server device.

Some different role names used in ICS protocols:

 
and a visual on the request/response aspect of ModbusTCP:

Since we are capturing the traffic in Wireshark now, we can see the Queries going out, and then we can see the responses coming back.  It's straightforward to calculate the time between the two, and we can call that the response time.  The time between queries for a given set of data we can call Query-Query time, or what is called RepRate in the graphic above (it's not really a repetition rate, but rather a repetition period - I didn't name it!)

For CIP (EtherNet/IP) the same concepts exist for explicit messaging.  This is actually what makes it explicit - a request comes in and then it is answered.  It's object based -yada, yada, yada, but fundamentally the communications profile is the same - a request is sent, and it is answered.  Now there are advances here.  There is Implicit messaging as well (often called Class1 or IO messaging), which is very different.  It uses UDP at the transport layer, and the logical connection is set up in advance in which both sides agree to what data will be sent, how often, and with what options (timeouts, etc).  In this case, then, there is no request - each side, independently, just publishes data to the wire (possibly using multicast, other times using unicast at the network layer).  But explicit is different - the specific request comes in, and then is answered, and uses TCP at the transport layer.  There is Class3 messaging which means a CIP connection is set up before hand and then explicit messages are sent and responded to.  There is UCMM, where the message is just sent and it's best effort - this is much like ModbusTCP where there is no concept of an application layer connection like with Class3 messaging.  There are unconnected_sends, which uses UCMM but the actual data request is encapsulated.  I expect to talk more about EtherNet/IP in the future.

For analysis, Class1 messaging has a free tool from NIST available on SourceForge.  It's called IENetP, and is available at http://sourceforge.net/projects/ienetp/.  A method of analysis will be presented here for explicit type messaging, though right now it is for ModbusTCP only.  It could be easily extended to EtherNet/IP, and will be in the future.  Note that this request/response type communication protocol is similar to others: SNMP has a similar structure with gets and get-responses (and has trap type extensions which are nice additions to improve performance with a report-by-exception mechanism).    

Next time we'll introduce the tool and show some of the results.

Industrial Control Protocols

As Ethernet takes over the Industrial Control connectivity space, I see more and more need for analysis tools to quickly and accurately benchmark ICS (Industrial Control Systems) networks and identify actual (or preferably) potential problems.

There are many Ethernet diagnostic tools, many in long use by IT groups to monitor and manage Ethernet networks.  Some have made it over to the ICS world, some have not.  Some various tools in use:

1. SNMP with network management software
2. Device webpages
3. Netflow and Openflow tools
4. Packet sniffers
5. etc.

There are many packet sniffers available, depending on the platform in use.  These sniffers are able to capture and timestamp Ethernet frames as they traverse the network - of course, putting a tap in an appropriate place is always useful, and this determines exactly which frames you will get...

Some different sniffers include tcpdump, windump, tethereal, Wireshark, etc.  One common way to capture some network data for later use is to use a CLI type tool, such as tcpdump, and then follow up with Wireshark as the analysis tool.  Other times, one can use Wireshark directly and sniff the network, and perhaps performing some analysis in real time.

Wireshark is maintained as a free tool at http://www.wireshark.org/, and a good site for follow advancements is http://www.lovemytool.com/ (no, it's not that kind of site!).

So you want to capture some industrial control system traffic, like ModbusTCP, or EtherNet/IP, or... whatever.  How to do it?  Download and install Wireshark, and tell which interface to listen on...

 
Easy.  No problem.  I want to sniff on my LAN port... to see some ModbusTCP traffic (for example).  This traffic is between two others devices on my network.  So Wireshark will try and and put my network interface (LAN) into promiscuous mode so the actual packets are sent up the stack to be captured by Wireshark.  What happens is that the Ethernet interface knows it's MAC address, so is listening on the wire for packets destined to this MAC, like this:

In this case the Dell PC has an IP 192.168.3.254 with the MAC shown.  As frames come from the network, the NIC card is looking for this address.  Other addresses will be dropped - the frames are not destined for this device, so why even deal with them?  There are exceptions to this, of course, namely broadcast traffic - frames that have a destination MAC address of ff:ff:ff:ff:ff:ff. (or Wireshark labels as Broadcast).  There are some other exceptions to this as well, but for now, this is enough.

So we want to capture and analyze ModbusTCP traffic between two devices using this Dell PC.  Turning on Wireshark, and waiting about a minute:

  
I know there is plenty of ModbusTCP traffic, but I can't see it.  The little LED lights are flashing away...  Why?  A number of reasons:

1. I am using an Ethernet switch
2. My Firewall is blocking traffic
3. There really is no traffic

So how to get traffic?  If using a switch, this presents a problem.  Switches operate at layer-2 of the OSI 7-layer model, so filter based on MAC address, much as the Ethernet NIC in the Dell PC does.  If two devices are communicating, the switch is smart and maintains a forwarding table, so only frames destined for a given device are sent to that port.  This is precisely what distinguishes a switch from a hub - a hub sends frames to every port, every time... a switch will only forward to ports that expect the data.

How to get around this switch limitation?  There are a number of ways:

1. Span or mirror port - allows for a given port on the switch to copy data over to another port, which we can then use to capture traffic.  Requires use of a managed switch that supports this feature, and the 'copy' operation is usually lower priority, so cannot guarantee that data is exact, with no delay.
2. Put a hub inline with one of the devices to be monitored.  Degrades link speed to 10Mbps/half duplex.
3. Add a special network tap inline with one of the devices to capture traffic.  Must purchase and install something, can be expensive and usually must break link to put inline.

The easiest method is to use a mirror port, with a setup that may look like this:

This says to take the traffic on port 1.15 (Source) and copy over to port 1.16 (destination).  You can also do many to one, so have multiple ports copy their traffic over to a single destination port.  Be careful: each port is likely full line rate capable, so trying to copy multiple ports into one under heavy load will saturate the destination, causing loss of data at the destination.

So once we know data is being copied, we can put our sniffer device onto port 16 to monitor the communications from port 15 (obviously, make sure the communication we want to observe is actually using port 15!)

Will we be able to see this data?  Maybe.  The NIC card attached to port 16 will drop the traffic because it is destined for other devices... not here.  So the sniffer tool has to put the interface into promiscuous mode.  Wireshark will do this automatically.  Is this enough... maybe.  In many cases, it is not.  Corporate firewalls can be installed on local machines so even though the NIC will pass up the traffic because it is in promiscuous mode, the firewall will drop the traffic before it gets into the sniffer tool.  Solution: disable the firewall (just don't tell IT I told you to do that...).  Better yet, work with IT to provide a long term solution to the problem.  Good luck.

On Linux, ifconfig can be used to tell if your in promiscuous mode:

eth2      Link encap:Ethernet  HWaddr 66:1C:A3:12:0D:5f

          inet6 addr: fe80::6a1c:a3ff:fe12:d5f/64 Scope:Link

          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:0 (0.0 b)  TX bytes:468 (468.0 b)

OK... we have mirroring, we started Wireshark so the interface is in promiscuous mode, we disabled the firewall... what do we get...  

    

ModbusTCP traffic!  By ensuring we are capturing correctly, we can see traffic between two devices on the network.  This is the start of our analysis - we need to be able to capture suitable data before can analyze it.