Faking services on an entire IP subnet – controllerless proxying.

In part II we demonstrated faking IP services for an entire subnet. Now, let’s eliminate the need for an interactive OpenFlow control entirely, with the magic OVS learn actions. These are almost like OVS assembly language instructions, that allow OVS rewrite packets or even add flows to itself. Once new pipette has programmed OVS, once, OVS can respond to ARP requests and do the L2/L3 NAT translation entirely by itself.

This means we don’t have to rely on the controller being up to handle new connections, and connection handling is fast.

Here’s how OVS looks after it has just been programmed by pipette:

# ovs-ofctl -OOpenFlow13 dump-flows copro0
 cookie=0x0, duration=6.758s, table=0, n_packets=0, n_bytes=0, priority=1,in_port=enx0023565c8859,vlan_tci=0x1000/0x1000 actions=pop_vlan,goto_table:1
 cookie=0x0, duration=6.758s, table=0, n_packets=0, n_bytes=0, priority=1,tcp,in_port=enx0023565c8859 actions=goto_table:1
 cookie=0x0, duration=6.758s, table=0, n_packets=0, n_bytes=0, priority=1,udp,in_port=enx0023565c8859 actions=goto_table:1
 cookie=0x0, duration=6.758s, table=0, n_packets=0, n_bytes=0, priority=1,tcp,in_port=ovsfake0 actions=push_vlan:0x8100,set_field:4098->vlan_vid,goto_table:2
 cookie=0x0, duration=6.758s, table=0, n_packets=0, n_bytes=0, priority=1,udp,in_port=ovsfake0 actions=push_vlan:0x8100,set_field:4098->vlan_vid,goto_table:2
 cookie=0x0, duration=6.757s, table=0, n_packets=0, n_bytes=0, priority=1,arp,in_port=ovsfake0,dl_src=0e:00:00:00:00:66 actions=goto_table:2
 cookie=0x0, duration=6.759s, table=0, n_packets=6, n_bytes=621, priority=0 actions=drop
 cookie=0x0, duration=6.758s, table=1, n_packets=0, n_bytes=0, priority=1,tcp actions=move:NXM_OF_IP_SRC[]->NXM_NX_REG0[],load:0xa0a->NXM_NX_REG0[16..31],move:NXM_NX_REG0[0..7]->NXM_NX_REG0[8..15],move:NXM_OF_IP_DST[0..7]->NXM_NX_REG0[0..7],load:0x2->NXM_NX_REG1[0..2],load:0x1->NXM_NX_REG2[0..2],learn(table=1,hard_timeout=300,priority=2,eth_type=0x800,NXM_OF_IP_SRC[],NXM_OF_IP_DST[],load:NXM_NX_REG0[]->NXM_OF_IP_SRC[],load:0xa0a0001->NXM_OF_IP_DST[],load:0xe0000000067->NXM_OF_ETH_SRC[],load:0xe0000000066->NXM_OF_ETH_DST[],output:NXM_NX_REG1[0..1]),learn(table=2,idle_timeout=300,priority=2,eth_type=0x800,ip_src=10.10.0.1,NXM_OF_IP_DST[]=NXM_NX_REG0[],load:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],load:NXM_OF_ETH_DST[]->NXM_OF_ETH_SRC[],load:NXM_OF_IP_SRC[]->NXM_OF_IP_DST[],load:NXM_OF_IP_DST[]->NXM_OF_IP_SRC[],output:NXM_NX_REG2[0..1]),set_field:0e:00:00:00:00:67->eth_src,set_field:0e:00:00:00:00:66->eth_dst,move:NXM_NX_REG0[]->NXM_OF_IP_SRC[],set_field:10.10.0.1->ip_dst,output:ovsfake0
 cookie=0x0, duration=6.758s, table=1, n_packets=0, n_bytes=0, priority=1,udp actions=move:NXM_OF_IP_SRC[]->NXM_NX_REG0[],load:0xa0a->NXM_NX_REG0[16..31],move:NXM_NX_REG0[0..7]->NXM_NX_REG0[8..15],move:NXM_OF_IP_DST[0..7]->NXM_NX_REG0[0..7],load:0x2->NXM_NX_REG1[0..2],load:0x1->NXM_NX_REG2[0..2],learn(table=1,hard_timeout=300,priority=2,eth_type=0x800,NXM_OF_IP_SRC[],NXM_OF_IP_DST[],load:NXM_NX_REG0[]->NXM_OF_IP_SRC[],load:0xa0a0001->NXM_OF_IP_DST[],load:0xe0000000067->NXM_OF_ETH_SRC[],load:0xe0000000066->NXM_OF_ETH_DST[],output:NXM_NX_REG1[0..1]),learn(table=2,idle_timeout=300,priority=2,eth_type=0x800,ip_src=10.10.0.1,NXM_OF_IP_DST[]=NXM_NX_REG0[],load:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],load:NXM_OF_ETH_DST[]->NXM_OF_ETH_SRC[],load:NXM_OF_IP_SRC[]->NXM_OF_IP_DST[],load:NXM_OF_IP_DST[]->NXM_OF_IP_SRC[],output:NXM_NX_REG2[0..1]),set_field:0e:00:00:00:00:67->eth_src,set_field:0e:00:00:00:00:66->eth_dst,move:NXM_NX_REG0[]->NXM_OF_IP_SRC[],set_field:10.10.0.1->ip_dst,output:ovsfake0
 cookie=0x0, duration=6.759s, table=1, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6.758s, table=2, n_packets=0, n_bytes=0, priority=1,arp,arp_op=1 actions=move:NXM_OF_ARP_TPA[]->NXM_NX_REG0[],load:0x2->NXM_OF_ARP_OP[0..2],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],move:NXM_NX_REG0[]->NXM_OF_ARP_SPA[],set_field:0e:00:00:00:00:67->eth_src,set_field:0e:00:00:00:00:67->arp_sha,IN_PORT
 cookie=0x0, duration=6.758s, table=2, n_packets=0, n_bytes=0, priority=0 actions=drop

While scary looking, the flows are mostly just rearranging fields. Let’s look at the flow that responds to ARP:

 cookie=0x0, duration=6.758s, table=2, n_packets=0, n_bytes=0, priority=1,arp,arp_op=1 actions=move:NXM_OF_ARP_TPA[]->NXM_NX_REG0[],load:0x2->NXM_OF_ARP_OP[0..2],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],move:NXM_NX_REG0[]->NXM_OF_ARP_SPA[],set_field:0e:00:00:00:00:67->eth_src,set_field:0e:00:00:00:00:67->arp_sha,IN_PORT

This flow matches ARP requests. Then, it turns the ARP packet into an ARP reply, rewrites source hardware addresses, and rewrites IP addresses (we temporarily store the ARP target address from the request in OVS’ software register reg0, so it doesn’t get overwritten and so we can use it at the end of the rewriting process). Finally, we say to output the packet on the same port it came in on.

Now let’s look at the really scary flow that implements learning:

cookie=0x0, duration=6.758s, table=1, n_packets=0, n_bytes=0, priority=1,tcp actions=move:NXM_OF_IP_SRC[]->NXM_NX_REG0[],load:0xa0a->NXM_NX_REG0[16..31],move:NXM_NX_REG0[0..7]->NXM_NX_REG0[8..15],move:NXM_OF_IP_DST[0..7]->NXM_NX_REG0[0..7],load:0x2->NXM_NX_REG1[0..2],load:0x1->NXM_NX_REG2[0..2],learn(table=1,hard_timeout=300,priority=2,eth_type=0x800,NXM_OF_IP_SRC[],NXM_OF_IP_DST[],load:NXM_NX_REG0[]->NXM_OF_IP_SRC[],load:0xa0a0001->NXM_OF_IP_DST[],load:0xe0000000067->NXM_OF_ETH_SRC[],load:0xe0000000066->NXM_OF_ETH_DST[],output:NXM_NX_REG1[0..1]),learn(table=2,idle_timeout=300,priority=2,eth_type=0x800,ip_src=10.10.0.1,NXM_OF_IP_DST[]=NXM_NX_REG0[],load:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],load:NXM_OF_ETH_DST[]->NXM_OF_ETH_SRC[],load:NXM_OF_IP_SRC[]->NXM_OF_IP_DST[],load:NXM_OF_IP_DST[]->NXM_OF_IP_SRC[],output:NXM_NX_REG2[0..1]),set_field:0e:00:00:00:00:67->eth_src,set_field:0e:00:00:00:00:66->eth_dst,move:NXM_NX_REG0[]->NXM_OF_IP_SRC[],set_field:10.10.0.1->ip_dst,output:ovsfake0

First, we calculate the source NAT address and put it in reg0. Then we have two learn() sections – one adds a flow to table 1 to implement inbound translation, and the other adds a flow to table 2 for outbound. Finally, we have to do NAT translation on the packet we just got and send it.

making docker applications build faster – caching packages

If you’re like me – you live in the middle of nowhere, in New Zealand, without fiber let alone any DSL, and you develop a lot of software using docker – you might find your build/deploy cycle a bit painful, since many docker builds download packages.

Even if you are an unusual case – you live in an urban area with good espresso on every corner, scarce parking, and fiber to the couch (complete with 5G) – retrieving lots of packages from disk/SSD is still faster than fiber.

Running a docker pull through cache can help if you have a few machines pulling the same images. However, you sometimes you still need to actually build those images locally, and the build process can involve a lot of apt-get’ing or apk add’ing.

To help with package retrieval over HTTP, you can run apt-cacher-ng. That’s great if you have a way to convince your docker build to use the cache (e.g. via an environment variable). However it’d be even nicer not to have to do anything special.

The missing piece in the puzzle, is a transparent HTTP proxy (like squid), that knows how to redirect requests to apt-cacher-ng. This is what I do and it makes docker builds really fast (at least, the package retrieval part). Using a relatively old squid redirector, jesred, squid intercepts common package retrieval URLs and pass them to apt-cacher-ng.

Here’s part of my /etc/jesred.conf:

regex ^http://((.*)archive.ubuntu.com/ubuntu/(dists|pool)/.*)$    http://localho
st:3142/\1
regex ^http://(security.ubuntu.com/ubuntu/(dists|pool)/.*)$    http://localhost:
3142/\1
regex ^http://(dl-cdn.alpinelinux.org/alpine.+)$    http://localhost:3142/\1
regex ^http://(.*cdn.*.debian.org/.+)$	http://localhost:3142/\1
regex ^http://(deb.debian.org/.+)$  http://localhost:3142/\1
regex ^http://(archive.raspberrypi.org/debian/.+)$    http://localhost:3142/\1

Of course, your squid installation has to be set up to do transparent caching on port 80, and has to reference jesred in /etc/squid/squid.conf:

redirect_program /usr/lib/squid/jesred

faking services on an entire IP subnet – part II (L3 NAT and fake services in docker)

In our last post, we used a new L2 OVS proxy to fake TCP services on an entire IP subnet. We used network namespaces for isolation. However, using namespaces can make running fake services under docker somewhat inconvenient. So this time around, we’re going to have pipette do L3 NAT for us as well, so we can just attach services as we please. We’re going to fake a webserver on port 80 (as docker based webservers are easy to come by).

We are using the same physical setup as last time. However, we are now using a slightly different FAUCET ACL. We are going to work with TCP port 80 only, and only on 192.168.2.0/24. Note that we are also instructing FAUCET to always add a VLAN VID tag when it dispatches the intercepted traffic to the coprocessor (not strictly necessary but does make things more consistent).

acls:  
  coprocessssh:
  - rule:
      dl_type: 0x800
      ip_proto: 6
      ipv4_src: 192.168.2.0/24
      ipv4_dst: 192.168.2.0/24
      tcp_dst: 80
      actions:
        output:
          vlan_vid: 2
          ports: [18]
  - rule:
      actions:
        allow: 1

We’ll also need a new version of pipette – this one is a bit smarter – it can do L3 NAT. Specifically, it will NAT 192.168.2.0/24 to 192.168.101.0/24 on the coprocessor. That way we can just run services that listen on 192.168.101.1/24, and they’ll appear in the real 192.168.2.0/24 network, magically with the right MAC addresses.

We’ll start pipette like this:

#!/bin/bash

# interface connected to FAUCET coprocessor port.
COPROINT=enx0023565c8859
# interface that will be created for fake services to run on.
FAKEINT=fake0
# Reserved MAC addresses for fake services to use to talk to clients.
FAKEHW=0e:00:00:00:00:66
FAKECLIENTHW=0e:00:00:00:00:67
# address fake services will be run on (will be proxied from real IPs)
FAKEIP=192.168.101.1/24
# OVS bridge name
BR=copro0
# pipette OF port
OF=6699

# Configure pipette's OVS switch.
# Remove all IP addresses, disable IPv6.
ip link add dev $FAKEINT type veth peer name ovs$FAKEINT
for i in $COPROINT $FAKEINT ovs$FAKEINT ovs-system ; do
  echo 1 > /proc/sys/net/ipv6/conf/$i/disable_ipv6
  ifconfig $i 0.0.0.0
done
ifconfig $COPROINT up
ifconfig ovs$FAKEINT up
ifconfig $FAKEINT hw ether $FAKEHW $FAKEIP up
ovs-vsctl del-br $BR 
ovs-vsctl add-br $BR
ovs-ofctl del-flows $BR
for i in $COPROINT ovs$FAKEINT ; do
  ovs-vsctl add-port $BR $i
done
ovs-vsctl set-controller $BR tcp:127.0.0.1:$OF

# Run pipette.
ryu-manager pipette.py --ofp-tcp-listen-port $OF  --verbose

Now, we can start a webserver on 192.168.101.1:

pi@coprocessor:~ $ docker run -d -p 192.168.101.1:80:80 hypriot/rpi-busybox-httpd
89425b1198b3eb30267f05fa10bc3691efc7b36710297e4695030705ee09c9bb
pi@coprocessor:~ $ docker ps
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS                      NAMES
89425b1198b3        hypriot/rpi-busybox-httpd   "/bin/busybox httpd …"   11 seconds ago      Up 9 seconds        192.168.101.1:80->80/tcp   keen_goldberg

Now all it remains for us to do, is from outside the coprocessor, try to access the webserver “allegedly” running on 192.168.2.1:

pi@pi8021x:~ $ wget -q -O- 192.168.2.1 80
<html>
<head><title>Pi armed with Docker by Hypriot</title>
  <body style="width: 100%; background-color: black;">
    <div id="main" style="margin: 100px auto 0 auto; width: 800px;">
      <img src="pi_armed_with_docker.jpg" alt="pi armed with docker" style="width: 800px">
    </div>
  </body>
</html>

With pipette now doing L3 NAT, we can start pretty much whatever services we like listening on fake0, and provided we use the right FAUCET ACL to intercept the traffic, those services will appear in the dataplane.

faking services on an entire IP subnet

In a previous post, we introduced coprocessing and faking a TCP service on a single IP. The FAUCET coprocessor feature allows an external host to inject packets to a given VLAN (or even port). FAUCET ACLs are used to select traffic to be sent to the coprocessor (by default nothing is sent for coprocessing).

In this post we will develop the idea further – we will fake a TCP service on all hosts in a subnet, and we will do away with the need to deal with ARP.

Consider this topology. The two real hosts are at the bottom. We want to cause, from 192.168.2.5’s point of view, 192.168.2.1 to have an SMTP service it doesn’t really have. Further, from 192.168.2.5’s point of view, we want any host within 192.168.2.0/24 to appear to have that service.

The first step is to have FAUCET apply a port ACL to 192.168.2.5’s port, saying we want TCP port 25 (we’re faking SMTP) for all of 192.168.2.0/24 to be coprocessed. We could also use a VLAN ACL (pipette would need a small modification to allow it to receive tagged VLAN packets in that case).

Port 13 will have the host with 192.168.2.5 – port 18 is where the coprocessor is connected.

vlans:
  trusted:
    vid: 2
interfaces:
  13:
    acls_in:
      - coprocessssh
    description: 802.1x Pi
    native_vlan: trusted
  18:
    coprocessor:
      strategy: vlan_vid
    description: coprocessor NFV
acls:
  coprocessssh:
  - rule:
      dl_type: 0x800
      ip_proto: 6
      ipv4_src: 192.168.2.0/24
      tcp_dst: 25
      actions:
        output:
          ports: [18]
  - rule:
      actions:
        allow: 1

Now, on the coprocessor, we’ll need OVS and Ryu installed, and pipette. pipette patches TCP traffic in and out of a namespace where the fake service will run while maintaining the L2 illusion that the fake service is running on the real host.

# interface connected to FAUCET switch
NFVINT=enx0023565c8859
# used privately by the fake service.
FAKEHW=0e:00:00:00:00:66
FAKECLIENTHW=0e:00:00:00:00:67
# all 192.168.2.0/24 port 25 will be redirected to this fake IP.
FAKEIP=192.168.2.1/24
NS=copro

ip netns add $NS
ifconfig $NFVINT up
ip link add dev fake0 type veth peer name fakeovs0
ifconfig fakeovs0 up
ip link set fake0 netns $NS
ip netns exec $NS ifconfig fake0 hw ether $FAKEHW $FAKEIP up
for i in $NFVINT fakeovs0 ovs-system ; do
  echo 1 > /proc/sys/net/ipv6/conf/$i/disable_ipv6
done
ifconfig ovs-system 0.0.0.0

ovs-vsctl del-br br0
ovs-vsctl add-br br0
ovs-ofctl del-flows br0

for i in $NFVINT fakeovs0 ; do
  ovs-vsctl add-port br0 $i
done

ovs-vsctl set-controller br0 tcp:127.0.0.1:6653
ryu-manager pipette.py --verbose

Now we can run the fake SMTP service in the fake namespace. Note, even though the fake service is only running on 192.168.2.1, pipette is arranging for TCP traffic for the entire subnet to be mapped to that IP address as below.

# while /bin/true ; do echo completely legit SMTP service| ip netns exec copro nc -l 192.168.2.1 25 ; done

With everything ready, let’s try to connect to the fake SMTP server, apparently on 192.168.2.1, but in reality inside the coprocessor:

pi@pi8021x:~ $ telnet 192.168.2.1 25
Trying 192.168.2.1...
Connected to 192.168.2.1.
Escape character is '^]'.
completely legit SMTP service

How odd! What about 192.168.2.2?

pi@pi8021x:~ $ telnet 192.168.2.2 25
Trying 192.168.2.2...
Connected to 192.168.2.2.
Escape character is '^]'.
completely legit SMTP service

It appears there too! What’s going on? pipette is doing some TCP MAC and IP translation and mapping that into an isolated namespace, while also proxying the real host’s MAC address dynamically.


root@coprocessor:/home/pi# ovs-ofctl -OOpenFlow13 dump-flows br0
 cookie=0x0, duration=178.489s, table=0, n_packets=6, n_bytes=428, priority=1,in_port=enx0023565c8859 actions=goto_table:1
 cookie=0x0, duration=178.489s, table=0, n_packets=9, n_bytes=600, priority=1,in_port=fakeovs0 actions=goto_table:2
 cookie=0x0, duration=178.490s, table=0, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=14.819s, table=1, n_packets=1, n_bytes=70, idle_timeout=30, priority=2,tcp,dl_src=b8:27:eb:d6:31:ca,nw_dst=192.168.2.1,tp_dst=25 actions=set_field:0e:00:00:00:00:67->eth_src,set_field:0e:00:00:00:00:66->eth_dst,set_field:192.168.2.1->ip_dst,output:fakeovs0
 cookie=0x0, duration=178.490s, table=1, n_packets=2, n_bytes=146, priority=1,tcp actions=CONTROLLER:65509
 cookie=0x0, duration=178.490s, table=1, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=14.819s, table=2, n_packets=1, n_bytes=66, idle_timeout=30, priority=2,tcp,nw_dst=192.168.2.5,tp_src=25 actions=set_field:aa:56:63:87:9e:e9->eth_src,set_field:b8:27:eb:d6:31:ca->eth_dst,set_field:192.168.2.1->ip_src,push_vlan:0x8100,set_field:4098->vlan_vid,output:enx0023565c8859
 cookie=0x0, duration=178.490s, table=2, n_packets=2, n_bytes=84, priority=1,arp actions=CONTROLLER:65509
 cookie=0x0, duration=178.490s, table=2, n_packets=4, n_bytes=280, priority=0 actions=drop

Building a cost-effective OpenFlow wireless development lab

Choosing the right hardware

As they say when purchasing new hardware, you can have two of the following: good, cheap or fast. When building a development lab environment we want cheap and can sacrifice on fast. We won’t be getting line rate gigabit, instead in the order of 800 Mbit/s which will still be plenty sufficient for many applications.

Today we are going to be focusing specifically on the $69 GL-AR750S wireless travel router from GL-iNet since I have tested it and know it works well, however many other GL-iNet devices as well as many other devices supported by OpenWrt should work as well. The GL-AR750S has 3 gigabit ports and a dual-band 2.4GHz/5GHz wireless radio. The firmware it ships with is a modified version of OpenWrt without OpenFlow support but we can easily flash a version of vanilla OpenWrt with the latest version of Open vSwitch which will let us do OpenFlow on this device.

GL-AR750S Lab
Faucet WiFi lab with three GL-AR750S travel routers and one management switch

Performance of the GL-AR750S is pretty good considering all the switching happens in software on the embedded CPU (no hardware offloads). With iperf I could achieve 820 Mbits/sec between the two LAN ports when using advanced faucet features such as inter-VLAN routing. WiFi performance was a bit more variable with iperf results in the range of 80 – 180 Mbits/sec.

Device setup

The following steps will walk you through setting up the GL-AR750S for use in an OpenFlow development lab. First, we will flash the latest copy of vanilla OpenWrt, then we will configure it to work with OpenFlow by adding Open vSwitch which will be our OpenFlow Agent. We will then install and configure faucet to be our OpenFlow controller.

1. Flash OpenWrt

The first step is to replace the firmware version shipped with the device with the latest version of OpenWrt. The GL-AR750S uses both NAND and NOR memory which isn’t supported by the latest stable version of OpenWrt, so we will be using a nightly snapshot of OpenWrt (we need to use this anyway to get the latest version of Open vSwitch). The process for flashing the latest OpenWrt snapshot version on the GL-AR750S is documented in this forum post, however we will detail the steps in this blog post as well:

  • Download the latest OpenWrt snapshot NOR firmware for GL-AR750S
  • Put the GL-AR750S into debrick mode and flash NOR firmware file:
    • Connect your computer to one of the LAN Ethernet ports of the router. You must leave the other ports unconnected.
    • Fold the antenna so they are pointing up and hold down the reset button (which is on the side of the device) and plug the power cable in.
    • Watch the 5G LED on the front, once it has flashed 5 times you can release the reset button.
    • Set your computer’s IP address to 192.168.1.2.
    • Use Firefox or Chrome to visit http://192.168.1.1.
    • Upload the NOR firmware file openwrt-ath79-nand-glinet_gl-ar750s-nor-squashfs-sysupgrade.bin (that you downloaded in the earlier step) to the webpage and click Update Firmware button.
    • Wait for around 3 minutes. Don’t power off your device when updating. The router is ready when the power LED is solid and not blinking.
  • Download the latest OpenWrt snapshot NOR/NAND firmware for GL-AR750S
  • Use the sysupgrade method to flash the NOR/NAND firmware to gain access to entire flash filesystem space, the steps for doing this are as follows:
    • Connect your computer to one of the LAN Ethernet ports of the router (you can’t use the WAN port).
    • Set your computer to use DHCP, it should receive an address inside of 192.168.1.0/24.
    • Copy the NOR/NAND firmware file openwrt-ath79-nand-glinet_gl-ar750s-nor-nand-squashfs-sysupgrade.bin from your computer to /tmp on the router with SCP:
$ scp openwrt-ath79-nand-glinet_gl-ar750s-nor-nand-squashfs-sysupgrade.bin root@192.168.1.1:/tmp/
    • SSH to the router and run the sysupgrade command:
$ ssh root@192.168.1.1

root@OpenWrt:~# sysupgrade -v /tmp/openwrt-ath79-nand-glinet_gl-ar750s-nor-nand-squashfs-sysupgrade.bin
    • Wait approximately 1 minute for the upgrade to finish before logging back in and completing the next steps of configuring the device.

2. Configure OpenWrt

OpenWrt uses a system called UCI for configuration. You can also install the web interface LUCI if you want a web GUI for configuring things how you like, but I’ll just use the UCI CLI for this tutorial.

Before we start configuring the device you will need to SSH into the router from your computer:

$ ssh root@192.168.1.1

Now we will start by doing a few basic things such as setting a password and hostname by running the following commands:

root@OpenWrt:~# passwd

root@OpenWrt:~# uci set system.@system[0].hostname='openflow-ap'

root@OpenWrt:~# uci commit
root@OpenWrt:~# reload_config

Now we want to delete the default networking configuration that comes with OpenWrt and add our own so that the router will do what we want.

First, let’s go ahead and delete the default firewall configuration so that it doesn’t get in our way:

root@openflow-ap:~# while $(uci delete firewall.@rule[-1] 2>/dev/null); do :; done
root@openflow-ap:~# while $(uci delete firewall.@zone[-1] 2>/dev/null); do :; done
root@openflow-ap:~# while $(uci delete firewall.@forwarding[-1] 2>/dev/null); do :; done

root@openflow-ap:~# uci commit
root@openflow-ap:~# reload_config

Now we need to reconfigure the network so that the physical ports are mapped to the right virtual interfaces and that we have an IP address assigned for managing the router. The default network configuration for this device is the physical wan port (1) is mapped to virtual interface eth0.2 and both the physical lan ports (2 and 3) are both mapped to the same virtual interface eth0.1.

What we want to do is remap these physical ports so that they are all exposed as separate virtual network interfaces in OpenWrt which will give us full control over the switching behaviour. We also need to setup a network interface that can be used to talk to our OpenFlow controller. I personally use out of band control since it is a lot simpler, but it does mean I will have to reserve one of my physical network ports for this. If you want to keep all three network ports you could look at Open vSwitch’s in-band mode, but I won’t be covering this here.

First, we will clear the current physical port and virtual interfaces mapping, then redefine the mapping so that:

Physical PortVirtual Interface
waneth0.1
lan1eth0.2
lan2eth0.3

Run the following commands to redefine the port mappings to match the table above:

root@openflow-ap:~# while $(uci delete network.@switch_vlan[-1] 2>/dev/null); do :; done

root@openflow-ap:~# uci add network switch_vlan
root@openflow-ap:~# uci set network.@switch_vlan[-1]=switch_vlan
root@openflow-ap:~# uci set network.@switch_vlan[-1].device='switch0'
root@openflow-ap:~# uci set network.@switch_vlan[-1].vlan='1'
root@openflow-ap:~# uci set network.@switch_vlan[-1].ports='1 0t'

root@openflow-ap:~# uci add network switch_vlan
root@openflow-ap:~# uci set network.@switch_vlan[-1]=switch_vlan
root@openflow-ap:~# uci set network.@switch_vlan[-1].device='switch0'
root@openflow-ap:~# uci set network.@switch_vlan[-1].vlan='2'
root@openflow-ap:~# uci set network.@switch_vlan[-1].ports='2 0t'

root@openflow-ap:~# uci add network switch_vlan
root@openflow-ap:~# uci set network.@switch_vlan[-1]=switch_vlan
root@openflow-ap:~# uci set network.@switch_vlan[-1].device='switch0'
root@openflow-ap:~# uci set network.@switch_vlan[-1].vlan='3'
root@openflow-ap:~# uci set network.@switch_vlan[-1].ports='3 0t'

Now what we will do is delete the predefined lan, wan and wan6 interfaces and create our own new lan1 and lan2 interfaces:

root@openflow-ap:~# uci delete network.lan
root@openflow-ap:~# uci delete network.wan
root@openflow-ap:~# uci delete network.wan6

root@openflow-ap:~# uci set network.lan1=interface
root@openflow-ap:~# uci set network.lan1.ifname=eth0.2

root@openflow-ap:~# uci set network.lan2=interface
root@openflow-ap:~# uci set network.lan2.ifname=eth0.3

I also re-assigned the wan interface to use as an out-of-band management interface (for SSH management and OpenFlow control channel), we can configure the new interface as DHCP:

root@openflow-ap:~# uci set network.mgmt=interface
root@openflow-ap:~# uci set network.mgmt.ifname=eth0.1
root@openflow-ap:~# uci set network.mgmt.proto=dhcp

Or with a static IP address (change ipaddr/netmask/gateway/dns to match your network):

root@openflow-ap:~# uci set network.mgmt=interface
root@openflow-ap:~# uci set network.mgmt.ifname=eth0.1
root@openflow-ap:~# uci set network.mgmt.proto=static
root@openflow-ap:~# uci set network.mgmt.ipaddr=172.16.0.1
root@openflow-ap:~# uci set network.mgmt.netmask=255.255.255.0
root@openflow-ap:~# uci set network.mgmt.gateway=172.16.0.254
root@openflow-ap:~# uci add_list network.mgmt.dns=172.16.0.254

Finally, let’s make sure the SSH daemon is only listening on the new management interface and that all the DHCP servers are disabled:

root@openflow-ap:~# uci set dropbear.@dropbear[0].Interface='mgmt'
root@openflow-ap:~# uci delete dhcp.lan
root@openflow-ap:~# uci delete dhcp.wan

Finally, we will commit all these network changes together:

root@openflow-ap:~# uci commit
root@openflow-ap:~# reload_config

Note: If you mess up your settings or lock yourself out of the router you can use the OpenWrt failsafe mode to gain access to your device again.

3. Configure WiFi

In this step we will configure the 2.4GHz/5GHz WiFi radios so that they start broadcasting an SSID, in this case faucetsdn and using the Pre-Shared Key faucetsdn.

root@openflow-ap:~# for radio in 0 1; do
root@openflow-ap:~# uci delete wireless.radio${radio}.disabled
root@openflow-ap:~# uci set wireless.default_radio${radio}.encryption=psk2
root@openflow-ap:~# uci set wireless.default_radio${radio}.ssid=faucetsdn
root@openflow-ap:~# uci set wireless.default_radio${radio}.key=faucetsdn
root@openflow-ap:~# uci delete wireless.default_radio${radio}.network
root@openflow-ap:~# done

root@openflow-ap:~# uci commit
root@openflow-ap:~# reload_config

4. Configure Open vSwitch

Now let’s install Open vSwitch and configure it so that our device can start speaking OpenFlow.

First, install the openvswitch package with the opkg package manager:

root@openflow-ap:~# opkg update
root@openflow-ap:~# opkg install openvswitch

Now we need enable Open vSwitch so that it starts on boot and configure a virtual network interface for our Open vSwitch bridge (ovsbr) so that OpenWrt knows it exists and can interact with it.

root@openflow-ap:~# uci set openvswitch.ovs.disabled=0
root@openflow-ap:~# uci set network.ovsbr=interface
root@openflow-ap:~# uci set network.ovsbr.ifname=ovsbr
root@openflow-ap:~# uci set network.ovsbr.proto=static

root@openflow-ap:~# uci commit
root@openflow-ap:~# reload_config
root@openflow-ap:~# /etc/init.d/openvswitch start

Now that Open vSwitch is installed we need to configure it by creating a bridge and attaching our two lan ports (eth0.2 and eth0.3) and our two WiFi ports (wlan0 and wlan1):

root@openflow-ap:~# ovs-vsctl --may-exist add-br ovsbr -- \
    set bridge ovsbr other-config:disable-in-band=true -- \
    set bridge ovsbr fail_mode=secure

root@openflow-ap:~# ovs-vsctl --may-exist add-port ovsbr eth0.2 -- set Interface eth0.2 ofport_request=1
root@openflow-ap:~# ovs-vsctl --may-exist add-port ovsbr eth0.3 -- set Interface eth0.3 ofport_request=2
root@openflow-ap:~# ovs-vsctl --may-exist add-port ovsbr wlan0 -- set Interface wlan0 ofport_request=3
root@openflow-ap:~# ovs-vsctl --may-exist add-port ovsbr wlan1 -- set Interface wlan1 ofport_request=4

We also need to add an additional patch to hostapd to add support for attaching to Open vSwitch bridges. WAND maintains a third party package repo which has a patched hostapd for OpenWrt, let’s add that repo:

root@openflow-ap:~# opkg install uclient-fetch libustream-openssl ca-bundle ca-certificates
root@openflow-ap:~# . /etc/os-release && echo "src/gz hostapd https://packages.wand.net.nz/openwrt/hostapd/${OPENWRT_ARCH}" >> /etc/opkg/customfeeds.conf
root@openflow-ap:~# wget https://packages.wand.net.nz/openwrt/hostapd/mips_24kc/repo-key.pub -O /tmp/hostapd-repo-key.pub
root@openflow-ap:~# opkg-key add /tmp/hostapd-repo-key.pub

When the repo is added we can then upgrade hostapd to the patched version and restart hostapd:

root@openflow-ap:~# opkg update
root@openflow-ap:~# opkg list-upgradable | grep -E '(wpad|hostapd)' | awk '{print $1}' | xargs opkg upgrade
root@openflow-ap:~# wifi up

5. Configure Faucet

Now that OpenWrt and Open vSwitch is all installed and configured our router is ready to receive its instructions from an OpenFlow controller.

I won’t fully cover installing and configuring faucet here, if you aren’t familiar with how to do that, follow the installing faucet for the first time tutorial.

To configure our router in faucet we will need its datapath id, this can be obtained from Open vSwitch with this command:

root@openflow-ap:~# ovs-ofctl show ovsbr | grep dpid
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000e4956e4a732a

We will also need to configure Open vSwitch to speak to our faucet controller:

root@openflow-ap:~# ovs-vsctl set-controller ovsbr tcp:172.16.0.10:6653 tcp:172.16.0.10:6654

Below is a very basic faucet configuration I used to bring up the router with all ports on the same VLAN. Some interesting things to note about the configuration is the eapol-to-local ACL which sends EAPOL packets (which are used for WiFi authentication) to the OFPP_LOCAL port which is where our patched hostapd is listening and can reply (we could also use faucet to NFV these packets to a central hostapd server for processing). We will also see hairpin mode is enabled on the WiFi ports, this allows packets that enter the WiFi interfaces to be sent back out the same interface (useful if you don’t want client isolation enabled on your radios). If you want to play with more faucet features try the faucet tutorial series.

faucet.yaml
vlans:
    office:
        description: "office vlan"
        vid: 10

acls:
    # Default allow
    default_allow:
        - rule:
            actions:
                allow: 1                 # allow

    # Default drop
    default_drop:
        - rule:
            actions:
                allow: 0                 # drop

    # Send EAPOL packets to OFPP_LOCAL port (so hostapd can process)
    eapol-to-local:
        - rule:
            dl_type: 0x888e              # EAPOL
            actions:
                output:
                    port: 0xfffffffe     # OFPP_LOCAL

dps:
    openflow-ap:
        dp_id: 0xe4956e4a732a
        description: "openflow-ap"
        hardware: "Open vSwitch"
        interfaces:
            1:
                name: "lan1"
                native_vlan: office
            2:
                name: "lan2"
                native_vlan: office
            3:
                name: "wlan0"
                native_vlan: office
                acls_in: [eapol-to-local, default_allow]
                hairpin: True
            4:
                name: "wlan1"
                native_vlan: office
                acls_in: [eapol-to-local, default_allow]
                hairpin: True
            0xfffffffe:
                description: "OFPP_LOCAL"
                output_only: True

deny but mirror

FAUCET ACLs are great for locking things down. And it is possible to add FAUCET ACL rules, that count traffic and let it through (instead of counting and denying it).

But sometimes that’s not enough – you want to deny the traffic, but you also want to see what it was. FAUCET’s ACL language lets you do this.

  - rule:
      actions:
        allow: 0
        output:
          ports: [99]

This rule, placed at the end of an ACL, will deny all traffic – but – before it denies it, it will make a copy and output it to port 99. In this way you can run a tcpdump on a host connected to port 99, and observe the denied traffic. Handy!

caching with docker-registry

Docker based systems are great, but if you live in New Zealand (and in rural New Zealand, like I do), repeated docker pulls to the Internet can be a little painful. Fortunately you can configure docker-registry to locally cache.

First install docker-registry and configure it as a proxy.

# apt-get install docker-registry
# vi /etc/docker/registry/config.yml
# cat /etc/docker/registry/config.yml 
version: 0.1
log:
fields:
service: registry
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /var/lib/docker-registry
delete:
enabled: true
http:
addr: :5000
headers:
X-Content-Type-Options: [nosniff]
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
proxy:
remoteurl: https://registry-1.docker.io
# service docker-registry restart

Now configure docker to use the proxy.

# vi /etc/docker/daemon.json 
# cat /etc/docker/daemon.json 
{
"storage-driver": "overlay2",
"registry-mirrors": ["http://localhost:5000"]
}
# service docker restart

Now when you do a docker pull, the cache will be used if the latest image is available locally.

# docker pull faucet/gauge
Using default tag: latest
latest: Pulling from faucet/gauge
Digest: sha256:26e2c56800df26deccc6a1d8377c697d7ca2a38cf2d8fc52739bb6b634028298
Status: Image is up to date for faucet/gauge:latest
docker.io/faucet/gauge:latest
# grep docker-registry /var/log/syslog|grep gauge
Nov 5 22:29:21 finf-gw docker-registry[29624]: time="2019-11-05T22:29:21Z" level=info msg="Challenge established with upstream : {https registry-1.docker.io /v2/ %!s(bool=false) } &{{{%!s(int32=0) %!s(uint32=0)} %!s(uint32=0) %!s(uint32=0) %!s(int32=0) %!s(int32=0)} map[https://registry-1.docker.io:443/v2/:[{bearer map[realm:https://auth.docker.io/token service:registry.docker.io]}]]}" go.version=go1.8.1 http.request.host="localhost:5000" http.request.id=fb8d44dd-193a-4a18-8625-35dc3ea015eb http.request.method=GET http.request.remoteaddr="[::1]:49450" http.request.uri=/v2/faucet/gauge/manifests/latest http.request.useragent="docker/19.03.4 go/go1.12.10 git-commit/9013bf583a kernel/5.3.0-19-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.4 (linux))" instance.id=9bdb53c3-1d48-4015-9708-8f1ddc816651 vars.name=faucet/gauge vars.reference=latest version=v2.6.2+debian
Nov 5 22:29:23 finf-gw docker-registry[29624]: time="2019-11-05T22:29:23Z" level=info msg="Adding new scheduler entry for faucet/gauge@sha256:26e2c56800df26deccc6a1d8377c697d7ca2a38cf2d8fc52739bb6b634028298 with ttl=167h59m59.999997987s" go.version=go1.8.1 instance.id=9bdb53c3-1d48-4015-9708-8f1ddc816651 version=v2.6.2+debian
Nov 5 22:29:23 finf-gw docker-registry[29624]: time="2019-11-05T22:29:23Z" level=info msg="response completed" go.version=go1.8.1 http.request.host="localhost:5000" http.request.id=fb8d44dd-193a-4a18-8625-35dc3ea015eb http.request.method=GET http.request.remoteaddr="[::1]:49450" http.request.uri=/v2/faucet/gauge/manifests/latest http.request.useragent="docker/19.03.4 go/go1.12.10 git-commit/9013bf583a kernel/5.3.0-19-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.4 (linux))" http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=3.545613795s http.response.status=200 http.response.written=1582 instance.id=9bdb53c3-1d48-4015-9708-8f1ddc816651 version=v2.6.2+debian
Nov 5 22:29:23 finf-gw docker-registry[29624]: ::1 - - [05/Nov/2019:22:29:20 +0000] "GET /v2/faucet/gauge/manifests/latest HTTP/1.1" 200 1582 "" "docker/19.03.4 go/go1.12.10 git-commit/9013bf583a kernel/5.3.0-19-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.4 \(linux\))"
Nov 5 22:31:51 finf-gw docker-registry[29624]: time="2019-11-05T22:31:51Z" level=info msg="response completed" go.version=go1.8.1 http.request.host="localhost:5000" http.request.id=013a90e2-11e3-44d8-8875-03aca0c54be5 http.request.method=GET http.request.remoteaddr="[::1]:49702" http.request.uri=/v2/faucet/gauge/manifests/latest http.request.useragent="docker/19.03.4 go/go1.12.10 git-commit/9013bf583a kernel/5.3.0-19-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.4 (linux))" http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=2.237893853s http.response.status=200 http.response.written=1582 instance.id=9bdb53c3-1d48-4015-9708-8f1ddc816651 version=v2.6.2+debian
Nov 5 22:31:51 finf-gw docker-registry[29624]: ::1 - - [05/Nov/2019:22:31:49 +0000] "GET /v2/faucet/gauge/manifests/latest HTTP/1.1" 200 1582 "" "docker/19.03.4 go/go1.12.10 git-commit/9013bf583a kernel/5.3.0-19-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.4 \(linux\))"