DokuWiki

Fair traffic shaping an ADSL line for a local network using Linux

I originally intended this to be a fully usable script to shape an ADSL line. However, given the amount of development work I have done to improve the script and the associated added complications, I now see this as a good tutorial instead. It can be used in its current form and should work quite well, but for production use I recommend the scripts located in the full web-portal instructions: here

Traffic shaping a standard ADSL link in order to share it with a couple of hundred users is a common problem. There are dozens of bits of software and firewall scripts out there already to do this. This particular script is one that I have written. It aims to be as simple as possible, is easily customised, and uses connlimit to identify P2P users. Although the latter is not 100% reliable, it seems to work pretty well and does not fall foul of any new/changed P2P software that happens to some of the other scripts.

Software required

Any recent Linux distribution should be able to be used for these scripts. The example shown here uses Debian.

The following packages are required:

Kernel 2.6.26
IPset

IPset is not yet in the stable kernel, so the easiest way to install it (in Debian Squeeze) is:

aptitude install ipset ipset-source module-assistant
module-assistant auto-install ipset-source

Principle of operation

All traffic is given an iptables MARK depending on its type:
- 10 for low latency traffic such as SSH shells
- 20 for outbound ack packets
- 30 for normal web browsing
- 40 for any other traffic
- 50 for bulk traffic
- 666 for P2P traffic
HTB is used to create classes for all this traffic
All traffic shaping is hashed by client, not by connection. This means that one user with several large download connections will get the same amount of total bandwidth as one user with one connection. The default within Linux and routers is normally to do it per connection.
ipset is used to generate a list of client IP addresses that are using P2P software.
connlimit is used to detect P2P software. The firewall rules below look for multiple connections to high port numbers from a client.

Assumptions

ppp0 is the internet connection
eth0 is the local network connection

Firewall rules required

I will first describe some of the rules that form the basis of the traffic shaping. Later in this page the full script is shown.

The first task is to create an ipset for storing the IP addresses of all our naughty users slurping up the bandwidth with constantly downloading P2P software. A timeout of 60 seconds is used, so that as soon as they turn off their software their IP address is removed. The current IP addresses in the IPset can be monitored with the 'ipset -L' comand from the bash prompt

# Create a set called p2p with 60 second timeout
ipset -N p2p iptree --timeout 60

Next we need to mark traffic as required, using the principles set out earlier. The following code contains some examples.

# Set default mark for forwarded packets
$IPTABLES -t mangle -A FORWARD -j MARK --set-mark 40

# Mark http and https traffic as 30, both in and out
$IPTABLES -t mangle -A FORWARD -p tcp --sport 80 -i ppp0 -j MARK --set-mark 30
$IPTABLES -t mangle -A FORWARD -p tcp --dport 80 -o ppp0 -j MARK --set-mark 30
$IPTABLES -t mangle -A FORWARD -p tcp --sport 443 -i ppp0 -j MARK --set-mark 30
$IPTABLES -t mangle -A FORWARD -p tcp --dport 443 -i eth0 -j MARK --set-mark 30

# Mark in and out SSH traffic as high priority
$IPTABLES -t mangle -A FORWARD -p tcp --sport 22 -i ppp0 -j MARK --set-mark 10
$IPTABLES -t mangle -A FORWARD -p tcp --dport 22 -o ppp0 -j MARK --set-mark 10

# Mark DNS traffic from localhost
$IPTABLES -t mangle -A OUTPUT -m udp -p udp --dport 53 -o ppp0 -j MARK --set-mark 10
$IPTABLES -t mangle -A OUTPUT -m udp -p udp --sport 53 -o eth0 -j MARK --set-mark 10

# IMAP
$IPTABLES -t mangle -A FORWARD -p tcp --sport 993 -i ppp0 -j MARK --set-mark 30
$IPTABLES -t mangle -A FORWARD -p tcp --dport 993 -o ppp0 -j MARK --set-mark 30

# Mark any large downloads as 40 (they may have been marked 30 or 10 earlier)
$IPTABLES -t mangle -A FORWARD -m connbytes --connbytes 504857: --connbytes-dir both \\
  --connbytes-mode bytes -j MARK --set-mark 40

# To speed up downloads while an upload is going on, put short ACK
# packets in their own class:
$IPTABLES -t mangle -A FORWARD -o ppp0 -p tcp -m tcp --tcp-flags \\
  FIN,SYN,RST,ACK ACK -m length --length :64 -j MARK --set-mark 20

Now we need to look out for all those P2P connections. We're going to find these out by looking for a client on the network making lots of connections to high port numbers, which is generally what P2P software does. This isn't foolproof of course: I have seen P2P software start to use port 80, and there could be false negatives, but on the whole it seems to work better than any other solutions out there that I have tried.

# The next few lines pick out P2P packets
# These are characterised by lots of connections to high port numbers
# First look for the packets and log to IPSET p2p
# Detects traffic from users using >8 ports above 1024
# and adds the source address to the P2P list.
$IPTABLES -t mangle -A FORWARD -o ppp0 -p tcp --dport 1024: \\
       -m connlimit --connlimit-above 8 -j SET --add-set p2p src

# Detects traffic from users using >4 UDP ports above 1024
# and adds the source address to the P2P list.
$IPTABLES -t mangle -A FORWARD -o ppp0 -p udp --dport 1024: \\
       -m connlimit --connlimit-above 4 -j SET --add-set p2p src

# Detects traffic to users using >8 ports above 1024
# and adds the dst address (ie the user) to the P2P list.
$IPTABLES -t mangle -A FORWARD -i ppp0 -p tcp --sport 1024: \\
       -m connlimit --connlimit-above 8 -j SET --add-set p2p dst

# Detects traffic to users using >4 UDP ports above 1024
# and adds the dst address (ie the user) to the P2P list.
$IPTABLES -t mangle -A FORWARD -i ppp0 -p udp --sport 1024: \\
       -m connlimit --connlimit-above 4 -j SET --add-set p2p dst

The above rules just add the client IP address to the ipset. We now need to mark the traffic, which the following rules do. On one network, traffic became so slow that I marked ALL traffic to and from those clients as '60'. This certainly sped the network up, but of course the entire internet connection for that client became really slow. If you want to do that just remove the destination port parameters.

# Once a user is in the p2p IPSET, these rules mark their packets
# Any packets above 1024 are marked as the lowest priority
$IPTABLES -t mangle -A FORWARD -o ppp0 -p tcp --dport 1024: \\
       -m set --set p2p src -j MARK --set-mark 666
$IPTABLES -t mangle -A FORWARD -i ppp0 -p tcp --sport 1024: \\
       -m set --set p2p dst -j MARK --set-mark 666
$IPTABLES -t mangle -A FORWARD -o ppp0 -p udp --dport 1024: \\
       -m set --set p2p src -j MARK --set-mark 666
$IPTABLES -t mangle -A FORWARD -i ppp0 -p udp --sport 1024: \\
       -m set --set p2p dst -j MARK --set-mark 666

Shaping the traffic using HTB

Now that we have got to this stage, we have all our traffic nicely marked. All we need to do now is shape it. Unfortunately ingress shaping on Linux is somewhat limited, so instead we only do egress shaping. To limit traffic coming in from the internet, we shape the traffic as it leaves the server. Not ideal, but it seems to work okay.

In these scripts I limit the downlink at 3600 kbps and the uplink at 550 kpbs. These figures *must* be less that your connection is capable of, otherwise no shaping will take place as the buffer at the other end of the ADSL link will do the shaping instead. The generally accepted approach is to set the values to 90% the theoretical maximum, but I advise you to experiment.

Downlink

First, the downlink. Shaping is done on eth0 as the traffic leaves for the network.

We use HTB to do the shaping. It can be difficult to understand HTB and tc in general, but if you look hard enough there is some good documentation out there, it's just hard to find. I will place some links at the end of this document.

First, some basics. A qdisc is a whole set of shaping rules that we apply, normally to a whole interface. Here we add a HTB qdisc to the interface eth0. This is known as the root for the interface.

We don't set a default class; this is so that local eth0 traffic is not shaped. As already stated, we have to shape at eth0 and not ppp0 as we can only do egress shaping decently.

# Create the root qdisc on the interface
tc qdisc add dev eth0 root handle 1: htb r2q 1

# Add some overall rate limits as defined above. Parent number 1:
# says to apply it to the root above. Call this child 1:1
tc class add dev eth0 parent 1: classid 1:1 htb rate 3600kbit ceil 3600kbit

We now have one root HTB qdisc with a rate limit of 3600kbps. To this we add 6 further children. Note the numbers. 1: (or 1:0) is the root. 1:1 is the first child with the overall rate limit. Each child of this is 1:10, 1:20 and so on. To make things simpler, I have numbered the children below to align with the MARK numbers. Note that all the rates of the children should add up to the single rate limit of the parent.

# Add a number of classes to the root qdisc.%%''\\
# With some qdiscs the classes are automatically%%''\\
# created. With HTB they are not so we add 4 in total%%''\\
# with different rate limits for each%%''\\

# interactive traffic
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 720kbit ceil 720kbit prio 0

# Note: no class 20. This is only used in the upload for ack packets

# web browsing
tc class add dev eth0 parent 1:1 classid 1:30 htb rate 1008kbit ceil 2880kbit prio 4

# default traffic
tc class add dev eth0 parent 1:1 classid 1:40 htb rate 792kbit ceil 2880kbit prio 4

# bulk
tc class add dev eth0 parent 1:1 classid 1:50 htb rate 720kbit ceil 3420kbit prio 4

# bad boys
tc class add dev eth0 parent 1:1 classid 1:666 htb rate 360kbit ceil 3600kbit prio 7

We now have the HTB qdisc fully set up. However, no traffic will be sent to it yet, and the traffic will not be shaped within each class.

The next set of rules shape traffic in each class. If we don't do this, then all the traffic for a particular class (such as all the webbrowsing traffic - 30) will be piled into the class on a fifo basis. We want to be more intelligent than this. SFQ does some nice fair shaping.

tc qdisc add dev eth0 parent 1:10 handle 4210: sfq perturb 10 limit 50
tc qdisc add dev eth0 parent 1:30 handle 4230: sfq perturb 10 limit 64
tc qdisc add dev eth0 parent 1:40 handle 4240: sfq perturb 10 limit 128
tc qdisc add dev eth0 parent 1:50 handle 4250: sfq perturb 10 limit 64
tc qdisc add dev eth0 parent 1:666 handle 666: sfq perturb 10 limit 128

So, everything is set up and ready to go. We just need to divert some traffic into each of the classes. We do this by attaching a filter to the class. A filter looks for traffic of a particular type and sucks it into the class. In this example, we use the MARK of the traffic (called flowid here).

tc filter add dev eth0 parent 1:0 protocol ip handle 10 fw flowid 1:10
tc filter add dev eth0 parent 1:0 protocol ip handle 30 fw flowid 1:30
tc filter add dev eth0 parent 1:0 protocol ip handle 40 fw flowid 1:40
tc filter add dev eth0 parent 1:0 protocol ip handle 50 fw flowid 1:50
tc filter add dev eth0 parent 1:0 protocol ip handle 666 fw flowid 1:666

Everything will be working nicely at this point. However, we have one more tweak to do. We want to share traffic between clients (by IP address) not by connection. This means that if one client has 4 downloads on the go, and another has only one, that traffic will be split 50/50, as opposed to the first client getting 80%. We do this by applying more filters to the existing ones.

tc filter add dev eth0 parent 4210: protocol ip handle 10 flow hash keys nfct-dst divisor 1024
tc filter add dev eth0 parent 4230: protocol ip handle 30 flow hash keys nfct-dst divisor 1024
tc filter add dev eth0 parent 4240: protocol ip handle 40 flow hash keys nfct-dst divisor 1024
tc filter add dev eth0 parent 4250: protocol ip handle 50 flow hash keys nfct-dst divisor 1024
tc filter add dev eth0 parent 666: protocol ip handle 666 flow hash keys nfct-dst divisor 1024

Uplink

The uplink is almost identical, except that it puts ACK packets into a separate class. This is so that when the upload link is saturated, download speeds are not affected.

tc qdisc add dev ppp0 root handle 1: htb r2q 1
tc class add dev ppp0 parent 1: classid 1:1 htb rate 550kbit ceil 550kbit

tc class add dev ppp0 parent 1:1 classid 1:10 htb rate 76kbit ceil 110kbit prio 0
tc class add dev ppp0 parent 1:1 classid 1:20 htb rate 170kbit ceil 495kbit prio 1
tc class add dev ppp0 parent 1:1 classid 1:30 htb rate 106kbit ceil 440kbit prio 4
tc class add dev ppp0 parent 1:1 classid 1:40 htb rate 84kbit ceil 440kbit prio 4
tc class add dev ppp0 parent 1:1 classid 1:50 htb rate 76kbit ceil 522kbit prio 4
tc class add dev ppp0 parent 1:1 classid 1:666 htb rate 38kbit ceil 550kbit prio 7

tc qdisc add dev ppp0 parent 1:10 handle 4210: sfq perturb 10 limit 50
tc qdisc add dev ppp0 parent 1:20 handle 4220: sfq perturb 10 limit 50
tc qdisc add dev ppp0 parent 1:30 handle 4230: sfq perturb 10 limit 64
tc qdisc add dev ppp0 parent 1:40 handle 4240: sfq perturb 10 limit 128
tc qdisc add dev ppp0 parent 1:50 handle 4250: sfq perturb 10 limit 64
tc qdisc add dev ppp0 parent 1:666 handle 666: sfq perturb 10 limit 128

tc filter add dev ppp0 parent 1:0 protocol ip prio 20 handle 20 fw flowid 1:20
tc filter add dev ppp0 parent 1:0 protocol ip prio 10 handle 10 fw flowid 1:10
tc filter add dev ppp0 parent 1:0 protocol ip prio 30 handle 30 fw flowid 1:30
tc filter add dev ppp0 parent 1:0 protocol ip prio 40 handle 40 fw flowid 1:40
tc filter add dev ppp0 parent 1:0 protocol ip prio 50 handle 50 fw flowid 1:50

tc filter add dev ppp0 parent 1:0 protocol ip prio 666 handle 666 fw flowid 1:666
tc filter add dev ppp0 parent 4210: protocol ip handle 10 flow hash keys nfct-src divisor 1024
tc filter add dev ppp0 parent 4220: protocol ip handle 20 flow hash keys nfct-src divisor 1024
tc filter add dev ppp0 parent 4230: protocol ip handle 30 flow hash keys nfct-src divisor 1024
tc filter add dev ppp0 parent 4240: protocol ip handle 40 flow hash keys nfct-src divisor 1024
tc filter add dev ppp0 parent 4250: protocol ip handle 50 flow hash keys nfct-src divisor 1024
tc filter add dev ppp0 parent 666: protocol ip handle 666 flow hash keys nfct-src divisor 1024