Lightning fast web server designed for security, streaming and static content
For commercial use, please contact the author gvalkov@gmail.com

Routing traffic over a non-preferred gateway

Preface
When multiple routes are available to a specific destination, such as when multiple default gateways are configured, the IP protocol is designed to output traffic over the route with lowest metric. This article describes the configuration required to define routes based on IP address and port.

Why would you route traffic over a non-preferred GW?
Assume the scenario where you have a private network with one or more hosts providing services such as HTTP, HTTPS, etc. The ISP might only provide a dynamic IP or no public IP, so forwarding services is not possible. We can use a VPN to obtain a public IP, then assign the VPN GW as preferred, and forward a few ports to make our LAN servers available on the Internet.

This configuration however has one limitation:
All traffic from all hosts on the LAN will go through the VPN, which may be slower than the direct Internet connection.

Routing traffic based on source IP or port:
To override the global routing, we define IP routing tables. Each table specify source, target, gateway, and metric. Next we define IP rules, which assign a table based on custom rules such as source IP address or firewall mark. When routing traffic based on source IP address, it is sufficient to create an IP rule which assigns a table to that source address.

Since rules operate on IP traffic, they cannot apply more precise filtering based on the source port. This limitation can be overcome by using firewall marks: in this case the rule specifies that packets with a specific mark should be assigned a particular routing table. The final step is to create precise filters which mark packets using iptables. iptables can mark packets based on source and destination IP and port, protocol, etc.

Example: output from router
Assume we have a private network NET_LAN='192.168.192.0/24', Internet connection is provided over NET_WAN='172.20.10.0/24', and the VPN network is NET_VPN='5.5.5.0/24'. If we want PORT_VPN_WAN='80' on the router to be accessible on the Internet over the VPN, we can mark the traffic using the following command: 

iptables -t mangle -A OUTPUT      ! -d $NET_LAN -p tcp -m multiport --sports $PORT_VPN_WAN -j $MARK_VPN

Note that we also want to exclude any traffic with destination NET_LAN, otherwise clients on NET_LAN will not be able to access our router using its public IP address on the VPN. We also need to explicitly mark output traffic from IP_VPN to go over the VPN interface MARK_VPN, while traffic from IP_WAN should go over the WAN interface MARK_WAN. Traffic from IP_VPN to NET_VPN should not be marked. The same applies to any traffic from IP_WAN to IP_WAN.

Example: output from devices on the private LAN
If we want PORT_VPN_LAN='80,443' on NET_LAN to be accessible on the Internet over the VPN, we can mark the traffic using the following command: 

iptables -t mangle -A PREROUTING  ! -d $NET_LAN -s $NET_LAN -p tcp -m multiport --sports $PORT_VPN_LAN -j $MARK_VPN

Note that we also want to exclude any traffic with destination NET_LAN, otherwise clients on NET_LAN will not be able to access services using our public IP address on the VPN. Traffic from NET_LAN to NET_VPN and NET_WAN, should be excluded, so we clear the firewall mark.


IP table for VPN 
ip route add default via 5.5.5.1 src $IP_VPN dev br-vpn table vpn metric 10

# OpenWRT: /etc/config/network
# table vpn: output to any destination via interface vpn, gateway 5.5.5.1,
# using source 5.5.5.2 and metric 10
config route
	option table 'vpn'
	option interface 'vpn'
	option target '0.0.0.0/0'
	option gateway '5.5.5.1'
	option source '5.5.5.2'
	option metric '10'

IP rule matching source address 
ip rule add from 5.5.5.2/32 table tu

# OpenWRT: /etc/config/network
# use lookup table vpn for packets marked 0xf2
config rule
	option source '5.5.5.2'
	option lookup 'vpn'

IP rule matching a firewall mark 
ip rule add fwmark 0xf2 table vpn

# OpenWRT: /etc/config/network
# use lookup table vpn for packets marked 0xf2
config rule
	option mark '0xf2'
	option lookup 'vpn'

Complete configuration 
# define route tables and rules
ip route add default via $GW_VPN src $IP_VPN dev br-vpn table vpn metric 10
ip route add default via $GW_WAN src $IP_WAN dev eth0   table wan metric 2
ip rule add fwmark 0xf2 table vpn
ip rule add fwmark 0xf1 table wan

# OpenWRT: /etc/firewall.user
GW_WAN='172.20.10.1'
GW_VPN='5.5.5.1'
IP_WAN='172.20.10.2'
IP_VPN='5.5.5.2'
NET_LAN='192.168.192.0/24'
NET_WAN='172.20.10.0/24'
NET_VPN='5.5.5.0/24'
PORT_VPN_LAN='80,443'
PORT_VPN_WAN='80'
MARK_VPN='MARK --set-mark 0xf2'
MARK_WAN='MARK --set-mark 0xf1'
MARK_CLR='MARK --set-mark 0x00'

# replay from VPN IP address (non-preferred gateway)
# output from router with
# source ports PORT_VPN_WAN should go through VPN, unless destination is NET_LAN
# source IP_VPN should go through VPN, unless destination is NET_LAN
# source IP_WAN should go through WAN, unless destination is NET_LAN
# source IP_VPN and destination NET_VPN: clear MARK and use default
# source IP_WAN and destination NET_WAN: clear MARK and use default
iptables -t mangle -A OUTPUT      ! -d $NET_LAN -p tcp -m multiport --sports $PORT_VPN_WAN -j $MARK_VPN
iptables -t mangle -A OUTPUT      ! -d $NET_LAN -s $IP_VPN -j $MARK_VPN
iptables -t mangle -A OUTPUT      ! -d $NET_LAN -s $IP_WAN -j $MARK_WAN
iptables -t mangle -A OUTPUT        -d $NET_VPN -s $IP_VPN -j $MARK_CLR
iptables -t mangle -A OUTPUT        -d $NET_WAN -s $IP_WAN -j $MARK_CLR

# output from devices on NET_LAN with
# source ports PORT_VPN_LAN should go through VPN, unless destination is NET_LAN
# destination NET_VPN: clear MARK and use default
# destination NET_WAN: clear MARK and use default
iptables -t mangle -A PREROUTING  ! -d $NET_LAN -s $NET_LAN -p tcp -m multiport --sports $PORT_VPN_LAN -j $MARK_VPN
iptables -t mangle -A PREROUTING  ! -d $NET_LAN -s $NET_LAN -p udp -m multiport --sports $PORT_VPN_LAN -j $MARK_VPN
iptables -t mangle -A PREROUTING    -d $NET_VPN -s $NET_LAN -j $MARK_CLR
iptables -t mangle -A PREROUTING    -d $NET_WAN -s $NET_LAN -j $MARK_CLR


# OpenWRT: /etc/config/network

# bridges
config device
	option name 'br-lan'
	option type 'bridge'
	option stp '1'
	list ports 'lan'

config device
	option name 'br-vpn'
	option type 'bridge'
	option stp '0'
	list ports 'wan'
	list ports 'tap_vpn'

# interfaces
config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option metric '0'
	option force_link '1'
	option netmask '255.255.255.0'
	option broadcast '192.168.192.255'
	option dns '8.8.8.8 8.8.4.4 1.1.1.1'
	option ipaddr '192.168.192.2'
	option ip6addr 'ddaa::2/64'
	option ip6prefix 'ddaa::/64'

config interface 'wan'
	option device 'eth0'
	option proto 'static'
	option metric '2'
	option force_link '0'
	option netmask '255.255.255.240'
	option broadcast '172.20.10.15'
	option ipaddr '172.20.10.2'
	option gateway '172.20.10.1'
	list dns '172.20.10.1'

config interface 'vpn'
	option device 'br-vpn'
	option proto 'static'
	option metric '10'
	option force_link '0'
	option netmask '255.255.255.0'
	option broadcast '5.5.5.255'
	option dns '8.8.8.8 8.8.4.4 1.1.1.1'
	option ipaddr '5.5.5.2'
	option gateway '5.5.5.1'
	option defaultroute '1'
	option peerdns '0'
	option delegate '0'

# IP rules
# use IP table wan for packets marked 0xf1
config rule
	option mark '0xf1'
	option lookup 'wan'

# use IP table vpn for packets marked 0xf2
config rule
	option mark '0xf2'
	option lookup 'vpn'

# IP routes
# table wan: output to any destination via interface wan, gateway 172.20.10.1,
# using source 172.20.10.2 and metric 2
config route
	option table 'wan'
	option interface 'wan'
	option target '0.0.0.0/0'
	option gateway '172.20.10.1'
	option source '172.20.10.2'
	option metric '2'

# table vpn: output to any destination via interface vpn, gateway 5.5.5.1,
# using source 5.5.5.2 and metric 10
config route
	option table 'vpn'
	option interface 'vpn'
	option target '0.0.0.0/0'
	option gateway '5.5.5.1'
	option source '5.5.5.2'
	option metric '10'

# if we have a local link over the VPN to the VPN server, force traffic over
# the wan interface
config route
	option interface 'wan'
	option target '5.5.5.5'
	option gateway '172.20.10.1'
	option metric '0'
© 2015-2021 George Valkov •
httpstorm web server