iptables routing based on destination port
Hi, I am trying to route any tcp openvpn connection from localhost via a particular network which is on eth1, and have all other traffic go via the default gateway on eth0. Just wondering if anyone knows if this is possible or why the below is not working for me? I've tried using the technique specified in [1]. The diagram in [2] also seems to support that this should work. The technique is basically: 1) Add a mark to the packet # iptables -t mangle -A PREROUTING -p tcp --dport 1194 -j MARK --set-mark 0x4aa 2) Verify the PREROUTING table has the mark # iptables t mangle -L PREROUTING -v Chain PREROUTING (policy ACCEPT 126K packets, 87M bytes) pkts bytes target prot opt in out source destination 0 0 MARK tcp - any any anywhere anywhere tcp dpt:openvpn MARK set 0x4aa 3) Add a routing rule that tells a packet with that mark to use a specific routing table. # ip rule add from all fwmark 0x4aa lookup vlan156 4) Verify the rule has higher priority than the other rules # ip rule show 0: from all lookup local 32764: from all fwmark 0x4aa lookup vlan156 32765: from 172.26.10.0/24 lookup vlan156 32766: from all lookup main 32767: from all lookup default 5) Add the above routing table (already added to /etc/iproute2/rt_tables) # ip route add default via 172.26.10.1 dev eth1 table vlan156 6) Verify the routing table exists # ip route show table vlan156 default via 172.26.10.1 dev eth1 So to my mind, any packet with destination port of tcp 1194, should get the 0x4aa mark and then be routed using the routing table vlan156 which tells it to use the default gateway of 172.26.10.1. However this is not happening, all traffic is still being routed using the default gateway in the main routing table (ip route show). Any suggestions as to why it's not working? Regards, Marcus. [1] http://lartc.org/howto/lartc.netfilter.html [2] http://inai.de/images/nf-packet-flow.png -- Marcus Furlong
1) Add a mark to the packet # iptables -t mangle -A PREROUTING -p tcp --dport 1194 -j MARK --set-mark 0x4aa
2) Verify the PREROUTING table has the mark # iptables t mangle -L PREROUTING -v Chain PREROUTING (policy ACCEPT 126K packets, 87M bytes) pkts bytes target prot opt in out source destination 0 0 MARK tcp - any any anywhere anywhere tcp dpt:openvpn MARK set 0x4aa
That "0 0" means no bytes have been satisfied by that rule. Can you check after you have tested a packet that should satisfy the rule to confirm that the counters are increasing?
...
So to my mind, any packet with destination port of tcp 1194, should get the 0x4aa mark and then be routed using the routing table vlan156 which tells it to use the default gateway of 172.26.10.1. However this is not happening, all traffic is still being routed using the default gateway in the main routing table (ip route show).
Any suggestions as to why it's not working?
How are you determining that the packets aren't going the right way? Is OpenVPN definitely using TCP? Default is UDP. Did you flush the route cache after - "ip route flush cache"? James
On Wed, Oct 2, 2013 at 2:30 PM, James Harper <james.harper@bendigoit.com.au> wrote:
1) Add a mark to the packet # iptables -t mangle -A PREROUTING -p tcp --dport 1194 -j MARK --set-mark 0x4aa
2) Verify the PREROUTING table has the mark # iptables t mangle -L PREROUTING -v Chain PREROUTING (policy ACCEPT 126K packets, 87M bytes) pkts bytes target prot opt in out source destination 0 0 MARK tcp - any any anywhere anywhere tcp dpt:openvpn MARK set 0x4aa
That "0 0" means no bytes have been satisfied by that rule. Can you check after you have tested a packet that should satisfy the rule to confirm that the counters are increasing?
The counters are not increasing.
So to my mind, any packet with destination port of tcp 1194, should get the 0x4aa mark and then be routed using the routing table vlan156 which tells it to use the default gateway of 172.26.10.1. However this is not happening, all traffic is still being routed using the default gateway in the main routing table (ip route show).
Any suggestions as to why it's not working?
How are you determining that the packets aren't going the right way?
tcpdump, lsof -i, conntrack -L and checking the connected IP address on the openvpn server.
Is OpenVPN definitely using TCP? Default is UDP.
Yes, definitely using TCP.
Did you flush the route cache after - "ip route flush cache"?
No I hadn't, but I just tried this now, and it hasn't made any difference. I also tried with conntack -F to flush the conntrack caches. Marcus. -- Marcus Furlong
2) Verify the PREROUTING table has the mark # iptables t mangle -L PREROUTING -v Chain PREROUTING (policy ACCEPT 126K packets, 87M bytes) pkts bytes target prot opt in out source destination 0 0 MARK tcp - any any anywhere anywhere tcp dpt:openvpn MARK set 0x4aa
That "0 0" means no bytes have been satisfied by that rule. Can you check after you have tested a packet that should satisfy the rule to confirm that the counters are increasing?
The counters are not increasing.
If the counters are not increasing then your rule isn't being hit, so nothing else is going to work. Are the packets being generated on the same box as is running the iptables rule? I just did a test: # iptables -t mangle -I PREROUTING -p tcp --dport 1194 # telnet 1.2.3.4 1194 # iptables -t mangle -vnL PREROUTING And the counters are 0, indicating that the rule is not being hit. If I try the telnet from a machine behind that one, the counters do increase. So it would seem that PREROUTING doesn't get hit for locally generated packets. If you put the iptables rule on the OUTPUT table the rule will get hit (I just tested this), but that might be too late for routing to be affected. Give it a go though as it should be easy to test. I think I'm doing that on my router. James
On 2013-10-02 06:35, James Harper wrote:
If the counters are not increasing then your rule isn't being hit, so nothing else is going to work.
Are the packets being generated on the same box as is running the iptables rule?
I just did a test:
# iptables -t mangle -I PREROUTING -p tcp --dport 1194 # telnet 1.2.3.4 1194 # iptables -t mangle -vnL PREROUTING
And the counters are 0, indicating that the rule is not being hit. If I try the telnet from a machine behind that one, the counters do increase. So it would seem that PREROUTING doesn't get hit for locally generated packets.
If you put the iptables rule on the OUTPUT table the rule will get hit (I just tested this), but that might be too late for routing to be affected. Give it a go though as it should be easy to test. I think I'm doing that on my router.
Page 3 of the O'Reilly Linux iptables Pocket Reference shows how packets traverse the system, and confirmd that in the mangle table, the first thing that a local packet hits is the OUTPUT, and it never hits PREROUTING: http://techedu.cu.cc/linux/OReilly%20Linux%20iptables,%20Pocket%20Reference%... -- Regards, Matthew Cengia
On Wed, Oct 2, 2013 at 5:00 PM, Matthew Cengia <mattcen@gmail.com> wrote:
On 2013-10-02 06:35, James Harper wrote:
If the counters are not increasing then your rule isn't being hit, so nothing else is going to work.
Are the packets being generated on the same box as is running the iptables rule?
I just did a test:
# iptables -t mangle -I PREROUTING -p tcp --dport 1194 # telnet 1.2.3.4 1194 # iptables -t mangle -vnL PREROUTING
And the counters are 0, indicating that the rule is not being hit. If I try the telnet from a machine behind that one, the counters do increase. So it would seem that PREROUTING doesn't get hit for locally generated packets.
If you put the iptables rule on the OUTPUT table the rule will get hit (I just tested this), but that might be too late for routing to be affected. Give it a go though as it should be easy to test. I think I'm doing that on my router.
Page 3 of the O'Reilly Linux iptables Pocket Reference shows how packets traverse the system, and confirmd that in the mangle table, the first thing that a local packet hits is the OUTPUT, and it never hits PREROUTING:
http://techedu.cu.cc/linux/OReilly%20Linux%20iptables,%20Pocket%20Reference%...
Slowly coming to the same same conclusion myself, but I was hoping this was out of date: http://www.faqs.org/docs/iptables/traversingoftables.html Table 3-2. Source local host (our own machine), at Step 2 the routing decision is taken before the OUTPUT chain of the mangle table. Grr. Does anyone have any other ideas how I might achieve this? Thanks, Marcus. -- Marcus Furlong
http://techedu.cu.cc/linux/OReilly%20Linux%20iptables,%20Pocket%20Refer ence%20(2004).pdf
Slowly coming to the same same conclusion myself, but I was hoping this was out of date:
http://www.faqs.org/docs/iptables/traversingoftables.html
Table 3-2. Source local host (our own machine), at Step 2 the routing decision is taken before the OUTPUT chain of the mangle table. Grr.
Does anyone have any other ideas how I might achieve this?
Did you actually try putting it in the OUTPUT chain? I have rules for that on my router and it is definitely working (just checked with tcpdump). I have 2 DSL connections and a 3G connection. One DSL is for web browsing etc, the other is for SSH, RDP, and other low volume latency sensitive traffic, and the 3G is for failover. I have another rule that sets the connection mark for incoming connections and then reflects that in the outgoing connections so a connection stays with the right DSL, so I can come in on either DSL if one is playing up. James
On Wed, Oct 2, 2013 at 5:43 PM, James Harper <james.harper@bendigoit.com.au> wrote:
http://techedu.cu.cc/linux/OReilly%20Linux%20iptables,%20Pocket%20Refer ence%20(2004).pdf
Slowly coming to the same same conclusion myself, but I was hoping this was out of date:
http://www.faqs.org/docs/iptables/traversingoftables.html
Table 3-2. Source local host (our own machine), at Step 2 the routing decision is taken before the OUTPUT chain of the mangle table. Grr.
Does anyone have any other ideas how I might achieve this?
Did you actually try putting it in the OUTPUT chain? I have rules for that on my router and it is definitely working (just checked with tcpdump).
I have 2 DSL connections and a 3G connection. One DSL is for web browsing etc, the other is for SSH, RDP, and other low volume latency sensitive traffic, and the 3G is for failover.
I have another rule that sets the connection mark for incoming connections and then reflects that in the outgoing connections so a connection stays with the right DSL, so I can come in on either DSL if one is playing up.
I just tried this now. Chain OUTPUT (policy ACCEPT 4504 packets, 857K bytes) pkts bytes target prot opt in out source destination 170 27734 MARK tcp -- any any anywhere anywhere tcp dpt:openvpn MARK set 0x4aa So this time the packets are actually getting marked, but they still go out over the wrong interface. It looks like because the routing decision has already been made, it doesn't bother to look up the routing tables. Marcus. -- Marcus Furlong
Did you actually try putting it in the OUTPUT chain? I have rules for that on my router and it is definitely working (just checked with tcpdump).
I just tried this now.
Chain OUTPUT (policy ACCEPT 4504 packets, 857K bytes) pkts bytes target prot opt in out source destination 170 27734 MARK tcp -- any any anywhere anywhere tcp dpt:openvpn MARK set 0x4aa
So this time the packets are actually getting marked, but they still go out over the wrong interface. It looks like because the routing decision has already been made, it doesn't bother to look up the routing tables.
What kernel? It is definitely working for me. Are you using tcpdump to determine that the packets are going out over the wrong interface? James
On Wed, Oct 2, 2013 at 6:12 PM, James Harper <james.harper@bendigoit.com.au> wrote:
Did you actually try putting it in the OUTPUT chain? I have rules for that on my router and it is definitely working (just checked with tcpdump).
I just tried this now.
Chain OUTPUT (policy ACCEPT 4504 packets, 857K bytes) pkts bytes target prot opt in out source destination 170 27734 MARK tcp -- any any anywhere anywhere tcp dpt:openvpn MARK set 0x4aa
So this time the packets are actually getting marked, but they still go out over the wrong interface. It looks like because the routing decision has already been made, it doesn't bother to look up the routing tables.
What kernel? It is definitely working for me.
# uname -r 3.2.0-53-generic
Are you using tcpdump to determine that the packets are going out over the wrong interface?
For the last test I only checked with lsof and conntrack: # lsof -i -n | grep openvpn openvpn 21972 nobody 4u IPv4 129321 0t0 TCP 115.146.92.84:51520->172.26.10.100:openvpn (ESTABLISHED) # conntrack -d 172.26.8.100 -L tcp 6 431995 ESTABLISHED src=115.146.92.84 dst=172.26.10.100 sport=51520 dport=1194 src=172.26.8.100 dst=115.146.92.84 sport=1194 dport=51520 [ASSURED] mark=0 use=2 conntrack v1.0.0 (conntrack-tools): 1 flow entries have been shown. But now using tcpdump, I see that packets ARE leaving via eth1. But there is only outgoing packets, no incoming packets at all. Whereas on the default gateway device, packets are only incoming, no outgoing packets. The openvpn tunnel seems fine with that. So I need to change the source address of the packets for it to come back on the correct interface? -- Marcus Furlong
Are you using tcpdump to determine that the packets are going out over the wrong interface?
For the last test I only checked with lsof and conntrack:
# lsof -i -n | grep openvpn openvpn 21972 nobody 4u IPv4 129321 0t0 TCP 115.146.92.84:51520->172.26.10.100:openvpn (ESTABLISHED)
# conntrack -d 172.26.8.100 -L tcp 6 431995 ESTABLISHED src=115.146.92.84 dst=172.26.10.100 sport=51520 dport=1194 src=172.26.8.100 dst=115.146.92.84 sport=1194 dport=51520 [ASSURED] mark=0 use=2 conntrack v1.0.0 (conntrack-tools): 1 flow entries have been shown.
But now using tcpdump, I see that packets ARE leaving via eth1. But there is only outgoing packets, no incoming packets at all. Whereas on the default gateway device, packets are only incoming, no outgoing packets. The openvpn tunnel seems fine with that. So I need to change the source address of the packets for it to come back on the correct interface?
Yes that's exactly what you need to do, or the other end will just reply with the original (wrong) destination address. Something like: iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE you can use -j SNAT if you want to explicitly specify the source address (eg because you have several), but MASQUERADE should suffice. James
On Wed, Oct 2, 2013 at 7:25 PM, James Harper <james.harper@bendigoit.com.au> wrote:
Are you using tcpdump to determine that the packets are going out over the wrong interface?
For the last test I only checked with lsof and conntrack:
# lsof -i -n | grep openvpn openvpn 21972 nobody 4u IPv4 129321 0t0 TCP 115.146.92.84:51520->172.26.10.100:openvpn (ESTABLISHED)
# conntrack -d 172.26.8.100 -L tcp 6 431995 ESTABLISHED src=115.146.92.84 dst=172.26.10.100 sport=51520 dport=1194 src=172.26.8.100 dst=115.146.92.84 sport=1194 dport=51520 [ASSURED] mark=0 use=2 conntrack v1.0.0 (conntrack-tools): 1 flow entries have been shown.
But now using tcpdump, I see that packets ARE leaving via eth1. But there is only outgoing packets, no incoming packets at all. Whereas on the default gateway device, packets are only incoming, no outgoing packets. The openvpn tunnel seems fine with that. So I need to change the source address of the packets for it to come back on the correct interface?
Yes that's exactly what you need to do, or the other end will just reply with the original (wrong) destination address.
Something like:
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
you can use -j SNAT if you want to explicitly specify the source address (eg because you have several), but MASQUERADE should suffice.
Just got to test this today and it didn't seem to work. From the openvpn logs, I was getting the following: Wed Oct 9 13:49:06 2013 us=596232 TCP: connect to [AF_INET]172.26.8.100:1194 failed, will try again in 5 seconds: Connection timed out Although tcpdump showed the packets were now only using the non-default interface as they should. To get it working I also needed to do the following to disable reverse packet filtering, or to enable "loose" mode: echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter or echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter echo 2 > /proc/sys/net/ipv4/conf/eth1/rp_filter And it seems "all" doesn't enable it for all interfaces, as I thought it would. You need both the above lines to get this working. Regards, Marcus. -- Marcus Furlong
On 2 October 2013 17:36, Marcus Furlong <furlongm@gmail.com> wrote:
Slowly coming to the same same conclusion myself, but I was hoping this was out of date:
http://www.faqs.org/docs/iptables/traversingoftables.html
Table 3-2. Source local host (our own machine), at Step 2 the routing decision is taken before the OUTPUT chain of the mangle table. Grr.
Does anyone have any other ideas how I might achieve this?
Apologies for the late response. As per the diagram: http://inai.de/images/nf-packet-flow.png tproxy is the solution that allow things like transparent proxies to work with UDP as well as TCP and for IPv6 as well as IPv4, without relying on NAT. My understanding of tproxy is that it ends up with (note doing this with ipv6 appears to be broken in the standard wheezy kernel however that is another issue, if you have problems with the wheezy kernel you might want to try a latter kernel from back ports just in case): OUTPUT(eth0) -> routing -> PREROUTING (lo) As per the diagram, after mangle:output rule, it does a "reroute check", which is capable of "jumping" back to the prerouting stage. I think. The best explanation I have been able to come up with anyway :-) When I asked about this before I got the response: “Because it's two different interfaces (eth0 vs lo). The diagram is for an single interface.“ Trying to understand how this works completely mangles my brain. Or nats my brain. Or something. However, will try: In the mangle:OUTPUT I have: MARK tcp -- 0.0.0.0/0 192.168.0.0/16 tcp MARK set 0x1 MARK udp -- 0.0.0.0/0 192.168.0.0/16 udp MARK set 0x1 This assigns all required packets the mark. Then, as above, it must look at my routing rules: aquitard# ip rule 0: from all lookup local 32764: from all fwmark 0x1 lookup 100 32765: from all fwmark 0x1 lookup 100 32766: from all lookup main 32767: from all lookup default aquitard# ip route show table 100 local default dev lo scope host Which reroutes it to lo. Then in mangle:PREROUTING I do the tproxy stuff. Which is probably not really relevant to your task. All seems a bit like black magic to me. However it does actually work. So in conclusion, while this seems very non-obvious, have you tried marking the packets in the OUTPUT chain? Thread where I try to understand this for the first time: https://groups.google.com/forum/#!msg/sshuttle/5U0JFrecHks/PhUGMa1RElMJ -- Brian May <brian@microcomaustralia.com.au>
On Wed, 2 Oct 2013, 14:17, Marcus Furlong wrote: } Hi, } } I am trying to route any tcp openvpn connection from localhost via a } particular network which is on eth1, and have all other traffic go via } the default gateway on eth0. } } Just wondering if anyone knows if this is possible or why the below is } not working for me? I've tried using the technique specified in [1]. } The diagram in [2] also seems to support that this should work. The } technique is basically: } } 1) Add a mark to the packet } # iptables -t mangle -A PREROUTING -p tcp --dport 1194 -j MARK --set-mark 0x4aa } } 2) Verify the PREROUTING table has the mark } # iptables t mangle -L PREROUTING -v } Chain PREROUTING (policy ACCEPT 126K packets, 87M bytes) } pkts bytes target prot opt in out source destination } 0 0 MARK tcp - any any anywhere anywhere tcp dpt:openvpn MARK set 0x4aa } } 3) Add a routing rule that tells a packet with that mark to use a } specific routing table. } # ip rule add from all fwmark 0x4aa lookup vlan156 } } 4) Verify the rule has higher priority than the other rules } # ip rule show } 0: from all lookup local } 32764: from all fwmark 0x4aa lookup vlan156 } 32765: from 172.26.10.0/24 lookup vlan156 } 32766: from all lookup main } 32767: from all lookup default } } 5) Add the above routing table (already added to /etc/iproute2/rt_tables) } # ip route add default via 172.26.10.1 dev eth1 table vlan156 } } 6) Verify the routing table exists } # ip route show table vlan156 } default via 172.26.10.1 dev eth1 } } So to my mind, any packet with destination port of tcp 1194, should } get the 0x4aa mark and then be routed using the routing table vlan156 } which tells it to use the default gateway of 172.26.10.1. However this } is not happening, all traffic is still being routed using the default } gateway in the main routing table (ip route show). } } Any suggestions as to why it's not working? Do you want udp 1194 instead of tcp ? T. } Regards, } Marcus. } } [1] http://lartc.org/howto/lartc.netfilter.html } [2] http://inai.de/images/nf-packet-flow.png } } -- } Marcus Furlong
participants (5)
-
Brian May -
James Harper -
Marcus Furlong -
Matthew Cengia -
Tony Crisp