How to forward packets from one network interface via another interface
Managed to find the issue.
Whilst adding the route for the container in the other subnet I haven't correctly specified the gateway. The gateway still points to the host machine in which docker is run (see above figure). So I added the correct routing rule specific to the two end-containers - c1 & c3.
c1 - ip route add 10.23.0.0/24 via 10.12.0.1 dev peervpn12
c3 - ip route add 10.12.0.0/24 via 10.23.0.1 dev peervpn23
In the meantime, had to add the correct FORWARD rules in the c2 container's iptables:
iptables -A FORWARD -s 10.12.0.2 -i peervpn12 -d 10.23.0.2 -o peervpn23 -j ACCEPT
iptables -A FORWARD -s 10.23.0.2 -i peervpn23 -d 10.12.0.2 -o peervpn12 -j ACCEPT
With this setup I was able to achieve the flow I expected.
Thank You and I don't know why it's down-voted.
Maybe if I know the reason I can correct myself in future :)
UDP connect doesn't route traffic for particular interface
So in my case, the issue was asymmetry of ip rules between the ethernet and wifi interface that cause routing problems. If you run into a similar thing, you can see some rules that are set with:
ip rule show
and/or
iptables --list
Packet forwarding between interfaces
N.B - edited the command (added eth2 as outgoing interface)
I understand from the setup described that the that the wan side on your router is eth2 which has address 10.2.20.3.
Hence I would expect outgoing packets (on eth2) to have the source IP 10.2.20.3 and not 10.2.40.4.
I think this indicates that you got your nat rules reversed or something.
In fact it seems that the only iptables rule that you need is:
iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE
I suggest you remove all other DNAT/SNAT/MASQUERADE rules and try this one.
How to route TCP/IP responses through a different interface?
You could add an additional address to the lo
interface on each system and use these new addresses as the TCP connection endpoints. You can then use static routes to direct which path each machine takes to get to the other machine's lo
address.
For example:
Machine A:
ip addr add 1.1.1.1/32 dev lo
ip route add 2.2.2.2/32 dev eth0 via <eth0 default gateway>
Machine B:
ip addr add 2.2.2.2/32 dev lo
ip route add 1.1.1.1/32 dev gr0
Then bind to 1.1.1.1 on machine A and connect to 2.2.2.2.
Can't establish connection over second NIC (two hops)
Not an authoritative answer, but my first working attempt (applying what I managed to understand):
sudo ip route add 192.168.1.0/24 via 192.168.222.254 from 192.168.222.200 dev eth1 table 253
sudo ip rule add from 192.168.222.200 table 253
Update:
from
anddev
arguments in theip route
command aren't required (it works perfetly well without them).
...after isuinng first command I couldn't connect yet, but after issuing second one yes.
The logic behind that comes from this text i found in this document:
Linux-2.x can pack routes into several routing tables identified by a number in the range from 1 to 255 or by name from the file /etc/iproute2/rt_tables By default all normal routes are inserted into the main table (ID 254) and the kernel only uses this table when calculating routes.
Actually, one other table always exists, which is invisible but even more important. It is the local table (ID 255). This table consists of routes for local and broadcast addresses. The kernel maintains this table automatically and the administrator usually need not modify it or even look at it.
In fact, I finally ended up using another routing table, identified by its id (253) instead of what I now understand it is just an alias (defined in /etc/iproute2/rt_tables file).
...and checking again that file, I now see that there was an alias ("default") already defined for that routing table (next to the "main" one which is indeed 254 as the text fragment I pasted previously says.
What I don't know yet is which is the logic behind this naming (the "default" for 253 routing table I mean) and if, for any reason, is better to use lower routing tables (1, 2, 3...) like this solution (already mentioned in the question) does.
But, for the sake of simplicity, if we aren't going to build complex routing policies and just want to fix this connectivity issue, I guess it could be a good solution to use something like (not yet tested):
gateway 192.168.222.254 table 253
post-up ip rule add from 192.168.222.200 table 253
I still need to test and check if I need an additional
via 192.168.222.254
in the gateway row or if it won't work at all and need to add it with another post-up command instead.I will update this answer with the results.
Edit 1: Same works with default routes:
sudo ip route add default from 192.168.222.200 via 192.168.222.254 table 253
sudo ip rule add from 192.168.222.200 table 253
Edit 2: First (now fully¹) working approach
After playing for a while with a testing machine, I think that the best solution is to add following rows to the second NIC configuration in /etc/network/interfaces
file:
gateway 192.168.222.254 table 1
post-up ip rule add from 192.169.222.200 table 1
pre-down ip rule del from 192.168.222.200 table 1
post-up ip route add 192.188.222.0/24 dev eth1 src 192.168.222.200 table 1
Comments:
Adding
table 1
to thegateway
keyword worked well so additional (less readable) post-up command to add that default route was not necessary.- ...in fact, using specific table (other than main) for first NIC together with a similar rule than what we used for our second NIC would be a bad idea because, that that rule will only apply when 192.168.111.200 is going to be used as source address so there will not be any "default default gateway". Leaving first NIC configuration in the main routing table, will make all ("locally generated") outgoing connections to remote LANs will go though our first default gateway by default.
First
post-up
command adds a rule that packets with the source address of that NIC, should be routed using table 1 (otherwise our new default gateway wouldn't be used).pre-down
command removes that rule. It is not mandatory but, without it, multiple network service restarts will duplicate this rule every time.I also tried to use
dev eth1
instead offrom 192.169.222.200
(to avoid having to duplicate network address) but it didn't work. I guess which NIC to use to for "response" packets were "not yet decided".I used
table 1
for eth1 (our second NIC) and I could usetable 2
for an eventual third one and so on. It wasn't needed to specify any table/rule for first NIC because it comes to the main table (not "default": see below note).Finally(¹) the second
post-up
command make all things work well because (as I now realize) only (first matching) one routing table is used so the default network route (automatically created when the interface brought up) doesn't apply because it was created in table main.- I still don't know if there is a way to force it to be crated directly into table 1.
NOTE: By command
sudo ip rule list
we can see current routing rules as follows:0: from all lookup local
32765: from 192.168.222.200 lookup 1
32766: from all lookup main
32767: from all lookup default
As I can understand, they are added decreasingly from 32767 to 0 and tried
increasingly until one matches. Last two ones and the "0" were already
defined by default. The former because of the logic I previously cited
from this document but that documents says that rules starts from "1"
so I guess "0" should also be some predefined "default starting point".
Edit 3:
As I said in the Edit 2 (of the question), I found this Linux Advanced Routing & Traffic Control HOWTO that helped me a lot in clarifying things.
Concretely the Routing for multiple uplinks/providers chapter was very useful to me in the task of understanding setups having "network loops" (even in our case we aren't acting as a router to Internet).
Related Topics
Mongodb (Result= Signal, Code = Killed, Signal = Ill
Nvcc Cuda Cross Compiling Cannot Find "-Lcudart"
Compiling and Linking a 32 Bit Application on Debian 64 Bit
Git Clone Using Ssh Failed in Windows Due to Permission Issue
Linux Support 802.1Ag and Y1731
How to Distribute C++11 Shared Library on Centos6
Shared Libraries (Dlopen) and Thread-Safety of Library Static Pointers
Why Isn't Git Bash Transforming The Path to *Nix Notation for My Python Installation
Ansible: Copying One Unique File to Each Server in a Group
Analyze Memory with Crash with Kdump
Proxmox with Opnsense as Firewall/Gw - Routing Issue
How to Install a Recent Version of Gdal on Amazon Linux
Error Installing Chrome on Aws Ec2 Linux Instance: Scaling_Cur_Freq & Scaling_Max_Freq Not Found
How The File Size Is Limited on a Specific File System
Library Path Order for Alternate Glibc Dynamic Linker (Ld.So)