Dynamic routing of friends' networks at home using BGP privately : building Intranets.
Intro
Once you are connected to the Internet using your own router and bypassing your poor CPE, and once you are eventually multihomed, then you can start mounting VPN tunnels with friends' routers, and exchange routing informations dynamically using BGP over your several ISP outputs. You may also trigger asymetric routing of your traffic.
As usual now, this blog entry will detail some scenarios about what my setup looks like actually (some part of it), for my own needs. We'll only rely on one thing : one/several Internet connection(s) up and running, and that's all : no intermediaries of any kind, but an intranet of friends' networks built on top of the Internet, totally private and fully encrypted. Ready ?
Quick reminder about the actual setup
You should read the other blog posts linked above in the intro. Here is a quick reminder of what my setup looks like today :
Like you can see, I got one router, with 2 WAN parts, as I have applied two ISPs, and with several LAN parts as I manage several VLANs on my private LAN part. The switch has two links to the router using LACP.
Mounting VPNs
Now, what we can do is having our router connect directly to some friends' routers, using VPN technologies. That way, me and some friends will be directly connected with just one router hop.
Having a direct route between friends and me, we'll be able to reach each other using this route, but only using the private VPN range endpoints, until we exchange routing informations to be able to exchange some traffic on top of the VPN connections. This is what we'll detail here : we are going to build an intranet of several networks joined together transparently over the VPN connections. That way, my private LAN networks will be able to access my friends private LAN networks using private IP ranges, and vice-versa. We'll experiment that on top of IPV4.
Remember that the network, should be configured at the network level, not at the client level. There will be nothing to do about all the machines residing into the LAN networks. They stay as-is, but the gateway they use will be added routes, thus they'll be able to use them.
VPN technologies
There exists several of them, the most known being PPTP, SSH, IPIP, SSTP, L2TP, GRE, OVPN. All those technologies are available using Mikrotik's RouterOS. We will show one example : using OVPN.
Client or server ? I'll act as a server, because I'm multihomed. As I manage my own DNS zone, here is what I publish to my peers :
- vpn.mydomain.com : entry point to my router through ISP#1 network.
- vpn2.mydomain.com : entry point to my router through ISP#2 network
If my ISP public IP address change - that happens something like once a year - then I would just need to update the DNS entries, and all my peers will get connected back soon.
Firewall
Acting as a server, the first thing to do is to go into the Firewall, and allow traffic from Internet to the ports or protocols used by the chosen VPN technologies. To ease things, I created interface groups, and grouped both ISP interfaces into a group, to apply the same rules on both :
> /interface list member print where list=ALL-WAN
# LIST INTERFACE
0 ALL-WAN vlan832-orange-internet
1 ALL-WAN sfp-wan-sfr
The list is named ALL-WAN.
For the firewall, I use custom chains. I get all the traffic coming from ALL-WAN, and put it in the INPUT_WAN chain. Then I filter the INPUT_WAN chain, and open access to the VPN technologies used :
> /ip firewall filter print where chain=INPUT_WAN
7 ;;; OpenVPN
chain=INPUT_WAN action=accept protocol=tcp dst-port=1194
8 X ;;; SSTP
chain=INPUT_WAN action=accept protocol=tcp dst-port=4430
9 ;;; PPTP
chain=INPUT_WAN action=accept protocol=tcp dst-port=1723
10 ;;; GRE
chain=INPUT_WAN action=accept protocol=gre
11 X ;;; L2TP
chain=INPUT_WAN action=accept protocol=udp dst-port=1701
12 X ;;; L2TP-IPSEC
chain=INPUT_WAN action=accept protocol=udp dst-port=500
13 X ;;; IPSEC
chain=INPUT_WAN action=accept protocol=ipsec-esp
14 X ;;; IPSEC NAT-T
chain=INPUT_WAN action=accept protocol=udp dst-port=4500Some rules are disabled because not used, or not needed here (and all the rules are not shown here, just a bunch for the example). Mikrotik provides some ALG, and sometimes not :-D Discover them by yourself (as the OS is closed-source).
OVPN
Once ports are open, we need to create endpoints and credentials on our side. We will play safely and use certificates. I won't detail certificate generation and signature here. Here is the OVPN server config :
> /interface ovpn-server server print
enabled: yes
port: 1194
mode: ip
netmask: 32
mac-address: FE:44:DD:XXXXXXXXXX
max-mtu: 1450
keepalive-timeout: 60
default-profile: default
certificate: server_cert
require-client-certificate: yes
auth: sha1
cipher: aes128,aes256
We require client certificate, we'll then have to issue one cert per client. This has been done and is not detailed here. We will clamp TCP MSS dynamicaly and we reduce the link MTU to 1450 which should be enough to tunnel OVPN and possibly tunnel other light tunneling protocols into OVPN as well.
Now we create a profile for users :
> /ppp profile print
2 name="VPN-interco" remote-ipv6-prefix-pool=*0 use-ipv6=no use-mpls=no use-compression=no use-encryption=yes only-one=yes change-tcp-mss=yes use-upnp=no address-list="" on-up="" on-down=""
And then we create user accounts that will use the latter created profile. Here is one detailed :
/ppp secret print detail
> 4 name="friend1-1" service=ovpn caller-id="" password="XXXXXXXX"
profile=VPN-interco local-address=172.16.10.1
remote-address=172.16.10.2 routes="" limit-bytes-in=0
limit-bytes-out=0
Notice a very important thing : It is here, in the user profile, that we tell the VPN interconnection IP addresses. As we're going to setup a PointToPoint connection, only involving two peers, we'll use a traditionnal /32 IPV4 private range network for interconnection. The netmask is passed in the server configuration.
So, I will be 172.16.10.1/32, and my peer "friend1" router will be given 172.16.10.2/32 address. Obviously, "friend1" has to replicate the configuration on his side ;-)
Now, the setup would be exactly the same for the second ISP connection, except that my friend would have to create a second connection, using my second entry point ISP#2, and myself I would have to create a second account, like "friend1-2", with a second /32 IP address. Like this :
> /ppp secret print detail
> 5 name="friend1-2" service=ovpn caller-id="" password="XXXXXXXX"
profile=VPN-interco local-address=172.16.11.1
remote-address=172.16.11.2 routes="" limit-bytes-in=0
limit-bytes-out=0Routing traffic over the VPN endpoints
So far so good, we end up with a schema like this :
If we create the server bindings, we'll be able to deal with clear interface names.
> /interface ovpn-server print detail
0 R name="friend1-1" user="friend1-1" mtu=1450
client-address="XXXXXXXXX" uptime=16h48m19s
encoding="AES-128-CBC/SHA1"
1 R name="friend1-2" user="friend1-2" mtu=1450
client-address="XXXXXXXXX" uptime=16h48m24s
encoding="AES-128-CBC/SHA1"
Like you can see, my friend 1 has two direct connections to me, and thus I got two IPs on my router to support those both :
> /ip address print
# ADDRESS NETWORK INTERFACE
14 D 172.16.10.1/32 172.16.10.2 friend1-1
15 D 172.16.11.1/32 172.16.11.2 friend1-2
Under the hood, the 172.16.10.1/32 interconnection flows using ISP#1 whereas the 172.16.11.1/32 interconnection flows through ISP#2. At this point, our routers can communicate with each other using one or the other address.
What we do is that we can each other log-in to the opposite router, with a read-only account, to be able to debug the "other side", when needed. All my friends are equipped with Mikrotik hardware as well.
Now, we are going to share our private LAN networks of choice, on top of those endpoints. Let's first do it statically.
I would like to share my 192.168.0.0/24 network to him. He will share to me his 192.168.42.0/24 network. Well, there is nothing complex at all : it is just about adding both routes on both ends, and that's done.
For me :
> /ip route add dst-address=192.168.42.0/24 gateway=172.16.10.2 distance=10 check-gateway=ping
> /ip route add dst-address=192.168.42.0/24 gateway=172.16.11.2 distance=20 check-gateway=pingFor him :
> /ip route add dst-address=192.168.0.0/24 gateway=172.16.10.1 distance=10 check-gateway=ping
> /ip route add dst-address=192.168.0.0/24 gateway=172.16.10.2 distance=20 check-gateway=pingThat's all. We simply tell each other that to reach the other's network, we must contact the other's router VPN address.
Like you can see, we got two routes to reach each other's foreign LAN network. We have a fail over case : the router will prefer using the lowest distance route, as this is the main routing rule (at least for static routing, BGP is different in best path selection behavior).
So, our routers will both send traffic using the 172.16.10.X endpoint, because of the lowest distance (which for me passes through ISP#1). If that route fails (it is checked using ping) then the router will setup the second path in the FIB, and the data flow will now use the 172.16.11.X endpoint through my ISP#2, without any interruption.
As an example, our both NAS (mine is 192.168.0.2 and his is 192.168.42.42) synchronise each other so that we can backup our numeric lives with each other; and the NAS devices have absolutely no idea on how the packets are routed.
Here is what we end up with :
If we swap the distance values on one end and not both ends, we then route the traffic asymetrically and the bytes I send to my friend could flow through ISP#1 whereas the ones he sends to me (TCP ACKs as well, obviously) would flow through ISP#2. Asymetric routing is also a big part of how the global Internet works. When you send a packet to FOO, the answer from FOO to you may use another path on the Internet. For asymetric routing, reverse path filtering should be taken care of.
This may be the very first network engineering rule to always keep in mind : traffic flows in both directions, and both directions may not be the same pipe. You must always keep in mind about back-traffic when thinking in networking.
Finally, here we talk about our networks, from my friends and I : We are building our own private Internet (called an Intranet), on top of the global Internet, and every byte we exchange flows though a crypted channel : our VPN tunnels.
Tip : All you need to build networks are basically physical {wires, routers, switches} and logical {routes, firewalls}.
Note that each of us could send Internet destinated traffic through the VPN, and then having it routed through our friend peer ISP. Why would someone pay for such a service, and send its traffic to an unknown provider that would be able to log/analyze/alter it ? Just find friends, and build networks by yourself ! Find a trusty friend on each continent, mount a VPN session with them, and you'll be able to originate your traffic from any continent on the planet, thus cancelling all those silly region locks some services make use of. Should I remind what the "Internet" word means, and what the Internet has been built for ?
I trust friends I'm connected with. If I send them traffic not originated to them, it will use their default gateway and they'll route it for me. We trust each others, but all of us are still free to change the routing rules and setup firewall rules as well, on the VPN; Thus restricting what the other part is allowed to do, or not.
Ok, what we did so far is cool, and could be well enough. We are able to send/receive traffic to each others, and we managed to connect some of our LANs together, creating a private, crypted intranet. We could end here.
But, we could also push this even further.
BGP
What if now an endpoint friend would like to route another network with me ? Or me with him ?. My real use case is up with my friend Rémi. Let's say for example I or Rémi created a new 192.168.200.0/24 network and want to share it with the other part.
We would then both need manual intervention : adding a static route to the RIB.
BGP is a protocol that is aimed to automate route annoucements between border routers. By setting up a BGP session between us, we would then be able to create new LAN networks on our respective parts, and have them automatically announced to the connected routers, in both directions, using the VPN tunnels. The routing table will be updated dynamically and the new machines will be able to reach each other from one end, to the other. Magic.
BGP is pretty complex to understand and master, and not mastering it leads quickly to security problems or performance problems in routing decisions. It is the protocol used to route the overall Internet. We will use it to a very little scale : a bunch of routers and networks.
Document yourself about BGP before experimenting it even on a private scale. We will not explain all BGP here, and BGP is huge.
BGP routes A.S borders. A.S are Autonomous Systems. To use BGP privately, we'll then need to create a private AS both Rémi and I. Private AS ranges are from 64512 to 65534 (unextended). We will also need to be bordered with each other, that means we'll need to be able to reach each other with just one router hop. And .... that's already the case, as the VPNs allow us to reach each other with one hop. Look at a traceroute from my LAN to the connected Rémi's LAN :
> mtr 192.168.42.252
HOST: 192.168.0.1 Loss% Snt Last Avg Best Wrst StDev
1.|-- 192.168.0.254 0.0% 10 0.7 0.7 0.5 0.8 0.1
2.|-- 172.16.11.1 0.0% 10 11.5 11.1 10.3 11.6 0.4
3.|-- 192.168.42.252 0.0% 10 10.8 11.1 10.4 12.0 0.4
We can then use BGP between our AS. I'll use AS 64514 and Rémi will use 64513.
Autonomous systems are what they are : autonomous systems. All the networks I do manage at home represent an AS, and all the networks some friend manages represent another AS. AS sizes doesn't matter. Those two AS are bordered : they have both one router that is directly connected to the other's one, in our case, the "directly" is logical : it is a VPN endpoint. For Internet operators, "directly" is physical : they interconnect in a POP on the same router, and thus they create an interconnection point into the Internet by using BGP to announce traffic they'll like to flow here. You may explore all of them using https://bgp.he.net/, for example.
So what we have to do is create a BGP instance with our AS, check the networks we want to sync, and connect our BGP session with friends' ones.
By default, BGP will redistribute any route. Peers must use input filter to filter what they want to receive and what they don't want. Peers can also make use of output filter to select what they'll announce to the others. If you don't filter, you can really mess up the entire network by announcing something that should not be announced, and then have the traffic de-routed from its normal routing schema. Take care of what you do.
BGP routing can get complex. At an ISP scale, it is awfully complex and require a ton of good practices not to mess up a whole part of the Internet. At our little ridiculously tiny scale : the risks are downed too, but still exist.
> /routing bgp network add network="192.168.0.0/24" synchronize="yes"
> /routing bgp instance add name="default" as="64514" redistribute-connected=no redistribute-static=no redistribute-rip=no redistribute-ospf=no redistribute-other-bgp=yes out-filter="" client-to-client-reflection=yes ignore-as-path-len=no routing-table=""Just above is the creation of a simple BGP instance that will announce to other BGP peers my 192.168.0.0/24 network, and that will redistribute to any friend, any route learnt from any other friend using BGP, not only Rémi. They will have to filter at input what they'll want to keep, and what they don't want.
Now, let's add Rémi BGP peers, I'll have two of them, one for each ISP, and I'll declare some input and output filters to change the route priority for the data flow to flow through ISP#1 in priority, and have ISP#2 as a backup.
/routing filter add chain="out-aspath" set-bgp-prepend=3 action="accept"
/routing filter add chain="remi-in-isp1" set-bgp-local-pref=200 jump-target="remi-in"
/routing filter add chain="remi-in" prefix=192.168.42.0/23 prefix-length=23-32 action="accept"
/routing bgp peer add name="remi-isp1" instance="default" remote-address="172.16.11.2" remote-as=64513 in-filter="remi-in-isp1"
/routing bgp peer add name="remi-isp2" instance="default" remote-address="172.16.10.2" remote-as=64513 in-filter="remi-in"If Rémi configures its BGP session on his side, then BGP should be up and running, announcing routes. Check :
> /ip route print detail where bgp
0 ADb dst-address=192.168.42.0/24 gateway=172.16.11.1 gateway-status=172.16.11.1 reachable via remi-out-orange distance=20 scope=40 target-scope=10 bgp-as-path="64513" bgp-local-pref=200 bgp-origin=igp received-from=remi-orange
1 Db dst-address=192.168.42.0/24 gateway=172.16.10.2 gateway-status=172.16.10.2 reachable via remi-in-sfr distance=20 scope=40 target-scope=10 bgp-as-path="64513" bgp-origin=igp received-from=remi-sfrBGP makes use of TCP port 179. You should open that in your firewall on the VPN interface that will support BGP sessions.
Like you can see, the BGP local preference allows me to have the router prefer this path (ISP#1) than the other one (ISP#2).
Depending on our filters, Rémi could now announce some more routes to me, or the opposite; and have all our related attached machines access those later, without any intervention. This is BGP.
If you are curious about the Full View (that is the full Internet BGP table), you may find it, and its evolution from 1994 to nowadays, at https://bgp.potaroo.net/.
Conclusion
Internet is a sum of many networks managed by Autonomous Systems. It is possible at our home-level scale to use BGP with private AS and private IP addresses, and inter-connect our home networks the same way the Internet is itself interconnected, BGP being the main dynamic routing protocol of the global Internet.
Play with the Internet, create your own, only depending on your ISP connectivity. That's why you should choose your ISP for its connectivity to the world point of view, and not only from a "bandwidth" point of view. Then, connect to friends using your own hardware, all privately and encrypted. With such a scenario, you can now consume your friends movie list from their NAS, you can listen to their music even remotely as phones talk IP nowadays (and support VPNs), you can play with them video games directly (not sending packets to some external host you don't know and having this latter route them for you), well, you can do your stuff without relying (and paying) on any external soul, just your ISP.
This, is the definition of Internet, the real one ;-)