This blog post will show you what I deployed at home, to get access to the Internet.
Also, flows go in two directions as the blog you're reading is self-hosted at my home as well, on a full custom hardware I bought and linked myself to have my tiny-production at home.
No deep technical details will be given here. Those will be subjects for next posts to come.
Well, for fun !
More seriously, I'm a computer scientist, and as such, I also have some networking, hardware and electronics/physics knowledge. Networking is really an interesting challenge to me, I like it a lot, sitting next to Web application development or more generally computer programming.
So, I wanted to start a project of customizing my home network stack, and pushing it to a professional stack, like one you could find in some datacenter, but scaled to my personnal little level and needs ;-)
I tend to be a do-it-yourself guy in some ways. I like self-hosting (I even self-host my DNS zone). I need nobody to be up, except of course my Internet Service Provider (ISP), but even here, I got a backup (as I apply two to three ISPs).
My blog, my pictures, my domain, everything about jpauli.tech is done by myself, except the application themselves that I did not develop from scratch. This is a hardware-and-network lab, not a coding lab here ;-)
In my past jobs, I've been working for a company which performed everything by itself internally. I've been involved in IP address buying (We were LIR from RIPE), BGP/OSPF routing of our IP blocks, IPV6 obviously, hardware purshasing, enclosing and wiring, etc... All done by hands.
Nowadays, things tend to get more and more fragmented, and less people have "the full picture" in mind, when it comes to explain how from your mouse click here, you see that appearing on your screen... And virtualization just adds some mess and complexity to the global scene understandings.
What (is needed) ?
All detailed later :
- A router
- A switch
- A Wifi Access point
- A general-purpose server
- A storage-dedicated server
- Some wires, no optical wires (not needed)
- A case to stack all that
- A battery-backed power inverter
You can see all this stuff running on the picture bellow.
The lines you are reading are delivered to you by this stack.
It is light, cannot handle thousands of parallel requests but its properly sized for my needs :-)
How much $$ for that ?
Well, the overall total cost is something like 1500€, brand-new.
It is not that much, I even like to say that it is not expensive at all, considering the power behind such a stack :-)
Details are barely (for new items, from traditionnal market places) : 450€ router, 250€ switch, 400€ server, 400€ NAS and some other stuff.
Things are always the same : instead of buying the "genious new gamer router that makes coffee as well" , and get trapped in marketing promises, push a little further your reflection, put a little bit more money on the table, and you'll end-up with fully professional hardware like the one you can find in datacenters (and the ones you may have had hands on in the past ?).
The quality is way above what you can find on the personnal-target market, and its marketing sharks fighting together to pull the meat (you) to them...
With professional hardware, you really can start toying with true concepts, for not that much of price.
Here is my stack, fitting my needs. There exists any prices, for any sizes of all the items presented here.
Not fully detailed here, but...
I first wanted an Internet access fail-over. ISP in France are good, of a good quality (compared to the world mass ISP average), and not expensive at all. So it is possible to apply to several of them, not just one. This is called MultiHoming.
When ISP#1 goes down (because that can happen, and everyone can understand that, no need to detail), I wanted to be able to still surf the Web and access the Internet, with nothing to do at all. This is just a simple network route fail-over, but you need some good hardware router to do that.
Also, if I can reach that target on the Internet better from ISP#2 than from ISP#1, then I want to change my routing and get the best one (this case is rare, as ISP in France are interconnected in such a way that there is barely no differences for any Internet target. More on that in a future blog post).
Second, as the bandwidth is crazy huge in France (1Gbps down, and 500Mbps up as of 2019), I can host services, files, blablabla , without suffering from the available bandwidth. My different ISPs ping are 2 to 15ms to most services based in France... My ISPs interconnexions to the world are pretty good (I always check that before applying) so you may access my servers and services in few milliseconds, with tons of bandwidth.
So the second need is : input traffic : host some services.
Third, as we are in 2019 now, I wanted a dual stack : that is IPV4 and IPV6. Seriously, the Internet is sick nowadays, sick with IPV4-illness. One should not apply to an ISP not providing IPV6... Here again, the router capabilities about IPV6 (tunnels, etc...) are crucial.
A lot of my hosted services are dual-stack compatible, and can be accessed with IPV6 only.
One cant be a network engineer, using IPV4 only. IPV4 belongs to the past. IPV6 is so much more well designed in its roots...
Fourth argument : VPN and interconnexions. I wanted to be able to create some networks, onto the Inter-net(work). With the help of general VPNs, IPV6, IPSEC and BGP I managed to actually build my own private networks with friends and familly, where each one can access each other resources, over the global Internet, but fully secured / authenticated / crypted.
So for example to access my brother's NAS, located somewhere in France, with IPV4 I don't use the public Internet IP, but I use a private range from RFC1918, for IPV6, we can get rid of those private ranges and we route directly the private IPs. Easy (and well firewalled).
Internet, is just a bunch of networks, interconnected together. Create your own network, and participate in the global Internet evolution ! This is how it's been built.
One cannot access my "private" resources without mounting some VPN access and exchanging routes with IPV4. In IPV6, I use my public range (a /56) but I protect it with IPSec. As little services as possible are directly accessible through the Internet. That is how I implemented security : Whether you auth with some strong crypto/algos, and you got an account with me, whether you won't be able to access anything, and don't try to force as that could lead to a ban of your IP, or worse : a ban of your IP AS announced range.
With such a network, you don't really need instagram/facebook and all, to share a lot of stuff with your familly and friends, you just need them to be equipped with a feature-rich router, and the lowest prices start below 100€.
The routers will exchange privates routes over a VPN, and everyone will be able to access everyone's resources, like if we were in the same room alltogether.
Exploiting the Internet, is building into/onto it, not buying a service from one other who builds stuff for you (and give access everyone your private resources)... it's been designed with that goal in mind 50 years ago, but the history has written another sheet...
Hardware in details
I wont give too many technical "secret" details here, but it all starts with a router implementing every needed technology.
I'm running a Mikrotik ccr1009-7g-1c-1s+ router.
Mikrotik is an outsider, nearly an alien in the worldwide network communications.
They are not very known, thus pretty mature as they started their adventure back in beginning of nineties ! Mikrotik is a Latvian company that produces both hardware and closed-source routing software known as RouterOS.
It seats somewhere around big heads : Cisco and Juniper, but it provides very affordable software and hardware.
The licencing fees for the RouterOS exploitation are ridiculous compared to Cisco's IOS policy, for example.
Same for the hardware prices : I would use such a brand if I had to grow an Internet business, at least at start. Mikrotik cannot handle backbone traffic, Mikrotik cannot push that far away, but it can handle everything else, including small/medium or even large companies networks.
So, that's enough for my needs.
I had to learn RouterOS as I knew nothing about it, but it's still a routing OS based on a Linux Kernel, so if you have strong networking (and Linux networking) knowledge, the step is not hard to take to master RouterOS.
Ubiquiti is once more an outsider in global networking technologies. Younger than Mikrotik, they provide as well nice references, closed-source but built on top of Cisco's IOS (at least for the hardware I got from them).
Here again, in term of pricing, nothing to share with big market colossus.
I got a EdgeSwitch Lite 24
It is enough for my needs. I needed more than 16 ports, not that I have that many devices, but some of them use 802.3ad link aggregation, and so RJ45 sockets tend to get used rapidly.
I don't really need active POE, that's a good-to-have feature but that costs so much more money and that heats a lot. I got passive POE however, just for one device.
All the stack is stored into my main living room, so as much as I can, I try to buy quiet hardware, with passive-only heat extraction. POE switchs do a lot of noise because of power generation and consumption over Ethernet, they require heavy active heat extraction.
I also needed VLANs as I exploit them a lot to segment the networks.
For example, it is out of questions that my IOTs (I got few of them) can communicate with my computer/servers stack. Also, I want to see and control every byte every IOT exchanges with those servers hosted on Chineses AS. Yes, I can't believe what happens to the data and the whole numeric life to someone that doesn't master his own network...
Then, if I could get some RADIUS support from my switch .... It does !
If you want to plug anything physically onto my stack, you'd better have an LDAP account with me, if not RADIUS will not allow you to reach any networks.
I love RADIUS. It all started with RADIUS, and so few people nowadays now what it is, what it does, etc...
I also needed port mirroring.
Because when you want to debug a network stack, be it complex, you need both TCPDump and port mirroring support from your switch.
Just a note about the recent SuperMicro hack chip : I don't suffer from paranoia. So no, I did not open my case to check if some soul added a special modchip that would allow the NSA to breach into my stack. That's so ridiculous.
So I bought a SuperChassis 504-203B. Here it is :
And I added a MBD-X10SBA-O motherboard to it :
With 8Gb of RAM and 3 BTRFS-balanced mirror SSD.
If you don't know BTRFS, I really recommand you to get some informations about one of the best FileSystem ever invented. Really, BTRFS is just the right FS for my needs here.
Why this model ? Well, first, I wanted 1U server, not a monster. And with that I have a deepness problem : my case is not like production cases : it's not 64U high, obviously, but also it is about 40cm deep. Which is not many. Many hardware for traditionnal production datacenters are at minimum 80cm deep, often even more.
Second, I dont need the power of a Xeon, and I mainly don't need the noise from fans to cool down such a CPU.
So this motherboard has a soldered-in CPU, an Intel Celeron that is not very powerful, but it has some passive heat extraction, and thus no fans, and thus no noise ;-)
Once more : do I need more power to server 20 people a day to read my blog or see my photos ? I don't think so. Let's keep the Earth a little bit cooler here, and my energy monthly bill a little bit lower as well.
Did I tell you that I even manage to run PC gaming multiplayer services on such a light stack ? Well yes, this physical server actually handles many services, and it does it just all right, with no virtualization and no other complexities of that kind, just a custom self-compiled-and-tuned Linux Kernel, like usual.
The main storage is provided by a NAS from Synology : RS816. It is easy to manage, nothing to do and that's what I wanted. No DIY for the NAS ;-)
Once more here, each material has its own responsibilities. It's not because the Synology OS can provide services (PHP, Apache, NodeJS, DHCP, Firewall, VPN, Python, and it makes coffee) that I will use them.
I only use it for storage (CIFS, NFS), and that's all.
I really don't care about its CPU being able to transcode video formats : I will do that by myself if I need.
So that's why here again : the NAS CPU is really tiny - not very powerful - but to just provide some data, there is once more no need of the power of a Xeon. If you start using the apps inside, yes, you'll need a much bigger hardware, but here I myself would buy a second Linux based machine, and serve those services by myself : optimized to my needs.
This NAS is 1U case, and it can be fed 4 disks. It is actually fed 4 disks, using RAID10 and ext4. I got all my data and my life in there, it just works right, it provides 1Gbps of throughput over the network, with two disks failover : I'm happy.
Whatever the firewall or the routing rules into the NAS OS itself : I dont use that. To route, I use .... a router. To serve some pictures, I setup a web stack on a server. I can understand why NAS OS nowadays are full of features (and security holes), but that's not how things must be designed IMHO.
The NAS is plugged to the switch with Link Aggregation LACP lvl 4, so it can throughtput twice one Gigabit per second to two different clients (That model of NAS has two Ethernet ports, that was a requirement to me so that it would be hard to saturate its connexion in case of several clients requesting data from it).
Also, the NAS is connected to a "Multimedia" VLAN, and so it provides movies to my connected TV, and music to my connected Sound Amplifier (Yamaha RN-602). I don't need Netflix's services (or FooBar's services).
When I started filling up my disk space, it was 1996 (my disk was 500Mb large). Netflix did not exist, and today it exists, but I don't need it (same for Spotify and every platform like that, I listen to the music from my NAS, even remotely).
As I got some small space, I don't need any mesh, so any self-controlled Wifi spot will do the job. Cisco's Meraki are definitely a no-go here, I turned to Mikrotik again. They got many references in term of Wifi, so does Ubiquiti.
So I bought just a basic Access Point I pluggued onto its VLAN, and I was done.
I tend not to use Wifi. I've known Internet access when Wifi did not exist yet (< year 2000) and guess what : we managed to make it, and with security hahhh. So I don't use Wifi a lot, I always got a wire somewhere I can connect my device to, and my phone is not connected neither to Wifi, nor to data networks (I dont need that to make a phone call, right ?).
That was a challenge. Not hard to find references for Datacenter housing, but harder for smaller stacks.
I turned to 4xracks, with the 4EW-1065. Just exactly the right dimensions needed to fit the place I want to store it at. Note it is 19" wide, like traditionnal DC racks, simply it is really narrow in depth.
All the stack is 1Gbps, which is way enough for the needs. The router allows one SFP port of 10Gbps (known as SFP+) though. So what I need, is simply RJ45 cat5E wires. that's what I bought (well, I borrowed them from different DC places ^^).
Why use some cat6 or cat7, when cat5E is enough ?
Why use light-based fiber-wires when electrical ones still have some place ?
Tip : electrical based wires can support up to 10Gbps (cat 6A). Above, you'll need fiber (light transport).
So you wonder how many power all the stack drains right ?
I got the number : 150 watts per hour when the four disks from the NAS are rotating. When sleeping, that's 100 watts per hour for the stack.
That's a little cost, that should be something like 40 to 50€ per year, for the energy ? (France prices, full 365d service time).
Yes, that's not a datacenter stack right! That's not a nuclear reactor stack. Small stack, small power consumption.
And thus : small power backup batteries. An APC UPS SC420I :
It can be charged up to 300 watts and provides to my consumption 10 to 15min of energy autonomy in case of main power failure, for the whole stack. It also prevents anti-surge from the main power line.
It is connected to the Mikrotik router (that supports APC-UPS hardware using RS-232, yep) so that the router can fall asleep when it knows that less than XYZ minutes batteries are available.
It switches back to full operation mode when the main power line is back up, without loosing configuration or without loosing its FIB
The whole stack just fits all right in my living room shelf. Look at that :
You may spot the sound amplifier, the TV, an ISP CSP(yes there is still one) and a mini-PC that allows me to play games on my TV and have some sort of Web access to show things on TV (although its a smart connected TV). Also, the inverter is not hidden in the back, but visible and its front panel is accessible.
I could buy a rackable inverter, but it would have been too heavy for my case (which has no rails and no back attach, only front one)
The noise is clearly managed. Only during the 2 hotest months of summer, I could hear it loud, in such a situation I open the case, or improve air extraction by giving more power to the case fans (this is manual).
I will next talk in details about many subjects related to general networking, more or less in deep :
- How to route, fail over, ECMP, Policy based route, BGP (many technologies) ...
- How to use VLANS to segment networks, isolate them and firewall them
- How to IPV6
- How to firewall
- How to QOS packets and flows
- How to use LACP (Link Aggregation Channel Protocol) to combine ports
- How to RADIUS
- How to VPN (many technologies)
Hope you'll enjoy like I do when I share stuff like that.
Once more, opinions are mine, ideas and needs are mine, you like them or not, it's just shared over the Web ;-)