Turning my SOHO ethernet network stack to 10Gbps.

Introduction

We are in 2020. Gigabit Ethernet, 1Gbps, is 20 years old (2000). In France, you can apply Internet Services (ISP) up to 8Gbps. At the moment (begining of 2020) only one ISP in France allows you to get over >1Gbps, but they all announced that within few years (2 to 3 years), they will all update their commercial offers to allow end customers to reach 10Gbps data transfer rates (mostly symetric).

Appart from the fact that this data flow rate is crazy fast as of 2020, and is theoretical maximum, if your Internet line goes above 1Gbps, that means that you will need to upgrade all your network stack, to be able to reach >1Gbps, hopefully 10Gbps, from the Internet link.

Also, as of today, we consume tons of data and need to transfer tons of data. You know about my actual network setup right ? I got a NAS, and even if that stays occasionnal, pushing several Gb of data to/from my NAS with a network at 1Gbps can take ages. Yes, it all evolves. I still remember the first time I plugged-in at 1Gbps (from 100Mbps upgrade), that was amazingly fast to me, but .... Time runs.
Nowadays, you may transfer several dozens of Gb, or even hundreds of Gb from time to time. With SSD reaching gigabyte-per-second transfer rate, the 1Gbits per seconds - aka 125MBytes per second data flow - starts becoming very limiting.

Ah yes, bytes versus bits... I guess you don't confuse them right ? There is a factor of eight in between ! 1Gbps is one gigabit per second, which represent 125 megabytes per seconds (1/8). What interest us are bytes, but what network domain (and marketers) uses are bits.
So when I express a network capacity, aka 100Mbps, those are 100 MegaBits Per Second, and thus represent only 12.5 MegaBytes per Seconds, a very poor capacity as of today. And yes, one byte equals eight bits (or digits) ;-)

1G, 2.5G, 5G, 10G, 40G... ?

I'm not a fan of "half standards". History : Ethernet and "LANs" all started at 1Mbps, then upgraded to 10Mbps, then 100Mbps, then 1000Mbps or 1Gbps... You smell the math series ? You multiply by 10 each time.

I myself started networking studies where 10Mbps were deployed (2000), using 10base2 BNC. You'd better not forget the 50 ohms network terminator back at this epoch !

Ok so nowadays "bastard" transfer rate start to appear in Ethernet, by "bastard" I mean not a multiple of ten. Those bastard transfer rate of 2.5Gbps and 5Gbps (and even 40Gbps exists) can find some fields of application. They are also present on the market because 10Gbps is expensive as of 2020. This is true, however prices can only decrease as the market demand will grow, and that demand is just now starting to appear as of 2020.

So, turning to intermediate, less expensive technology, is just a way for sellers to sell more and make more business ?

For my case, I bet that if today (begin of 2020) I turn to 2.5Gb, in about 3 years, I'll need to upgrade again. Same for 5Gbps? I will jump from 1Gbps to 10Gbps, with no intermediaries. It's like being safe for dozens of years. Do whatever you want once more, that's about needs : if you think you'll feel comfortable with 2.5Gbps, or even if 1Gbps is enough to you : take your own decisions right ? ;-) I'm here to express mine, which may be different from yours : I was just OK with my 1Gbps stack up until a precise need showed up, then I brained, then I decided I could start - and only start - a move to 10Gbps. It's still a little bit pricy for a full stack upgrade.

Making a whole home network evolve, smoothly

Because upgrading a network, is well .... about upgrading the full path network, right ? Routers, switches, wires or Wifi, and NICs. So you'd better be sure about it before investing.

I did jump from 100Mbps to 1Gbps around 2004, and thus my 1Gbps stack has been up since 15 years now, not bad for such a technology. I don't remember the exact price that the upgrade did cost back in that time, but that would be something like the 1G to 10G jump today, barely.

Multiple data rates

What is cool with Ethernet, is that the upgrade in data transfer rates can be done smoothly, as any 10Gb active device still supports 5Gbps, 2.5Gbps, 1Gbps, 100Mbps and sometimes even 10Mbps connectivity ! So we're not talking about a one-full-step upgrade here, but parts. Ethernet auto-negociation rates and duplex works well nowadays.

For example, I will still keep my 1Gbps router and my 1Gbps switch next to a new 10Gbps switch that will be connected to it.

I still have 100Mbps devices connected at home, the ones that don't need more bandwidth : my TV set for example. A TV reads multimedia from the network in real time : it streams it. And to stream video today (2020), you need between 1 and ~50Mbps. Far from a single Gbps right ?
Same for a home standard camera system : it needs a 100Mbps Ethernet connection, not more : it will never make use of such a bandwidth.

So upgrading to 10Gb, is about upgrading what needs it : NAS, fixed PC stations, and Internet connection when it will come. Plugging a 100Mbps device on a switch's 10Gbps socket is a pure waste, that device will stay plugged on the 1Gbps switch.

Ethernet auto-negociation allows several rates to exist on one switch or router. A 10Gbps switch would also support 5G, 2.5G, 1G and 100Mbps devices.

Prices

Well they evolve a lot, and prices are about your own appreciation.

• 10Gbps switches, about 150€ for 4 ports.
• 10Gbps routers start at about 1000€ for 2 ports.
• 10Gbps wires, cat 6A or cat 7, are affordable, same for patch panels, keystones, etc..., not a big deal compared to 1Gbps (cat 5E) wires. cat 5E wires may work for 10Gbps short distance links, but it's not guaranteed.
• 10Gbps NICs can be found at around 100€ cheapest price, but take care as you will need a PCIE port minimum X2 for 3.0 and X4 for 2.0 to support it.
• Wifi is complex, too complex to detail, there exists some norms today that allow one client to get over 1Gbps. Even with less single data rates, supporting several clients in activity, the hotspot could benefit from 10Gbps connectivity, but hotspots with 10G connectivity are still very expensive.

A first example : the CRS312-4C+8XG-RM Mikrotik switch

So I did recently invest into what is about to become my "backbone" : a CRS312-4C+8XG-RM 10 Gbps switch from Mikrotik. This is a first step to 10G. It fits my current needs, and my future needs for dozens of years (I bet).

My current goal is to have my two fixed PCs - used for streaming/gaming - and my NAS to be connected using 10Gbps. This is my hot path. Then, will come the Internet connection and the choice of a 10Gbps router (I got some references in mind as of today), then wifi.

So I don't need that much sockets, 3 to 5. The switch provides a total of 12 ports at 10Gbps, 4 of them being combo ports with 4 SFP+ cages. Hence when future needs will show up, I got plenty of spare sockets ready to be plugged in.

Mikrotik (some other brands too) start to provide cheap switches, with 4 sockets, then 8, then more. All at 10Gbps. I already detailed my thoughts about Mikrotik throughout several blog posts. Obviously the more sockets, the higher hardware price will be.

Some switch models only provide SFP+ cages and no RJ45 sockets : I don't want that for a switch designed to be connected mainly to end devices, using copper RJ45 cat 6/7 wires. The copper-based SFP+ modules cost a lot each (80€), take that into account. I will use the SFP+ cages I got with fiber wires for uplinks to the router and the 1Gb switch, and then plug in my devices using copper wires.

So I started playing with my new CRS312-4C+8XG-RM ! Here are the first things I hacked into :

• Plug-in on my home LAN with a fixed IP address and secure it (credentials, use strong SSH crypto, disable HTTP access etc...).
• General purpose config : Default route, DNS, NTP, SNMP.
• Check that by default, the device is wide open : no L2 rules of any kind, nor L3. That's what I want from now.
• Update to latest RouterOS.
• Disable all RouterOS packages I don't need (many of them). Routerboards come with many packages installed by default.
• Disable IP Forwarding and Firewall (connection tracking).

The hardware ships with many software packages that would let you use it like a router at first glance. Do not : uninstall all the routing features as the CPU is very small and won't route above a ridiculous 450Mbps throughput (according to vendor). If you buy a dedicated switch, it's not for its main CPU, but for its ASIC. For this reference : a 98DX8212

Then I play with the switch and especially the VLANs scenarios. I did secure it using a dedicated management VLAN I deployed on my stack. It uses RouterOS, so I wasn't very lost in the setup even if I haven't touch the switching purposes before, there was no real difficulty in having all my VLANs up.

I'm very happy of my new CRS312-4C+8XG-RM so far ! This really seems to be a killer product. It tends to make a lot of noise if it heats, that is if you push some heavy traffic to it, which is not my case at the moment so the fans are off most of the time. Among other features, I'd like next to try 802.1x.

You should probably watch also Home 10 Gig Network Upgrade for CHEAP from Linus Tech Tips on Youtube if you are interested.

And next

Well, when ISPs will start >1Gbps offers, I'll start looking at the market closer for a router. A Mikrotik I guess ;-) They already provide several references routing at > 10Gbps with 10Gbps ports.

I need to buy NICs also, that's the very next plan. Then upgrade the NAS. The server actually doesn't need over 1Gbps.