Replacing the ISP CPE ("box") : the Orange ISP case study.

Introduction

This post is close to the "Introducing my home network stack" one, but it will push further the technical details on how to get rid of your provider Customer Premises Equipment (CPE), often called - at least here in France - the "Internet box" or simply the "box". We'll detail how to plug-in a router in place of the CPE.

I suggest you read the "Introducing my home network stack" before this post. I will focus here on the "how-to" part.

As we will talk about network here, some network knowledges are required. You must master OSI, at minimum, and its common protocols for each layer at least 1 to 4.

Data network design is not complex, but yes it is a big/wide subject. It is layered. Study layers one after the other, understand each layer goal and not-goal, and how they are interconnected with each others. When operating, always think at what layer you sit, and what goal you have to achieve.

CPE ?

Customer Premise Equipment is delivered by the ISP (Internet Service Provider) to the customer applying to one of its commercial offers. It is the end device on the other side of the Internet line, from the ISP point of view. Your ISP's got equipement at one end, sending you frame informations providing you Internet access, and the other end-of-the-line is plugged into the CPE.

So, if your ISP updates its stack, it will send an update to the firmware of your CPE for it to be able to reach the new deployed infrastructure and eventually talk new protocols. That's why ISP usually provide a CPE so that end-users have nothing to care about on their side, but barely plug-in and go. You may find some more details into the Wikipedia article about CPE, for example.

CPE ends the line, and acts as a modem or kind of, but it usually provides more services - at least a router and a firewall - and may be more advanced providing some more features acting at every OSI layers.

Once more, if you need some further details about one point or another, get some documentation by yourself. I often find Wikipedia as being a good entry point to such knowledge, thus I often link to Wikipedia.

Why replace the ISP CPE ?

Well, to push things further.  In one sentence : to put your hand into OSI from 1 to 4 at least and master everything in your home network.

ISP CPE are usually closed black boxes equiped with light hardware, or at most medium, that talks in OSI layers 1 to 7 the way the ISP has programmed it to.

This is not to say they are bad, as-is, but they usually don't offer very advanced settings. Like for example, my router is connected to several friends' routers and networks through VPNs using GRE/IPSec or PPTP or OpenVPN. My Router then talks BGP onto those VPNs, to push/pull routes to/from my peers so that we can make our Internet with our rules on top of the global Internet. Not really a use case you may do with your ISP default CPE that usually can barely handle few devices connected to it on the LAN part, end-of-story.

Also, the firewall embeded into the CPE is really weak (in configuration I mean). You can't really play on the connection tracking (TCP timeouts, TCP SYN cookies, options, etc...), the rules are basic, etc... With IPV4, the NAT features also are just basic and can't be pushed any further (NAT Hairpinning, for example ?)

Example of a SOHO rackable Mikrotik router I own.

There are many reasons to use your own hardware instead of the one provided by your ISP. If you change ISP, you change the hardware ? The technology bringing Internet access is the same between ISPs, why should one change the hardware then, and re-do a complete configuration from nothing ?

But be warned if you replace your CPE, you take some responsibilities :

  • Patching your stack when your ISP will patch the other side (may happen, everything evolves)
  • Securing yourself, fully (firewall). If you make a breach, you may pay it or make other Internet users pay it (replay attacks)
  • Securing your router by updating its firmware and OS as often as needed

Last thing : You must absolutely keep the CPE somewhere. If the ISP updates its protocols, you are disconnected and in a black hole. You'll then need to reconnect the CPE, wait for its firmware to get patched upon connection, and then dump again the new network procedure and dialogue. The CPE in France is usually attributed to the customer as part of the monthly subscription fee.
You could also find some Web access and browse forums for help if you don't feel up for a network study case when ISP update happens ;-)

Choosing your hardware

Well I cannot choose for you. If you already feel comfortable in a brand, go for it. Check what it allows you to do, and how deep you are allowed to go. Some routers are just like ISP CPE : very cheap, not pushed very far.

It all depends on your own needs ;-)

I use Mikrotik

I did myself invest in Mikrotik hardwares some years ago, nearly for a decade now. I wanted to keep in the SOHO range with cool features, cool hardware, but not BIG price. Mikrotik does the job, also Ubiquiti is a known brand in the same range.

You may choose fully open source software solutions, or closed source ones. Mikrotik is closed source and it is known to be built on top of a Linux Kernel.

From my own point of view and experience, Mikrotik is wonderful as it offers so many advanced features, at every OSI layer, accessible through easy command line syntax, or pretty nice GUI; for a very interesting price, let's say it : a cheap price.
Not many bugs, not many security flaws in the OS or tools, a good wiki, a good forum, some rare books (but still), etc...
They mainly sell hardware with their software pre-installed, or licences for their OS standalone, which is called RouterOS. The OS can be tried for free with many features enabled, on an X86 machine. Other atchitectures are supported as well. Try it, I'm sure you will enjoy it like I do.

This blog post will describe scenarios using Mikrotik hardware and RouterOS software (ROS) version 6.45.7 from end of 2019.

Internet access technologies

Well today, there exists two main technologies : DSL, should it be Symetric or Asymetric, and fiber based access.

So, you have to connect your router weither to a telephone cable (DSL), or a light-based optical cable (fiber). For that, you will probably use SFP modules :

SFP module for fiber technology
SFP module for DSL technology

If not, you can use a transformer device that will turn the Internet socket (DSL or Fiber), into a more common Ethernet RJ45 cable. For DSL, those devices are modems. For the fiber case : they don't touch anything but just convert the signal from one OSI layer 1 to another. Like this one :

TP-LINK MC220L , known to work perfectly with Orange ISP fiber. Converts from SFP to RJ45

Layer 1 transformer devices are active devices thus requiring a power source.

Authentication protocols

The protocols used to authenticate your device against the ISP-side one will be weither PPP based, or DHCP based. In 2019 at least. There exists other protocols, pretty uncommon. I haven't got a lot of knowledge in those however.

You should apply an ISP that talks common protocols, RFC'ed. If the ISP stack is custom, then you may figure out if it is possible to connect to it using your own hardware. To help in this task and get the informations, turn to your ISP directly, or if that later is somehow keeping informations for itself, use public forums where tons of people experiment, discuss, dump, think, debate ... and above all : share knowledge. For France, lafibre.info is a nice resource.

The Orange case-study with fiber technology

Orange is the name of the ISP that will be studied here. I leave in France, as a reminder. We will study a fiber access case. DSL access is not that much different.

Orange uses common protocols, they did not build their owns, but they sometimes don't follow the RFCs or the norms, they differ a bit. Whatever. They make use of common stacks, with no encryption at all (from OSI 2 to 4 at least, what we are interested in).
The CPE talks to the other end with just classic readable OSI protocols, you just have to dump those to understand what happens, and replicate the scenarios.

A dumping session of the CPE talking on its Internet side interface

As there is no auth challenge yet (it is coming for Orange ^^), you get what you need from the network dump with no crypto challenge, easy piece of cake :-)

Here is a short story :

Fiber GPON

Orange makes use of GPON as OSI Layer 1 fiber technology to the customer, in France. They provide a CPE with an external SFP module, a SERCOM-FGS202. You must use the one provided by them as its ID will be checked by the Orange Infrastructure. Using any other GPON SFP module simply won't work. Thus you must buy a router with SFP sockets and compatibility with this module, or use an external converter like the TP-LINK MC220L which is known to work. Many Mikrotik hardwares are compatible with the SERCOM (but not all references), other router brands are as well as the SFP module has nothing really "special" or custom.

The Orange Sercom SFP module connected directly into a border router, thus completly bypassing the CPE

New Orange CPE hardware - from October 2019 - doesn't provide an external SFP module anymore. You must then buy a SERCOM-FGS202 on your own, and ask the customer support to register its ID (serial number) in the infrastructure so that it will be allowed to talk to the ISP hardware. You may also try devices like this one, from Ubiquiti.
The CPE mac address is not used for physical authentication, only the serial number in the SFP module, at least for the Internet part (for VOD part, MAC is used).

So for my use-case, that wasn't hard at all. I applied an Internet offer, they provided me with a SERCOM SFP they registered in their database, and I simply plugged it into my router instead of the CPE. The GPON layer 1 communication is then managed by the SFP module itself and that later is known by the ISP hardware as they registered its ID when applying (you then can't connect any other SFP module but this one : don't loose it or break it :-D)

802.1Q VLANs

Once the SFP is connected and accepted by the other end, we have access to OSI layer 2. Whatever layer 2 we'll send, will be turned to layer 1 optical signal by the SFP module. For layer 2 now, Orange makes use of classical Ethernet and also VLANs, 3 of them to be accurate :

  • VLAN 832 : Internet
  • VLAN 838 : VOD services
  • VLAN 840 : TV and IPTV

You can notice those VLANs from network dump analyzis. You may use more advanced sniffing setups using TZSP for example, Mikrotik supports it. That prevents you from moving wires in your network setup, for example (you mirror the CPE port to your machine using TZSP monitor).

I already talked about my own VLAN usage for my own network.

So, take care of the VLANs and don't forget about them. If you are wrong starting from layer 2, that's not a good scenario to climb the other layers right ?

Let's then create the VLAN on the interface supported by the SFP module in Layer 1 :

> /interfaces vlan add vlan-id=832 interface=sfp-wan-orange name=vlan832-orange-internet

You can dump your SFP stats if needed :

> /interface ethernet monitor sfp-wan-orange 
                name: sfp-wan-orange
              status: link-ok
    auto-negotiation: disabled
                rate: 1Gbps
         full-duplex: yes
     tx-flow-control: no
     rx-flow-control: no
  sfp-module-present: yes
         sfp-rx-loss: no
        sfp-tx-fault: no
            sfp-type: SFP-or-SFP+
  sfp-connector-type: SC
 sfp-link-length-9um: 20000m
     sfp-vendor-name: SERCOMM
sfp-vendor-part-number: FGS202
 sfp-vendor-revision: 0001
   sfp-vendor-serial: XXXXXXXXXXXXXXXX
sfp-manufacturing-date: 17-12-25
      sfp-wavelength: 1310nm
     sfp-temperature: 44C
  sfp-supply-voltage: 3.3V
 sfp-tx-bias-current: 13mA
        sfp-tx-power: 1.926dBm
        sfp-rx-power: -19.355dBm
     eeprom-checksum: good
              eeprom: 0000: 03 04 01 00 00 00 00 00  XXXXX
              XXXXXXXXXXXXXXXXXXXXXX

802.1P

Orange sometimes uses 802.1P COS priority of 6 for DHCP packets. We are still sitting in Layer 2 here. COS tagging depends on your geographic zone. I myself don't need it, but if you do, you must tag the frame. If not : no answer from the DHCP server.

Priority 6 in the 802.1P header. VLAN ID 832 as of 802.1Q

To do so, you can use the L2 filtering of Mikrotik, through a bridge. Put the VLAN832  interface as part of a bridge, alone, and add this filter :

> /interface bridge filter add chain=output action=set-priority new-priority=6 passthrough=yes out-interface=vlan832-orange-internet mac-protocol=ip dst-port=67 ip-protocol=udp log=no log-prefix="" 

You should hardware offload the bridge port to a switch chip in your Mikrotik. So yes : you must choose the right hardware ;-) Refer here. Remember : All the packets from Internet will go through this filter, so if you don't hardware offload it, you will pay a big performance penalty and won't reach 1Gb bandwidth (Orange offers mainly 1Gb of bandwidth  today 2019).

Also remember that now the VLAN interface is part of a bridge, the main interface becomes the bridge, not the VLAN anymore.

DHCP

Orange uses DHCP to authenticate and provide network configuration to its clients, at least for B2C commercial offers, so we have to create a DHCP client over the VLAN interface (or the bridge if 802.1P tagging is used). Here, we are climbing at OSI layer 3, and about to get some layer 3 configuration to setup IP.

3 exotic DHCPV4 options are required :

  • Option 77, userclass
  • Option 60, class-id
  • option 90, auth
Analysing DHCPV4 requests from the Orange CPE

We can analyse that, decode it, and replicate it in our router.

  • Option 60 should get the value "sagem"
  • Option 77 should get the value "+FSVDSL_livebox.Internet.softathome.Livebox4"
  • Option 90 should get the value  0x00000000000000000000001a0900000558010341010dXXXXXXXXXXX with XXXX being the ISP customer login
/ip dhcp-client options add name="vendorclass" code=60 value="'sagem'"
{same for the other options}

/ip dhcp-client print where status=bound
Flags: X - disabled, I - invalid, D - dynamic 
INTERFACE                   USE-PEER-DNS ADD-DEFAULT-ROUTE STATUS  ADDRESS
 0   vlan832-orange-internet   no           no              bound  xxxxx

If the options have the good values, and you talk over the right interface (VLAN and/or COS tagging), DHCP should get bound like shown above. If not, check the setting or increase DHCP logging :

/system logging add action=memory topics=dhcp

For IPV6 now, DHCP is used on Orange infrastructure. 3 DHCPV6 extra options are required :

  • Option 15, userclass, should get the value "+FSVDSL_livebox.Internet.softathome.Livebox4"
  • Option 16, class-id, should get the value 0x0000040e0005 followed by "sagem"
  • Option 11, auth,  should get the value  0x00000000000000000000001a0900000558010341010dXXXXXXXXXXX with XXXX being the ISP customer login
DHCPV6 options dumped

For IPV6, you must ask for a prefix, as Orange provides you a /56. If you request a single address, that won't work.

>  /ipv6 dhcp-client print detail

interface=vlan832-orange-internet status=bound duid="0xXXXXXXXXXXX" dhcp-server-v6=fe80::ba0:bab request=prefix add-default-route=yes 
  default-route-distance=2 use-peer-dns=yes pool-name="Orange" pool-prefix-length=64 prefix-hint=::/0 dhcp-options=auth,class-identifier,userclass 
  prefix=2a01:cb00:b6f:f900::/56, 2d1h37m37s 

The ISP part is neighbor-discovered on Link Local as being fe80::ba0:bab. Nice IPV6 naming isn't it ?

Orange provides to B2C customers "semi fixed" IP addresses. That means that if you release it (by turning off your router, for example), it will be released back in the available pool. The same applies for IPV6. If you want to keep your IP addresses and don't change, never disconnect your router, or not too long (20min to several hours have been reported on forums, that varies. you can afford a short disconnection).

A little bit of firewalling

Not talking about LAN segments yet to route, let's quickly protect ourselves as we are now wide open over the Internet :

/ipv6 firewall filter add 

chain=input action=accept protocol=icmpv6
chain=input action=accept connection-state=established,related in-interface=vlan832-orange-internet
chain=input action=accept protocol=udp src-address=fe80::ba0:bab/128 in-interface=vlan832-orange-internet dst-port=546 comment="DHCP allowed"
chain=input action=drop in-interface=vlan832-orange-internet
chain=forward action=accept protocol=icmpv6
chain=forward action=accept connection-state=established,related
chain=forward action=drop in-interface=vlan832-orange-internet

/ip firewall filter add
chain=forward action=accept protocol=icmp
chain=forward action=drop connection-state=invalid
chain=forward action=accept connection-state=established,related
chain=forward action=accept
chain=input action=accept protocol=icmp
chain=input action=drop connection-state=invalid
chain=input action=accept connection-state=established,related
chain=input action=accept in-interface=trunk
chain=input action=accept protocol=udp in-interface=vlan832-orange-internet dst-port=67,123
chain=input action=drop

I'm feeling better. That's the basic protection right : accept blindly any established forwarding or input, and drop any non-established forwarding or input from Internet. Accept input from the internal network (named trunk here), and accept to be queried for NTP and DHCP. Also, in any case, accept ICMP. As a reminder : ICMP is mandatory to IPV6, and strongly suggested for IPV4. You could implement rate control as well (always a good idea).
As you can see, we protect ourselves and our network peers against unknown incoming connections from the Internet. There should be no need to setup any firewalling on any device on the LANs part of the router, at least about incoming.

Let's go and check the connection now.

Routing and IPV4 NATing.

From this point, you should now be able to access the Internet securely using your router, both in IPV6 and IPV4. Check :

/ping 8.8.8.8     
  SEQ HOST                                     SIZE TTL TIME  STATUS
  0 8.8.8.8                                    56  56 1ms  
  1 8.8.8.8                                    56  56 1ms  


ping address=2001:4860:4860::8888
  SEQ HOST                                     SIZE TTL TIME  STATUS
  0 2001:4860:4860::8888                       56  55 1ms   echo reply
  1 2001:4860:4860::8888                       56  55 1ms   echo reply

Check OK, I assume pinging Google is enough to demonstrate. We could try HTTP or anything else, but would need some DNS resolution. If you use ISP DNS received from DHCP, that should already be all right, just try to ping a domain to check against DNS resolution as well.

Let's create a LAN segment for IPV6 now, advertise it, and route it to Internet :

> /ipv6 settings set forward=yes accept-router-advertisements=no
> /ipv6 address add from-pool=Orange advertise=yes interface=trunk
> /ipv6 nd add interface=trunk managed-address-configuration=no other-configuration=no advertise-mac-address=yes advertise-dns=yes

The default route should already be present as we asked for it in DHCP-client.

Now LAN devices should own a GUA IPV6 through SLAAC taken from the allocated pool from our ISP, and can access the Internet using IPV6 (as we requested a default route). Cool so far.

IPV4 will need NAT, because you know that for IPV4 we are globally running out of addresses, so our ISP only provided us with one little IPV4 address, whereas we have been given in the IPV6 world a /56 which is ... ? ah, yes, 256 times the number of total IPV4 addresses in the world, just for us. Amazing IPV6 isn't it ?

Let's NAT IPV4, because IPV6 today is still not enough to talk to the world !

> /ip firewall nat add chain=srcnat action=masquerade src-address=192.168.0.0/24 out-interface=vlan832-orange-internet

Done : we are connected to the Internet both in IPV4 and IPV6, without using our ISP CPE at all (called "LiveBox" for the Orange ISP, by the way), and with basic firewall rules. :-)

DNS

As you may know, I own my resolver so I use it and don't use DNS resolvers provided by my ISP or anyone else. This part could be detailed in a further article. I use bind with split horizon DNS and zones transfers with peers connected in VPN to my router.

/ip dhcp-client print where status=bound
Flags: X - disabled, I - invalid, D - dynamic 
INTERFACE                   USE-PEER-DNS ADD-DEFAULT-ROUTE STATUS  ADDRESS
 0   vlan832-orange-internet   no           no              bound  xxxxx

You can see on the example above that my DHCP client has "use-peer-dns" turned to off, which means that I ignore the DNS servers pushed through the DHCP answer and don't install them in the router.

Wifi

I did talk about it in my home network blog post. I use some wifi hotspots connected on the LAN part of the router, using VLANs on top of it to separate networks.

Some routers embed wifi chips and can act as hotspots by themselves, pretty much like ISP CPE does. I prefer a real Wifi topology planning with several hotspots in the house. I use Mikrotik or Ubiquiti hardware for that, but all is stand alone and obviously not cloud oriented (as that would leak all the traffic details to the provider, something which is not his business but my privacy).

More firewall :

If you host services like I do, I suggest implementing TCP SYN flood detection and ICMP flood detection at minima.

> /ip firewall filter add chain=forward action=drop tcp-flags=syn connection-state=new connection-limit=30,32 protocol=tcp in-interface=vlan832-orange-internet

chain=forward action=accept protocol=icmp
chain=input action=accept protocol=icmp icmp-options=0:0 comment="echo replies"
chain=input action=accept protocol=icmp limit=20,10:packet 
chain=input action=drop protocol=icmp log=no log-prefix="icmp_flood" 
chain=output action=accept protocol=icmp icmp-options=0:0 comment="echo replies"
chain=output action=accept protocol=icmp limit=40,10:packet
chain=output action=drop protocol=icmp log=no log-prefix="icmp_flood" 

> /ip settings set icmp-rate-limit=10

I myself don't make use of TCP SYN cookies but prefer a rate limit of opening connections. Mikrotik manual has other examples of such cases, and you should experiment as well. Firewall rules can really get pushed very far with Mikrotik, I only showed here some basics.

Orange TV

To be detailed in a next blog post.

You can read some rare uncrypted IPTV (from VLAN 840, once connected) streams as-is (enjoying TV with a client such as VLC), but most of them are encrypted and need the TV Box for decryption.

It is possible to connect the TV Box - normally connected to the main CPE box - to your router instead, and emulate the dialog with the main CPE forwarding it the IPTV streams from the ISP using IGMP. That works, here again, classic protocol stacks involved with no encryption.

I could detail that in the future.

Conclusion

Well, here was a tutorial about my Orange Internet setup at home.

Orange's infrastructure moves. They changed DHCP auth recently, and are still moving. Forum reads, shares, a good brain and some network capture dumps still allows us today to connect our own devices and build our own network(s) as part of the Internet.

Protocols should stay open and widely used, ISPs have no real interest in closing their public-faced stacks. They are secured enough, that would cost tons of money (think about some kind of proprietary encryption from layer 2) for ... what goal ?