Routed/Switched network hybrid or migrating from switched to routed network
About two years ago I’ve prepared a plan for a network which started out as a completely switched network using mix of Mikrotik, Motorola and other wireless equipment.
The goal was to advise a way to switch from switched to routed network with minimal impact on SLA.
Below I’m providing the guide I’ve prepared for that network with the approval of the customer who wishes to remain anonymous. The guide is very technical and prepared for that particular network but it has some useful tips on building hybrid (switched/routed) network using Mikrotik routers which on its own is quite interesting endeavor.
I hope this may be helpful to some of you out there…
=======================================
PPPoE & Routing – migration plan from switched network
Idea
Migration should be done in steps, one access point at a time.
A routed segment needs to be built from the network’s edge down to that access point.
In order to keep backward compatibility with the bridged devices we’ll setup EoIP tunnels at each router.
After one end router of the network (router with all access points attached to it) is migrated fully to routed network, the EoIP tunnel can be disabled from that router, and the bridge interface removed completely.
This process needs to be repeated for each end router.
Setting up routed network
The nice thing about having MikroTik routers is that we can have bridged and routed network at the same time on the same equipment.
Each point to point will receive /29 (to cover both end of the routes and optional radio equipment). Point to point links will start on IPs 10.15.0.0 and future allocation of network ranges will be decided later.
If Mikrotik is used also as the wireless link /32 point-to-point addressing can be used on both ends of the link to avoid wasting IPs on broadcast and network addresses.
The IP addresses need to be assigned directly to the interfaces they should be accessed from. Example:
SiteA-mt-rb1000
/ip address add address=10.15.0.17/29 interface=”ether3 – gateway”
SiteA-mt-router:
/ip address add address=10.15.0.18/29 interface=”ether2 – RB1000″
/ip address add address=10.15.0.1/29 interface=”ether3 – to SiteB”
SiteB-mt-router:
/ip address add address=10.15.0.4/29 interface=”ether6 – to SiteA”
/ip address add address=10.15.0.9/29 interface=”ether3 – to SiteC”
SiteC-mt-router
/ip address add address=10.15.0.12/29 interface=”ether1 – to SiteB”
[Note: each Site has access points and links to other sites. The above example focuses on one leg of the network]
The IP addresses on the radio equipment between these two routers should also be changed to 10.15.0.2/29 and 10.15.0.3/29, etc. The gateway for the backhaul should be the router connected to it (ie. 10.15.0.1 for the BH with IP 10.15.0.2, then 10.15.0.4 for BH 10.15.0.3, etc).
Next comes OSPF.
We need to create a backbone for our network. Backbone is a network on which the information about routes is being redistributed on. Which means that the /29 links have to be on the backbone in order to share routes on both ends with each other. If it is not possible to have a direct link between backbone routers a Virtual Link needs to be created between two backbone routers, but I won’t touch that topic here (this will be explained in a different doc).
Example: SiteA-mt-rb1000 (10.15.1.1):
/routing ospf set distribute-default=always-as-type-1 metric-bgp=20 metric-connected=20 metric-default=1 metric-rip=20 metric- static=20 mpls-te-area=unspecified mpls-te-router-id=unspecified redistribute-bgp=no redistribute-connected=as-type-1 redistribute-rip=no redistribute-static=as-type-1 router-id=10.15.1.1
/routing ospf network
add area=backbone disabled=no network=10.15.1.8/29
add area=backbone disabled=no network=10.15.1.0/29
add area=backbone disabled=no network=10.15.0.16/29
Note that this is the edge router so it redistributes the default route, no other router should redistribute default routes. SiteA-mt-router (10.15.0.1)
/routing ospf set distribute-default=never metric-bgp=20 metric-connected=20 metric-default=1 metric-rip=20 metric- static=20 mpls-te-area=unspecified mpls-te-router-id=unspecified redistribute-bgp=no redistribute-connected=as-type-1 redistribute-rip=no redistribute-static=as-type-1 router-id=10.15.0.1
/routing ospf network
add area=backbone disabled=no network=10.15.0.0/29
add area=backbone disabled=no network=10.15.0.16/29
add area=backbone disabled=yes network=10.15.0.24/29
Repeat this on the whole path. This will result in a OSPF network that will automatically redistribute connected and static routes.
Before you proceed check that you can ping the first router (ie 10.15.1.1) from the last router. If the routes are propagated correctly you should be able to. Without this EoIP won’t be possible.
Next step is to create EoIP bridges.
The EoIP bridge will ensure that customers still setup for bridged network will seamlessly operate even tho we’ll remove the bridge between the routers.
IMPORTANT NOTE: All the bridgess should use RSTP!
All of the tunnels need to connect to the edge router (connected to the gateway). Each EoIP tunnel on one router needs to have an unique tunnel ID and MAC address. MAC address is randomly generated in latest (3.20+) firmware so the only thing to watch out for is the tunnel ID. These tunnels need to be added to bridges on both routers. These bridges need to also include the ports on which the Access Points are connected.
Example:
SiteC (10.15.0.12):
/interface eoip add arp=enabled mac-address=02:56:1F:C2:84:D3 name=eoip3-SiteA remote-address=10.15.1.1 tunnel-id=3
/interface bridge ports add bridge=bridge1 interface=”ether4 – to AP” add bridge=bridge1 interface=eoip3-SiteA
SiteA (10.15.1.1):
/interface eoip
add mac-address=02:E4:16:48:3D:98 name=eoip3-SiteC remote-address=10.15.0.12 tunnel-id=3
add mac-address=02:D2:D2:90:43:77 name=eoip4-SiteB remote-address=10.15.0.4 tunnel-id=4
/interface bridge ports
add bridge=bridge1 interface=”ether4 – gateway”
add bridge=bridge1 interface=eoip3-SiteC
add bridge=bridge1 interface=eoip4-SiteB
Having this you can now take the interface linking the routers out of the bridge (note these interfaces are not listed in above examples as they are already taken out on the routers from which this example is exported).
Setting up PPPoE access
The edge router facing the gateway needs to have proxy-arp set on the interface (in this case the bridge) facing the gateway.
The router facing gateway needs to have an IP address that the gateway and the router can communicate with, then on the gateway a static route needs to be added for 10.0.0.0/8 range pointing at that router.
Example:
gateway: 172.16.155.1 <–> router: 172.16.155.134 on the gateway: route add -net 10.0.0.0/8 172.16.155.134
This will guarantee that all traffic to 10/8 network will go through that router. The 10.0.0.0/8 IP range is set to be used in the routed network.
The router closest to customers will have PPPoE Server will have two /25 IP ranges one for equipment (CPE) and one for client routers (or WAN on CPE if NAT is enabled there). Initially the IP range for equipment will start from 10.20.0.0, clients will ideally receive public IPs but for now private IPs will start 10.30.0.0.
Assuming that we’ll use 10.30.0.0 for the customer IPs the gateway router needs to be able to route traffic from such IPs.
Also make sure that router has
/routing ospf set redistribute-static-routes=as-type-1
The router running PPPoE (closest to the clients) needs to get that IP range assigned too:
Example:
/ip address add address=10.30.0.1/25 interface=”ether9 – AP”
Make sure that OSPF uses area range to limit the amount of routing table entries:
/routing ospf area range add advertise=yes area=area-name cost=default disabled=no range=10.30.0.0/25
Without this each PPPoE connection would add one entry to routing table across the OSPF network.
PPPoE Server and authentication is the last bit.
/ppp profile add bridge=bridge1 change-tcp-mss=yes local-address=10.30.0.1 name=pppoe only-one=yes use- encryption=yes
Note that since we still have the interface facing Access Point we need to attach the PPPoE server to the bridge.
Once everyone is migrated to PPPoE that interface MUST be taken out from the bridge (and EoIP removed) before you move onto other Mikrotik. This is because PPPoE handshake is Layer-2 based and does propagate over bridge, which can result in all weird results, even with customers from across the network registering with mikrotik on the other side of the network over the EoIP bridge.
/interface pppoe-server server add authentication=pap default-profile=pppoe interface=bridge1 one-session-per-host=yes service-name=internet
/ppp aaa set accounting=yes interim-update=1m use-radius=yes
/radius add accounting-backup=no accounting-port=1813 address=X.X.X.X authentication-port=1812 secret=secretpassword service=ppp timeout=10s
Now the RADIUS/UserManager has to be setup to allow this router to check authentication requests on the X.X.X.X address.
Our PPPoE server mikrotik needs to have access (either routed or via public IP) to the RADIUS/UserManager, and the IP that the UserManager will see as originator for incoming requests needs to be used as the router IP when you’re adding router to it.
Read MoreMikroTik – solution to machine running The Dude that hits 100% CPU
Recently I’ve noticed my Dude RouterOS machine that it hits 100% CPU very often and never goes down below 50% CPU.
Now, this is rather beefy machine with powerful two dual core Xeon CPUs and the RouterOS is set to multi-cpu=yes
I went through the settings on The Dude and what I’ve noticed was the default label refresh interval on the map was 1 second.
There’s at least 7 active sessions at any time, all looking at a map with 300+ network devices, all linked (which also show link usage etc).
Immediately after changing that value from 1 second to 1 minute the CPU usage dropped to 1-2% and only occasionally it jumps to 50%.
So if you see similar thing on your Dude monitoring machine try to set the Settings -> Map -> Label Refresh Interval, to something reasonable.
Member of
