Iasa Architect Core – Dublin, March 2011
This year I’m changing the way I work to be more meaningful for myself personally and for long term professional career.
Aside from reserving part of my time this year for consultancy I will spend more time learning. Last year had gone very fast and drained a lot of my energy. It was rewarding in experience, both good and bad.
The first training this year that I’ll attend is the Iasa Architect Core hosted by Irish Chapter of Iasa and presented by Jim Wilt.
The purpose of these two courses are to introduce students to the ITArchitecture skills that the IASA has determined are necessary for allarchitects, regardless of level or specialization, and to then apply those skillsin a simulated business environment.
Routed/Switched network hybrid or migrating from switched to routed network
About two years ago I’ve prepared a plan for a network which started out as a completely switched network using mix of Mikrotik, Motorola and other wireless equipment.
The goal was to advise a way to switch from switched to routed network with minimal impact on SLA.
Below I’m providing the guide I’ve prepared for that network with the approval of the customer who wishes to remain anonymous. The guide is very technical and prepared for that particular network but it has some useful tips on building hybrid (switched/routed) network using Mikrotik routers which on its own is quite interesting endeavor.
I hope this may be helpful to some of you out there…
=======================================
PPPoE & Routing – migration plan from switched network
Idea
Migration should be done in steps, one access point at a time.
A routed segment needs to be built from the network’s edge down to that access point.
In order to keep backward compatibility with the bridged devices we’ll setup EoIP tunnels at each router.
After one end router of the network (router with all access points attached to it) is migrated fully to routed network, the EoIP tunnel can be disabled from that router, and the bridge interface removed completely.
This process needs to be repeated for each end router.
Setting up routed network
The nice thing about having MikroTik routers is that we can have bridged and routed network at the same time on the same equipment.
Each point to point will receive /29 (to cover both end of the routes and optional radio equipment). Point to point links will start on IPs 10.15.0.0 and future allocation of network ranges will be decided later.
If Mikrotik is used also as the wireless link /32 point-to-point addressing can be used on both ends of the link to avoid wasting IPs on broadcast and network addresses.
The IP addresses need to be assigned directly to the interfaces they should be accessed from. Example:
SiteA-mt-rb1000
/ip address add address=10.15.0.17/29 interface=”ether3 – gateway”
SiteA-mt-router:
/ip address add address=10.15.0.18/29 interface=”ether2 – RB1000″
/ip address add address=10.15.0.1/29 interface=”ether3 – to SiteB”
SiteB-mt-router:
/ip address add address=10.15.0.4/29 interface=”ether6 – to SiteA”
/ip address add address=10.15.0.9/29 interface=”ether3 – to SiteC”
SiteC-mt-router
/ip address add address=10.15.0.12/29 interface=”ether1 – to SiteB”
[Note: each Site has access points and links to other sites. The above example focuses on one leg of the network]
The IP addresses on the radio equipment between these two routers should also be changed to 10.15.0.2/29 and 10.15.0.3/29, etc. The gateway for the backhaul should be the router connected to it (ie. 10.15.0.1 for the BH with IP 10.15.0.2, then 10.15.0.4 for BH 10.15.0.3, etc).
Next comes OSPF.
We need to create a backbone for our network. Backbone is a network on which the information about routes is being redistributed on. Which means that the /29 links have to be on the backbone in order to share routes on both ends with each other. If it is not possible to have a direct link between backbone routers a Virtual Link needs to be created between two backbone routers, but I won’t touch that topic here (this will be explained in a different doc).
Example: SiteA-mt-rb1000 (10.15.1.1):
/routing ospf set distribute-default=always-as-type-1 metric-bgp=20 metric-connected=20 metric-default=1 metric-rip=20 metric- static=20 mpls-te-area=unspecified mpls-te-router-id=unspecified redistribute-bgp=no redistribute-connected=as-type-1 redistribute-rip=no redistribute-static=as-type-1 router-id=10.15.1.1
/routing ospf network
add area=backbone disabled=no network=10.15.1.8/29
add area=backbone disabled=no network=10.15.1.0/29
add area=backbone disabled=no network=10.15.0.16/29
Note that this is the edge router so it redistributes the default route, no other router should redistribute default routes. SiteA-mt-router (10.15.0.1)
/routing ospf set distribute-default=never metric-bgp=20 metric-connected=20 metric-default=1 metric-rip=20 metric- static=20 mpls-te-area=unspecified mpls-te-router-id=unspecified redistribute-bgp=no redistribute-connected=as-type-1 redistribute-rip=no redistribute-static=as-type-1 router-id=10.15.0.1
/routing ospf network
add area=backbone disabled=no network=10.15.0.0/29
add area=backbone disabled=no network=10.15.0.16/29
add area=backbone disabled=yes network=10.15.0.24/29
Repeat this on the whole path. This will result in a OSPF network that will automatically redistribute connected and static routes.
Before you proceed check that you can ping the first router (ie 10.15.1.1) from the last router. If the routes are propagated correctly you should be able to. Without this EoIP won’t be possible.
Next step is to create EoIP bridges.
The EoIP bridge will ensure that customers still setup for bridged network will seamlessly operate even tho we’ll remove the bridge between the routers.
IMPORTANT NOTE: All the bridgess should use RSTP!
All of the tunnels need to connect to the edge router (connected to the gateway). Each EoIP tunnel on one router needs to have an unique tunnel ID and MAC address. MAC address is randomly generated in latest (3.20+) firmware so the only thing to watch out for is the tunnel ID. These tunnels need to be added to bridges on both routers. These bridges need to also include the ports on which the Access Points are connected.
Example:
SiteC (10.15.0.12):
/interface eoip add arp=enabled mac-address=02:56:1F:C2:84:D3 name=eoip3-SiteA remote-address=10.15.1.1 tunnel-id=3
/interface bridge ports add bridge=bridge1 interface=”ether4 – to AP” add bridge=bridge1 interface=eoip3-SiteA
SiteA (10.15.1.1):
/interface eoip
add mac-address=02:E4:16:48:3D:98 name=eoip3-SiteC remote-address=10.15.0.12 tunnel-id=3
add mac-address=02:D2:D2:90:43:77 name=eoip4-SiteB remote-address=10.15.0.4 tunnel-id=4
/interface bridge ports
add bridge=bridge1 interface=”ether4 – gateway”
add bridge=bridge1 interface=eoip3-SiteC
add bridge=bridge1 interface=eoip4-SiteB
Having this you can now take the interface linking the routers out of the bridge (note these interfaces are not listed in above examples as they are already taken out on the routers from which this example is exported).
Setting up PPPoE access
The edge router facing the gateway needs to have proxy-arp set on the interface (in this case the bridge) facing the gateway.
The router facing gateway needs to have an IP address that the gateway and the router can communicate with, then on the gateway a static route needs to be added for 10.0.0.0/8 range pointing at that router.
Example:
gateway: 172.16.155.1 <–> router: 172.16.155.134 on the gateway: route add -net 10.0.0.0/8 172.16.155.134
This will guarantee that all traffic to 10/8 network will go through that router. The 10.0.0.0/8 IP range is set to be used in the routed network.
The router closest to customers will have PPPoE Server will have two /25 IP ranges one for equipment (CPE) and one for client routers (or WAN on CPE if NAT is enabled there). Initially the IP range for equipment will start from 10.20.0.0, clients will ideally receive public IPs but for now private IPs will start 10.30.0.0.
Assuming that we’ll use 10.30.0.0 for the customer IPs the gateway router needs to be able to route traffic from such IPs.
Also make sure that router has
/routing ospf set redistribute-static-routes=as-type-1
The router running PPPoE (closest to the clients) needs to get that IP range assigned too:
Example:
/ip address add address=10.30.0.1/25 interface=”ether9 – AP”
Make sure that OSPF uses area range to limit the amount of routing table entries:
/routing ospf area range add advertise=yes area=area-name cost=default disabled=no range=10.30.0.0/25
Without this each PPPoE connection would add one entry to routing table across the OSPF network.
PPPoE Server and authentication is the last bit.
/ppp profile add bridge=bridge1 change-tcp-mss=yes local-address=10.30.0.1 name=pppoe only-one=yes use- encryption=yes
Note that since we still have the interface facing Access Point we need to attach the PPPoE server to the bridge.
Once everyone is migrated to PPPoE that interface MUST be taken out from the bridge (and EoIP removed) before you move onto other Mikrotik. This is because PPPoE handshake is Layer-2 based and does propagate over bridge, which can result in all weird results, even with customers from across the network registering with mikrotik on the other side of the network over the EoIP bridge.
/interface pppoe-server server add authentication=pap default-profile=pppoe interface=bridge1 one-session-per-host=yes service-name=internet
/ppp aaa set accounting=yes interim-update=1m use-radius=yes
/radius add accounting-backup=no accounting-port=1813 address=X.X.X.X authentication-port=1812 secret=secretpassword service=ppp timeout=10s
Now the RADIUS/UserManager has to be setup to allow this router to check authentication requests on the X.X.X.X address.
Our PPPoE server mikrotik needs to have access (either routed or via public IP) to the RADIUS/UserManager, and the IP that the UserManager will see as originator for incoming requests needs to be used as the router IP when you’re adding router to it.
Read MoreFreedom of choice in architecture design
I’ve been part of many projects where choice of language, frameworks and libraries were imposed by the client strictly. This became so natural that even for my own projects I used to look at designing them around single language (ie. Python). Usually JavaScript creeps here and there as well regardless it seems these days of what you are developing: web app (obviously!), data base views (think: CouchDB), game (Unity3D), desktop apps (Titanium/others?).
Time shows that the saying “if the only tool that you have is a hammer, everything starts to look like a nail” is absolutely correct. In every system design the right tools should be used for the job. This is obvious! – you may think. Sure, but how often did you end up anyway thinking in categories of single language?
To be really flexible think of mixing and matching different languages for your projects.
Node.js is maturing and becoming serious contender when it comes to writing web apps. Throwing Objective-J to the mix may give you huge productivity boost. Using Python on the backend to utilize its great amount of libraries to glue your app with whatever you may want or wish for or to handle some more advanced logic. And Erlang for the parts of your system that need concurrency and its async nature (ie. for streaming). There’s four languages right there! And who is not to say that you’ll want to boost some performance with C or C++ modules for Python, Erlang or Node.js in the above scenario.
Unfortunately not every project owner will give you such freedom. But it’s your task, as a software architect/lead programmer/etc to rationalize use of every and each tool at your disposal.
Time is most precious asset in every project – using right tools for the job means better productivity so delivery faster.
Read MoreVoronoi-based Overlay Network (VON) with super-nodes (node merging)
This is my first draft of an idea on how to design a scalable discrete (near real time) simulation using P2P architecture which will allow for scalability with little to no overhead as no server does have to be aware of the scale and all the nodes or users in simulation.
VAST is a VON implementation which brings a novel approach to scaling real time simulations using peer-to-peer approach where each node and server does not know the whole structure of the simulation. The way the nodes are connected ensures that any node can discover any other. The VON provides a self-healing, distributable, horizontally scalable data structure for geo-localized nodes where every node is connected to it’s neighbors and other nodes within its Area of Interest (AOI) and/or Area of Responsibility (AOR).
It becomes quickly obvious that above approach can lead to huge memory consumption when nodes get crowded. Crowding occurs when large amount of nodes gather within each others AOI which leads to situation where the amount of information about connections between nodes grows to N².
I’m not completely convinced by the Dynamic AOI (dAOI) solution proposed by Shun-Yun Hu et al. It does address somewhat the memory issue but the tradeoff is that implementing AOR becomes impossible and in many designs I think AOR is far more important than AOI. Take for example a scenario of a disaster, it’s not so important that a node sees the cause of the event but rather than it is affected by it. Hence the AOI of the node is not as important as AOR of the disaster that affects the node.
A solution to this problem would be to form a new super-node that would act as a proxy actor to the crowded nodes. A super-node should have an upper limit of how many nodes it can support to prevent creation of ‘ultra’ nodes which would make partitioning of the nodes between servers difficult.
Super-node needs to calculate its own super-AOI based on union of AOIs of all of the nodes it proxies for. Then the super nodes connects to the neighbors of the edge nodes it consumes and to nodes within its super-AOI. This way the super-node keeps connection to all nodes it consumed and its neighbors, and the consumed nodes only keep connection to the super-node. The neighbors of newly formed super-node benefit from this as well as from there on they only need to keep connection to the super-node rather to all the nodes their AOI would intersect with.
Nodes can join the super-node when their individual AOI intersects the super-node AOI. Node can leave a super-node when its AOI does not intersect with AOIs of other nodes within same super-node.
AOR can be of course used in above instead of AOI.
Nodes within one super-node can overlap with nodes within other super-node. Their communication will be relayed via their parent super-nodes.
A super-node can have either nodes or super-nodes as neighbors. It could be also made possible to create super-nodes that would consume other super-nodes in overly crowded areas and still have flexible enough structure that could be distributed easily and memory efficiently.
Read MoreGeneralization leads to unnecessary complexity
Every good programer likes the code to be clean and well organized. That leads very often to the temptation of making the code generic. Object Oriented designs are pushing us towards that every day, so it’s also very natural to start making wrappers, factories and interfaces. Use of inheritance encourages code re-use and generalization.
Unfortunately very quickly this can lead to the unnecessary complexity anti-pattern. When you’ll start seeing that navigating through your code takes you on a journey that resembles a maze with many twists and turns – you’ve hit this problem.
Getting there is usually very innocent at first. Making function size small and readable is always a good idea but it can be the first step that will lead you to a spaghetti code. When working on a complex problem and splitting it into smaller digestible chunks it’s important to keep the high level of data flow and processes in mind.
Read MoreThe shortest how-to install MySQL 5.x on Snow Leopard OSX
Every time I am configuring new Mac or doing a clean upgrade to new version I end up with the same batch of problems.
So this time let me do myself and any of you readers a favor and note this one crucial bit that keeps coming back to me on such occasions.
First, make sure you have installed XCode from the Installation DVD.
Next, download MySQL community server 5.x (the x86_64 version if you’re on Snow Leopard onwards).
Now, crucial bit: start Terminal and execute this command:
export PATH=$PATH:/usr/local/mysql-5.1.41-osx10.5-x86_64/bin/
Obviously your version may be different. If you skip this you will end up with an error message “EnvironmentError: mysql_config not found”
Then simply do:
sudo easy_install mysql-python
Et voila!
MikroTik – solution to machine running The Dude that hits 100% CPU
Recently I’ve noticed my Dude RouterOS machine that it hits 100% CPU very often and never goes down below 50% CPU.
Now, this is rather beefy machine with powerful two dual core Xeon CPUs and the RouterOS is set to multi-cpu=yes
I went through the settings on The Dude and what I’ve noticed was the default label refresh interval on the map was 1 second.
There’s at least 7 active sessions at any time, all looking at a map with 300+ network devices, all linked (which also show link usage etc).
Immediately after changing that value from 1 second to 1 minute the CPU usage dropped to 1-2% and only occasionally it jumps to 50%.
So if you see similar thing on your Dude monitoring machine try to set the Settings -> Map -> Label Refresh Interval, to something reasonable.
Member of
