Category Archives: Linux

Evaluating the world of WAN link-load-balancing (SD-WAN)

It is probably obvious from the postings I’ve made here at BraindeadProjects that my home is nothing more than a giant networking lab. When I wanted to learn how GPON worked, I prepped my “lab” by building a 12 strand fiber-optic ring through the walls of my home and connecting the five Cisco switches throughout the house together using bi-directional SFPs

12 strands of fiber-optic and some kevlar blonde hair

When I needed better wireless coverage, I built out a Ubiquiti Unifi wireless network and later rewired most of the light switches in my home with Wifi-enabled TP-Link switches so that I could voice control the home using Amazon Alexa Echo’s.

The Ubiquiti UniFi Controller

Wanting to centralize my firewall policies, long ago I routed each of the 12 production VLANs at home run through a Fortigate 60C High Availability cluster.

Buy what you need, not necessary what’s new.

The home has 4 Internet connections with 2 diverse paths: The 3rd floor terminates two 5Ghz microwave PtMP links from a Wireless ISP that I used to work for. The basement terminates a Verizon 5Mbps/760Kbps DSL line, and a Comcast 100Mbps cable link.

Install large 3 foot dish while wife is busy, ask forgiveness later.

So how do I maintain connectivity to the Internet if a connection goes down or if I lose power on a floor of my home? Previously I had a simple VRRP setup: Whichever connection was performing best I would manually set to be the VRRP master and fail over if connectivity went down. If I wanted to specify that email should operate over the microwave backhauls, I would create another VRRP group (so that I could have redundancy), policy-based route email traffic to that group, and setup an IP SLA to test the connection. This was a bit of an administrative nightmare, so I did so sparingly.

Ubiquiti UNMS – a dashboard to view all of your Edge Routers

Then the world became abuzz with “Software Defined Wide Area Networking“. To qualify as “SD-WAN” Gartner has four required characteristics: The ability to support multiple connection types (MPLS, LTE, Internet, etc), support for dynamic path selection, load sharing over the links, and simplified provisioning (Zero Touch Provisioning).

I’ve had the opportunity to evaluate a small handful of “SD-WAN” solutions, each with their own pros and cons: Some are surprisingly lacking in features (despite large sales footprints), some are full of features but have lackluster provisioning, and some are insanely expensive (at least for home use).

Initially I had settled on adding a different vendor’s SD-WAN appliance into the home network and purchased 3 of their devices. After waiting for the shipment for over a month, I received a full refund from the seller with little explanation. I seriously lucked out.

Long wait, no explanation. Oh well…

While waiting for my boxes to arrive, I had the chance to borrow and test the platform and found some limitations – namely only support for 2 WAN connections and no active-active support (so I couldn’t use my other 2 WAN connections) . Then I took a closer look at the Fortigate’s I already had in my network.

Fortigate supports re-configuring each of their 10 ethernet connections for various use. This allowed me to take ports that are typically used for LAN connections and re-purpose them into WAN connections. This is a major plus. The downside was my exisiting Fortigate 60C’s don’t support the lastest FortiOS (6.0) code.

One of the 3 racks of equipment at home.

For the price of the other vendor’s limited platform (x3), I could purchase 2 used Fortigate 60D’s off Ebay – plus purchase rack-mount trays for each unit. No more Fortigate sitting atop another device in the network racks. Since I don’t need the advanced features the platform provides (anti-virus, IPS/IDS, etc), the second-hand solution is perfect for my needs (Firewall policies, SD-WAN, VPNs).

So here’s how Fortinet does things:

Configure an IP on each of the WAN connections you intend to use. In my instance, VLAN 66 is my “Internet DMZ” where each of the 4 Ubiquiti EdgeRouter X SFPs bring the Internet connections into my network.

To allow the Fortigate to have multiple WAN interfaces in the same subnet, you have to override the system default preventing that:

flamethrowerX # show system settings
config system settings
set inspection-mode flow
set allow-subnet-overlap enable
set gui-fortiextender-controller enable
end

When creating the WAN interfaces, you’ll need to manually specify the bandwidth of each link. This is one unfortunate downside to the Fortigate solution – it cannot measure available bandwidth dynamically.

When selecting the members of the “SD-WAN” interface, you may find that you’re unable to include certain interfaces. The most likely cause of this is a firewall policy referencing that interface. If you don’t follow the cookbook, you’ll likely run into this frustrating problem, so RTFM.

Oh… so that’s why I couldn’t do that… Hmm…

When you aggregate interfaces into the SD-WAN interface, you’ll need to specify the gateway of each WAN link and the default load-balancing mechanism. In my instance I’m using “Volume-based” balancing.

Defining the pie.


Under the SD-WAN rules section you can further specify how you want the volume dispersed.

Slicing up the pie.

After creating the base settings you can have the real fun. The PBR rules that used to take additional thought and design are now the matter of a point and click solution. Making email route over the 5Ghz links by default is the simple matter of creating an SD-WAN rule. Video streaming services such as NetFlix and Hulu can simply be prioritized to run over the higher bandwidth cable connection – and failover to the other options when needed.

This is WAYYYY easier than the old way of doing things.

The SD-WAN SLA’s are somewhat simplistic. You have the option to either ping or pull a web request from a designated server. Neither solution detects MTU issues in a path. If I were to disable TCP MSS clamping on my DSL line the system continues to use it despite a user being unable to download content from websites correctly.

The SD-WAN SLA’s. Pingy, pingy, pingy, pingy, pingy.

One of my favorite features in the web interface is the ability to look at the logical topology and see which users in each VLAN are consuming what amount of traffic.

Lots of penguins heading to the cloud.

You can also drill into the flows determining which flow is using which WAN link.

You go this way, you go that way, you go this way, you go that way.

So, what do I not like about the solution? I’m able to rename an interface, but on some screens the GUI displays the interface name and NOT the alias. This requires additional thought “Oh, interface7 is the DSL”.

I also wish I had the ability in each flow to see which SD-WAN rule was hit. This is important since it can help you verify that things like Email are classified correctly (I found that IMAP wasn’t considered part of the “All Email” out-of-the box classification in the non-Fortinet solution I initially purchased).

I’m still working to perfect the HA failover on the system. The general idea is that if the one Fortigate can’t ping the VRRP addresses I had setup on the WAN routers or LAN switches the backup unit should take over. “Remote Link Monitoring” took me some time to get working on the former Fortigate 60C’s, so I’m not discouraged yet.

High Availability: When you need a backup flamethrower.

Overall you can certainly see the power of what Fortinet’s re-branded “WAN Link Load-balancing” has to offer. The ability to leverage redundant Internet links in such a simple manner places some serious power in the hands of companies with limited IT resources – and I’m only scratching the surface of the capabilities.

If you’re looking to test your own WAN load balancing, I’ve put together a webpage that will display your IP address, as seen from 5 different IP lookup sites on the Internet. Feel free to use it for testing. You can find it here.

Using ALSA plugins to stop scaring the cats.

The XMBCbuntu ZBOX server has really changed things in my house. It’s become hard to justify the $50+ cable bill seeing as how I only watch one television show when it originally airs (Breaking Bad).  I’m not invested in a Triple Play package – I get my Internet access from a 5Ghz Ubiquiti link to my office (in addition to an aDSL backup line), and for phone service I have an Asterisk server as a PBX with a SIP trunk to Teliax. Quite simply, cable doesn’t do much for me.

I’ve also cut down on shelves — my video and music collection is being converted to digital and packed away in the basement. Most books I purchase on my Nook. If I can’t buy a video or album new as  a DRM free digital download, I simply don’t.

The only issue so far with the XBMCbuntu set-top box has been audio – the Radeon HD 6250/6310 sound card seems to have no hardware feature to increase or decrease audio — only mute it. That means I turn the TV up really loud to watch an episode of The Wire, and when my wife switches back to the Nintendo Wii the sound is startling.

SilentLoudWire
A whisper then a YELL!

Thankfully ALSA is more than just the ALSA mixer, and has an entire plugin architecture behind it.

alsalogo

Using an .asoundrc file, I’m able to connect my HDMI audio output to a softvol plugin and increase audio that way. No more cats running from the upstairs loft scared half to death.

pcm.hdmi_formatted {
type plug
slave.pcm “dmix:0,3” #card 0, device 3
}

pcm.hdmi_complete {
type softvol
slave.pcm hdmi_formatted
control.name hdmi_volume
control.card 0
max_dB 50.0
}

pcm.!default pcm.hdmi_complete

 

Quite simply, I’m binding the sound card with a softvol plugin, setting it’s maximum gain to 50dB. Set the volume using alsamixer, and store your settings using “alsactl store” so they persist across reboots.

Voilà -- a way to boost audio.
Voilà — a way to boost audio.

 

While this is the first time I’ve utilized ALSA this way, I’ve long found that while watching movies on my netbook during long flights, mplayer’s softvol feature is extremely useful. You might find this trick handy when trying to hear a film over a loud jet engine:

mplayer -softvol-max 100 RobotAndFrank.avi

 

Robot and Frank
A funny and bittersweet movie I watched on a recent flight.

 

 

A New MediaCenter and how HDMI to RCA does and doesn’t work.

I’ve always referred to Ubuntu as “the bridge that let the Windows idiots into the Linux world.” Ubuntu has done a wonderful job with their distro of Linux, but for some reason it seems to be the place the newbies spring from. I’m guessing it’s because of their reputation for being easy to use.

Ughhhhbuntu
Ughhhhbuntu

“It all just works” isn’t something that I find to be the case too often with my preferred distro, Gentoo.  I tend to forget to rebuild dependencies after updating software. You often have to sort through package blockages that prevent you from upgrading one small piece of the puzzle, and then on occasion packages simply won’t compile.

All of this leads to a strength-training of sorts. Actually, even installing Gentoo teaches a new user some important things since it’s all done by hand. There’s no clicking “Next”, “Next”, “Next”   — Voilà, installed. Instead, you’re partitioning and formatting by hand, configuring LVM, mounting partitions,  learning how to chroot into a directory, rsyncing the portage tree to your machine, configuring and building a kernel (and usually missing a module that you need), etc. It’s a time-consuming process for a new user.

CompilerSpew
But you’re not smart just because you’re watching compiler spew.

Then something happened that’s softened my dislike for Ubuntu.

A few months ago my XBOX Classic media center (XMBC)  died. I’m pretty sure the problem was only with the power supply, but I was getting frustrated with how slow it would decode some videos. I was also tired of my latest storage setup (a USB attached 1TB drive served over the network via SMBFS), so I took the opportunity to upgrade things.

XBox Classic Power Supply
XBOX with dead power supply.

The first item to upgrade was my file storage. My workplace had a few Zyxel NSA-320′s laying around, but I wanted something that would give me JBOD and eSATA capabilities.  One of my fears is being locked into a particular vendor’s RAID controller, having it fail and being unable to access my data until I either buy ANOTHER exact model enclosure (or have to hunt for something with a similar controller). For that reason, I opted to go with software RAID – thus requiring JBOD support.

Zyxel NAS web interface
Zyxel NSA-320: It’s not a bad product at all, just not what I needed.

I finally settled on a Sans Digital TR4UTBPN, a 4 drive RAID enclosure with USB3.0 and eSATA support. I added a pair of 2-Terrabyte drives in a software RAID-1 configuration. Initially, I connected to the array via USB but found a few problems with that (mostly USB disconnects causing the software array to kick out a drive).  Thankfully BestBuy had a Dynex – 2-Port eSATA II PCI Express Adapter on the clearance rack. This card supports the “Port Multiplier” feature allowing me to access both drives in the enclosure individually, all over a single cable.

TR4UTBPN
It’s like a digital storage locker

I contemplated purchasing an $80 Roku box for streaming to my television. My only stipulation was that I wanted to be able to stream from my new storage array, and while the Roku does support this via HTTP ranges, the list of supported codecs left a bit to be desired. Not being interested in having to transcode a movie just to watch it, I opted to build another XMBC system. (While XBMC cannot stream Netflix, I have a Wii and DVD player that both support that functionality)

Initially I was going to purchase an Acer Revo for the system, but after a bit of discussion on IRC, a friend recommended that I look at something with an  E350/450/E2-1800 processor instead. After a bit of searching, I finally settled on a Zotac ZBOX nano AD10.

From Xbox to ZBox
From Xbox to ZBox

The video output options on the ZBOX are DisplayPort and HDMI. My television is an older model Sony Wega with no HDMI input (if it’s not broke, why replace it?). A few friends encouraged me to buy an HDMI to RCA cable, claiming it works fine for them. When my cable finally arrived, it didn’t work for me.

A co-worker was fighting through a similar predicament, having purchased a Stellar Labs 33-11610 for the same purpose. In his instance, a newer DVD player with HDMI-only output wouldn’t work with his RCA-only television. Searching through numerous HDMI to RCA  converters online I saw the same thing: Numerous people claiming a certain cable didn’t work for them, numerous others stating the exact same cable worked perfectly for them.

HDMI-to-RCA
This cable does NO frame resolution conversion.

Here’s why: A typical older model NTSC television only supports 480i. The conversion cables DO NOT downsample to 480i. In my case the ZBOX is outputting a 1080p-only signal, while my TV can’t handle that rate. Some TV’s can adapt, mine can’t. What you need is an HDMI to RCA downconverter.

The problem is that the downconverters aren’t cheap – typically they run about $200.00 on Ebay. I have located a Stellar Labs model VC11620 for $87, and lucked out and found a knock-off model for $30. Later I found that NewEgg also sells these inexpensive downconverters.

 

HDMI2AV
This image taken with a potato.

 

With all the parts in place and ready to go, it came time to install XBMC. When I noticed an XBMC based distro called “XMBCbuntu” I knew it would feature one major plus – it would just work, no need to fight with configuring the remote control, no fidgeting with network configurations, and no need to spend hours compiling or installing binary packages and building the media center by hand.

In fact, in 20 minutes I had XBMC installed to the ZBOX hard-drive and I love it. I’m not going to back away from my belief that Ubuntu is the “bridge that let the Windows idiots into the Linux world”, but I will say that Ubuntu has a tendency to make things easy for everyone. That’s a major plus.

Streaming Freaks and Geeks from my new storage array.
Streaming Freaks and Geeks from my new storage array.