Category Archives: What?!

StarTech PEX10000SFP and locating modules in the Linux source.

A friend contacted me recently with issues getting a new StarTech PCIe card with SFP+ slot working. He had hoped the card would work out of the box… but sometimes that doesn’t happen.

PEX10000SFP
Our test subject: The PEX10000SFP.

First off, let’s have a look at the PCI bus and see what the card has for a device ID number:

edge:~# lspci -k

01:06.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02)
Subsystem: Super Micro Computer Inc Device 1711
Kernel driver in use: radeon
03:00.0 Ethernet controller: Tehuti Networks Ltd. Device 4024
Subsystem: Tehuti Networks Ltd. Device 3015
edge:~#

As you can see, the VGA controller has a kernel module loaded and associated with it (radeon), however the Startech (Tehuti Networks) controller does not. With the device ID number in hand (0x4024), we can now look for it in the kernel source. If you don’t already have a copy of the Linux source, make sure to grab one via git:

edge:~# mkdir ~/git

edge:~# cd ~/git

edge: git# git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux

edge: git# cd ~/git/linux

edge:~# grep 0x4024 include/linux/pci_ids.h

edge:~#

Hmm…not a single hit. Let’s search for anything Tehuti related:

edge:~# grep DEVICE_ID_TEHUTI include/linux/pci_ids.h
#define PCI_DEVICE_ID_TEHUTI_3009 0x3009
#define PCI_DEVICE_ID_TEHUTI_3010 0x3010
#define PCI_DEVICE_ID_TEHUTI_3014 0x3014

So there’s device ID’s 0x3009, 0x3010, and 0x3014… but no 0x4024. So it doesn’t appear to be present in the source tree. But a quick search on the vendor website and the drivers are readily available for download – great news, but the running kernel (3.16.0-4-amd64) isn’t supported:

(From the Tehuti_TN4010.zip Readme file)

“- Supported kernels: 2.6.24 – 3.14.x”

edge:~# uname -r

3.16.0-4-amd64

And when trying to compile it, it fails:

/var/tmp/Linux/tn40.c: In function ‘bdx_ethtool_ops’:
/var/tmp/Linux/tn40.c:4021:5: error: implicit declaration of function ‘SET_ETHTOOL_OPS’ [-Werror=implicit-function-declaration]
SET_ETHTOOL_OPS(netdev, &bdx_ethtool_ops);
^
cc1: some warnings being treated as errors
/usr/src/linux-headers-3.16.0-4-common/scripts/Makefile.build:262: recipe for target ‘/var/tmp/Linux/tn40.o’ failed

So, let’s dig around and see if we can find the SET_ETHTOOL_OPS macro in the changelogs:

edge: git# cd ~/git/linux

edge:git# git log -S “#define SET_ETHTOOL_OPS”

commit 7ad24ea4bf620a32631d7b3069c3e30c078b0c3e
Author: Wilfried Klaebe <w-lkml@lebenslange-mailadresse.de>
Date: Sun May 11 00:12:32 2014 +0000

net: get rid of SET_ETHTOOL_OPS

net: get rid of SET_ETHTOOL_OPS

Dave Miller mentioned he’d like to see SET_ETHTOOL_OPS gone.
This does that.

Mostly done via coccinelle script:
@@
struct ethtool_ops *ops;
struct net_device *dev;
@@
– SET_ETHTOOL_OPS(dev, ops);
+ dev->ethtool_ops = ops;

Compile tested only, but I’d seriously wonder if this broke anything.

Suggested-by: Dave Miller <davem@davemloft.net>
Signed-off-by: Wilfried Klaebe <w-lkml@lebenslange-mailadresse.de>
Acked-by: Felipe Balbi <balbi@ti.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

Well, there’s the reason it won’t compile – the macro was recently removed. So how do we get the module to compile? Simple – just update the source to perform the same action that the macro used to do. Or to make things easy (although it’s overkill for a patch file), just apply a truly braindead patch:

edge: tmp# wget http://www.braindeadprojects.com/src/tn40.c.ethtool_ops.patch

edge: tmp# patch -p0 < tn40.c.ethtool_ops.patch

patching file Linux/tn40.c

And with a quick recompile, install and modprobe, we now have a working Startech card in our system:

edge: tmp# modprobe tn40xx

edge: tmp# lspci -k

03:00.0 Ethernet controller: Tehuti Networks Ltd. Device 4024
Subsystem: Tehuti Networks Ltd. Device 3015
Kernel driver in use: tn40xx

edge: tmp## ethtool eth2
Settings for eth2:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Advertised link modes: 10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Speed: Unknown!
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: external
Auto-negotiation: off
Link detected: no

I’ve passed along the information to Startech. It’s a pretty simple fix, so I’d expect to see it in their distributed source code soon. But in the meantime, if you’re working with this card and unable to get the kernel module to build, see if this solution will work for you.

The Sunset on Lambs Gap

A couple of months ago, I started to notice a few issues when watching YouTube videos at my home.  It seemed that the videos were taking longer to buffer, and a quick check of my Ubiquiti NanoBridge M5 (NMB5-25) revealed my signal had deteriorated from it’s usual -64dBm signal to around -75dBm. The Airmax Quality had also significantly tanked, now reading around 40%. An Airview scan didn’t seem to indicate any new interference:

Airview Before
The Access Point is in use.

Throughput tests to two on-network Speedtest.net Mini sites still showed decent rates, but it became obvious that TCP retransmissions were starting to significantly impact surfing the web. Unfortunately, I didn’t think to grab a tcpdump to support that theory.

Before Speedtest
Decent Throughput it would seem

I had been holding off installing the new NanoBeam M5 (NBE-M5-400) since I wanted to ensure they were stable. While I’ve not been burned by the problems with ToughCable or chain burnout problems on the Rocket Titanium M5’s, I’ve heard my fair share of grumbling from people that have. On the Ubiquiti forum people have mostly raved about the new units so I figured I’d see how well it performs.

On the roof of my home, I have a non-penetrating mount atop two anti-fatigue mats purchased from the Home Depot. The mats aren’t UV rated, something I didn’t consider until later. After a yearlong installation, the mats are in surprisingly good shape. I don’t anticipate having to replace them in the next two years based on their current wear.

The Mount.
Carrying 3 of these up a ladder makes you sleepy… that’s why there’s no fourth.

The NanoBridge M5 dish itself was still tightly connected to it’s mast, and the 3 concrete ballasts holding the mount in place seem to prevent any possible shifting.  Initially when my signal deteriorated, I thought a bird may have slammed into the dish or mount and somehow moved it, but I don’t believe this is the case. The more likely culprit is a tree.

Tree Clipping
A view from behind the new NanoBeam — I’m fairly certain I’m clipping the tree on the right.
Hey What's That Picture
Courtesy of heywhatsthat.com – the line to the tower.

I knew when I installed the link that it was very likely clipping the tree on the right. Our tower is slightly to the right of the valley.  Not particularly liking heights and trying to limit visibility from the street, I opted not to move the mount closer to the front of my home.  I may revisit that decision in the future. Before I removed the NanoBridge I did try to correct it’s position but was only able to receive around an average of -70dBm at best.

The NanoBridge M5
This is about where my signal settled.

Losing sunlight, I upgraded the dish to the NanoBeam M5. The only thing about the dish I dislike is that it has one U-bolt instead of two. I’m sure it’s ample but the NanoBridge’s 2 U-bolt configuration really seemed to hold it to the pole so that it couldn’t possibly be knocked out of alignment. My roof is mostly flat with a slight angle, so I wasn’t able to fully straighten the dish.

The Level
Do Not Point the Laser into Someone’s Eyes.

Installing the NanoBeam was quick and easy. The first thing that was obvious was the noise floor had gone from -93dBm to -103dBm. The signal was significantly improved, now running in the -63dBm range, and overall stats were a remarkable improvement over the NanoBridge.

NanoBeam Stats
A bit of a difference

 

Post Upgrade Speedtests
Zrrrrrrrrrroooooomm!

The week before I performed the upgrade, I finally added the Ubiquiti radio to my home Cacti installation. Forgive the ugly colors of the graphs (I plan of making them more visually appealing one day), but as Adam Savage says “The only difference between screwing around and science is writing it down“.  These graphs display a pretty clean-cut before and after of the upgrade, which took place at about 20:00.

Frequency in Use
I didn’t change frequencies, but here’s a graph.
Mo Memory
The NanoBeam has more memory than the NanoBridge.
CCQ
Client Connection Quality (not much of a change here)
Signal Graph
The signal is substantially improved.
Data Rates
My data rates improved.

I incorrectly saved the Airmax graph after the upgrade and only noticed the next day (that’s why this graph is hours after the others).

Airmax
Much better Airmax Quality.

I’m extremely pleased with the upgrade. My initial problems with YouTube buffering have gone away and three days later things are still going strong. In the end I had a chance to see a pretty cool sunset on Lambs Gap.

SunsetOnLambsGap
The Sunset on Lambs Gap

Wireless Iperf Graphing

About a year ago, we were doing some wireless throughput/interference testing at the office  to acclamate ourselves to the wireless world. Most importantly, we were trying to answer the question “Why can’t I just lease a 10 foot section on a tower and  install a zillion radios there?”

Despite having a handful of BCP design documents at hand, as well as input from established and respected members of the wireless community, the question kept seeming to pop up. So how does one present a worthwhile answer?

My solution was to build a small section of a tower leg on our back lot using a non-penetrating roof mount, 2 Ubiquiti Rocket M5 90 5G-20-90 degree sectors (mounted 90 degrees from each other, without shielding), 2 Nanostion M5 CPE’s, 4 laptops, and a couple of inverters and car batteries to power 2 of the laptops (the other 2 were run off extension cables).  The two laptops connected at the Access Point side ran as Iperf servers, each laptop ran as a client.

UbiquiperfLayout
Plug this in here, that in over there, and this cable.. here.

The thought was to grab Iperf and AirOS datapoints at every couple of seconds  and see “what happens when I do this?” “THIS ” generally entailed doing something BCPs warned against doing.

To gather the Iperf data and AirOS statistics, I put together a quick script i called “Ubiquiperf“. Basically it polls an iperf test every second, pulls the stats off the radio (Capacity, Quality, RSSI, etc) , generates a CSV file of all the data, and finally creates a graph using the jpgraph PHP libraries.

Due to weather conditions our testing was cut short and our initial results had a simple problem: we needed to turn the transmit power down on all the radios.

For a baseline, we performed a quick throughput test one at a time, where each Access Point operating on the same frequency could perform at about 85Mbps:

ap1-test1
AP1 Baseline
ap2-test1
AP2 Baseline

So what happens if I perform the same throughput test on both units but at the exact same time, over a longer duration?

ap1-test2
AP1 concurrent iperf test
ap2-test2
AP2 concurrent iperf

Well, it obviously cut off about 20Mbps from the first access point. Of course, the power on the radios really needs to be turned down so this test is unfortunately not worth much. We’ll be doing additional testing in the near future, and hope to have meaningful results that include mounts with and without RF Armor, Titanium model equipment, etc. I’ll publish our results when I can.

In the meantime, feel free to play with it yourself. Full PHP source is available on GitHub.

Cisco Switch and Route Completed!

I recently took my CCNP Switch Exam (642-813), passing on my first attempt. While I mostly used GNS3/dynamips to simulate routers for my CCNP Route Exam (642-902), I actually put together physical hardware for the Switch Exam. My switch lab consists of a Cyclades ACS48 terminal server, an APC 7900 switched PDU, 3 Cisco 3550-EMI multilayer switches, a 24 port patch panel, a Cisco SPA942 IP phone, and 2 ZNYX PCI 4-port ethernet cards for my desktop (for a combined 8 NICs dedicated to virtualization).

SwitchLab0
8 strands of cat5e can fit in a 3/4″ conduit, but it’s a tight fit

Initially I had thought there were no emulators for Cisco switches, but upon news that GNS3 would be supporting switching in their upcoming 1.0 release, I started digging. Apparently it’s a well established fact that Cisco switching can indeed be emulated and there are leaked binaries compiled for Linux for a handful of layer 2 IOS images. Hewlett Packard also has a 64-bit VM emulator for their VSR1000 Virtual Services Router Series.

 

GNS3 CrowdHoster
I think it’s safe to say GNS3’s fundraising will be successful when it ends.

 

While IOU is nice to work with, there are a number of bugs in the images. First of all, while you can configure private VLAN’s, they don’t appear to work (and they’re not supported on my physical 3550’s). Testing dynamic arp inspection in the following images also doesn’t seem to work:

  • i86bi_linuxl2-upk9-ms-15.0.bin
  • i86bi_linuxl2-ipbasek9-ms-15.1B.bin
  • i86bi_linuxl2-ipbasek9-ms-15.1A.bin
  • i86bi_linuxl2-upk9-ms-12.2.bin

The utility iou2net (which is used to connect the unix socket to physical hardware) is sometimes wonky,  sending unidirectional traffic much of the time.  So for my exam, I stuck mostly with my physical hardware for switching, emulated routers via dynamips (in GNS3), and hosts spawned by VirtualBox Manager (to make moving them around to different switchports easier).

VirtualBoxNICs
Lots ‘o NICs and Bridge Interfaces

I’m connecting my emulated routers to my physical switches and virtual machines using GNS3’s Generic NIO ethernet interface. Of course, my physical setup isn’t without it’s own issues – the tulip drivers (or my motherboard) appear to have occassional complications, flooding the kernel ring buffer with annoying “slowpath” messages and also preventing traffic. Kernel upgrades haven’t resolved the problem (there were some fixes between kernel 3.9.11 and 3.10.1, but they don’t appear to resolve my issues). Thankfully bouncing the problematic ports will fix the issue.

[2842987.323263] Call Trace:
[2842987.323268]  [<c0455341>] dump_stack+0x16/0x18
[2842987.323272]  [<c012796c>] warn_slowpath_common+0x48/0x5f
[2842987.323275]  [<c013284b>] ? del_timer_sync+0x2b/0x3d
[2842987.323277]  [<c0127992>] warn_slowpath_null+0xf/0x13
[2842987.323280]  [<c013284b>] del_timer_sync+0x2b/0x3d
[2842987.323284]  [<f8606ace>] t21142_lnk_change+0x331/0x507 [tulip]
[2842987.323294]  [<f86021e4>] tulip_interrupt+0x5d0/0x784 [tulip]
[2842987.323298]  [<c015c538>] ? clockevents_program_event+0xe5/0x103
[2842987.323302]  [<c0176516>] handle_irq_event_percpu+0x4d/0x158
[2842987.323305]  [<c0176647>] handle_irq_event+0x26/0x3f
[2842987.323307]  [<c01785fa>] handle_fasteoi_irq+0x63/0x8b
[2842987.323310]  [<c0102862>] handle_irq+0x67/0x71
[2842987.323313]  [<c01024ce>] do_IRQ+0x35/0x8e
[2842987.323317]  [<c045deec>] common_interrupt+0x2c/0x31
[2842987.323319] —[ end trace 80bd835791c79500 ]—

 

For studying, I worked off Chris Bryant’s really solid Bryant Advantage lectures, the official certification guide, and labwork, labwork, labwork. To play with some of the layer 2 and 3 protocols outside of IOS, I put together a small TinyCore LiveCD [gpg] that includes yersinia, hping2, scapy, and tcpdump. (I’ve packaged yersinia separately if anyone would like it here).

TinyCore Yersinia
Versions Prior to 0.7.3 can be problematic and hang in VM environments

I find it’s extremely helpful to see traffic in action and having 8 NICs helps to facilitate that. While you can easily sniff HSRP, VRRP, and GLBP traffic inside GNS3, seeing VTP and DTP frames can be accomplished by creating bridge interfaces on my desktop OS and using the patch-panel to place myself in the middle of switch-to-switch connections.

The exam itself was pretty typical. I find the simulations to be fun although I’m surprised at how bad the Cisco interpreter is. At one point I had placed a VACL on the wrong piece of equipment, went to remove it and place it on the correct switch but couldn’t remove it on the original switch. Trying every variation of “no” proved fruitless, and in the end I’m sure I lost points due to that mistake. What’s annoying is that I tried to correct it. Thankfully, I’m not alone and a LOT of people complain about the interpreter.

If you’re studying for the exam, here are my pieces of advice:

  • You’re paying to take the exam, so get the best grade possible. That means study, study, study.
  • Find a good practice exam engine (like Boson’s ExSim) and take a bunch of practice exams. You’ll find out what you don’t know, then lab, lab, lab.
  • Take some time off work to study if you can. I dedicated 12 hours a day 5 days (over a long weekend) before the exam to studying. Was it excessive, definitely, but there’s no harm in being over prepared (you’re paying to take the exam, work your hardest).
  • If you truly know the material, you know the wrong answers sometimes more clearly than you know the right ones. Always remove the wrong answers from your available options while taking the exam. “No, GLBP uses UDP port 3222 so that answer isn’t for HSRP”
  • Even when you’re absolutely certain of the right answer, read all of the answers before going to the next question. On multiple choice questions, I typically say in my head “No that’s not it, that’s it, that’s not it, that’s not it. Second answer is correct”.
  • Double check your work on the simulations before going to the next question. Make sure you do everything and completely, and test your configurations.
  • Lab work means breaking things. Don’t just configure a working protocol, purposely misconfigure the protocol. Make sure you see the nuances that no book will dedicate the column inches to detailing.
  • Look over the exam topics. I didn’t realize they existed until after I took my CCNA and the CCNP Route and Switch exams, and it really led to a lot of unnecessary anxiety (“What’s on this exam, do I know everything???? HELP!”)

So now, I’m in the process of studying for my final CCNP exam – the TSHOOT. Wish me luck!