My top 2 articles on the internals of broadcom-based routers

Embedded devices are energy efficient, and their limited memory and storage present some satisfying challenges. How have I missed this for all these years, I don't know, but it is time to play catch up. For a few weeks, I've been spending some of my free time reading on Openwrt and the WRT54GL and other devices of similar design.

Of course, flashing a custom firmware can be daunting enough for an average user, but an average geek like myself will want to know how exactly these devices work. There is a published book on the subject but upon reading the table of contents, I decided against buying it as I deemed it not technical enough.

So, I assumed there was information somewhere for people like me, info that didn't require me to read source code. And there is, but you have to search for it.

And what are the top two articles I found on the subject? Here they are.

1. First of all, there is a three year-old post entitled "Everything you need to know about Broadcom hardware" here in the Openwrt forums: https://forum.openwrt.org/viewtopic.php?id=11304

It is extremely informative on the boot process of these devices and it finally clarified for me how the CFE bootloader works. I didn't understand the SquashFS + JFFS2 combination details, thanks to this post now I do. You also should read about union mounts if you're not familiar with the topic (I wasn't, HP-UX does not support this!).

2. Then you should read about how VLANs and network interfaces work on the Broadcom platform -- this article is from the old OpenWRT wiki but well done, too. It is specifically for the Asus WL-500g but being a Broadcom design I can only suspect other routers are very similar, if not identical:

http://wiki.openwrt.org/oldwiki/openwrtdocs/networkinterfaces

This article explains why you have a bridge interface on your router. It especially shines in explaining how these routers isolate the interfaces cheaply using VLANs. In fact, there is only one twisted pair interface (eth0), the other one being the wireless antenna (eth2). I was under the impression that firewalls needed different and isolated interfaces, but the VLAN trick lets you do something similar on cheaper designs. And I guess it is good enough!

I might replace my m0n0wall PC soon in order to reclaim real estate in my utility room and save a few bucks on electricity. I just need a router, and disable the wireless radio. But I will NOT be purchasing a WRT54GL. Why? I'll tell you in a future post!

O.

Ah, Mom, I bricked my router!

Remember I was running OpenWRT on a Bufallo WLI-TX4-G54HP ? After playing with OpenWrt a bit I spent some time setting up a Ubuntu-based build environment to be able to build my own custom firmware.

There's no particular reason for building a custom firmware since the package I want to ultimately run fits on the JFFS2 partition. I just wanted to get my hands dirty. But gasp... the documentation for OpenWRT is sparse and disseminated in four areas: an unfinished HTML manual, an old Wiki, a new Wiki being slowly migrated from the old one, and the OpenWRT forums. One really has to spend some time reading through all of them to understand how everything works, and I've covered maybe 10% of that. And that's okay. I don't pay a cent for OpenWRT and it's a distribution targeted to power-users... If I wasn't looking for a challenge, I'd be running Tomato instead.

Anyway. It turns out that I bricked my device yesterday after flashing that darn custom firmware. I didn't solder a serial or JTAG port, wasn't really looking into doing this, so I couldn't do much to troubleshoot. One thing that no longer worked, and this was supposed to be my planned way out, was the 2-3 seconds it pings on 192.168.1.1 when it's at the CFE bootloader so that I can TFTP a correct firmware. Since it no longer did this, I couldn't do that to unbrick it.

I was about to throw it in the garbage but found somewhere in the DD-WRT forums that some Buffalo routers listen to 192.168.11.1. I tried this address and it worked! Now as to why it decided to listen to 192.168.11.1 while the first time it was 192.168.1.1, I don't know. I must have pressed the reset button 50 times on this thing so who knows what it ended up doing.

Back to square one, I have a router that works, but no custom firmware.

O.

VMware player... what have I been missing?

Up until now, after all these years, I never considered using VMware Player because as its name implies, it used to be be a player for virtual appliances. That's what it was initially designed to do, but it looks like it can do much more than I thought.

Since I used to be an ESX admin, I don't feel the need to learn to use something else and the only solution I knew to virtualize random stuff for free on my low-end laptop, if I stuck with VMware's offering, was to use VMware Server. I tried Server a few times over the years, but must admit that it isn't really interesting when compared to ESX. The web interface is clunky, and it does not integrate well on a "home" PC.

Little did I know that versions 3 and up of VMware Player now let you create virtual machines at your liking, without needing to use pre-made appliances. I've tried it and it works well, and it is much easier to use than Server's web interface.

I've been "converted".

O.

Looking for my next technical venture

My regular readers know that I used to be an HP-UX admin until a few months ago, and I threw the towel on my sysadmin career.

I'm left with an interesting-enough IT-related job at work, but I can't blog about it easily. The work I'm doing cannot be easily contributed back in a generic way. And the technologies I'm learning are mostly specific industrial stuff and I don't think anyone would be interested in reading about this. Case in point: Is there anybody here, expect perhaps me, interested in IRIG-B ? Yeah, I thought so.

So I'm slowly investigating where I should spend maybe 2 hours a week learning the ins and outs of a new technical thing, and possibly start contributing to the community in whatever means I can, once I feel good enough with it (which might end up taking years). Since I work with embedded devices at work, I've been getting interested in the embedded market and it looks like my next venture might be with OpenWrt but as I don't have many practical uses for it at home, I'm still not sure.

I just found out that a smaller-than-netbook device named běn NanoNote went into production recently, and it runs OpenWrt. What is special with the NanoNote is that both the software and the hardware schematics are completely open designs. At 99$, it's as cheap as it can get. It's almost the same price as a dumb HP digital picture frame I purchased a few days ago, yet I can do whatever I want with it.


Whatever I want with it... I've been spending half an hour staring at the NanoNote specs and pictures, thinking about what I could use it for. Remember that it's very limited in memory and flash, and you can't fatten it up with thousands of bloated applications. In our iPhone/Android era, I simply don't know what people would think of this. But I'll admit it has one advantage against smartphones up its sleeves: it is cheap, and can be mass-produced unencumbered by patents and legal restrictions (at least, I think so).

One can only hope that someone smarter than me will invent a proper use for this device.

O.

What running backfire on a WLI-TX4-G54HP *really* looks like

The 10.03 release of OpenWrt is named "Backfire", which is a cocktail made of the following ingredients according to the login banner:

I happen to have these three drinks at home, so I fixed myself one to celebrate yesterday's experience with this device. Here's what it looks like, with a WLI-TX4-G54HP:

I thought it would have been a three-layer drink, but it gets mixed up quickly in tha glass. I never tried a Backfire before, and it does taste good. Both in the glass, and in the router.

O.

Running OpenWrt on a WLI-TX4-G54HP

I have an on hand a Buffalo WLI-TX4-G54HP. This is a wireless-to-ethernet bridge. What that bridge does is acually the reverse of an access point: it lets you plug any device that doesn't support wireless, such as an old Xbox, and connect it to a wireless network. I actually used it with my locked-down corporate laptop which had its wireless fuction "deactivated for security reasons". :-)

I was thinking of purchasing a WRT-54GL or an Alix board. The WRT54GL, being a hobbyist device, is pricey for what you get (even on eBay) and I was hesitant. Since I had that Buffalo bridge doing nothing, I thought that I might as well hack it with an alternate firmware and see what I can do with it.

The WLI-TX4-G54HP is not specifically documented as runnable with a third party firmware, my take is that not many of these bridges have been sold so nobody reported it. Yet I found some specs hinting me that it was running on a Broadcom 5352 which is the same as the chip used in the WRT54GL. It also has the same amount of RAM and flash, which is a good thing. So sure enough, there were some Buffalo routers based on the 5352 that were officially supported by OpenWrt, but no mention of the WLI-TX4-G54HP. I decided to take a chance and flash it anyway. And it worked:

The only method of flashing OpenWRT on this device is to use the TFTP method. There are no signed firmware available that you can install from the router's webpage. It worked on my first try using tftp.

That's it for now. I'll do some more tests as time permits, and will see how I can submit that device in OpenWrt's compatibility lists.

Why OpenWrt? And why not Tomato or DD-WRT? Because from what I've seen until now, OpenWrt seems to be the most "open" solution available. All source code is available and GNU licensed. Furthermore, it has a lot of command-line interfaces and is targeted to experienced Linux admins.

N.B. If you're running Windows 7, you'll notice there is no longer a tftp or telnet client included as with Windows XP. Look here for a quick fix to include them:

http://www.leateds.com/2009/telnet-for-windows-vista-windows-7/

O.

Thoughts on NTP and a possible homebrew project

After two months without working with UNIX I'm already missing it, and I'm looking for a project to have fun at home and remain technical -- all I do at work is using Word and Excel, and it's starting to make me crazy.

After accumulating them for over 20 years, I recently gave or threw away most of my computer parts (which included two vintage 5 1/4 drives, what a shame). The only PCs remaining are various laptops which are used by my family. I also have a rock solid m0n0wall appliance hidden in my utility room, and I don't want to zap it right away because all my network depends on it.

Those who know me well are aware that I've had a personal fascination for years with NTP. Being a licenced ham operator, building a public stratum-1 server synchronized to CHU using Linux's CHU driver would have been a kickass project to undertake 10 years years ago, with the satisfaction later down the road of contributing to the NTP pool when it became mainstream. However, owning a fixed IP address is costly, not counting bandwidth, and limiting the usefulness of a time server to my own internal network wouldn't give me much. And what's the purpose of using CHU or WWVB when you can sync using a GPS anyway; the only situation I can think of is if you can't have a clear path to a satellite, or need a cheap solution to extract the time from a reliable source. That's the premise those nice radio clocks that set their time automatically are built on.

My second fascination is with embedded devices. They don't consume much power, they're small and fun to work with. The cheapest way to own one to play with would logically be to buy a WRT54GL and flash it with a third party power-user firmware such as OpenWRT. However, the WRT54GL is based on old technology (2002), and thus fairly expensive for what you get. To add basics such as a serial port, you need to crack open the case and solder wires. Fixed storage is limited to 4Mb, that's not a lot of space in 2010 numbers. Bottom line, using the WRT54GL for a homebrew project can get expensive and cumbersome quickly, and paying big bucks for an underpowered device doesn't excite me much. Thus, I'm almost ready to purchase an ALIX 2d13. It's around double the price of a WRT54GL when you count shipping, a case and a CF card. But but it packs a LOT more power and expandability and this embedded device should be able to offer enough power to last 10 years.

One thing I was thinking about is to combine both by adding a soundcard (miniPCI or USB) to the ALIX board, plugging in a shortwave radio, and building a homemade CHU-compatible NTP time server. And why not try WWVB as a "part two". It could have been used in areas where satellite is not accessible. But I quickly found out I wouldn't have been the first one to think of this -- Meinberg already has one. Their only drawback is that it's fixed to one station (they're German, so of course they offer one synced to DCF77). But once your choice is made, you can't change it.

So back to square one. I won't invest tons of money to recreate what has already been done. And this is where I'm at, as of today, with my research on embedded x86 devices.

I think I'll end up building a generic server on the ALIX. Some tasks such as PXE installs don't seem to be well documented on this platform, and I think that testing and documenting a PXE environment could be of benefit to whoever has a bunch of these to flash. We'll see.

Take care

O.