Making (fax)modems work under VMware ESX

Yesterday I was asked if it was possible to make an external faxmodem work under a Windows Server 2003 VM. The customer has an application that is linked against the Microsoft Fax Service DLL, and they have to use a honest-to-goodness modem. The answer was: probably. I checked and indeed, serial ports can virtualized in a VM.

That's nice, but...

• The modem being physically connected to a host, you can't put the VM in a cluster (unless maybe you have identical modems hooked up on each of your hosts, but I didn't try it)
  
  
• We have blade servers, and to use their serial ports we need to hook up a three-way cable in front of the blade, which has a USB, serial and video connectors. This is for diagnostic purposes only. Hooking a modem on this will work, but that's not really pretty to look at.
  

The solution I found was this: we already have Digi PortServers in production, which we use to access consoles on Alpha systems, and service a few modems as well. By installing RealPort inside a Windows VM, one can redirect a COM port, through the lan, to a port on the PortServer. Windows thinks it's speaking to a physically attached serial device, while in fact it's hooked up on a PortServer.

Tada! The modem now works inside a VM. And it will also work if I VMotion it to another host. Case closed.

RealPort exists on Linux and Solaris as well (no mention if it's x86 or SPARC, though...) so these VMs can also use a similar solution as well.

Sidenote: You can even assign a virtualized serial port on a named pipe on the ESX host. I initally thought of writing a perl script that would have opened the pipe, and use Net:Telnet to telnet to the PortServer. That was before I found out that RealPort existed.