I'm guessing a bit, but ... I don't think this will work.
It sounds like you have a PC with a Winmodem for dial-up connection to the
Internet, right? You can use the modem okay from the Windows Vista host,
and you'd also like to use the same modem from the Linux guest, running in
VirtualBox, to connect to the Internet - right?
The guest OS in VirtualBox cannot see all the hardware which is installed
in the host machine. It can only see the hardware which is emulated by
VirtualBox's virtual machine. I'm fairly sure the VM will have no
awareness of the Winmodem hardware on the host machine; it can only "see"
the hardware in the VM.
If you dual-boot between Windows and Linux, the Linmodem software (which
scanModem.gz is used to configure) can "see" the Winmodem hardware,
because it is running on the physical host, not in a VM.
Probably what you need to do is to create a network connection between the
Linux VM and the Windows host. This would be a normal TCP/IP network
connection over the guest's emulated network card. Then, connect to the
Internet using the dial-up Winmodem from Windows. As long as you have your
networking and routing configured correctly, the Linux guest should be
able to get an Internet connection via the host's dialup connection.
You're probably more likely to get good advice by asking in the VirtualBox
forums:
http://forums.virtualbox.org
More folks there will have a detailed knowledge of the VirtualBox
environment, than you'd find in this newsgroup. Also try the Linmodem
discussion list: see
http://linmodems.org
Hope this helps a bit,