I have 4GB of RAM, and a nVidia 8600M with 256MB of RAM.
However, the system allocated 800MB for the video, leaving only 3.2GB
for the system.
No it didn't. Here's the situation:
All 32-bit client versions of Windows (not just Vista/XP) have a 4GB
address space. That's the theoretical upper limit beyond which you can
not go.
But you can't use the entire 4GB of address space. Even though you
have a 4GB address space, you can only use *around* 3.1GB of RAM.
That's because some of that space is used by hardware and is not
available to the operating system and applications. The amount you can
use varies, depending on what hardware you have installed, but can
range from as little as 2GB to as much as 3.5GB. It's usually around
3.1GB.
Note that the hardware is using the address *space*, not the actual
RAM itself. The rest of the RAM goes unused because there is no
address space to map it too.
May I know how can I reduce the shared memory?
There's nothing to reduce. The only way to use more memory is to use
64-bit Vista.
Why do you need so much memory? Unless you use particularly
memory-hungry programs, it's unlikely that you would see any
performance improvement by going above 3.2GB.