The debate occurred again last night, with a Cinematics group, about how best to configure their linux render nodes. We're running a bare metal hypervisor, with two guests running RHEL 5, on a pair of xeon 5520 processors, with 12 Gb RAM and a slow-mo SATA disk. How should we allocate RAM to the guests, and do we want swap?
Sven immediately offered this article on how mysql behaves on a NUMA CPU architecture, which suggests to me that I might as well turn swap off. The counterpoint (and traditional view) is that a linux machine must have swap, as a part of its normal modus operandi. So far, the render nodes show no difference with/without swap (so I turned it off) and two virtual guests outperform one OS running on bare metal. I suspect the virtualization layer is sticking one guest on each of the processors, so their RAM is staying local to the CPU they're on.
I don't know the answer, but it's gonna be interesting. If you're in the same boat, you'll appreciate this handy secret-decoder-ring for vmstat. And here's a primer on "swappiness." Also, here's the middle road solution, enable swap, but just a pittance of it.
Finally, if you're running a render/compute farm, you'll find these bits handy ... there's a lovely o'reilly article on how the linux out-of-memory killer works. You'll also be interested in this article on how linux memory overcommit works. And finally, if you're trying with and without swap, here's a quick bit on how to use a normal file as swap, so you donut have to repartition anything to test the on/off configs.
Hopefully this helps someone!