Linux, Swap and Render Farms

The debate occurred again last night, with a Cinematics group, about how best to configure their linux render nodes.  We're running a bare metal hypervisor, with two guests running RHEL 5, on a pair of xeon 5520 processors, with  12 Gb RAM and a slow-mo SATA disk.  How should we allocate RAM to the guests, and do we want swap?

Sven immediately offered this article on how mysql behaves on a NUMA CPU architecture, which suggests to me that I might as well turn swap off.  The counterpoint (and traditional view) is that a linux machine must have swap, as a part of its normal modus operandi.  So far, the render nodes show no difference with/without swap (so I turned it off) and two virtual guests outperform one OS running on bare metal.  I suspect the virtualization layer is sticking one guest on each of the processors, so their RAM is staying local to the CPU they're on.

I don't know the answer, but it's gonna be interesting.  If you're in the same boat, you'll appreciate this handy secret-decoder-ring for vmstat.  And here's a primer on "swappiness."  Also, here's the middle road solution, enable swap, but just a pittance of it.

 Finally, if you're running a render/compute farm, you'll find these bits handy ... there's a lovely o'reilly article on how the linux out-of-memory killer works.  You'll also be interested in this article on how linux memory overcommit works.  And finally, if you're trying with and without swap, here's a quick bit on how to use a normal file as swap, so you donut have to repartition anything to test the on/off configs.

Hopefully this helps someone!