If your KVM virtual machine crashes in Ubuntu 10.04 LTS because it is out of memory try to increase the assigned memory first. If that does not help, you can try to add these two lines to /etc/sysctl.conf (don’t forget to restart your computer for the changes to become effective).
vm.overcommit_memory = 2 vm.overcommit_ratio = 80
This reduces the risk of memory overcommitment and made my virtual machines run stable again. Before Apache and MySQL crashed under load, the log file would only show “apache2 invoked oom-killer”.
Here the article that pointed me in the right direction: http://opsmonkey.blogspot.com/2007/01/linux-memory-overcommit.html
And here a page that explains more about Linux and memory usage: http://www.win.tue.nl/~aeb/linux/lk/lk-9.html