Caution: my English is far from perfect. (Русский тоже не всегда хорош).

Monday, 3 September 2012

Linux memory over-commit.

Linux memory over-commit. Previously I've only read about it, but now I am bitten by it. When cl-test-grid agent (a CCL program) starts ECL and ECL starts gcc, linux on my VPS kills someone of them (usually CCL). Sometimes it just doesn't allow to fork new process. Why do CCL and other Lisps allocate so large heap at startup?


Faré said...

Did you disable overcommit?

CCL reserves a large address space but doesn't actually touch it unless needed.

If that bothers you a LOT, you could replace that by mmap'ing /dev/zero with mprotect and remapping to anonymous pages on demand - hopefully Linux is clever enough to merge the remapped areas.

Anton Vodonosov said...

Yes, I've tried to disable overcommit, but then certain things doesn't work even before I start ECL - namely CMUCL doesn't work.

I am just a little bit curious, why CCL and other programs allocate that large heap at risk of being killed, instead of growing heap incrementally during runtime and garbage collecting when memory is not available (instead of being killed).

Anton Vodonosov said...

BTW, I have worked around the issue by restricting CCL heap size using --heap-reserve 58M option. With that option it seems to survive together with ECL and gcc running in parallel.