This issue was reported to Quality Engineering as SPR# NGRT67EGPM, and was fixed in Domino 6.5.4 Fix Pack 2.
The fix introduces two new server notes.ini parameters to control the behavior. Use either of the parameters below to specify the maximum amount of memory to be used by concurrent NoteOpen requests processed by the Domino server: