The Notes.ini variable "NSF_Backup_Memory_Constrained=1" enables the feature with a default size of 20Mb, the Notes.ini variable "NSF_Backup_Memory_Limit" is provided to tune that size.
The default is NSF_Backup_Memory_Limit=20000000.
Fixed in Release 6.5.5 FP2