[Patch] Allocating sparse files on Windows

[Colin LeMahieu]

Apologies if this arrives twice, I sent it once two days ago before
subscribing to the list and it hasn't appeared yet so I'm resending after
subscribing.

Memory mapping files on Windows requires the file size to match the size of
the memory mapped region however in the current implementation this
requires creating a file the full size of the environment which may be much
larger than the actual amount of data in the database.

This patch creates the database as a sparse file on Windows. Windows
Explorer will report the file size as the size of the entire memory mapped
region but the size on disk will be proportional to the size of committed
data in the database. This can be checked by checking the "size on disk"
property of the file.

Let me know if this is posted incorrectly, otherwise feel free to modify
the patch in whatever way is necessary for approval.

[Howard Chu]

Content preview: Colin LeMahieu wrote: > Apologies if this arrives twice, I
sent it once two days ago before > subscribing to the list and it hasn't
appeared yet so I'm resending > after subscribing. Thanks, but as noted on
http://www.openldap.org/devel/contributing.html patches should be submitted
to the ITS. [...]

Content analysis details: (-4.2 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, medium
trust
[69.43.206.106 listed in list.dnswl.org]
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked.
See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: highlandsun.com]
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]

Post by Colin LeMahieu
Apologies if this arrives twice, I sent it once two days ago before
subscribing to the list and it hasn't appeared yet so I'm resending
after subscribing.

Thanks, but as noted on http://www.openldap.org/devel/contributing.html patches should be submitted to the ITS.

Post by Colin LeMahieu
Memory mapping files on Windows requires the file size to match the size
of the memory mapped region however in the current implementation this
requires creating a file the full size of the environment which may be
much larger than the actual amount of data in the database.
This patch creates the database as a sparse file on Windows. Windows
Explorer will report the file size as the size of the entire memory
mapped region but the size on disk will be proportional to the size of
committed data in the database. This can be checked by checking the
"size on disk" property of the file.
Let me know if this is posted incorrectly, otherwise feel free to modify
the patch in whatever way is necessary for approval.

[Howard Chu]

Content preview: Howard Chu wrote: > Colin LeMahieu wrote: >> Apologies if
this arrives twice, I sent it once two days ago before >> subscribing to the
list and it hasn't appeared yet so I'm resending >> after subscribing. >

Post by Howard Chu
Thanks, but as noted on http://www.openldap.org/devel/contributing.html
patches should be submitted to the ITS. [...]

Content analysis details: (-4.2 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, medium
trust
[69.43.206.106 listed in list.dnswl.org]
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]

Post by Howard Chu

Post by Colin LeMahieu
Apologies if this arrives twice, I sent it once two days ago before
subscribing to the list and it hasn't appeared yet so I'm resending
after subscribing.

Thanks, but as noted on http://www.openldap.org/devel/contributing.html
patches should be submitted to the ITS.

Initial feedback from some folks testing this patch are that there's a
significant drop in performance. Most likely this is because Windows
NTFS allocates sparse file space in 64KB chunks, while pages are still
4KB, and so filling empty space with data involves a number of RMW
iterations.

Did you compare before/after performance in your own testing?

Post by Howard Chu

Post by Colin LeMahieu
Memory mapping files on Windows requires the file size to match the size
of the memory mapped region however in the current implementation this
requires creating a file the full size of the environment which may be
much larger than the actual amount of data in the database.
This patch creates the database as a sparse file on Windows. Windows
Explorer will report the file size as the size of the entire memory
mapped region but the size on disk will be proportional to the size of
committed data in the database. This can be checked by checking the
"size on disk" property of the file.
Let me know if this is posted incorrectly, otherwise feel free to modify
the patch in whatever way is necessary for approval.