openldap.git branch master updated. b7a291a488dff3902931559670cf94c7abe2655b

[Howard Chu]

Content preview: openldap-***@OpenLDAP.org wrote: > A ref change was
pushed to the OpenLDAP (openldap.git) repository. > It will be available
in the public mirror shortly. > > The branch, master has been updated > via
b7a291a488dff3902931559670cf94c7abe2655b (commit) > from 1ab08d2f8e4c9baf71e3c146e95d013d7b8bd656
(commit) > > Those revisions listed above that are new to this repository
have > not appeared on any other notification email; so we list those > revisions
in full, below. > > - Log > commit b7a291a488dff3902931559670cf94c7abe2655b

Experimental syslog() replacement > > 2-3x faster than libc. Add it to

servers/slapd/overlays/syncprov.c | 3 +- [...]

Content analysis details: (-1.9 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked.
See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: highlandsun.com]
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]

A ref change was pushed to the OpenLDAP (openldap.git) repository.
It will be available in the public mirror shortly.
The branch, master has been updated
via b7a291a488dff3902931559670cf94c7abe2655b (commit)
from 1ab08d2f8e4c9baf71e3c146e95d013d7b8bd656 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
commit b7a291a488dff3902931559670cf94c7abe2655b
Date: Wed Jul 8 14:22:29 2015 +0100
Experimental syslog() replacement
2-3x faster than libc. Add it to the Makefile yourself if you want to test it.
-----------------------------------------------------------------------
servers/slapd/overlays/syncprov.c | 3 +-

Oops. Ignore the above, reverted already.

servers/slapd/syslog.c | 329 +++++++++++++++++++++++++++++++++++++

The most obvious braindeadedness in glibc syslog() is that it acquires a mutex
before writing to the log socket. This is utterly inane; the log socket is
datagram-based and as such all writes to it are already atomic and need no
mutex protection.

The other big problem is that the function that generates the log timestamp
also acquires 2 mutexes. I avoid all that crap too by writing my own
implementation of localtime(). This overhead can be reduced further still by
simply omitting the timestamp - it turns out that modern syslogds (rsyslog,
syslog-ng) actually ignore the timestamp in the message and generate their own
anyway. (Note that this localtime implementation is broken wrt DST.)

There are a few other areas to explore here, including a log socket per
thread, which would improve throughput even more in heavy workloads. These
things can be explored without major overhauls of existing code.

For greater throughput we need a logging mechanism that doesn't rely on
sprintf. I.e., we should consider a binary log format where we simply pass
around short message IDs (and relevant message parameters) and use a compiled
message catalog that maps IDs to text. This would require a post-processor to
read the binary log and generate human-readable messages somewhere else. (It's
obviously not a new concept, IBM mainframes have done this for decades; the
Microsoft system logger does this as well.)

[Florian Weimer]

Content preview: * Howard Chu: > For greater throughput we need a logging mechanism
that doesn't rely > on sprintf. I.e., we should consider a binary log format
where we > simply pass around short message IDs (and relevant message parameters)

and use a compiled message catalog that maps IDs to text. This would >

require a post-processor to read the binary log and generate > human-readable
messages somewhere else. (It's obviously not a new > concept, IBM mainframes
have done this for decades; the Microsoft > system logger does this as well.)
[...]

Content analysis details: (-1.9 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
domain
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]

For greater throughput we need a logging mechanism that doesn't rely
on sprintf. I.e., we should consider a binary log format where we
simply pass around short message IDs (and relevant message parameters)
and use a compiled message catalog that maps IDs to text. This would
require a post-processor to read the binary log and generate
human-readable messages somewhere else. (It's obviously not a new
concept, IBM mainframes have done this for decades; the Microsoft
system logger does this as well.)

Current systems have journald, which does something like that.

[Howard Chu]

Content preview: Florian Weimer wrote: > * Howard Chu: > >> For greater throughput
we need a logging mechanism that doesn't rely >> on sprintf. I.e., we should
consider a binary log format where we >> simply pass around short message
IDs (and relevant message parameters) >> and use a compiled message catalog
that maps IDs to text. This would >> require a post-processor to read the
binary log and generate >> human-readable messages somewhere else. (It's
obviously not a new >> concept, IBM mainframes have done this for decades;
the Microsoft >> system logger does this as well.) > > Current systems have
journald, which does something like that. [...]

Content analysis details: (-1.9 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked.
See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: freedesktop.org]
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]

Post by Florian Weimer

Post by Howard Chu
For greater throughput we need a logging mechanism that doesn't rely
on sprintf. I.e., we should consider a binary log format where we
simply pass around short message IDs (and relevant message parameters)
and use a compiled message catalog that maps IDs to text. This would
require a post-processor to read the binary log and generate
human-readable messages somewhere else. (It's obviously not a new
concept, IBM mainframes have done this for decades; the Microsoft
system logger does this as well.)

Current systems have journald, which does something like that.

I had thought of journald, but looking at its API, it seems to still primarily
work with fully-formatted messages being sent from the logging clients.

http://www.freedesktop.org/software/systemd/man/sd_journal_print.html

We want to be able to dump unformatted data items, and incur all the
formatting cost in the post-processor. Otherwise, we haven't gained anything
in terms of performance.