Discussion:
slaptest-ldif -l <ldif> tool or slapadd option
Hallvard Breien Furuseth
2015-03-24 16:51:12 UTC
Permalink
Content preview: I'd like a slap tool which verifies an LDIF before I try to
ldapadd/slapadd it. "slapadd -u -o value-check=yes" is fairly close. What
does it fail to catch? I can think of: - Duplicate entries. - Missing entries
(if the initial DB is expected to be empty). - Child entries before parents
(OK for slapadd to at least back-<bdb,hdb,mdb>). [...]

Content analysis details: (-4.2 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, medium
trust
[129.240.10.15 listed in list.dnswl.org]
-0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
domain
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]

I'd like a slap tool which verifies an LDIF before I try to ldapadd/slapadd it.
"slapadd -u -o value-check=yes" is fairly close. What does it fail to catch?
I can think of:

- Duplicate entries.
- Missing entries (if the initial DB is expected to be empty).
- Child entries before parents (OK for slapadd to at least back-<bdb,hdb,mdb>).

- Issues which the tool can only catch if it opens the database, like attempts
to add already-existing entries. I probably don't want to do that.

- Issues which overlays like slapo-unique would reject. Can't do that,
since the overlay won't have a non-empty DB to check against and slap
tools do not use overlays anyway. Might special-case "unique" though,
since the "duplicate entries" check will need uniqueness code anyway.
--
Hallvard
Howard Chu
2015-03-24 22:56:03 UTC
Permalink
Content preview: Hallvard Breien Furuseth wrote: > I'd like a slap tool which
verifies an LDIF before I try to ldapadd/slapadd it. > "slapadd -u -o value-check=yes"
is fairly close. What does it fail to catch? > I can think of: > > - Duplicate
entries. [...]

Content analysis details: (-4.2 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, medium
trust
[69.43.206.106 listed in list.dnswl.org]
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked.
See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: highlandsun.com]
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
Post by Hallvard Breien Furuseth
I'd like a slap tool which verifies an LDIF before I try to ldapadd/slapadd it.
"slapadd -u -o value-check=yes" is fairly close. What does it fail to catch?
- Duplicate entries.
Quite an unrealistic requirement. You need to store the set of entryDNs to achieve this, and for a large LDIF you may need an actual database to manage this. Might as well just do a normal slapadd.
Post by Hallvard Breien Furuseth
- Missing entries (if the initial DB is expected to be empty).
- Child entries before parents (OK for slapadd to at least back-<bdb,hdb,mdb>).
- Issues which the tool can only catch if it opens the database, like attempts
to add already-existing entries. I probably don't want to do that.
- Issues which overlays like slapo-unique would reject. Can't do that,
since the overlay won't have a non-empty DB to check against and slap
tools do not use overlays anyway. Might special-case "unique" though,
since the "duplicate entries" check will need uniqueness code anyway.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
Emmanuel Lécharny
2015-03-24 23:56:00 UTC
Permalink
Content preview: Le 24/03/15 23:56, Howard Chu a écrit : > Hallvard Breien
Furuseth wrote: >> I'd like a slap tool which verifies an LDIF before I try
to >> ldapadd/slapadd it. >> "slapadd -u -o value-check=yes" is fairly close.
What does it fail >> to catch? >> I can think of: >> >> - Duplicate entries.
Post by Howard Chu
Quite an unrealistic requirement. You need to store the set of > entryDNs
to achieve this, and for a large LDIF you may need an actual > database to
manage this. Might as well just do a normal slapadd. Let me disagree. A simple
merge sort will do the trick. No need of a database to deal with that, even
for a large LDIF. We have done some tests with the tool we use to handle
bulkloads at ApacheDS, and that works quite fine, and quite fast enough. [...]


Content analysis details: (-2.0 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(elecharny[at]gmail.com)
0.0 RCVD_IN_DNSWL_BLOCKED RBL: ADMINISTRATOR NOTICE: The query to DNSWL
was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[209.85.212.178 listed in list.dnswl.org]
-0.0 SPF_PASS SPF: sender matches SPF record
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's
domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-BeenThere: openldap-***@openldap.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: OpenLDAP development discussion list <openldap-devel.openldap.org>
List-Unsubscribe: <http://www.openldap.org/lists/mm/options/openldap-devel>,
<mailto:openldap-devel-***@openldap.org?subject=unsubscribe>
List-Archive: <http://www.openldap.org/lists/openldap-devel/>
List-Post: <mailto:openldap-***@openldap.org>
List-Help: <mailto:openldap-devel-***@openldap.org?subject=help>
List-Subscribe: <http://www.openldap.org/lists/mm/listinfo/openldap-devel>,
<mailto:openldap-devel-***@openldap.org?subject=subscribe>
Errors-To: openldap-devel-***@openldap.org
Sender: "openldap-devel" <openldap-devel-***@openldap.org>
X-Spam-Score: -2.0 (--)
X-Spam-Report: Spam detection software, running on the system "gauss.openldap.net", has
identified this incoming email as possible spam. The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email. If you have any questions, see
the administrator of that system for details.

Content preview: Le 24/03/15 23:56, Howard Chu a écrit : > Hallvard Breien
Furuseth wrote: >> I'd like a slap tool which verifies an LDIF before I try
to >> ldapadd/slapadd it. >> "slapadd -u -o value-check=yes" is fairly close.
What does it fail >> to catch? >> I can think of: >> >> - Duplicate entries.
Post by Howard Chu
Quite an unrealistic requirement. You need to store the set of > entryDNs
to achieve this, and for a large LDIF you may need an actual > database to
manage this. Might as well just do a normal slapadd. Let me disagree. A simple
merge sort will do the trick. No need of a database to deal with that, even
for a large LDIF. We have done some tests with the tool we use to handle
bulkloads at ApacheDS, and that works quite fine, and quite fast enough. [...]


Content analysis details: (-2.0 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.0 RCVD_IN_DNSWL_BLOCKED RBL: ADMINISTRATOR NOTICE: The query to DNSWL
was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[209.85.212.178 listed in list.dnswl.org]
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(elecharny[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's
domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
Post by Howard Chu
I'd like a slap tool which verifies an LDIF before I try to
ldapadd/slapadd it.
"slapadd -u -o value-check=yes" is fairly close. What does it fail to catch?
- Duplicate entries.
Quite an unrealistic requirement. You need to store the set of
entryDNs to achieve this, and for a large LDIF you may need an actual
database to manage this. Might as well just do a normal slapadd.
Let me disagree. A simple merge sort will do the trick. No need of a
database to deal with that, even for a large LDIF. We have done some
tests with the tool we use to handle bulkloads at ApacheDS, and that
works quite fine, and quite fast enough.
Hallvard Breien Furuseth
2015-03-25 10:45:14 UTC
Permalink
Content preview: On 25. mars 2015 00:56, Emmanuel Lécharny wrote: > Le 24/03/15
23:56, Howard Chu a écrit : >> Hallvard Breien Furuseth wrote: >>> I'd like
a slap tool which verifies an LDIF before I try to >>> ldapadd/slapadd it.
Post by Emmanuel Lécharny
Post by Howard Chu
"slapadd -u -o value-check=yes" is fairly close. What does it fail >>>
to catch? >>> I can think of: [...]

Content analysis details: (-1.9 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.0 RCVD_IN_DNSWL_BLOCKED RBL: ADMINISTRATOR NOTICE: The query to DNSWL
was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[129.240.10.17 listed in list.dnswl.org]
-0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
domain
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
Cc: openldap-***@openldap.org
X-BeenThere: openldap-***@openldap.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: OpenLDAP development discussion list <openldap-devel.openldap.org>
List-Unsubscribe: <http://www.openldap.org/lists/mm/options/openldap-devel>,
<mailto:openldap-devel-***@openldap.org?subject=unsubscribe>
List-Archive: <http://www.openldap.org/lists/openldap-devel/>
List-Post: <mailto:openldap-***@openldap.org>
List-Help: <mailto:openldap-devel-***@openldap.org?subject=help>
List-Subscribe: <http://www.openldap.org/lists/mm/listinfo/openldap-devel>,
<mailto:openldap-devel-***@openldap.org?subject=subscribe>
Errors-To: openldap-devel-***@openldap.org
Sender: "openldap-devel" <openldap-devel-***@openldap.org>
X-Spam-Score: -1.9 (-)
X-Spam-Report: Spam detection software, running on the system "gauss.openldap.net", has
identified this incoming email as possible spam. The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email. If you have any questions, see
the administrator of that system for details.

Content preview: On 25. mars 2015 00:56, Emmanuel Lécharny wrote: > Le 24/03/15
23:56, Howard Chu a écrit : >> Hallvard Breien Furuseth wrote: >>> I'd like
a slap tool which verifies an LDIF before I try to >>> ldapadd/slapadd it.
Post by Emmanuel Lécharny
Post by Howard Chu
"slapadd -u -o value-check=yes" is fairly close. What does it fail >>>
to catch? >>> I can think of: [...]

Content analysis details: (-1.9 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.0 RCVD_IN_DNSWL_BLOCKED RBL: ADMINISTRATOR NOTICE: The query to DNSWL
was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[129.240.10.17 listed in list.dnswl.org]
-0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
domain
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
Post by Emmanuel Lécharny
Post by Howard Chu
I'd like a slap tool which verifies an LDIF before I try to
ldapadd/slapadd it.
"slapadd -u -o value-check=yes" is fairly close. What does it fail to catch?
Another requirement:

- An attribute should not occur several times in an entry with
other attributes in between. E.g. "cn", "title", then "cn".
That gives broken entries at least with slapadd -q.
Post by Emmanuel Lécharny
Post by Howard Chu
- Duplicate entries.
Quite an unrealistic requirement. You need to store the set of
entryDNs to achieve this, and for a large LDIF you may need an actual
database to manage this. Might as well just do a normal slapadd.
No, I want it to be faster than that. Also I might do someting else
with the verified LDIF anyway. Generate a diff against the current DB
and feed it to ldapmodify.

DN checking would need to be an option, which could just document
"you asked for unbounded memory usage, you got it". Though I suppose
it could slapadd to a DB with just the DNs, not any attributes.
Post by Emmanuel Lécharny
Let me disagree. A simple merge sort will do the trick. No need of a
database to deal with that, even for a large LDIF. We have done some
tests with the tool we use to handle bulkloads at ApacheDS, and that
works quite fine, and quite fast enough.
Nice idea.

A reliable sorter would still need to link slapd or something else
knowing the schema though, so it'll normalize the DNs correctly. Not
quite sure if I'll bother to worry about that. Don't need it with
LDIFs we generate ourselves, but LDIFs from elsewhere are less fun.
--
Hallvard
Mathieu
2015-03-24 23:19:56 UTC
Permalink
Content preview: These requirements are definitely too loosely described to
be implemented. But maybe prerequisites could be added to the slapadd operation,
mimicking dynamic dns (rfc2136)? That way all prerequisites would be sent
by the client, and the server would "just" have too check them before applying
the changes. I know it looks a like as a ldapsearch (to check these prerequisites)
followed by a ldapadd, but if implemented server side it could be done in
an atomic way? [...]

Content analysis details: (-2.7 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low
trust
[209.85.192.43 listed in list.dnswl.org]
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked.
See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: highlandsun.com]
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(mathieu.baeumler[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's
domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature

These requirements are definitely too loosely described to be
implemented. But maybe prerequisites could be added to the slapadd
operation, mimicking dynamic dns (rfc2136)?
That way all prerequisites would be sent by the client, and the server
would "just" have too check them before applying the changes. I know
it looks a like as a ldapsearch (to check these prerequisites)
followed by a ldapadd, but if implemented server side it could be done
in an atomic way?
Post by Howard Chu
Post by Hallvard Breien Furuseth
I'd like a slap tool which verifies an LDIF before I try to
ldapadd/slapadd it.
"slapadd -u -o value-check=yes" is fairly close. What does it fail to catch?
- Duplicate entries.
Quite an unrealistic requirement. You need to store the set of entryDNs to
achieve this, and for a large LDIF you may need an actual database to manage
this. Might as well just do a normal slapadd.
Post by Hallvard Breien Furuseth
- Missing entries (if the initial DB is expected to be empty).
- Child entries before parents (OK for slapadd to at least
back-<bdb,hdb,mdb>).
- Issues which the tool can only catch if it opens the database, like attempts
to add already-existing entries. I probably don't want to do that.
- Issues which overlays like slapo-unique would reject. Can't do that,
since the overlay won't have a non-empty DB to check against and slap
tools do not use overlays anyway. Might special-case "unique" though,
since the "duplicate entries" check will need uniqueness code anyway.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
--
Mathieu
Loading...