Enhancing back-sock to use JSON

[Dagobert Michelsen]

Hi,

I have made some enhancements to back-sock to use JSON for the passed data and JSON-RPC
to map LDAP calls to method invocations. The function signatures of the JSON-RPC calls
are modeled to be similar to the ones used in json2ldap (which does just the opposite
direction to talk LDAP via JSON-RPC) [1]. The previous hand-crafted format passed on the
socket was harder to parse and needed a manually built parser whereas now a standard
library can be used. However, handling the JSON data structures now imposes an additional
dependency to Jansson (a JSON access library in C) [2]. Jansson itself is leightweight and
has no dependencies itself. Due to the limited use of back-sock and the enhanced ease of
use I think it would be acceptable to add this dependency.

I would be glad if the modification would be possible to be applied to OpenLDAP and I
happily submit a patch.


Best regards

-- Dago


[1] JSON2LDAP interface from JSON-RPC to LDAP
http://connect2id.com/products/json2ldap/web-api#ldap-compare

[2] Jansson, a C library for reading and writing JSON data structures
http://www.digip.org/jansson/

[Howard Chu]

Content preview: Dagobert Michelsen wrote: > Hi, Hi my initial reaction: the
current format is just a tweaked LDIF. LDIF itself is still a more compact
format than JSON. I personally am opposed to adding any JSON dependencies
to our code base. Anyone else have an opinion? > > I have made some enhancements
to back-sock to use JSON for the passed data and JSON-RPC > to map LDAP calls
to method invocations. The function signatures of the JSON-RPC calls > are
modeled to be similar to the ones used in json2ldap (which does just the
opposite > direction to talk LDAP via JSON-RPC) [1]. The previous hand-crafted
format passed on the > socket was harder to parse and needed a manually built
parser whereas now a standard > library can be used. However, handling the
JSON data structures now imposes an additional > dependency to Jansson (a
JSON access library in C) [2]. Jansson itself is leightweight and > has no
dependencies itself. Due to the limited use of back-sock and the enhanced
ease of > use I think it would be acceptable to add this dependency. > >
I would be glad if the modification would be possible to be applied to OpenLDAP
and I > happily submit a patch. > > > Best regards > > -- Dago > > > [1]
JSON2LDAP interface from JSON-RPC to LDAP > http://connect2id.com/products/json2ldap/web-api#ldap-compare

Post by Dagobert Michelsen
[2] Jansson, a C library for reading and writing JSON data structures

http://www.digip.org/jansson/ > [...]

Content analysis details: (-1.9 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked.
See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: connect2id.com]
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]

Hi,

Hi

my initial reaction: the current format is just a tweaked LDIF. LDIF itself is still a more compact format than JSON. I personally am opposed to adding any JSON dependencies to our code base. Anyone else have an opinion?

I have made some enhancements to back-sock to use JSON for the passed data and JSON-RPC
to map LDAP calls to method invocations. The function signatures of the JSON-RPC calls
are modeled to be similar to the ones used in json2ldap (which does just the opposite
direction to talk LDAP via JSON-RPC) [1]. The previous hand-crafted format passed on the
socket was harder to parse and needed a manually built parser whereas now a standard
library can be used. However, handling the JSON data structures now imposes an additional
dependency to Jansson (a JSON access library in C) [2]. Jansson itself is leightweight and
has no dependencies itself. Due to the limited use of back-sock and the enhanced ease of
use I think it would be acceptable to add this dependency.
I would be glad if the modification would be possible to be applied to OpenLDAP and I
happily submit a patch.
Best regards
-- Dago
[1] JSON2LDAP interface from JSON-RPC to LDAP
http://connect2id.com/products/json2ldap/web-api#ldap-compare
[2] Jansson, a C library for reading and writing JSON data structures
http://www.digip.org/jansson/

[Dagobert Michelsen]

Hi Howard,

Post by Howard Chu

Post by Dagobert Michelsen
I have made some enhancements to back-sock to use JSON for the passed data and JSON-RPC
to map LDAP calls to method invocations. The function signatures of the JSON-RPC calls
are modeled to be similar to the ones used in json2ldap (which does just the opposite
direction to talk LDAP via JSON-RPC) [1]. The previous hand-crafted format passed on the
socket was harder to parse and needed a manually built parser whereas now a standard
library can be used. However, handling the JSON data structures now imposes an additional
dependency to Jansson (a JSON access library in C) [2]. Jansson itself is leightweight and
has no dependencies itself. Due to the limited use of back-sock and the enhanced ease of
use I think it would be acceptable to add this dependency.

my initial reaction: the current format is just a tweaked LDIF. LDIF itself is still a more
compact format than JSON. I personally am opposed to adding any JSON dependencies to our
code base. Anyone else have an opinion?

Well, of course you are right that the LDAP presentation is more efficient.
However I think from a client perspective it would be easier not to deal
with LDIF, especially as you can choose a JSON-RPC server suitable for your
needs and have the data already available for the function and concentrate
on implementing the functionality:
https://en.wikipedia.org/wiki/JSON-RPC#Implementations

I see back-sock not as a solution for super high performance, but as a
tool to build somewhat exotic adapters and proof of concepts. Allowing
a quick implementation is at least for me the primary goal here.


Best regards

— Dago

[Michael Ströder]

Post by Dagobert Michelsen

Post by Howard Chu

Post by Dagobert Michelsen
I have made some enhancements to back-sock to use JSON for the passed data and JSON-RPC
to map LDAP calls to method invocations.

my initial reaction: the current format is just a tweaked LDIF. LDIF itself is still a more
compact format than JSON. I personally am opposed to adding any JSON dependencies to our
code base. Anyone else have an opinion?

Well, of course you are right that the LDAP presentation is more efficient.
However I think from a client perspective it would be easier not to deal
with LDIF, especially as you can choose a JSON-RPC server suitable for your
needs and have the data already available for the function and concentrate
https://en.wikipedia.org/wiki/JSON-RPC#Implementations

You assume that many people want to do JSON-RPC. IMHO that's only your
specific need. And it shouldn't be too hard for you to write an external
generic back-sock listener which translates this custom LDIF to JSON and
provide it as separate open source project.

The point with choosing a data format is that there are so many formats.
During the last 15 years there were various major formats en vogue also with
lots of proprietary flavors. I'm pretty sure JSON is not the last one, e.g.
personally I lost interest in JSON-SCIM pretty quickly.

Ciao, Michael.

[Dagobert Michelsen]

Hi Michael,

Post by Michael Ströder

Post by Dagobert Michelsen

Post by Howard Chu

Post by Dagobert Michelsen
I have made some enhancements to back-sock to use JSON for the passed data and JSON-RPC
to map LDAP calls to method invocations.

my initial reaction: the current format is just a tweaked LDIF. LDIF itself is still a more
compact format than JSON. I personally am opposed to adding any JSON dependencies to our
code base. Anyone else have an opinion?

Well, of course you are right that the LDAP presentation is more efficient.
However I think from a client perspective it would be easier not to deal
with LDIF, especially as you can choose a JSON-RPC server suitable for your
needs and have the data already available for the function and concentrate
https://en.wikipedia.org/wiki/JSON-RPC#Implementations

You assume that many people want to do JSON-RPC. IMHO that's only your
specific need. And it shouldn't be too hard for you to write an external
generic back-sock listener which translates this custom LDIF to JSON and
provide it as separate open source project.

I would happily do that, however I need some extra fields to be passed.
Without a standardized extendable format that doesn’t break existing clients
this inconveniences existing users, be it XML or JSON or YAML or whatever else
that doesn’t require a custom parser.

Post by Michael Ströder
The point with choosing a data format is that there are so many formats.
During the last 15 years there were various major formats en vogue also with
lots of proprietary flavors. I'm pretty sure JSON is not the last one, e.g.
personally I lost interest in JSON-SCIM pretty quickly.

Certainly not, but sticking to a hardly extendable custom format doesn’t seem
very useful either.


Best regards

— Dago

[Michael Ströder]

Post by Dagobert Michelsen
Hi Michael,

Post by Michael Ströder

Post by Dagobert Michelsen

Post by Howard Chu

Post by Dagobert Michelsen
I have made some enhancements to back-sock to use JSON for the passed data and JSON-RPC
to map LDAP calls to method invocations.

my initial reaction: the current format is just a tweaked LDIF. LDIF itself is still a more
compact format than JSON. I personally am opposed to adding any JSON dependencies to our
code base. Anyone else have an opinion?

Well, of course you are right that the LDAP presentation is more efficient.
However I think from a client perspective it would be easier not to deal
with LDIF, especially as you can choose a JSON-RPC server suitable for your
needs and have the data already available for the function and concentrate
https://en.wikipedia.org/wiki/JSON-RPC#Implementations

You assume that many people want to do JSON-RPC. IMHO that's only your
specific need. And it shouldn't be too hard for you to write an external
generic back-sock listener which translates this custom LDIF to JSON and
provide it as separate open source project.

I would happily do that, however I need some extra fields to be passed.

It seems we're getting to the interesting part.

I'd also like to see back-sock used as overlay to pipe requests and responses
through an external process. E.g. I'd like to rewrite filters based on
authz-DN or similar but let slapd itself process the (modified) request.

Ciao, Michael.

[Howard Chu]

Content preview: Michael Ströder wrote: > Dagobert Michelsen wrote: >> Hi Michael,

Post by Michael Ströder

Post by Dagobert Michelsen

Post by Michael Ströder

Post by Howard Chu

Dagobert Michelsen wrote: >>>>> Am 19.02.2015 um 18:05 schrieb Howard

Chu <***@symas.com>: >>>>>> Dagobert Michelsen wrote: >>>>>> >>>>>> I have
made some enhancements to back-sock to use JSON for the passed data and JSON-RPC
the current format is just a tweaked LDIF. LDIF itself is still a more >>>>>
compact format than JSON. I personally am opposed to adding any JSON dependencies
to our >>>>> code base. Anyone else have an opinion? >>>> >>>> Well, of course
you are right that the LDAP presentation is more efficient. >>>> However
I think from a client perspective it would be easier not to deal >>>> with
LDIF, especially as you can choose a JSON-RPC server suitable for your >>>>
needs and have the data already available for the function and concentrate

Post by Michael Ströder

Post by Dagobert Michelsen

Post by Michael Ströder

on implementing the functionality: >>>> https://en.wikipedia.org/wiki/JSON-RPC#Implementations

Post by Howard Chu

You assume that many people want to do JSON-RPC. IMHO that's only

your >>> specific need. And it shouldn't be too hard for you to write an
external >>> generic back-sock listener which translates this custom LDIF
to JSON and >>> provide it as separate open source project. >> >> I would
happily do that, however I need some extra fields to be passed. > > It seems
we're getting to the interesting part. [...]

Content analysis details: (-1.9 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked.
See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: wikipedia.org]
0.0 RCVD_IN_DNSWL_BLOCKED RBL: ADMINISTRATOR NOTICE: The query to DNSWL
was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[69.43.206.106 listed in list.dnswl.org]
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
Cc: openldap-***@openldap.org
X-BeenThere: openldap-***@openldap.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: OpenLDAP development discussion list <openldap-devel.openldap.org>
List-Unsubscribe: <http://www.openldap.org/lists/mm/options/openldap-devel>,
<mailto:openldap-devel-***@openldap.org?subject=unsubscribe>
List-Archive: <http://www.openldap.org/lists/openldap-devel/>
List-Post: <mailto:openldap-***@openldap.org>
List-Help: <mailto:openldap-devel-***@openldap.org?subject=help>
List-Subscribe: <http://www.openldap.org/lists/mm/listinfo/openldap-devel>,
<mailto:openldap-devel-***@openldap.org?subject=subscribe>
Errors-To: openldap-devel-***@openldap.org
Sender: "openldap-devel" <openldap-devel-***@openldap.org>
X-Spam-Score: -1.9 (-)
X-Spam-Report: Spam detection software, running on the system "gauss.openldap.net", has
identified this incoming email as possible spam. The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email. If you have any questions, see
the administrator of that system for details.

Content preview: Michael Ströder wrote: > Dagobert Michelsen wrote: >> Hi Michael,

Post by Michael Ströder

Post by Dagobert Michelsen

Post by Michael Ströder

Post by Howard Chu

Dagobert Michelsen wrote: >>>>> Am 19.02.2015 um 18:05 schrieb Howard

Chu <***@symas.com>: >>>>>> Dagobert Michelsen wrote: >>>>>> >>>>>> I have
made some enhancements to back-sock to use JSON for the passed data and JSON-RPC
the current format is just a tweaked LDIF. LDIF itself is still a more >>>>>
compact format than JSON. I personally am opposed to adding any JSON dependencies
to our >>>>> code base. Anyone else have an opinion? >>>> >>>> Well, of course
you are right that the LDAP presentation is more efficient. >>>> However
I think from a client perspective it would be easier not to deal >>>> with
LDIF, especially as you can choose a JSON-RPC server suitable for your >>>>
needs and have the data already available for the function and concentrate

Post by Michael Ströder

Post by Dagobert Michelsen

Post by Michael Ströder

on implementing the functionality: >>>> https://en.wikipedia.org/wiki/JSON-RPC#Implementations

Post by Howard Chu

You assume that many people want to do JSON-RPC. IMHO that's only

your >>> specific need. And it shouldn't be too hard for you to write an
external >>> generic back-sock listener which translates this custom LDIF
to JSON and >>> provide it as separate open source project. >> >> I would
happily do that, however I need some extra fields to be passed. > > It seems
we're getting to the interesting part. [...]

Content analysis details: (-1.9 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.0 RCVD_IN_DNSWL_BLOCKED RBL: ADMINISTRATOR NOTICE: The query to DNSWL
was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[69.43.206.106 listed in list.dnswl.org]
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked.
See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: highlandsun.com]
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]

Post by Michael Ströder

Post by Dagobert Michelsen
Hi Michael,

Post by Michael Ströder

Post by Howard Chu

I have made some enhancements to back-sock to use JSON for the passed data and JSON-RPC
to map LDAP calls to method invocations.

my initial reaction: the current format is just a tweaked LDIF. LDIF itself is still a more
compact format than JSON. I personally am opposed to adding any JSON dependencies to our
code base. Anyone else have an opinion?

Well, of course you are right that the LDAP presentation is more efficient.
However I think from a client perspective it would be easier not to deal
with LDIF, especially as you can choose a JSON-RPC server suitable for your
needs and have the data already available for the function and concentrate
https://en.wikipedia.org/wiki/JSON-RPC#Implementations

You assume that many people want to do JSON-RPC. IMHO that's only your
specific need. And it shouldn't be too hard for you to write an external
generic back-sock listener which translates this custom LDIF to JSON and
provide it as separate open source project.

I would happily do that, however I need some extra fields to be passed.

It seems we're getting to the interesting part.

Indeed.

Post by Michael Ströder
I'd also like to see back-sock used as overlay to pipe requests and responses
through an external process.

It already works as an overlay.

Post by Michael Ströder
E.g. I'd like to rewrite filters based on
authz-DN or similar but let slapd itself process the (modified) request.

Sounds like you two could start enumerating all of the parameters you would like to see added for each operation type. A final list should be submitted to the ITS as an enhancement request. Patches adding such params would be even better.

[Conrad Steenberg]

Content preview: Hi Dago, So would the use case for this be to talk to a JSON
database like MongoDB via the LDAP protocol? Thanks! Conrad [...]

Content analysis details: (-2.0 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked.
See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: connect2id.com]
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(conrad.steenberg[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's
domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature

Hi Dago,

So would the use case for this be to talk to a JSON database like
MongoDB via the LDAP protocol?

Thanks!
Conrad

Post by Dagobert Michelsen
Hi,
I have made some enhancements to back-sock to use JSON for the passed data and JSON-RPC
to map LDAP calls to method invocations. The function signatures of the JSON-RPC calls
are modeled to be similar to the ones used in json2ldap (which does just the opposite
direction to talk LDAP via JSON-RPC) [1]. The previous hand-crafted format passed on the
socket was harder to parse and needed a manually built parser whereas now a standard
library can be used. However, handling the JSON data structures now imposes an additional
dependency to Jansson (a JSON access library in C) [2]. Jansson itself is leightweight and
has no dependencies itself. Due to the limited use of back-sock and the enhanced ease of
use I think it would be acceptable to add this dependency.
I would be glad if the modification would be possible to be applied to OpenLDAP and I
happily submit a patch.
Best regards
-- Dago
[1] JSON2LDAP interface from JSON-RPC to LDAP
http://connect2id.com/products/json2ldap/web-api#ldap-compare
[2] Jansson, a C library for reading and writing JSON data structures
http://www.digip.org/jansson/

[Dagobert Michelsen]

Hi Conrad,

Post by Conrad Steenberg
So would the use case for this be to talk to a JSON database like
MongoDB via the LDAP protocol?

Not directly. I didn’t change the transport from back-sock which is still
a local socket. You can now write a simple JSON-RPC server taking method
calls send from OpenLDAP according to the incoming native LDAP requests.
The implementation is connection-aware via cookies and multithreaded.
This allows to glue on legacy data systems easily in any language that
understands JSON-RPC. Of course you could also write a translator to
MongoDB. In the future it may be enhanced to use a real HTTPS connection.


Best regards

— Dago