Discussion:
lmdb: is read operation permitted inside the write transaction?
Dmytro Milinevskyy
2015-05-12 16:08:27 UTC
Permalink
Hello,

this might be a trivial question for you but I didn't find anything in the
documentation nor any mention in the examples.

Basically I need to have an opportunity to call mdb_get while in write
transaction.
The sequence is smth like:
- txn = mdb_txn_begin(flags=0)
for i in 0..x:
- v = mdb_get(txn, i)
- mdb_put(txn, x+i, v)
- mdb_put(txn, i, v+1)
- mdb_txn_commit(txn)

Will it be always valid data?
I'm running a stress test and didn't spot any issues so far. Though I might
be extremely lucky today.

The DB is used in multi-threaded environment but the code illustrated above
is protected with a mutex.

I tried to dig a bit into the lmdb internals but unfortunately didn't get
any glue.

Thanks for your help,
Dmytro
Howard Chu
2015-05-12 17:36:19 UTC
Permalink
Content preview: Dmytro Milinevskyy wrote: > Hello, > > this might be a trivial
question for you but I didn't find anything in the > documentation nor any
mention in the examples. Which examples did you look at? mtest.c (and others)
all operate this way already. [...]

Content analysis details: (-4.2 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, medium
trust
[69.43.206.106 listed in list.dnswl.org]
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
Post by Dmytro Milinevskyy
Hello,
this might be a trivial question for you but I didn't find anything in the
documentation nor any mention in the examples.
Which examples did you look at? mtest.c (and others) all operate this way already.
Post by Dmytro Milinevskyy
Basically I need to have an opportunity to call mdb_get while in write
transaction.
- txn = mdb_txn_begin(flags=0)
- v = mdb_get(txn, i)
- mdb_put(txn, x+i, v)
- mdb_put(txn, i, v+1)
- mdb_txn_commit(txn)
Will it be always valid data?
Yes.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
Hallvard Breien Furuseth
2015-05-18 21:03:10 UTC
Permalink
Content preview: On 12/05/15 19:36, Howard Chu wrote: > Dmytro Milinevskyy
wrote: >> Basically I need to have an opportunity to call mdb_get while in
write >> transaction. >> The sequence is smth like: >> - txn = mdb_txn_begin(flags=0)
for i in 0..x: >> - v = mdb_get(txn, i) >> - mdb_put(txn, x+i, v) >> -
mdb_put(txn, i, v+1) >> - mdb_txn_commit(txn) >> >> Will it be always valid
data? > > Yes. [...]

Content analysis details: (-4.2 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, medium
trust
[129.240.10.15 listed in list.dnswl.org]
-0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
domain
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
Basically I need to have an opportunity to call mdb_get while in write
transaction.
- txn = mdb_txn_begin(flags=0)
- v = mdb_get(txn, i)
- mdb_put(txn, x+i, v)
- mdb_put(txn, i, v+1)
- mdb_txn_commit(txn)
Will it be always valid data?
Yes.
...No. I expect he means v = the MDB_val returned by mdb_get().
mdb_put() can modify the data it points at.

See the MDB_val documentation in ldmb.h:
* Values returned from the database are valid only until a subsequent
* update operation, or the end of the transaction. Do not modify or
* free them, they commonly point into the database itself.
Timur Kristóf
2015-05-19 15:39:44 UTC
Permalink
Content preview: On Mon, 2015-05-18 at 23:03 +0200, Hallvard Breien Furuseth
Post by Hallvard Breien Furuseth
Basically I need to have an opportunity to call mdb_get while in >
write > > > transaction. > > > The sequence is smth like: > > > - txn
= mdb_txn_begin(flags=0) > > > for i in 0..x: > > > - v = mdb_get(txn, i)
Post by Hallvard Breien Furuseth
- mdb_put(txn, x+i, v) > > > - mdb_put(txn, i, v+1) > > > - mdb_txn_commit(txn)
Will it be always valid data? > > > > Yes. > > ...No. I expect
he means v = the MDB_val returned by mdb_get(). > mdb_put() can modify the
data it points at. > > See the MDB_val documentation in ldmb.h: > * Values
returned from the database are valid only until a > subsequent > * update
operation, or the end of the transaction. Do not modify or > * free them,
they commonly point into the database itself. > [...]

Content analysis details: (-2.5 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low
trust
[209.85.212.181 listed in list.dnswl.org]
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(timur.kristof[at]gmail.com)
0.0 DKIM_ADSP_CUSTOM_MED No valid author signature, adsp_override is
CUSTOM_MED
-0.0 SPF_PASS SPF: sender matches SPF record
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid
0.0 T_DKIM_INVALID DKIM-Signature header exists but is not valid
Post by Hallvard Breien Furuseth
Basically I need to have an opportunity to call mdb_get while in write
transaction.
- txn = mdb_txn_begin(flags=0)
- v = mdb_get(txn, i)
- mdb_put(txn, x+i, v)
- mdb_put(txn, i, v+1)
- mdb_txn_commit(txn)
Will it be always valid data?
Yes.
...No. I expect he means v = the MDB_val returned by mdb_get().
mdb_put() can modify the data it points at.
* Values returned from the database are valid only until a
subsequent
* update operation, or the end of the transaction. Do not modify or
* free them, they commonly point into the database itself.
What exactly does a subsequent update operation mean? Overwriting the
value of the same key, or any mdb_put or mdb_cursor_put operation at
all, regardless of which key they touch?
Howard Chu
2015-05-19 16:05:04 UTC
Permalink
Content preview: Timur Kristóf wrote: > On Mon, 2015-05-18 at 23:03 +0200,
Hallvard Breien Furuseth wrote: >> On 12/05/15 19:36, Howard Chu wrote: >>>
Dmytro Milinevskyy wrote: >>>> Basically I need to have an opportunity to
call mdb_get while in >>>> write >>>> transaction. >>>> The sequence is smth
like: >>>> - txn = mdb_txn_begin(flags=0) >>>> for i in 0..x: >>>> - v =
mdb_get(txn, i) >>>> - mdb_put(txn, x+i, v) >>>> - mdb_put(txn, i, v+1) >>>>
- mdb_txn_commit(txn) >>>> >>>> Will it be always valid data? >>> >>> Yes.
Post by Timur Kristóf
Post by Hallvard Breien Furuseth
...No. I expect he means v = the MDB_val returned by mdb_get(). >>
mdb_put() can modify the data it points at. [...]

Content analysis details: (-4.2 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, medium
trust
[69.43.206.106 listed in list.dnswl.org]
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
Cc: Dmytro Milinevskyy <***@forgetbox.com>, openldap-***@openldap.org
X-BeenThere: openldap-***@openldap.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: OpenLDAP development discussion list <openldap-devel.openldap.org>
List-Unsubscribe: <http://www.openldap.org/lists/mm/options/openldap-devel>,
<mailto:openldap-devel-***@openldap.org?subject=unsubscribe>
List-Archive: <http://www.openldap.org/lists/openldap-devel/>
List-Post: <mailto:openldap-***@openldap.org>
List-Help: <mailto:openldap-devel-***@openldap.org?subject=help>
List-Subscribe: <http://www.openldap.org/lists/mm/listinfo/openldap-devel>,
<mailto:openldap-devel-***@openldap.org?subject=subscribe>
Errors-To: openldap-devel-***@openldap.org
Sender: "openldap-devel" <openldap-devel-***@openldap.org>
X-Spam-Score: -4.2 (----)
X-Spam-Report: Spam detection software, running on the system "gauss.openldap.net", has
identified this incoming email as possible spam. The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email. If you have any questions, see
the administrator of that system for details.

Content preview: Timur Kristóf wrote: > On Mon, 2015-05-18 at 23:03 +0200,
Hallvard Breien Furuseth wrote: >> On 12/05/15 19:36, Howard Chu wrote: >>>
Dmytro Milinevskyy wrote: >>>> Basically I need to have an opportunity to
call mdb_get while in >>>> write >>>> transaction. >>>> The sequence is smth
like: >>>> - txn = mdb_txn_begin(flags=0) >>>> for i in 0..x: >>>> - v =
mdb_get(txn, i) >>>> - mdb_put(txn, x+i, v) >>>> - mdb_put(txn, i, v+1) >>>>
- mdb_txn_commit(txn) >>>> >>>> Will it be always valid data? >>> >>> Yes.
Post by Timur Kristóf
Post by Hallvard Breien Furuseth
...No. I expect he means v = the MDB_val returned by mdb_get(). >>
mdb_put() can modify the data it points at. [...]

Content analysis details: (-4.2 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, medium
trust
[69.43.206.106 listed in list.dnswl.org]
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
Post by Timur Kristóf
Post by Hallvard Breien Furuseth
Basically I need to have an opportunity to call mdb_get while in write
transaction.
- txn = mdb_txn_begin(flags=0)
- v = mdb_get(txn, i)
- mdb_put(txn, x+i, v)
- mdb_put(txn, i, v+1)
- mdb_txn_commit(txn)
Will it be always valid data?
Yes.
...No. I expect he means v = the MDB_val returned by mdb_get().
mdb_put() can modify the data it points at.
Since he's using "v+1" in his example I assumed he's making a local copy of
the returned values. Anyway, the plain answer to the question in the Subject
is Yes.
Post by Timur Kristóf
Post by Hallvard Breien Furuseth
* Values returned from the database are valid only until a
subsequent
* update operation, or the end of the transaction. Do not modify or
* free them, they commonly point into the database itself.
What exactly does a subsequent update operation mean? Overwriting the
value of the same key,
If we meant "same key" we would have said so.
Post by Timur Kristóf
or any mdb_put or mdb_cursor_put operation at
all, regardless of which key they touch?
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
Timur Kristóf
2015-05-19 16:23:12 UTC
Permalink
Content preview: On Tue, 2015-05-19 at 17:05 +0100, Howard Chu wrote: > Timur
Kristóf wrote: > > On Mon, 2015-05-18 at 23:03 +0200, Hallvard Breien Furuseth
wrote: > > > On 12/05/15 19:36, Howard Chu wrote: > > > > Dmytro Milinevskyy
wrote: > > > > > Basically I need to have an opportunity to call mdb_get
while > > > > > in > > > > > write > > > > > transaction. > > > > > The sequence
is smth like: > > > > > - txn = mdb_txn_begin(flags=0) > > > > > for i in
0..x: > > > > > - v = mdb_get(txn, i) > > > > > - mdb_put(txn, x+i, v) >
Post by Howard Chu
Post by Timur Kristóf
Post by Hallvard Breien Furuseth
- mdb_put(txn, i, v+1) > > > > > - mdb_txn_commit(txn) > > > > > >
Will it be always valid data? > > > > > > > > Yes. > > > > > > ...No.
I expect he means v = the MDB_val returned by mdb_get(). > > > mdb_put()
can modify the data it points at. > > Since he's using "v+1" in his example
I assumed he's making a local > copy of > the returned values. Anyway, the
plain answer to the question in the > Subject > is Yes. > > > > See the MDB_val
documentation in ldmb.h: > > > * Values returned from the database are valid
only until a > > > subsequent > > > * update operation, or the end of the
transaction. Do not > > > modify or > > > * free them, they commonly point
into the database itself. > > > What exactly does a subsequent update operation
mean? Overwriting > > the > > value of the same key, > > If we meant "same
key" we would have said so. [...]

Content analysis details: (-2.7 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(timur.kristof[at]gmail.com)
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low
trust
[209.85.212.174 listed in list.dnswl.org]
-0.0 SPF_PASS SPF: sender matches SPF record
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's
domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
Cc: Hallvard Breien Furuseth <***@usit.uio.no>,
Dmytro Milinevskyy <***@forgetbox.com>, openldap-***@openldap.org
X-BeenThere: openldap-***@openldap.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: OpenLDAP development discussion list <openldap-devel.openldap.org>
List-Unsubscribe: <http://www.openldap.org/lists/mm/options/openldap-devel>,
<mailto:openldap-devel-***@openldap.org?subject=unsubscribe>
List-Archive: <http://www.openldap.org/lists/openldap-devel/>
List-Post: <mailto:openldap-***@openldap.org>
List-Help: <mailto:openldap-devel-***@openldap.org?subject=help>
List-Subscribe: <http://www.openldap.org/lists/mm/listinfo/openldap-devel>,
<mailto:openldap-devel-***@openldap.org?subject=subscribe>
Errors-To: openldap-devel-***@openldap.org
Sender: "openldap-devel" <openldap-devel-***@openldap.org>
X-Spam-Score: -2.7 (--)
X-Spam-Report: Spam detection software, running on the system "gauss.openldap.net", has
identified this incoming email as possible spam. The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email. If you have any questions, see
the administrator of that system for details.

Content preview: On Tue, 2015-05-19 at 17:05 +0100, Howard Chu wrote: > Timur
Kristóf wrote: > > On Mon, 2015-05-18 at 23:03 +0200, Hallvard Breien Furuseth
wrote: > > > On 12/05/15 19:36, Howard Chu wrote: > > > > Dmytro Milinevskyy
wrote: > > > > > Basically I need to have an opportunity to call mdb_get
while > > > > > in > > > > > write > > > > > transaction. > > > > > The sequence
is smth like: > > > > > - txn = mdb_txn_begin(flags=0) > > > > > for i in
0..x: > > > > > - v = mdb_get(txn, i) > > > > > - mdb_put(txn, x+i, v) >
Post by Howard Chu
Post by Timur Kristóf
Post by Hallvard Breien Furuseth
- mdb_put(txn, i, v+1) > > > > > - mdb_txn_commit(txn) > > > > > >
Will it be always valid data? > > > > > > > > Yes. > > > > > > ...No.
I expect he means v = the MDB_val returned by mdb_get(). > > > mdb_put()
can modify the data it points at. > > Since he's using "v+1" in his example
I assumed he's making a local > copy of > the returned values. Anyway, the
plain answer to the question in the > Subject > is Yes. > > > > See the MDB_val
documentation in ldmb.h: > > > * Values returned from the database are valid
only until a > > > subsequent > > > * update operation, or the end of the
transaction. Do not > > > modify or > > > * free them, they commonly point
into the database itself. > > > What exactly does a subsequent update operation
mean? Overwriting > > the > > value of the same key, > > If we meant "same
key" we would have said so. [...]

Content analysis details: (-2.7 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low
trust
[209.85.212.174 listed in list.dnswl.org]
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(timur.kristof[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's
domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
Post by Howard Chu
Post by Timur Kristóf
Post by Hallvard Breien Furuseth
Post by Dmytro Milinevskyy
Basically I need to have an opportunity to call mdb_get while
in
write
transaction.
- txn = mdb_txn_begin(flags=0)
- v = mdb_get(txn, i)
- mdb_put(txn, x+i, v)
- mdb_put(txn, i, v+1)
- mdb_txn_commit(txn)
Will it be always valid data?
Yes.
...No. I expect he means v = the MDB_val returned by mdb_get().
mdb_put() can modify the data it points at.
Since he's using "v+1" in his example I assumed he's making a local copy of
the returned values. Anyway, the plain answer to the question in the Subject
is Yes.
Post by Timur Kristóf
Post by Hallvard Breien Furuseth
* Values returned from the database are valid only until a subsequent
* update operation, or the end of the transaction. Do not modify or
* free them, they commonly point into the database itself.
What exactly does a subsequent update operation mean? Overwriting the
value of the same key,
If we meant "same key" we would have said so.
Okay, thanks for the clarification!
Post by Howard Chu
Post by Timur Kristóf
or any mdb_put or mdb_cursor_put operation at
all, regardless of which key they touch?
Loading...