[LMDB] inconsistent mutex behavior and problems for M:N threading

[David Barbour]

The code setting up the locks is `mdb_env_setup_locks` circa line 4250 of
mdb.c. This code has a bunch of #ifdefs for posix vs win32. A major
difference is that locks for posix are non-recursive by default (unless you
set a specific flag), whereas the CreateMutex operation used for win32
creates recursive locks.

The MDB_NOTLS flag only affects readers.

When two writers operate in the same thread using M:N threading, a bug will
occur in Windows whenever two different lightweight 'writer' threads grab
the same mutex in the same OS thread. Because both writers will succeed.

Conversely, in posix systems, a different bug will occur: the second writer
will attempt to grab the mutex; this will lock up the thread... which,
unfortunately, happens to be the same thread that the first writer needs to
use to unlock the mutex. There is no way for a lightweight writer to
migrate to another thread and unlock it there.

I believe both systems would behave more consistently and usefully if we
switch from mutexes to semaphores (of size 1). The difference is that a
semaphore doesn't track an 'owner' thread. This is also the suggestion to
avoid the recursive mutex behavior of windows [1].

I would like to hear other opinions on this matter.

Regards,

Dave

[1]
http://stackoverflow.com/questions/1988324/how-to-alter-the-recursive-locking-behaviour-of-windows-mutex

[Howard Chu]

Content preview: David Barbour wrote: > The code setting up the locks is `mdb_env_setup_locks`
circa line 4250 > of mdb.c. This code has a bunch of #ifdefs for posix vs
win32. A major > difference is that locks for posix are non-recursive by
default (unless > you set a specific flag), whereas the CreateMutex operation
used for > win32 creates recursive locks. > > The MDB_NOTLS flag only affects
readers. > > When two writers operate in the same thread using M:N threading,
a bug > will occur in Windows whenever two different lightweight 'writer'

threads grab the same mutex in the same OS thread. Because both writers

the second > writer will attempt to grab the mutex; this will lock up the
thread... > which, unfortunately, happens to be the same thread that the
first > writer needs to use to unlock the mutex. There is no way for a > lightweight
writer to migrate to another thread and unlock it there. > > I believe both
systems would behave more consistently and usefully if we > switch from mutexes
to semaphores (of size 1). The difference is that a > semaphore doesn't track
an 'owner' thread. This is also the suggestion > to avoid the recursive mutex
behavior of windows [1]. > > I would like to hear other opinions on this
matter. [...]

Content analysis details: (-3.2 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, medium
trust
[69.43.206.106 listed in list.dnswl.org]
1.0 SINGLE_HEADER_2K A single header contains 2K-3K characters
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]

The code setting up the locks is `mdb_env_setup_locks` circa line 4250
of mdb.c. This code has a bunch of #ifdefs for posix vs win32. A major
difference is that locks for posix are non-recursive by default (unless
you set a specific flag), whereas the CreateMutex operation used for
win32 creates recursive locks.
The MDB_NOTLS flag only affects readers.
When two writers operate in the same thread using M:N threading, a bug
will occur in Windows whenever two different lightweight 'writer'
threads grab the same mutex in the same OS thread. Because both writers
will succeed.
Conversely, in posix systems, a different bug will occur: the second
writer will attempt to grab the mutex; this will lock up the thread...
which, unfortunately, happens to be the same thread that the first
writer needs to use to unlock the mutex. There is no way for a
lightweight writer to migrate to another thread and unlock it there.
I believe both systems would behave more consistently and usefully if we
switch from mutexes to semaphores (of size 1). The difference is that a
semaphore doesn't track an 'owner' thread. This is also the suggestion
to avoid the recursive mutex behavior of windows [1].
I would like to hear other opinions on this matter.

Interesting. The downside of your semaphore suggestion on Windows is
that means any thread can also unlock it, regardless of current owner.
This is also detected as an error in POSIX mutexes.

LMDB is documented to be a single-writer design. I don't see any sane
way for us to support M:N threading models ourselves; not portably to
all the possible runtimes out there. I suggest you wrap your own mutex
mechanism around your wrapper for mdb_txn_begin().

Regards,
Dave
[1]
http://stackoverflow.com/questions/1988324/how-to-alter-the-recursive-locking-behaviour-of-windows-mutex

[Hallvard Breien Furuseth]

Content preview: On 12/11/2014 02:21 AM, Howard Chu wrote: > David Barbour
wrote: >> When two writers operate in the same thread using M:N threading,
a bug >> will occur in Windows whenever two different lightweight 'writer'

threads grab the same mutex in the same OS thread. Because both writers
will succeed. [...]

Content analysis details: (-1.9 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
domain
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]

When two writers operate in the same thread using M:N threading, a bug
will occur in Windows whenever two different lightweight 'writer'
threads grab the same mutex in the same OS thread. Because both writers
will succeed.

Yes - after the code which locks the write mutex, mdb.c should do:
if (env->me_txn) {
unlock the mutex;
return MDB_DEADLOCK;
}
That also helps with MDB_NOLOCK.

(...)
Interesting. The downside of your semaphore suggestion on Windows is that
means any thread can also unlock it, regardless of current owner. This is
also detected as an error in POSIX mutexes.

If mdb.c uses error-checking mutexes
pthread_mutexattr_settype(&mattr, PTHREAD_MUTEX_ERRORCHECK)
and checks for the error, yes:-)

OTOH, I can't spot a problem with write transactions being thread-
independent, as long as the user serializes the operations within
these transactions.

On the third hand, semaphores will block before it can get to the
env->me_txn error check above.

We could save the thread ID in the mdb_env and check it before locking,
but that could give false positives because thread IDs need not be
atomic-sized.

LMDB is documented to be a single-writer design. I don't see any sane way for
us to support M:N threading models ourselves; not portably to all the
possible runtimes out there. I suggest you wrap your own mutex mechanism
around your wrapper for mdb_txn_begin().

[Howard Chu]

Content preview: David Barbour wrote: > OTOH, I want to move on to using LMDB,
rather than working on it. My > Haskell bindings to LMDB [1] are now in a
usable condition, albeit only > at the lowest level. You can add it to the
list. :) > > [1] http://hackage.haskell.org/package/lmdb-0.2.0 > Great! Done.
http://symas.com/mdb/#wrappers [...]

Content analysis details: (-1.9 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]

OTOH, I want to move on to using LMDB, rather than working on it. My
Haskell bindings to LMDB [1] are now in a usable condition, albeit only
at the lowest level. You can add it to the list. :)
[1] http://hackage.haskell.org/package/lmdb-0.2.0

Great! Done. http://symas.com/mdb/#wrappers