Re: SIGSEGV on mdb_txn_begin / __pthread_mutex_lock_full

[Howard Chu <hyc@symas.com>]

opensource@gmx-topmail.de wrote:
> We are using LMDB and are running a big number of tests on a CI server (Debian via Docker, Xenon CPU).

Are you running multiple Docker instances on the same database? AFAIK that 
won't work because of Docker's munging with POSIX namespaces.

> The crash happens during opening the database. This is done a lot in the tests without problems (and on Windows this problem does not exist). However, during the CI process we get the crash quite reproducible (~80% crash rate when running the full CI test suite).
>
> This is our basic code triggering the SIGSEGV (minus error checking for simplicity):
>
>     mdb_env_create(&dbEnv_);
>     mdb_env_set_maxdbs(dbEnv_, 8);
>     mdb_env_set_mapsize(dbEnv_, maxDbSizeInKByte * 1024);
>     mdb_env_open(dbEnv_, directory, 0, fileMode);
>     MDB_txn* txn;
>     mdb_txn_begin(dbEnv_, NULL, 0, &txn);
>
> It crashes inside mdb_txn_begin with this:
> SIGSEGV (0xb) at pc=0x00007fd2607e4b58, pid=23603, tid=0x00007fd228633700
> C  [libpthread.so.0+0x5b58]  __pthread_mutex_lock_full+0x1a8
>
> I digged into it a bit, and could tracd it down to this LOCK_MUTEX call in mdb_txn_renew0:
>
>         /* Not yet touching txn == env->me_txn0, it may be active */
>         if (ti) {
>             if (LOCK_MUTEX(rc, env, env->me_wmutex))
>                 return rc;
>
> I also logged some data about the env and the mutex (which looks normal however):
> PID: 23603 ENV: 0x7fd1e03e9ab0
> TXN: 0x7fd22811b000 COUNT: 0 KIND: 144 LOCK: 0 OWNER: 0
>
> Any idea what could go wrong here?
>
> Thanks!
> Markus
>
>
> Some additional crash info:
>
> siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007fd22801a058
>
> Registers:
> RAX=0x00007fd22801a060, RBX=0x00007fd22811b040, RCX=0x0000000000000000, RDX=0x0000000000000000
> RSP=0x00007fd22862ff50, RBP=0x0000000000005c4f, RSI=0x0000000000000080, RDI=0x0000000000005c4f
> R8 =0x0000000000000080, R9 =0x0000000000000000, R10=0x00007fd22811b060, R11=0x0000000000000000
> R12=0x0000000000000000, R13=0x00007fd22813b0f0, R14=0x00007fd2286309d0, R15=0x00007fd25848b000
> RIP=0x00007fd2607e4b58, EFLAGS=0x0000000000010206, CSGSFS=0x0000000000000033, ERR=0x0000000000000006
>   TRAPNO=0x000000000000000e
>
> Top of Stack: (sp=0x00007fd22862ff50)
> 0x00007fd22862ff50:   00007fd22813b0f0 00007fd228630040
> 0x00007fd22862ff60:   0000000000000000 00007fd22813b0f0
> 0x00007fd22862ff70:   00007fd2286309d0 00007fd20c2d05e4
> 0x00007fd22862ff80:   00007fd228630050 00007fd22862ff90
> 0x00007fd22862ff90:   0000000000001020 000000000000003c
> 0x00007fd22862ffa0:   0000000a00000023 0000000a00000000
> 0x00007fd22862ffb0:   00007fd228633700 00007fd25fee1c29
> 0x00007fd22862ffc0:   0000000000000000 00007fd25fe7a423
> 0x00007fd22862ffd0:   00007fd2286309d0 0000000900000000
> 0x00007fd22862ffe0:   0000000000000008 00007fd244000000
> 0x00007fd22862fff0:   00007fd22813b0f0 00007fd25fe79ae3
> 0x00007fd228630000:   0000000000000000 00000000286300c0
> 0x00007fd228630010:   0000000000000000 00007fd25fe7b255
> 0x00007fd228630020:   00007fd22811b000 00007fd1e03e9ab0
> 0x00007fd228630030:   00007fd1e08f3260 00007fd25fe6fb22
> 0x00007fd228630040:   00007fd2286300c0 00007fd20c2d0e8a
> 0x00007fd228630050:   00007fd2286300c0 00007fd2601ac2a0
> 0x00007fd228630060:   0000000000000020 00007fd244000000
> 0x00007fd228630070:   0000000800000000 00007fd25fe79ae3
> 0x00007fd228630080:   0000000000000020 00007fd228630660
> 0x00007fd228630090:   00007fd1e08f3260 00007fd2286301c8
> 0x00007fd2286300a0:   00000000601ac2a0 0000000000000000
> 0x00007fd2286300b0:   00007fd1e03e9ab0 00007fd25fe6fb22
> 0x00007fd2286300c0:   00007fd228630660 00007fd20c23a768
> 0x00007fd2286300d0:   0000000200f01b00 00007fd20000003b
> 0x00007fd2286300e0:   00007fd20c2eb3fd 00007fd24921da4c
> 0x00007fd2286300f0:   00000005cac01420 00007fd22814c210
> 0x00007fd228630100:   00007fd228630460 00007fd24921f99f
> 0x00007fd228630110:   00007fd2286301c8 0000002000000000
> 0x00007fd228630120:   0000000000000000 0000000000000000
> 0x00007fd228630130:   0000000000000000 0000000000000020
> 0x00007fd228630140:   0000000000000000 0000000000000000
>
> Instructions: (pc=0x00007fd2607e4b58)
> 0x00007fd2607e4b38:   00 a9 00 00 00 40 0f 85 02 ff ff ff c7 43 04 01
> 0x00007fd2607e4b48:   00 00 00 64 48 8b 04 25 e0 02 00 00 48 83 e0 fe
> 0x00007fd2607e4b58:   4c 89 50 f8 64 48 8b 04 25 e0 02 00 00 48 89 43
> 0x00007fd2607e4b68:   20 64 48 8b 04 25 10 00 00 00 48 05 e0 02 00 00
>
> Register to memory mapping:
>
> RAX=0x00007fd22801a060 is an unknown value
> RBX=0x00007fd22811b040 is an unknown value
> RCX=0x0000000000000000 is an unknown value
> RDX=0x0000000000000000 is an unknown value
> RSP=0x00007fd22862ff50 is pointing into the stack for thread: 0x00007fd25848b000
> RBP=0x0000000000005c4f is an unknown value
> RSI=0x0000000000000080 is an unknown value
> RDI=0x0000000000005c4f is an unknown value
> R8 =0x0000000000000080 is an unknown value
> R9 =0x0000000000000000 is an unknown value
> R10=0x00007fd22811b060 is an unknown value
> R11=0x0000000000000000 is an unknown value
> R12=0x0000000000000000 is an unknown value
> R13={method} {0x00007fd22813b0f0} 'nativeCreate' '(Ljava/lang/String;J[B)J' in 'MyDb'
> R14=0x00007fd2286309d0 is pointing into the stack for thread: 0x00007fd25848b000
> R15=0x00007fd25848b000 is a thread


--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/