[Date Prev][Date Next] [Chronological] [Thread] [Top]

RE: SLAPD WON'T START ON ONE OF THE MULTIMASTERS



I have upgraded to 2.4.44, ran:

# db_recover -v -c -h /var/lib/openldap/accesslog
Finding last valid log LSN: file: 5 offset 4694358
Recovery starting from [4][28]
Recovery complete at Mon May 16 15:12:49 2016
Maximum transaction ID 8000329e Recovery checkpoint [5][4694980]

Then attempted to start and received the following:
573a1bb7 bdb_db_open: database "dc=example,dc=com": unclean shutdown
detected; attempting recovery.
573a1bb7 bdb_db_open: database "cn=accesslog": unclean shutdown detected;
attempting recovery.
573a1bb7 slapd starting
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
Aborted

Same error(s) as before.


John D. Borresen (Dave)
Ph: (781) 981-1609
Email: john.borresen@ll.mit.edu


-----Original Message-----
From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] 
Sent: Monday, May 16, 2016 1:47 PM
To: Borresen, John - 0444 - MITLL; openldap-technical@openldap.org
Subject: RE: SLAPD WON'T START ON ONE OF THE MULTIMASTERS

Actually, never mind, it is not specific to mdb, I was thinking of the wrong
thing.  You'll have to build OpenLDAP 2.4.44 + that fix to recover.

--Quanah

--On Monday, May 16, 2016 6:21 PM +0000 "Borresen, John - 0444 - MITLL" 
<John.Borresen@ll.mit.edu> wrote:

> I've run db_recover on ldapserver2, to no avail:
>
># db_recover -v -h /var/lib/openldap/accesslog  Finding last valid log 
>LSN: file: 5 offset 4659816  Recovery starting from [5][4655494]  
>Recovery complete at Mon May 16 13:03:19 2016  Maximum transaction ID 
>8000000c Recovery checkpoint [5][4660438]
>
># db_recover -v -h /var/lib/openldap/openldap-data  Finding last valid 
>log LSN: file: 3 offset 7169862  Recovery starting from [3][7169734]  
>Recovery complete at Mon May 16 13:03:33 2016  Maximum transaction ID 
>800035d9 Recovery checkpoint [3][7169862]
>
> Same error:
># /usr/local/openldap/libexec/slapd -u ldap -h  
>ldap://ldapserver2.example.come -F  
>/usr/local/openldap/etc/openldap/slapd.d -d 256
> 5739fe11 @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $
> 	
> clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2.4.4
> 0/s
> erv ers/slapd
> 5739fe11 bdb_db_open: database "dc=group42,dc=ldap": unclean shutdown 
> detected; attempting recovery.
> 5739fe11 bdb_db_open: database "cn=accesslog": unclean shutdown 
> detected; attempting recovery.
> 5739fe11 slapd starting
> slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
> Aborted
>
> I found references to remove *.bdb in the accesslog and openldap-data 
> directories and the __db.*.  Could I run a slapcat of the dbase on
> ldapserver1 and copy that over to ldapserver2?   Is that a viable option?
>
>
>
> John D. Borresen (Dave)
> Ph: (781) 981-1609
> Email: john.borresen@ll.mit.edu
>
>
> -----Original Message-----
> From: Quanah Gibson-Mount [mailto:quanah@zimbra.com]
> Sent: Monday, May 16, 2016 11:55 AM
> To: Borresen, John - 0444 - MITLL; openldap-technical@openldap.org
> Subject: Re: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
>
> That was specific to back-mdb.  Your logs showed corruption with BDB.  
> Are you using mdb, bdb, or both?
>
> --Quanah
>
> --On Monday, May 16, 2016 3:22 PM +0000 "Borresen, John - 0444 - MITLL"
> <John.Borresen@ll.mit.edu> wrote:
>
>>
>>
>> I've noticed this error/warning that keeps standing out when starting 
>> slapd on ldapserver2:
>>
>>
>>
>> slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
>>
>>
>>
>> In my google searches I found this post from Quanah as a possible bug 
>> in
>> 2.4.44 (we're running 2.4.40):
>>
>>
>>
>>
>>
>> This list is for discussing reported issues in OpenLDAP Software ()
>>
>> headers
>>
>> quanah | 27 Apr 16:56 2016
>>
>> (ITS#8413) Assertion in back-mdb/search.c during replication
>>
>>
>>
>> Full_Name: Quanah Gibson-Mount
>>
>> Version: 2.4.44
>>
>> OS: Linux
>>
>> URL: ftp://ftp.openldap.org/incoming/
>>
>> Submission from: (NULL) (75.111.52.177)
>>
>>
>>
>> During replication from the accesslog DB, in a 4-way MMR setup, 
>> various masters
>>
>> periodically crash with slapd: search.c:1246: oc_filter: Assertion `f 
>> != ((void
>>
>> *)0)' failed.
>>
>>
>>
>> This is back-mdb/search.c, not slapd-search.c
>>
>>
>>
>> This is triggered when a NULL filter is passed through.  However, it 
>> should be
>>
>> impossible for the filter generated by str2filter to ever fail.
>>
>>
>>
>> Permalink | Reply |
>>
>> Navigate
>>
>> Go to gmane.network.openldap.bugs.
>>
>> Topic
>>
>> Go to the topic.
>>
>> Advertisement
>>
>> Project Web Page
>>
>> This list is for discussing reported issues in OpenLDAP Software ()
>>
>> Search Archive
>>
>>
>>
>> Language
>>
>> Change language
>>
>> Options
>>
>> Current view: Threads only / Showing whole messages / Not hiding 
>> cited text.
>>
>> Change to All messages, shortened messages, or hide cited text.
>>
>>
>>
>> Post a message
>>
>> NNTP Newsgroup
>>
>> Classic Gmane web interface
>>
>> XML RSS Feed
>>
>> List Information
>>
>>
>>
>> About Gmane
>>
>>
>>
>> Gmane
>>
>>
>>
>> Again, if anyone has any suggestions as to a workaround or a 
>> resolution that would be most appreciative.
>>
>>
>>
>> Thanks,
>>
>>
>>
>> John D. Borresen (Dave)
>>
>> Email: john.borresen@ll.mit.edu
>>
>>
>>
>>
>> From: openldap-technical
>> [mailto:openldap-technical-bounces@openldap.org]
>> On Behalf Of Borresen, John - 0444 - MITLL
>> Sent: Friday, May 13, 2016 11:13 AM
>> To: openldap-technical@openldap.org
>> Subject: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
>>
>>
>>
>> We have a 3-way multimaster configuration running on CentOS 5.11, 
>> OpenLDAP 2.4.40.  All three have been up for years, until the other day:
>>
>>
>>
>> Slapd is running on two of the three (server names:  ldapserver1, 
>> ldapserver2, and ldapserver3).  Slapd stopped and won't restart on 
>> ldapserver2.
>>
>>
>>
>> From Logs on ldapserver2:
>>
>> May 10 04:02:13 gp42-admin4 slapd[4541]: slapd shutdown: waiting for 
>> 0 operations/tasks to finish
>>
>> May 10 04:02:19 gp42-admin4 slapd[15633]: @(#) $OpenLDAP: slapd 
>> 2.4.40 (Sep 30 2014 16:49:45) 
>> $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open
>> l
>> dap
>> -2.4.40/servers/slapd
>>
>> May 10 04:02:19 gp42-admin4 slapd[15633]: nss-ldap: do_open:
>> do_start_tls
>> failed:stat=-1
>>
>> May 10 04:02:19 gp42-admin4 slapd[15633]: nss_ldap: reconnected to 
>> LDAP server ldap://ldapserver1.example.com
>>
>> May 10 04:02:21 gp42-admin4 slapd[15634]: bdb_db_open: database
>> "cn=accesslog": database already in use.
>>
>> May 10 04:02:21 gp42-admin4 slapd[15634]: backend_startup_one 
>> (type=bdb,
>> suffix="cn=accesslog"): bi_db_open failed! (-1)
>>
>> May 10 04:02:21 gp42-admin4 slapd[15634]: slapd stopped.
>>
>> May 10 04:02:22 gp42-admin4 slapd[4541]: slapd stopped.
>>
>>
>>
>> When attempting to restart slapd on server2:
>>
>> May 13 10:13:54 gp42-admin4 slapd[12085]: @(#) $OpenLDAP: slapd 
>> 2.4.40 (Sep 30 2014 16:49:45) 
>> $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open
>> l
>> dap
>> -2.4.40/servers/slapd
>>
>> May 13 10:13:54 gp42-admin4 slapd[12085]: nss-ldap: do_open:
>> do_start_tls
>> failed:stat=-1
>>
>> May 13 10:13:54 gp42-admin4 slapd[12085]: nss_ldap: reconnected to 
>> LDAP server ldap://ldapserver1.example.com
>>
>> May 13 10:13:56 gp42-admin4 slapd[12086]: slapd starting
>>
>> May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=002 (4096) 
>> Content Sync Refresh Required
>>
>> May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=001 (4096) 
>> Content Sync Refresh Required
>>
>> May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_insert_key: 
>> c_put id
>> failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock 
>> (-30995)
>>
>> May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_add 0xfc6:
>> parent
>> (cn=accesslog) insert failed: -30995
>>
>> May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_delete_key: 
>> c_del id
>> failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock 
>> (-30995)
>>
>> May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_delete 0xf50:
>> parent (cn=accesslog) delete failed: -30995
>>
>> May 13 10:15:55 gp42-admin4 slapd[12106]: @(#) $OpenLDAP: slapd 
>> 2.4.40 (Sep 30 2014 16:49:45) 
>> $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open
>> l
>> dap
>> -2.4.40/servers/slapd
>>
>> May 13 10:15:55 gp42-admin4 slapd[12106]: nss-ldap: do_open:
>> do_start_tls
>> failed:stat=-1
>>
>> May 13 10:15:55 gp42-admin4 slapd[12106]: nss_ldap: reconnected to 
>> LDAP server ldap://ldapserver1.example.com
>>
>> May 13 10:15:55 gp42-admin4 slapd[12106]: bdb_db_open: database
>> "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
>>
>> May 13 10:15:57 gp42-admin4 slapd[12106]: bdb_db_open: database
>> "cn=accesslog": unclean shutdown detected; attempting recovery.
>>
>> May 13 10:15:58 gp42-admin4 slapd[12106]: slapd starting
>>
>> May 13 10:28:49 gp42-admin4 slapd[12255]: @(#) $OpenLDAP: slapd 
>> 2.4.40 (Sep 30 2014 16:49:45) 
>> $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open
>> l
>> dap
>> -2.4.40/servers/slapd
>>
>> May 13 10:28:49 gp42-admin4 slapd[12255]: nss-ldap: do_open:
>> do_start_tls
>> failed:stat=-1
>>
>> May 13 10:28:49 gp42-admin4 slapd[12255]: nss_ldap: reconnected to 
>> LDAP server ldap://ldapserver1.example.com
>>
>> May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database
>> "dc=example,dc=com": unclean shutdown detected; attempting recovery.
>>
>> May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database
>> "cn=accesslog": unclean shutdown detected; attempting recovery.
>>
>> May 13 10:28:52 gp42-admin4 slapd[12255]: slapd starting
>>
>> May 13 10:29:24 gp42-admin4 slapd[12264]: @(#) $OpenLDAP: slapd 
>> 2.4.40 (Sep 30 2014 16:49:45) 
>> $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open
>> l
>> dap
>> -2.4.40/servers/slapd
>>
>> May 13 10:29:24 gp42-admin4 slapd[12264]: nss-ldap: do_open:
>> do_start_tls
>> failed:stat=-1
>>
>> May 13 10:29:24 gp42-admin4 slapd[12264]: nss_ldap: reconnected to 
>> LDAP server ldap://ldapserver1.example.com
>>
>> May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database
>> "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
>>
>> May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database
>> "cn=accesslog": unclean shutdown detected; attempting recovery.
>>
>> May 13 10:29:24 gp42-admin4 slapd[12264]: slapd starting
>>
>> May 13 10:29:53 gp42-admin4 slapd[12280]: @(#) $OpenLDAP: slapd 
>> 2.4.40 (Sep 30 2014 16:49:45) 
>> $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open
>> l
>> dap
>> -2.4.40/servers/slapd
>>
>> May 13 10:29:53 gp42-admin4 slapd[12280]: nss-ldap: do_open:
>> do_start_tls
>> failed:stat=-1
>>
>> May 13 10:29:53 gp42-admin4 slapd[12280]: nss_ldap: reconnected to 
>> LDAP server ldap://ldapserver1.example.com
>>
>> May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database
>> "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
>>
>> May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database
>> "cn=accesslog": unclean shutdown detected; attempting recovery.
>>
>> May 13 10:29:53 gp42-admin4 slapd[12280]: slapd starting
>>
>> May 13 10:32:35 gp42-admin4 slapd[12345]: @(#) $OpenLDAP: slapd 
>> 2.4.40 (Sep 30 2014 16:49:45) 
>> $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open
>> l
>> dap
>> -2.4.40/servers/slapd
>>
>>
>>
>> Attempting to restart slapd from the command-line:
>>
>> 5735ed50 slapd starting
>>
>> 5735ed50 => bdb_entry_get: ndn: "cn=accesslog"
>>
>> 5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
>>
>> 5735ed50 bdb_idl_fetch_key: %cn=accesslog
>>
>> 5735ed50 bdb_idl_fetch_key: [b49d1940]
>>
>> 5735ed50 bdb_idl_fetch_key:
>>
>> 5735ed50 send_ldap_result: err=0 matched="" text=""
>>
>> 5735ed50 => bdb_entry_get: ndn: "dc=example,dc=com"
>>
>> 5735ed50 => bdb_entry_get: oc: "(null)", at: "contextCSN"
>>
>> ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN 
>> reqDeleteOldRDN reqNewSuperior entryCSN
>>
>> ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN 
>> reqDeleteOldRDN reqNewSuperior entryCSN
>>
>> => ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
>>
>> <= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
>>
>> => ldap_dn2bv(272)
>>
>> <= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
>>
>> => ldap_dn2bv(272)
>>
>> <= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
>>
>> => ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
>>
>> <= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
>>
>> => ldap_dn2bv(272)
>>
>> <= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
>>
>> => ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
>>
>> <= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
>>
>> => ldap_dn2bv(272)
>>
>> <= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
>>
>> 5735ed50 => bdb_entry_get: ndn: "uid=jdoe,ou=Users,dc=example,dc=com"
>>
>> 5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
>>
>> slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
>>
>> Aborted
>>
>>
>>
>>
>>
>> I have run db_recover on the dbase(s) on ldapserver2 but to no avail.
>>
>>
>>
>> Does anyone have any suggestions?
>>
>>
>>
>> Thank you in advance for any assistance.
>>
>>
>>
>>
>>
>>
>>
>> John D. Borresen (Dave)
>>
>> Linux/Unix Systems Administrator
>>
>> MIT  Lincoln Laboratory
>>
>> Humanitarian Assistance and Disaster Relief (HADR) Systems
>>
>> 244 Wood St
>>
>> Lexington, MA  02420
>>
>> Email: john.borresen@ll.mit.edu
>>
>>
>
>
>
> --
>
> Quanah Gibson-Mount
> Platform Architect
> Zimbra, Inc.
> --------------------
> Zimbra ::  the leader in open source messaging and collaboration A 
> division of Synacor, Inc
>



--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration A division
of Synacor, Inc

Attachment: smime.p7s
Description: S/MIME cryptographic signature