[Date Prev][Date Next] [Chronological] [Thread] [Top]

RE: SLAPD WON'T START ON ONE OF THE MULTIMASTERS

Actually, never mind, it is not specific to mdb, I was thinking of the wrong thing. You'll have to build OpenLDAP 2.4.44 + that fix to recover. 

--Quanah
--On Monday, May 16, 2016 6:21 PM +0000 "Borresen, John - 0444 - MITLL" <John.Borresen@ll.mit.edu> wrote: 


I've run db_recover on ldapserver2, to no avail:

# db_recover -v -h /var/lib/openldap/accesslog
Finding last valid log LSN: file: 5 offset 4659816
Recovery starting from [5][4655494]
Recovery complete at Mon May 16 13:03:19 2016
Maximum transaction ID 8000000c Recovery checkpoint [5][4660438]

# db_recover -v -h /var/lib/openldap/openldap-data
Finding last valid log LSN: file: 3 offset 7169862
Recovery starting from [3][7169734]
Recovery complete at Mon May 16 13:03:33 2016
Maximum transaction ID 800035d9 Recovery checkpoint [3][7169862]

Same error:
# /usr/local/openldap/libexec/slapd -u ldap -h
ldap://ldapserver2.example.come -F
/usr/local/openldap/etc/openldap/slapd.d -d 256
5739fe11 @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $
	
clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2.4.40/s
erv ers/slapd
5739fe11 bdb_db_open: database "dc=group42,dc=ldap": unclean shutdown
detected; attempting recovery.
5739fe11 bdb_db_open: database "cn=accesslog": unclean shutdown detected;
attempting recovery.
5739fe11 slapd starting
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
Aborted

I found references to remove *.bdb in the accesslog and openldap-data
directories and the __db.*.  Could I run a slapcat of the dbase on
ldapserver1 and copy that over to ldapserver2?   Is that a viable option?



John D. Borresen (Dave)
Ph: (781) 981-1609
Email: john.borresen@ll.mit.edu


-----Original Message-----
From: Quanah Gibson-Mount [mailto:quanah@zimbra.com]
Sent: Monday, May 16, 2016 11:55 AM
To: Borresen, John - 0444 - MITLL; openldap-technical@openldap.org
Subject: Re: SLAPD WON'T START ON ONE OF THE MULTIMASTERS

That was specific to back-mdb.  Your logs showed corruption with BDB.  Are
you using mdb, bdb, or both?

--Quanah

--On Monday, May 16, 2016 3:22 PM +0000 "Borresen, John - 0444 - MITLL"
<John.Borresen@ll.mit.edu> wrote:




I've noticed this error/warning that keeps standing out when starting
slapd on ldapserver2:



slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.



In my google searches I found this post from Quanah as a possible bug
in
2.4.44 (we're running 2.4.40):





This list is for discussing reported issues in OpenLDAP Software ()

headers

quanah | 27 Apr 16:56 2016

(ITS#8413) Assertion in back-mdb/search.c during replication



Full_Name: Quanah Gibson-Mount

Version: 2.4.44

OS: Linux

URL: ftp://ftp.openldap.org/incoming/

Submission from: (NULL) (75.111.52.177)



During replication from the accesslog DB, in a 4-way MMR setup,
various masters

periodically crash with slapd: search.c:1246: oc_filter: Assertion `f
!= ((void

*)0)' failed.



This is back-mdb/search.c, not slapd-search.c



This is triggered when a NULL filter is passed through.  However, it
should be

impossible for the filter generated by str2filter to ever fail.



Permalink | Reply |

Navigate

Go to gmane.network.openldap.bugs.

Topic

Go to the topic.

Advertisement

Project Web Page

This list is for discussing reported issues in OpenLDAP Software ()

Search Archive



Language

Change language

Options

Current view: Threads only / Showing whole messages / Not hiding cited
text.

Change to All messages, shortened messages, or hide cited text.



Post a message

NNTP Newsgroup

Classic Gmane web interface

XML RSS Feed

List Information



About Gmane



Gmane



Again, if anyone has any suggestions as to a workaround or a
resolution that would be most appreciative.



Thanks,



John D. Borresen (Dave)

Email: john.borresen@ll.mit.edu




From: openldap-technical
[mailto:openldap-technical-bounces@openldap.org]
On Behalf Of Borresen, John - 0444 - MITLL
Sent: Friday, May 13, 2016 11:13 AM
To: openldap-technical@openldap.org
Subject: SLAPD WON'T START ON ONE OF THE MULTIMASTERS



We have a 3-way multimaster configuration running on CentOS 5.11,
OpenLDAP 2.4.40.  All three have been up for years, until the other day:



Slapd is running on two of the three (server names:  ldapserver1,
ldapserver2, and ldapserver3).  Slapd stopped and won't restart on
ldapserver2.



From Logs on ldapserver2:

May 10 04:02:13 gp42-admin4 slapd[4541]: slapd shutdown: waiting for 0
operations/tasks to finish

May 10 04:02:19 gp42-admin4 slapd[15633]: @(#) $OpenLDAP: slapd 2.4.40
(Sep 30 2014 16:49:45)
$#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl
dap
-2.4.40/servers/slapd

May 10 04:02:19 gp42-admin4 slapd[15633]: nss-ldap: do_open:
do_start_tls
failed:stat=-1

May 10 04:02:19 gp42-admin4 slapd[15633]: nss_ldap: reconnected to
LDAP server ldap://ldapserver1.example.com

May 10 04:02:21 gp42-admin4 slapd[15634]: bdb_db_open: database
"cn=accesslog": database already in use.

May 10 04:02:21 gp42-admin4 slapd[15634]: backend_startup_one
(type=bdb,
suffix="cn=accesslog"): bi_db_open failed! (-1)

May 10 04:02:21 gp42-admin4 slapd[15634]: slapd stopped.

May 10 04:02:22 gp42-admin4 slapd[4541]: slapd stopped.



When attempting to restart slapd on server2:

May 13 10:13:54 gp42-admin4 slapd[12085]: @(#) $OpenLDAP: slapd 2.4.40
(Sep 30 2014 16:49:45)
$#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl
dap
-2.4.40/servers/slapd

May 13 10:13:54 gp42-admin4 slapd[12085]: nss-ldap: do_open:
do_start_tls
failed:stat=-1

May 13 10:13:54 gp42-admin4 slapd[12085]: nss_ldap: reconnected to
LDAP server ldap://ldapserver1.example.com

May 13 10:13:56 gp42-admin4 slapd[12086]: slapd starting

May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=002 (4096)
Content Sync Refresh Required

May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=001 (4096)
Content Sync Refresh Required

May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_insert_key: c_put
id
failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)

May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_add 0xfc6:
parent
(cn=accesslog) insert failed: -30995

May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_delete_key: c_del
id
failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)

May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_delete 0xf50:
parent (cn=accesslog) delete failed: -30995

May 13 10:15:55 gp42-admin4 slapd[12106]: @(#) $OpenLDAP: slapd 2.4.40
(Sep 30 2014 16:49:45)
$#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl
dap
-2.4.40/servers/slapd

May 13 10:15:55 gp42-admin4 slapd[12106]: nss-ldap: do_open:
do_start_tls
failed:stat=-1

May 13 10:15:55 gp42-admin4 slapd[12106]: nss_ldap: reconnected to
LDAP server ldap://ldapserver1.example.com

May 13 10:15:55 gp42-admin4 slapd[12106]: bdb_db_open: database
"dc=example,dc=ldap": unclean shutdown detected; attempting recovery.

May 13 10:15:57 gp42-admin4 slapd[12106]: bdb_db_open: database
"cn=accesslog": unclean shutdown detected; attempting recovery.

May 13 10:15:58 gp42-admin4 slapd[12106]: slapd starting

May 13 10:28:49 gp42-admin4 slapd[12255]: @(#) $OpenLDAP: slapd 2.4.40
(Sep 30 2014 16:49:45)
$#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl
dap
-2.4.40/servers/slapd

May 13 10:28:49 gp42-admin4 slapd[12255]: nss-ldap: do_open:
do_start_tls
failed:stat=-1

May 13 10:28:49 gp42-admin4 slapd[12255]: nss_ldap: reconnected to
LDAP server ldap://ldapserver1.example.com

May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database
"dc=example,dc=com": unclean shutdown detected; attempting recovery.

May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database
"cn=accesslog": unclean shutdown detected; attempting recovery.

May 13 10:28:52 gp42-admin4 slapd[12255]: slapd starting

May 13 10:29:24 gp42-admin4 slapd[12264]: @(#) $OpenLDAP: slapd 2.4.40
(Sep 30 2014 16:49:45)
$#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl
dap
-2.4.40/servers/slapd

May 13 10:29:24 gp42-admin4 slapd[12264]: nss-ldap: do_open:
do_start_tls
failed:stat=-1

May 13 10:29:24 gp42-admin4 slapd[12264]: nss_ldap: reconnected to
LDAP server ldap://ldapserver1.example.com

May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database
"dc=example,dc=ldap": unclean shutdown detected; attempting recovery.

May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database
"cn=accesslog": unclean shutdown detected; attempting recovery.

May 13 10:29:24 gp42-admin4 slapd[12264]: slapd starting

May 13 10:29:53 gp42-admin4 slapd[12280]: @(#) $OpenLDAP: slapd 2.4.40
(Sep 30 2014 16:49:45)
$#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl
dap
-2.4.40/servers/slapd

May 13 10:29:53 gp42-admin4 slapd[12280]: nss-ldap: do_open:
do_start_tls
failed:stat=-1

May 13 10:29:53 gp42-admin4 slapd[12280]: nss_ldap: reconnected to
LDAP server ldap://ldapserver1.example.com

May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database
"dc=example,dc=ldap": unclean shutdown detected; attempting recovery.

May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database
"cn=accesslog": unclean shutdown detected; attempting recovery.

May 13 10:29:53 gp42-admin4 slapd[12280]: slapd starting

May 13 10:32:35 gp42-admin4 slapd[12345]: @(#) $OpenLDAP: slapd 2.4.40
(Sep 30 2014 16:49:45)
$#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl
dap
-2.4.40/servers/slapd



Attempting to restart slapd from the command-line:

5735ed50 slapd starting

5735ed50 => bdb_entry_get: ndn: "cn=accesslog"

5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"

5735ed50 bdb_idl_fetch_key: %cn=accesslog

5735ed50 bdb_idl_fetch_key: [b49d1940]

5735ed50 bdb_idl_fetch_key:

5735ed50 send_ldap_result: err=0 matched="" text=""

5735ed50 => bdb_entry_get: ndn: "dc=example,dc=com"

5735ed50 => bdb_entry_get: oc: "(null)", at: "contextCSN"

ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN
reqDeleteOldRDN reqNewSuperior entryCSN

ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN
reqDeleteOldRDN reqNewSuperior entryCSN

=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)

<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0

=> ldap_dn2bv(272)

<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0

=> ldap_dn2bv(272)

<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0

=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)

<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0

=> ldap_dn2bv(272)

<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0

=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)

<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0

=> ldap_dn2bv(272)

<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0

5735ed50 => bdb_entry_get: ndn: "uid=jdoe,ou=Users,dc=example,dc=com"

5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"

slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.

Aborted





I have run db_recover on the dbase(s) on ldapserver2 but to no avail.



Does anyone have any suggestions?



Thank you in advance for any assistance.







John D. Borresen (Dave)

Linux/Unix Systems Administrator

MIT  Lincoln Laboratory

Humanitarian Assistance and Disaster Relief (HADR) Systems

244 Wood St

Lexington, MA  02420

Email: john.borresen@ll.mit.edu






--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration A
division of Synacor, Inc





--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration
A division of Synacor, Inc