"Blank" node in multi-master OpenLDAP 2.4.21 setup (findbase failed)

[Mark Cairney <Mark.Cairney@ed.ac.uk>]

Hi,

Apache Directory Studio appears to have FUBAR'ed one of the nodes in our multi-master OpenLDAP setup and I'd appreciate some help or pointers.

Im running OpenLDAP 2.4.21 and BDB 4.8.26 with Kerberos 5 and GSSAPI SASL.

On the "afflicted" node the DIT is empty other than the Root DSE and ldapsearch returns "32" no such object.

The logs contain the following:

** ld 0x2aab4c4b7c70 Outstanding Requests:
connection_get(44): got connid=1095
connection_read(44): checking for input on id=1095
 * msgid 4,  origid 4, status InProgress
   outstanding referrals 0, parent count 0
ldap_pvt_sasl_generic_install
  ld 0x2aab4c4b7c70 request count 1 (abandoned 0)
** ld 0x2aab4c4b7c70 Response Queue:
ber_get_next
   Empty
  ld 0x2aab4c4b7c70 response count 0
ldap_chkResponseList ld 0x2aab4c4b7c70 msgid 4 all 0
ldap_chkResponseList returns ld 0x2aab4c4b7c70 NULL
ldap_int_select
ber_get_next: tag 0x30 len 292 contents:
op tag 0x63, time 1286205781
ber_get_next
conn=1095 op=3 do_search
ber_scanf fmt ({miiiib) ber:
>>> dnPrettyNormal: <dc=authorise,dc=ed,dc=ac,dc=uk>
<<< dnPrettyNormal: <dc=authorise,dc=ed,dc=ac,dc=uk>, <dc=authorise,dc=ed,dc=ac,dc=uk>
ber_scanf fmt (m) ber:
ber_scanf fmt ({M}}) ber:
=> get_ctrls
ber_scanf fmt ({m) ber:
ber_scanf fmt (m) ber:
=> get_ctrls: oid="1.3.6.1.4.1.4203.1.9.1.1" (noncritical)
ber_scanf fmt ({i) ber:
ber_scanf fmt (m) ber:
ber_scanf fmt (b) ber:
ber_scanf fmt (}) ber:
ber_scanf fmt ({m) ber:
ber_scanf fmt (b) ber:
=> get_ctrls: oid="2.16.840.1.113730.3.4.2" (critical)
<= get_ctrls: n=2 rc=0 err=""
==> limits_get: conn=1095 op=3 self="uid=replicator.authorise.is.ed.ac.uk,ou=people,ou=central,dc=authorise,dc=ed,dc=ac,dc=uk" this="dc=authorise,dc=ed,dc=ac,dc=uk"
=> bdb_search
bdb_dn2entry("dc=authorise,dc=ed,dc=ac,dc=uk")
bdb_dn2entry("cn=admins,ou=group,ou=central,dc=authorise,dc=ed,dc=ac,dc=uk")
bdb_entry_get: rc=0
send_ldap_result: conn=1095 op=3 p=3
findbase failed! 32
send_ldap_result: conn=1095 op=3 p=3
send_ldap_response: msgid=4 tag=101 err=32
ber_flush2: 14 bytes to sd 44
connection_get(43): got connid=0
=>do_syncrepl rid=030
=>do_syncrep2 rid=030
ldap_result ld 0x2aab4c4b7c70 msgid 4
wait4msg ld 0x2aab4c4b7c70 msgid 4 (timeout 0 usec)
wait4msg continue ld 0x2aab4c4b7c70 msgid 4 all 0
** ld 0x2aab4c4b7c70 Connections:
* host: alder.authorise.is.ed.ac.uk  port: 636  (default)
  refcnt: 2  status: Connected
  last used: Mon Oct  4 16:23:01 2010


** ld 0x2aab4c4b7c70 Outstanding Requests:
 * msgid 4,  origid 4, status InProgress
   outstanding referrals 0, parent count 0
  ld 0x2aab4c4b7c70 request count 1 (abandoned 0)
** ld 0x2aab4c4b7c70 Response Queue:
   Empty
  ld 0x2aab4c4b7c70 response count 0
ldap_chkResponseList ld 0x2aab4c4b7c70 msgid 4 all 0
ldap_chkResponseList returns ld 0x2aab4c4b7c70 NULL
ldap_int_select
read1msg: ld 0x2aab4c4b7c70 msgid 4 all 0
ber_get_next
ber_get_next: tag 0x30 len 12 contents:
read1msg: ld 0x2aab4c4b7c70 msgid 4 message type search-result
ber_scanf fmt ({eAA) ber:
read1msg: ld 0x2aab4c4b7c70 0 new referrals
read1msg:  mark request completed, ld 0x2aab4c4b7c70 msgid 4
request done: ld 0x2aab4c4b7c70 msgid 4
res_errno: 32, res_error: <>, res_matched: <>
ldap_free_request (origid 4, msgid 4)
ldap_parse_result
ber_scanf fmt ({iAA) ber:
ber_scanf fmt (}) ber:
ldap_err2string
do_syncrep2: rid=030 LDAP_RES_SEARCH_RESULT (32) No such object
ldap_err2string
ldap_err2string
do_syncrep2: rid=030 (32) No such object
ldap_err2string
ldap_msgfree
connection_get(43): got connid=0
ldap_free_connection 1 1
ldap_send_unbind
ber_flush2: 7 bytes to sd 43
connection_get(44): got connid=1095
connection_read(44): checking for input on id=1095
ber_get_next
TLS trace: SSL3 alert write:warning:close notify
ldap_free_connection: actually freed
ber_get_next: tag 0x30 len 5 contents:
op tag 0x42, time 1286205781
ber_get_next
do_syncrepl: rid=030 rc -2 retrying
TLS trace: SSL3 alert read:warning:close notify
ber_get_next on fd 44 failed errno=0 (Success)
conn=1095 op=4 do_unbind
connection_close: conn=1095 sd=44
TLS trace: SSL3 alert write:warning:close notify

So far to fix it I've tried running slapd with the "-c rid=" option, deleting the contents of /var/openldap-data, running db_verify and db_recover (with and without the -c flag) and doing a slapadd from one of the other working nodes but nothing has worked.

Interestingly the file sizes in /var/openldap-data/authorise look OK but the LDAP tree appears to have vanished without a trace.

Any ideas?

Kind regards,

Mark


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.