Re: hdb and mdb dereferencing aliases differently

If I may add some of my observations,

Problem (for me) is starting to show when I have more than 20000 users inserted in LMDB database.

I am testing with latest (almost) OpenLDAP compiled from source (git pull 24.04.2013).

Test was done by adding 10k users first then adding 5k at a time. Between 20k and 25k ldapsearch started to return only this:

root@test7kde:~# ldapsearch -x -h 172.17.103.200 -D cn=admin,dc=spr -s sub -a always -w secret -b pfUsername=user20000,dc=USERNAME,dc=spr
# extended LDIF
#
# LDAPv3
# base <pfUsername=user20000,dc=USERNAME,dc=spr> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# search result
search: 2
result: 0 Success

# numResponses: 1

I have attached ldapsearch results for reference.

You can see data structure from ldapsearch.

LMDB is configured like this:

dn: olcDatabase={1}mdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcMdbConfig
olcDatabase: {1}mdb
olcDbDirectory: /opt/openldap/var/openldap-data
olcSuffix: dc=spr
olcAccess: {0}to attrs=userPassword,shadowLastChange by self write by anonymou
s auth by dn="cn=admin,dc=spr" write by * none
olcAccess: {1}to attrs=shadowLastChange by self write by * read
olcAccess: {2}to dn.base="" by * read
olcAccess: {3}to * by self write by dn="cn=admin,dc=spr" write by * read
olcLastMod: TRUE
olcRootDN: cn=admin,dc=spr
olcRootPW:: e1NTSEF9eDRVRTBiV2Z4YnFSNnZDVDdKRWEwSWRhWFRhMDN2M3I=
olcDbCheckpoint: 4096 10
olcDbNoSync: TRUE
olcDbMaxSize: 38654705664
structuralObjectClass: olcMdbConfig
entryUUID: 804a8ede-2cc3-1032-9b7a-c7b2eb845e03
creatorsName: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
createTimestamp: 20130329135129Z
olcDbIndex: objectClass eq
olcDbIndex: uid eq
olcDbIndex: MSISDN eq
olcDbIndex: IMSI eq
olcDbIndex: pfUsername eq
olcDbIndex: entryCSN eq
olcDbIndex: entryUUID eq
olcDbIndex: contextCSN eq
olcMirrorMode: TRUE
olcSyncrepl: {0}rid=003 provider=ldap://spr1.lab.os binddn="cn=admin,dc=spr" b
indmethod=simple credentials=siemens searchbase="dc=spr" type=refreshOnly int
erval=00:00:00:10 retry="5 5 300 5" timeout=1
olcSyncrepl: {1}rid=004 provider=ldap://spr2.lab.os binddn="cn=admin,dc=spr" b
indmethod=simple credentials=siemens searchbase="dc=spr" type=refreshOnly int
erval=00:00:00:10 retry="5 5 300 5" timeout=1
entryCSN: 20130329141637.442296Z#000000#002#000000
modifiersName: cn=admin,cn=config
modifyTimestamp: 20130329141637Z

If you need more data I wiil be more than happy to provide.

On Wed, Apr 24, 2013 at 11:06 AM, Howard Chu <hyc@symas.com> wrote:

Juergen.Sprenger@swisscom.com wrote:

Hi Michael,

NSS results must not be dependent on the backend database a directory service uses.

I activated connection logging and here's the proof that NSS is not the culprit.

Searches initiated by NSS are identical and exactly this behavior can also be seen when using ldapsearch from command line with parameters from the log:

# running 'getent passwd' with hdb backend:
Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=0 BIND dn="cn=itsAgent,ou=customerAgent,dc=scom" method=128
Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=0 BIND dn="cn=itsAgent,ou=customerAgent,dc=scom" mech=SIMPLE ssf=0
Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=0 RESULT tag=97 err=0 text=
Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=1 SRCH base="ou=account,dc=its,dc=scom" scope=1 deref=3 filter="(objectClass=posixAccount)"
Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=1 SRCH attr=uid userPassword uidNumber gidNumber cn homeDirectory x-LinuxLoginShell gecos description objectClass
Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=1 SEARCH RESULT tag=101 err=0 nentries=656 text=
Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 fd=13 closed (connection lost)

# running 'getent passwd' with mdb backend:
Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=0 BIND dn="cn=itsAgent,ou=customerAgent,dc=scom" method=128
Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=0 BIND dn="cn=itsAgent,ou=customerAgent,dc=scom" mech=SIMPLE ssf=0
Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=0 RESULT tag=97 err=0 text=
Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=1 SRCH base="ou=account,dc=its,dc=scom" scope=1 deref=3 filter="(objectClass=posixAccount)"
Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=1 SRCH attr=uid userPassword uidNumber gidNumber cn homeDirectory x-LinuxLoginShell gecos description objectClass
Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=1 SEARCH RESULT tag=101 err=0 text=
Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 fd=13 closed (connection lost)

I suspect that aliases are not handled the same way in hdb and mdb as I am using aliases here and deref=3 in both searches, example:

dn: uid=joe,ou=Account,dc=its,dc=scom
objectClass: alias
objectClass: extensibleObject
uid: joe
aliasedObjectName: uid=joe,ou=Person,dc=its,dc=scom
structuralObjectClass: alias

When using hdb, the alias is dereferenced correctly (nentries=656), when using mdb it seems not to be dereferenced at all (nentries=0).

Maybe there's a parameter around for mdb which I couldn't find in the docs

to prevent this, but if not I consider this as a bug.

There is no parameter. Seems like you've found a bug, please submit the info to the ITS.

Regards

Juergen

michael@stroeder.com wrote:

It would certainly help if you could examine the issue with pure LDAP search
operations preferably with OpenLDAP's ldapsearch command-line tool.

When looking at NSS results too many things can go wrong with other
components' configuration.

Ciao, Michael.

Juergen.Sprenger@swisscom.com wrote:

Hi,

I have running OpenDLAP 2.4.35 on Gentoo Linux and wanted to make some tests with mdb.

Slapd was running fine with hdb, no problems so far.
Then I exported contents via slapcat and switched config to mdb.
When slapd started using mdb no users from directory were shown by 'getent passwd':

### hdb part ####
# using hdb parameters
database hdb
dirtyread
cachesize 150000
cachefree 100
idlcachesize 450000
dncachesize 100000

# slapadd from backup and run slapd with hdb backend
/etc/init.d/unscd stop
/etc/init.d/slapd stop
rm /var/lib/openldap-data/*
rm -rf /etc/openldap/slapd.d/*
cp -p /etc/openldap/DB_CONFIG /var/lib/openldap-data/
cp -p /etc/openldap/slapd.conf.hdb /etc/openldap/slapd.conf
su ldap -c '/usr/sbin/slapadd -f /etc/openldap/slapd.conf -l odsldap-dev.ldif'
/etc/init.d/slapd start
/etc/init.d/unscd start
slapcat -f /etc/openldap/slapd.conf -b dc=scom | md5sum
# 73850f9a3f7ff9d3d1ddb7663cd046a6 -

getent passwd
# all users shown, everything ok

### mdb part ####
# using mdb paramters
database mdb
dbnosync
maxsize 2094967296
searchstack 64

# slapadd from backup and run slapd with mdb backend
/etc/init.d/unscd stop
/etc/init.d/slapd stop
rm /var/lib/openldap-data/*
rm -rf /etc/openldap/slapd.d/*
cp -p /etc/openldap/slapd.conf.mdb /etc/openldap/slapd.conf
su ldap -c '/usr/sbin/slapadd -f /etc/openldap/slapd.conf -l odsldap-dev.ldif'
/etc/init.d/slapd start
/etc/init.d/unscd start
slapcat -f /etc/openldap/slapd.conf -b dc=scom | md5sum
# 73850f9a3f7ff9d3d1ddb7663cd046a6 -

<>

getent passwd
# no users from ldap shown

Am I missing something when setting up and using mdb?
Both backends have exactly the same content, and so the results for searches should also be identical.

Regards

Jürgen Sprenger

--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/