Re: Problem with back-mdb

Hello,

thanks for the reply.

The script works fine with the use of a 2nd connection for the modify request but ... during the execution of my script, the file data.mdb grows up considerably.

First time, I configured maxsize value with a 3GB in slapd.conf.

Then I modified the script to use 2 connections.

After that, I ran the script. Everything went right until and I was confronted to a MDB_MAP_FULL error in ldap.log.

Indeed, data.mdb reaches the max size.

So, I stopped slapd, I modified the maxsize to 5Gb value in slapd.conf and then, I restarted slapd.

I ran the script again and I checked the size of data.mdb.

Before runnig the script, the real size of data.mdb is about 700M.

Once the script finished, the real size of data.mdb is about 4.5G.

During execution, memory consumption of slapd increases up progressively to reach 83% at the end of execution.

Of course, if I make a slapcat then a slapadd, the size of data.mdb return to 696M.

I run mdb_stat -f before the use of script :

mdb_stat -f /var/lib/ldap

Freelist Status

Tree depth: 1

Branch pages: 0

Leaf pages: 1

Overflow pages: 35

Entries: 14

Free pages: 17300

Status of Main DB

Tree depth: 1

Branch pages: 0

Leaf pages: 1

Overflow pages: 0

Entries: 25

And then after the execution of script :

mdb_stat -f /var/lib/ldap

Freelist Status

Tree depth: 3

Branch pages: 12

Leaf pages: 2418

Overflow pages: 7

Entries: 55122

Free pages: 986942

Status of Main DB

Tree depth: 1

Branch pages: 0

Leaf pages: 1

Overflow pages: 0

Entries: 25

After some search, I found the message "(ITS#7565) unacceptable growth on delete with back-mdb" which looks like my problem.

Have you got any suggestions ?

Thanks,

Grégory Trucy

2014/1/6 Howard Chu <hyc@symas.com>

Grégory T wrote:

Hello,

before starting, we wish a happy new year to the community !

Now, we start with presentation.
We have compiled the Openldap build version 2.4.38 on a ubuntu 12.04.
This openldap server is a dual processor Intel Xeon with 4 Gb Memory. This
machine is virtualized (vmware).
Openldap is configured with back-mdb, overlays memberof and syncprov, and
envflags writemap and nosync.

We encounter a problem with a callback procedure in a perl script.
This script have to remove a couple of attributes for each entry (~120 000).
In order to save memory consumption, we use a callback function to treat each
entry one after one.

Everything works with back-bdb but ... there are mistakes with back-mdb.

In ldap.log, this kind of message appears many times :
ldapserver slapd[7264]: connection_input: conn=1014 deferring operation:
awaiting write
ldapserver last message repeated 55 times

This is an interesting problem. The above message indicates that a slapd thread on that connection has been blocked writing a result to the client. I.e., back-mdb is so much faster than back-bdb that it's returning results faster than your client can read them. Once it gets blocked like this, it defers any new requests on that connection until it gets unblocked. (I.e., it waits until the client catches up on reading.)

And suddenly,
ldapserver slapd[7264]: conn=1014 fd=14 closed (connection lost)

connection lost actually means the client closed the connection without performing an Unbind.

Our script result in "connection reset by peer" message.
The script stop around the 1040 entry.

Although the connection was closed by server, it still continue to run.
However, no explicit message in ldap.log.

If we run the script once again, the next ~1040 entries will be treated and
the script stop.

We have try with envflags writemap and nosync deactivated ... but unsuccessful.

This is the ldapsearch request :
my
$result=$ldapMaster->search(base=>$baseDN,scope=>'sub',filter=>$filter,typesonly=>'1',callback=>\&callback);

And this is the callback function :
sub callback{
my ($search,$ent)=@_;
return unless defined $ent;
my $entry = $search->shift_entry();
my @tabAttribToDel=();
if (defined $entry){
my $dn=$entry->dn();
$countDN++;
foreach my $attrib($entry->attributes){
if ($attrib ~~ @tabOldAttrib){
$countAttrib++;
push (@tabAttribToDel,$attrib);
}
}
if (($SUPPR==1) && (@tabAttribToDel)) {
$countTrace++;
my $result = $ldapMaster->modify($dn,delete => \@tabAttribToDel);
warn "(nbEntry:$countTrace, nbAttr:$countAttrib) Error with $dn:
".$result->error."\n" if $result->code;
}
}
}

Why is this script working well with back-bdb but not with back-mdb ?

Working with back-bdb because it's slow enough that your client never falls behind. With back-mdb your perl script is too slow to keep up.

One workaround, the avoid the deferring - use a 2nd connection for your modify requests.

Thanks,

--
Grégory Trucy
Ingénieur système et réseau
Aix-Marseille Université

--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/