Re: slapadd and attribute value length limit

[Howard Chu <hyc@symas.com>]

Jon Roberts wrote:

> Howard Chu wrote:
>
> > Judging from the error message you received, since the BDB library 
> > ran out of memory you may be able to resolve this by increasing your 
> > BDB cache size. Also you may get some more explicit information from 
> > BDB if you enable debugging (-d1) while running slapadd.
>
>
> I tried this. Cache size wasn't the problem. The gotcha:
>
> bdb(o=mentata.com): Lock table is out of available locks
>
> So now with DB_CONFIG:
>
> set_lg_dir      /var/log/openldap
> set_lg_max      104857600
> set_lg_bsize    26214400
> set_lk_max_locks 2000
> set_lk_max_lockers 2000
> set_lk_max_objects 2000
> # set_cachesize 0 20971520 1
> set_cachesize 0 419430400 1
> set_flags       DB_TXN_NOSYNC
>
> Now the load completes fine in under an hour. Much better than loading 
> just the english with slapadd and then using ldapmodify LDIFs for each 
> additional language (hours plural).
>
> I know Quanah might add that I should now recomment:
>
> #set_flags       DB_TXN_NOSYNC
>
> Already done, along with the cache switch. Would I need this many 
> locks/lockers in a runtime scenario, or should I comment out those as 
> well?
slapadd stores entries one at a time in the database, and back-bdb 
basically uses one transaction per entry. Within this one transaction, 
it will acquire locks for the entry itself and all of the indexing 
records that are needed to write the entry. If your indexing has 
generated so many records that you exhausted the default of 1000 locks, 
then I expect you will need many more locks in a runtime scenario. Of 
course, this assumes that you're adding/modifying entries like this at 
runtime. I.e., if it takes over 1000 locks to write just one entry, and 
if you have the possibility of writing two or more entries at runtime, 
then you will need far more locks at runtime.

I strongly doubt that you needed to raise the max_lockers.

As to why you need so many locks - the substring indexing you're 
performing, with the long strings, is definitely hitting a large number 
of records in all of the index databases, so this lock situation is not 
too surprising.

--
 -- Howard Chu
 Chief Architect, Symas Corp.       Director, Highland Sun
 http://www.symas.com               http://highlandsun.com/hyc
 Symas: Premier OpenSource Development and Support