RE: scope not ok errors on very large databases (ITS#3343)

[hyc@symas.com]

Armbrust, Daniel C. wrote:

> 
>Followup to the last post - when I reloaded my problematic database on openldap 2.2.17 using these parameters:
>
>#define BDB_IDL_DB_SIZE         (1<<17) /* 64K IDL on disk - dan modified to 128K */
>#define BDB_IDL_UM_SIZE         (1<<18) /* 128K IDL in memory - dan modified to 256K*/
>
>The problem that I reported in the initial post went away.  I am now able to view/search, etc, on this large node in my database.  This surprised me, because I didn't think that I had increased the size of the key enough to fix the problem.  It must be because my ~260,000ish entries are being split across multiple keys (I'm not sure why, the are almost all aliases)
>
>Am I likely to run into any other problems by using these larger values?  If not, is there a reason not to update openldap itself to use these larger values?
>  
>
The main issue is memory usage. Every IDL slot is 4 bytes, so 256K of 
them is 1024KB of memory. back-bdb preallocates a search stack for every 
thread; this stack is configurable in slapd.conf but defaults to 8 
chunks, so that's 8*1024KB = 8MB. Also, one or two of them may need to 
fit on the regular thread stack, as you already saw. All of this adds up 
quickly, especially if you have a large number of threads configured. 
These default values were chosen a long time ago, when slapd still 
defaulted to 32 threads (as opposed to 16 now), and were reasonable for 
a typical 32-bit machine. But obviously there's room here for tuning, 
and if you were to create a 64-bit build you'd probably want even larger 
sizes.

-- 
  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support