[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
need suggestion on indexing big directory
Hallo everybody,
I am trying to populate a directory (openldap-2.1.30, bdb-4.2.52-p2)
containing about 80000 entries. Each entry has about 50 attributes, most
of them must be indexed like
eq,pres,sub
some like eq,pres,sub,approx.
I am doing test on sun v100 (sparc2 600 MHz, 1Gb memory, only one
harddisk).
I have started with DB_CONFIG containing only set_cachesize parameter,
nothing about checkpoints and dbnosync in slapd.conf. I tried ldapadd,
with rather bad results: after adding ca. 1000 entries slapd went down
to ca. 5% CPU usage, rest was iowait. I checked iostat, it has shown
that I was writing ca. 3 Mb/sec with 3 seconds wait time. Just to be on
safe side I have installed all solaris 8 recommended patches, which
however did not change anything.
As next, I added
set_lg_dir
set_flags DB_LOG_AUTOREMOVE
to my DB_CONFIG and started to write log files on file system in RAM.
With checkpoint in slapd.conf I made sure that I was not running out of
memory due to log files.
Whole business went a bit further, however after sometime I came again
to the same result - everything what I was doing with slapd was iowait.
I have populated directory with just one index, shuted down slapd and
tried slapindex. Funny enough when using slapindex berkeley db ignored
presence of "set_flags DB_LOG_AUTOREMOVE" in my DB_CONFIG. It was just
creating one log file after another, which was rather bad for me as I
expect about 8 Gb of log files, 8 Gb which do not fit in any of mount
points that i have on this solaris box, and definitely not in memory. In
order to workaround this I have started db_checkpoint, which checkpoints
db every 100 MGb of data.
slapdindex was running rather well first 1 Gb of log files, however it
started to run rather slow afterwards, and I have noticed that db was
writing on disk like crazy, causing again huge iowait.
Thus, I will be really happy to get any suggestions on how can I index
my directory in such was that will take hours but not weeks to complete.
Thanx a lot in advance,
best regards, vadim tarassov.