[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
RE: Corrupt index files
Good day,
Despite the undeserved criticism, thank you for your reply.
> > > > Do you get these problems if you
> > > > use the ldbm
> > > > backend in 2.0.25 with BDB?
> > >
> > > Yes. I got exactly the same results with 2.0.21, 2.0.23 and
> > > 2.0.25 (yes, I
> > > have been trying very, very hard to get something working).
>
> You apparently don't read documentation very well.
It appears that you have not read and understood the entire thread and/or
the message you replied to. I will refrain from making a derogatory
comment, though. Comments follow.
> While the database formats generally don't change, this is
> not the documented
> way to migrate databases between software upgrades. You
> should always dump
> the
> database into LDIF format before upgrading to a new version.
That's precisely what I did when migrating between the backend versions. I
described the procedure in great detail in this thread. Thanks to John
Dalbec for pointing out possible pitfalls in the procedure.
> On little-endian (e.g., Intel) boxes, the byte-order of index
> keys changed
> between 2.0 and 2.1, so using your gdbm backups between 2.0
> and 2.1 will
> fail.
Please note again that I was rolling back to the original version of the
server that I had used (i.e. the one that generated those files). That's
the only reason I considered using my backup db files. I also described
that very well. I am not sure why you even mentioned 2.1.X ; I have not
even tried it yet.
> Since other people have reported indexing problems with gdbm
> that disappeared
> when switching to BDB, I am very skeptical about your
> conclusion. Since you
> claim to never have used slapadd before, then it's clear that
> you have never
> actually executed a slapd with BDB running in the backend,
As stated in this message and previous posts to this thread, slapadd (which
I presume you meant) has actually been run many, many, many times on both
backend types in an effort to get something to work.
I had never before used slapadd on THIS system with THIS version. I have
used it on a prior project- however that was over a year ago, on Solaris
instead of Linux, using bdb, with a much earlier version of OpenLDAP. I did
not have a problem with it then.
Additionally, please note that I have reported a problem that is
significantly different from the problems that "other people" have reported.
I would guess that if I were to get a proper set of indexes when I do an
import, I would not encounter my original index-corruption problem again
with bdb while the server is running.
[snip]
> Both slapadd and slapindex are additive programs. They will
> not overwrite
> data that already exists, they will only insert new data. If
> you want to
> recreate your indices from scratch, delete everything but the
> id2entry and
> the dn2id databases, then run slapindex.
Thanks for this information. I am not sure if your reindexing procedure is
correct, though. See results below.
Testing yesterday afternoon turns out slapindex does change my index files,
though. And, I think I have actually identified my problem (the problem
that causes the import indexes to go bad). In all of my tests, I've run
slapindex after a slapadd. Why? That's how I was instructed to do it from
this mailing list away-back when I first started to use OpenLDAP.
http://www.openldap.org/lists/openldap-software/200111/msg00373.html
Here's the results from my test. The problem does not appear to be adding,
but indexing afterwards.
- Start with no database files and slapd off.
- Populate the database:
/usr/sbin/slapadd -f /etc/openldap/slapd.conf < backup.ldif
( no errors )
- Change the owner of the files to the ldap owner
chown ldap:ldap *.dbb
(files now owned by ldap user)
- Start slapd (starts OK)
- search (objectclass=device)
# search result
search: 2
result: 0 Success
# numResponses: 2467
# numEntries: 2466
( All entries are correct and distinct. Yay! )
- shut down slapd (shuts down OK)
- run slapindex (This SHOULD not change anything; nothing has been changed).
/usr/sbin/slapindex -f /etc/openldap/slapd.conf
( completes OK, no new files )
- start slapd (starts OK)
- search (objectclass=device)
# search result
search: 2
result: 0 Success
# numResponses: 2603
# numEntries: 2602
( Uh oh. 136 new entries. They are 136 distinct entires that have
been duplicated once.)
- stop slapd (stops OK)
- run slapindex (This SHOULD not change anything; nothing has been changed).
/usr/sbin/slapindex -f /etc/openldap/slapd.conf
( completes OK, no new files )
- start slapd (starts OK)
- search (objectclass=device)
# search result
search: 2
result: 0 Success
# numResponses: 2881
# numEntries: 2880
( 278 (136*2) additional entries. The same 136 entries duplicated
before have been duplicated again, plus some other entries have been
duplicated. )
** Now testing index recreation
- Remove database and recreate it
rm -f *.dbb
/usr/sbin/slapadd -f /etc/openldap/slapd.conf < backup.ldif
(completes OK)
- Change the owner of the files to the ldap owner
chown ldap:ldap *.dbb
(files now owned by ldap user)
- start slapd (starts OK)
- search (objectclass=device)
# search result
search: 2
result: 0 Success
# numResponses: 2467
# numEntries: 2466
(back to normal)
- stop slapd (stops OK)
- destroy index files, as instructed
mv id2entry.dbb id2entry.dbb.bak
mv dn2id.dbb dn2id.dbb.bak
rm -f *.dbb
mv id2entry.dbb.bak id2entry.dbb
mv dn2id.dbb.bak dn2id.dbb
(all complete OK)
- run slapindex
/usr/sbin/slapindex -f /etc/openldap/slapd.conf
( completes OK, new files created )
- Change the owner of the files to the ldap owner
chown ldap:ldap *.dbb
(files now owned by ldap user)
- start slapd (starts OK)
- search (objectclass=device)
# search result
search: 2
result: 0 Success
# numResponses: 1
(No results (?!?). No errors in ldap log.).
- Remove database and recreate it
rm -f *.dbb
/usr/sbin/slapadd -f /etc/openldap/slapd.conf < backup.ldif
(completes OK)
- Change the owner of the files to the ldap owner
chown ldap:ldap *.dbb
(files now owned by ldap user)
- start slapd (starts OK)
- search (objectclass=device)
# search result
search: 2
result: 0 Success
# numResponses: 2467
# numEntries: 2466
(back to normal)
- stop slapd (stops OK)
- destroy index files using alternate procedure (keep the nextid.dbb file
too)
mv id2entry.dbb id2entry.dbb.bak
mv dn2id.dbb dn2id.dbb.bak
mv nextid.dbb nextid.dbb.bak
rm -f *.dbb
mv id2entry.dbb.bak id2entry.dbb
mv dn2id.dbb.bak dn2id.dbb
mv nextid.dbb.bak nextid.dbb
(all complete OK)
- run slapindex
/usr/sbin/slapindex -f /etc/openldap/slapd.conf
( completes OK, new files created )
- Change the owner of the files to the ldap owner
chown ldap:ldap *.dbb
(files now owned by ldap user)
- start slapd (starts OK)
- search (objectclass=device)
# search result
search: 2
result: 0 Success
# numResponses: 2467
# numEntries: 2466
(still normal. Huzzah!)
Mr. Chu, this is not meant to be offensive in any way, but, are you certain
your procedure is the correct one? The second procedure I detailed seems to
work fine while yours does not. But, considering the other problems that I
have been having with slapindex, I can not say for certain.
Now, this information gives me a solution to my first problem (use bdb
instead of gdbm) and a workaround to my second problem (don't run slapindex
unless all of the index files have been removed), so I will do further
testing with that server/backend combination.
In the event that I consider it, though, could someone please provide some
advice on what to expect if I were to upgrade to OpenLDAP 2.1.X?
Mr. Chu, I feel that an apology is in order. I realize that there are many
problems caused by individuals who do not read the documentation and who
deserve the response that you gave, but this has not yet happened here, and
you would have known that if you would have read the message and thread more
carefully before replying.
Thanks again to everyone for their help, including Mr. Chu .
============================
Darren Gamble
Planner, Regional Services
Shaw Cablesystems GP
630 - 3rd Avenue SW
Calgary, Alberta, Canada
T2P 4L4
(403) 781-4948