Re: BDB corruption on windows port of 2.2.19

[Safdar Kureishy <safdar.kureishy@gmail.com>]

Hi,

As a follow up, I found that the db_recover utility works in certain
cases, but it doesn't fix one particular issue, as below:

After an abnormal server shutdown (on Windows), I have seen this
problem occur where restart of the OpenLDAP service works fine.
However, there is data loss which cannot be recovered. If I run the
db_recover utility on the bdb files under OpenLDAP, the surprising
thing is that even more data gets deleted! I found that pretty
strange. I tried using the "-c" option for db_recover after the first
attempt without -c, and that didn't help either.

That's not it. It seems that only certain data (i.e subtrees) are lost
every time - I mean, a certain subtree of data is lost, and no other
part of the directory is affected. Each time this problem happens, it
is the same subtree that is affected. Also, if I try to re-import the
data that was lost (I had a backup of that data), I cannot start the
service after that, with error code 21.

If I try to run db_recover again, again the data that I just imported
gets deleted when I launch the service next. Mind you this data was
succesfully imported before the system shutdown occured, and I was
able to browse the data correctly. Now, if I delete all the .bdb files
and setup all the data from scratch, then it all works fine again.

I was wondering if anyone had seen something like this before. Any
insight or advise about this would be greatly appreciated.

Thanks,
Safdar

On 7/25/05, Safdar Kureishy <safdar.kureishy@gmail.com> wrote:
> Thanks Quanah. I have already started using db_recover to solve this
> problem after it happens. However, I was hoping to find out what
> configuration settings I could use to prevent the problem from
> happening in the first place. I was told that with appropriate
> configuration settings for the bdb back-end (hopefully via slapd.conf
> itself), it would be possible to avoid the data corruption issue
> alltogether in version 2.2.x...
> 
> Regards,
> Safdar
> 
> On 7/22/05, Quanah Gibson-Mount <quanah@stanford.edu> wrote:
> >
> >
> > --On Friday, July 22, 2005 4:42 PM -0700 Safdar Kureishy
> > <safdar.kureishy@gmail.com> wrote:
> >
> >
> > > My usage pattern is as follows:
> > > - At install time, the installer for our product sets up OpenLDAP and
> > > imports some seed data into it (using slapadd), under a specific base
> > > dn. This imported data (LDIF file) could be pretty large. This same
> > > import will happen again at a periodic rate of X hours (most likely 24
> > > hours). Each time the import occurs, it involves the deletion of the
> > > previous imported data (if any) using ldapdelete, followed by
> > > re-import of the updated LDIF data file (using slapadd), followed by a
> > > re-start of the OpenLDAP server.
> > > - Apart from this import, there will be very rare
> > > modification/addition of entries in a different subtree. This is
> > > anticipated to involve very little data and the server will not be
> > > restarted during this type of access.
> > > - The rest of the time, the directory is only accessed for running
> > > searches across all the data contained under it.
> > >
> > > Given this usage pattern, would you or anyone else be able to
> > > suggested some appropriate bdb configuration settings (inside
> > > slapd.conf) that I could use to prevent data corruption from an
> > > abnormal system shutdown?
> >
> > You must run db_recover after an abnormal shutdown before starting slapd.
> > In OpenLDAP 2.3, this is done for you automatically.  I have no idea what
> > Windows does in the case of abnormal shutdowns vs. Unix however.  Perhaps
> > it just permanently corrupts things.
> >
> > --Quanah
> >
> >
> > --
> > Quanah Gibson-Mount
> > Principal Software Developer
> > ITSS/Shared Services
> > Stanford University
> > GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
> >
> > "These censorship operations against schools and libraries are stronger
> > than ever in the present religio-political climate. They often focus on
> > fantasy and sf books, which foster that deadly enemy to bigotry and blind
> > faith, the imagination." -- Ursula K. Le Guin
> >
> >
>