[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: BDB 4.3.21 DB_LOG_INMEMORY bug on Debian Woody [#11505] (fwd)

Here is sleepycat's analysis of the issue I've seen loading large data sets with 4.3.21. It looks like at least part of the issue then, is on our side.

--Quanah

------------ Forwarded Message ------------
Date: Monday, December 20, 2004 11:16 PM +1100
From: Michael Cahill <mjc@sleepycat.com>
To: Quanah Gibson-Mount <quanah@symas.com>
Cc: hyc@highlandsun.com, support@sleepycat.com, Keith Bostic <bostic@abyssinian.sleepycat.com>
Subject: Re: BDB 4.3.21 DB_LOG_INMEMORY bug on Debian Woody [#11505]

Hi Quanah,

I'm sorry we went quiet on you -- I got tied up with release tasks for
our XML product last week.

I've spent some time running slapadd with the data you provided, and I
now understand the issue.  In the process I discovered two bugs with
in-memory logs: one minor and one serious, but neither one was the
source of the problem.  A patch for the two issues is attached anyway,
and they will be fixed in our next release.

The problem is that OpenLDAP's IDL database maintains large sets of
duplicates (up to BDB_IDL_DB_SIZE, which is 64K).  When the maximum is
reached, the duplicate set is deleted using DB->del (at idl.c:718).
That operation creates many log records -- far more log data is required
than the volume of data being deleted (it happens to be 176 bytes per
record deleted, so 11MB in this case).

In the data set you sent, some lines end up causing this to happen
several times in one transaction, with the result that the volume of log
space required for those lines is *much* higher than others.

Reducing BDB_IDL_DB_SIZE to 4K instead of 64K allows the test to
complete with a modest log buffer (about 4MB, possibly smaller), but I
don't know enough about OpenLDAP to have any idea what that does to
general performance.

I hope this helps -- please let me know if you need any more
information.

Regards,
Michael.


--
Michael Cahill                          Sleepycat Software
mailto:mjc@sleepycat.com                http://www.sleepycat.com

---------- End Forwarded Message ----------



--
Quanah Gibson-Mount
Product Engineer
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
--- Begin Message --- 
Hi Quanah,

I'm sorry we went quiet on you -- I got tied up with release tasks for
our XML product last week.

I've spent some time running slapadd with the data you provided, and I
now understand the issue.  In the process I discovered two bugs with
in-memory logs: one minor and one serious, but neither one was the
source of the problem.  A patch for the two issues is attached anyway,
and they will be fixed in our next release.

The problem is that OpenLDAP's IDL database maintains large sets of
duplicates (up to BDB_IDL_DB_SIZE, which is 64K).  When the maximum is
reached, the duplicate set is deleted using DB->del (at idl.c:718).
That operation creates many log records -- far more log data is required
than the volume of data being deleted (it happens to be 176 bytes per
record deleted, so 11MB in this case).

In the data set you sent, some lines end up causing this to happen
several times in one transaction, with the result that the volume of log
space required for those lines is *much* higher than others.

Reducing BDB_IDL_DB_SIZE to 4K instead of 64K allows the test to
complete with a modest log buffer (about 4MB, possibly smaller), but I
don't know enough about OpenLDAP to have any idea what that does to
general performance.

I hope this helps -- please let me know if you need any more
information.

Regards,
Michael.


--
Michael Cahill                          Sleepycat Software
mailto:mjc@sleepycat.com                http://www.sleepycat.com

*** log/log.c	2004-10-19 02:44:49.000000000 +1000
--- log/log.c	2004-12-20 21:14:43.000000000 +1100
***************
*** 1288,1303 ****
  	DB_ASSERT(lp->db_log_inmemory);
  
  	/*
  	 * If transactions are enabled and we're about to fill available space,
  	 * update the active LSN and recheck.  If transactions aren't enabled,
  	 * don't even bother checking: in that case we can always overwrite old
  	 * log records, because we're never going to abort.
- 	 *
- 	 * Allow room for an extra header so that we don't need to check for
- 	 * space when switching files.
  	 */
  	while (TXN_ON(dblp->dbenv) &&
! 	    RINGBUF_LEN(lp, lp->b_off, lp->a_off) <= len + sizeof(HDR)) {
  		old_active_lsn = lp->active_lsn;
  		active_lsn = lp->lsn;
  
--- 1288,1306 ----
  	DB_ASSERT(lp->db_log_inmemory);
  
  	/*
+ 	 * Allow room for an extra header so that we don't need to check for
+ 	 * space when switching files.
+ 	 */
+ 	len += sizeof (HDR);
+ 
+ 	/*
  	 * If transactions are enabled and we're about to fill available space,
  	 * update the active LSN and recheck.  If transactions aren't enabled,
  	 * don't even bother checking: in that case we can always overwrite old
  	 * log records, because we're never going to abort.
  	 */
  	while (TXN_ON(dblp->dbenv) &&
! 	    RINGBUF_LEN(lp, lp->b_off, lp->a_off) <= len) {
  		old_active_lsn = lp->active_lsn;
  		active_lsn = lp->lsn;
  
*** log/log_get.c	2004-09-18 23:22:27.000000000 +1000
--- log/log_get.c	2004-12-20 16:13:57.000000000 +1100
***************
*** 453,462 ****
  			goto err;
  		if (rp != NULL)
  			goto cksum;
! 		if (lp->db_log_inmemory) {
! 			ret = DB_NOTFOUND;
! 			goto err;
! 		}
  	}
  
  	/*
--- 453,460 ----
  			goto err;
  		if (rp != NULL)
  			goto cksum;
! 		if (lp->db_log_inmemory)
! 			goto nohdr;
  	}
  
  	/*
***************
*** 525,531 ****
  	 * going to have to retry.
  	 */
  	if (hdr.len == 0) {
! 		switch (flags) {
  		case DB_FIRST:
  		case DB_NEXT:
  			/* Zero'd records always indicate the end of a file. */
--- 523,529 ----
  	 * going to have to retry.
  	 */
  	if (hdr.len == 0) {
! nohdr:		switch (flags) {
  		case DB_FIRST:
  		case DB_NEXT:
  			/* Zero'd records always indicate the end of a file. */

--- End Message ---