[Date Prev][Date Next] [Chronological] [Thread] [Top]

RE: checkpointing BDB transactions and deleting BDB transaction l ogfiles



This particular topic has been discussed quite a bit on this list.  

Here are a couple of links:


http://www.stanford.edu/services/directory/openldap/configuration/bdb-build-
42.html
http://www.stanford.edu/services/directory/openldap/configuration/openldap-b
uild-42.html

http://www.openldap.org/lists/openldap-devel/200407/msg00068.html

It comes down to applying a couple of custom patches to bdb-5.2.52 and
openldap 2.2.>=15.  I believe this problem goes away with bdb 4.3, but there
seem to be other issues with using that release.

John Fortin
PBG Middleware and Web Services
(914) 767-7844


>-----Original Message-----
>From: owner-openldap-software@OpenLDAP.org 
>[mailto:owner-openldap-software@OpenLDAP.org] On Behalf Of 
>Bernhard Erdmann
>Sent: Tuesday, March 29, 2005 6:07 AM
>To: openldap-software@OpenLDAP.org
>Subject: checkpointing BDB transactions and deleting BDB 
>transaction logfiles
>
>
>Hi,
>
>an OpenLDAP server (Linux, OpenLDAP 2.2.24, BDB 4.2.52 + 2 patches by 
>Sleepycat, bdb backend) writes lots of BDB transaction logfiles. The 
>data gets pushed on a regular basis into the directory and is deleted 
>after a while based on createTimestamp. I don't have long running 
>sessions. New data is just quickly pushed into it by a PHP driven web 
>application.
>
>The transaction logfiles do not get removed by "db_archive 
>-d". Usually 
>they don't, sometimes they do, mainly after a server restart.
>
>I already tried
>
>database        bdb
>checkpoint      512 20
>
>in /etc/openldap/slapd.conf, but it does not work as expected. 
>Should I 
>use BDB's own checkpointing utility?
>
>http://www.sleepycat.com/docs/utility/db_checkpoint.html
>
>Yet to notice: the process lists holds several long running 
>slapds (for 
>two days now).
>
>Should I use OpenLDAP's monitor backend to get an idea what all these 
>slapds are doing?
>
>
>OpenLDAP backend directory:
>
>total 1729588
>-rw-------    1 ldap     ldap       684032 2005-03-29 12:58 
>createTimestamp.bdb
>-rw-------    1 ldap     ldap         8192 2005-03-17 08:31 __db.001
>-rw-------    1 ldap     ldap       270336 2005-03-17 08:31 __db.002
>-rw-------    1 ldap     ldap        98304 2005-03-17 08:31 __db.003
>-rw-------    1 ldap     ldap       368640 2005-03-17 08:31 __db.004
>-rw-------    1 ldap     ldap        24576 2005-03-17 08:31 __db.005
>-rw-------    1 ldap     ldap     64106496 2005-03-29 12:58 dn2id.bdb
>-rw-------    1 ldap     ldap     208961536 2005-03-29 12:58 
>id2entry.bdb
>-rw-------    1 ldap     ldap     10475445 2005-03-27 17:15 
>log.0000000564
>-rw-------    1 ldap     ldap     10485736 2005-03-27 17:41 
>log.0000000565
>-rw-------    1 ldap     ldap     10485731 2005-03-27 18:06 
>log.0000000566
>-rw-------    1 ldap     ldap     10485711 2005-03-27 18:18 
>log.0000000567
>-rw-------    1 ldap     ldap     10485730 2005-03-27 18:24 
>log.0000000568
>-rw-------    1 ldap     ldap     10485737 2005-03-27 18:29 
>log.0000000569
>-rw-------    1 ldap     ldap     10485305 2005-03-27 18:34 
>log.0000000570
>-rw-------    1 ldap     ldap     10470369 2005-03-27 18:39 
>log.0000000571
>-rw-------    1 ldap     ldap     10485501 2005-03-27 18:44 
>log.0000000572
>-rw-------    1 ldap     ldap     10485400 2005-03-27 18:49 
>log.0000000573
>-rw-------    1 ldap     ldap     10485669 2005-03-27 18:54 
>log.0000000574
>-rw-------    1 ldap     ldap     10477015 2005-03-27 18:59 
>log.0000000575
>-rw-------    1 ldap     ldap     10485598 2005-03-27 19:04 
>log.0000000576
>-rw-------    1 ldap     ldap     10485720 2005-03-27 19:09 
>log.0000000577
>-rw-------    1 ldap     ldap     10485559 2005-03-27 19:14 
>log.0000000578
>-rw-------    1 ldap     ldap     10485750 2005-03-27 19:19 
>log.0000000579
>-rw-------    1 ldap     ldap     10485145 2005-03-27 19:24 
>log.0000000580
>-rw-------    1 ldap     ldap     10485746 2005-03-27 19:29 
>log.0000000581
>-rw-------    1 ldap     ldap     10485705 2005-03-27 19:34 
>log.0000000582
>-rw-------    1 ldap     ldap     10485721 2005-03-27 19:39 
>log.0000000583
>-rw-------    1 ldap     ldap     10485746 2005-03-27 19:44 
>log.0000000584
>-rw-------    1 ldap     ldap     10485733 2005-03-27 19:49 
>log.0000000585
>-rw-------    1 ldap     ldap     10485646 2005-03-27 19:54 
>log.0000000586
>-rw-------    1 ldap     ldap     10485726 2005-03-27 19:59 
>log.0000000587
>-rw-------    1 ldap     ldap     10485743 2005-03-27 20:04 
>log.0000000588
>-rw-------    1 ldap     ldap     10485304 2005-03-27 20:09 
>log.0000000589
>-rw-------    1 ldap     ldap     10485719 2005-03-27 20:14 
>log.0000000590
>-rw-------    1 ldap     ldap     10485748 2005-03-27 20:19 
>log.0000000591
>-rw-------    1 ldap     ldap     10485344 2005-03-27 20:24 
>log.0000000592
>-rw-------    1 ldap     ldap     10485510 2005-03-27 20:29 
>log.0000000593
>-rw-------    1 ldap     ldap     10485665 2005-03-27 20:34 
>log.0000000594
>-rw-------    1 ldap     ldap     10485743 2005-03-27 20:39 
>log.0000000595
>-rw-------    1 ldap     ldap     10469499 2005-03-27 20:44 
>log.0000000596
>-rw-------    1 ldap     ldap     10485372 2005-03-27 20:49 
>log.0000000597
>-rw-------    1 ldap     ldap     10483551 2005-03-27 20:54 
>log.0000000598
>-rw-------    1 ldap     ldap     10485667 2005-03-27 20:58 
>log.0000000599
>-rw-------    1 ldap     ldap     10485752 2005-03-27 21:03 
>log.0000000600
>-rw-------    1 ldap     ldap     10485759 2005-03-27 21:08 
>log.0000000601
>-rw-------    1 ldap     ldap     10485713 2005-03-27 21:13 
>log.0000000602
>-rw-------    1 ldap     ldap     10485456 2005-03-27 21:18 
>log.0000000603
>-rw-------    1 ldap     ldap     10485606 2005-03-27 21:23 
>log.0000000604
>-rw-------    1 ldap     ldap     10471924 2005-03-27 21:28 
>log.0000000605
>-rw-------    1 ldap     ldap     10485746 2005-03-27 21:33 
>log.0000000606
>-rw-------    1 ldap     ldap     10485113 2005-03-27 21:38 
>log.0000000607
>-rw-------    1 ldap     ldap     10485726 2005-03-27 21:43 
>log.0000000608
>-rw-------    1 ldap     ldap     10485691 2005-03-27 21:48 
>log.0000000609
>-rw-------    1 ldap     ldap     10485751 2005-03-27 21:53 
>log.0000000610
>-rw-------    1 ldap     ldap     10485301 2005-03-27 21:58 
>log.0000000611
>-rw-------    1 ldap     ldap     10485753 2005-03-27 22:03 
>log.0000000612
>-rw-------    1 ldap     ldap     10485753 2005-03-27 22:07 
>log.0000000613
>-rw-------    1 ldap     ldap     10485488 2005-03-27 22:12 
>log.0000000614
>-rw-------    1 ldap     ldap     10485707 2005-03-27 22:17 
>log.0000000615
>-rw-------    1 ldap     ldap     10475661 2005-03-27 22:42 
>log.0000000616
>-rw-------    1 ldap     ldap     10485710 2005-03-27 23:12 
>log.0000000617
>-rw-------    1 ldap     ldap     10485737 2005-03-27 23:44 
>log.0000000618
>-rw-------    1 ldap     ldap     10485760 2005-03-28 00:16 
>log.0000000619
>-rw-------    1 ldap     ldap     10485731 2005-03-28 00:21 
>log.0000000620
>-rw-------    1 ldap     ldap     10485746 2005-03-28 00:26 
>log.0000000621
>-rw-------    1 ldap     ldap     10485116 2005-03-28 00:31 
>log.0000000622
>-rw-------    1 ldap     ldap     10485453 2005-03-28 00:35 
>log.0000000623
>-rw-------    1 ldap     ldap     10485714 2005-03-28 01:04 
>log.0000000624
>-rw-------    1 ldap     ldap     10485544 2005-03-28 02:06 
>log.0000000625
>-rw-------    1 ldap     ldap     10485521 2005-03-28 02:18 
>log.0000000626
>-rw-------    1 ldap     ldap     10485745 2005-03-28 02:23 
>log.0000000627
>-rw-------    1 ldap     ldap     10485732 2005-03-28 03:02 
>log.0000000628
>-rw-------    1 ldap     ldap     10485745 2005-03-28 04:16 
>log.0000000629
>-rw-------    1 ldap     ldap     10485748 2005-03-28 04:21 
>log.0000000630
>-rw-------    1 ldap     ldap     10485760 2005-03-28 04:53 
>log.0000000631
>-rw-------    1 ldap     ldap     10485725 2005-03-28 06:15 
>log.0000000632
>-rw-------    1 ldap     ldap     10485723 2005-03-28 06:19 
>log.0000000633
>-rw-------    1 ldap     ldap     10485730 2005-03-28 06:49 
>log.0000000634
>-rw-------    1 ldap     ldap     10485738 2005-03-28 07:59 
>log.0000000635
>-rw-------    1 ldap     ldap     10485573 2005-03-28 08:18 
>log.0000000636
>-rw-------    1 ldap     ldap     10485405 2005-03-28 08:23 
>log.0000000637
>-rw-------    1 ldap     ldap     10475099 2005-03-28 08:53 
>log.0000000638
>-rw-------    1 ldap     ldap     10476362 2005-03-28 09:33 
>log.0000000639
>-rw-------    1 ldap     ldap     10469950 2005-03-28 10:01 
>log.0000000640
>-rw-------    1 ldap     ldap     10485668 2005-03-28 10:17 
>log.0000000641
>-rw-------    1 ldap     ldap     10474836 2005-03-28 10:22 
>log.0000000642
>-rw-------    1 ldap     ldap     10479838 2005-03-28 10:37 
>log.0000000643
>-rw-------    1 ldap     ldap     10485744 2005-03-28 11:14 
>log.0000000644
>-rw-------    1 ldap     ldap     10485725 2005-03-28 11:41 
>log.0000000645
>-rw-------    1 ldap     ldap     10485712 2005-03-28 12:05 
>log.0000000646
>-rw-------    1 ldap     ldap     10485119 2005-03-28 12:18 
>log.0000000647
>-rw-------    1 ldap     ldap     10485693 2005-03-28 12:33 
>log.0000000648
>-rw-------    1 ldap     ldap     10485705 2005-03-28 12:49 
>log.0000000649
>-rw-------    1 ldap     ldap     10485730 2005-03-28 13:05 
>log.0000000650
>-rw-------    1 ldap     ldap     10485582 2005-03-28 13:21 
>log.0000000651
>-rw-------    1 ldap     ldap     10485738 2005-03-28 13:36 
>log.0000000652
>-rw-------    1 ldap     ldap     10485563 2005-03-28 13:58 
>log.0000000653
>-rw-------    1 ldap     ldap     10485666 2005-03-28 14:16 
>log.0000000654
>-rw-------    1 ldap     ldap     10477969 2005-03-28 14:33 
>log.0000000655
>-rw-------    1 ldap     ldap     10485388 2005-03-28 14:58 
>log.0000000656
>-rw-------    1 ldap     ldap     10485514 2005-03-28 15:28 
>log.0000000657
>-rw-------    1 ldap     ldap     10485673 2005-03-28 16:00 
>log.0000000658
>-rw-------    1 ldap     ldap     10485438 2005-03-28 16:17 
>log.0000000659
>-rw-------    1 ldap     ldap     10485717 2005-03-28 16:38 
>log.0000000660
>-rw-------    1 ldap     ldap     10485728 2005-03-28 17:02 
>log.0000000661
>-rw-------    1 ldap     ldap     10485735 2005-03-28 17:25 
>log.0000000662
>-rw-------    1 ldap     ldap     10485716 2005-03-28 17:48 
>log.0000000663
>-rw-------    1 ldap     ldap     10485742 2005-03-28 18:11 
>log.0000000664
>-rw-------    1 ldap     ldap     10485725 2005-03-28 18:19 
>log.0000000665
>-rw-------    1 ldap     ldap     10485726 2005-03-28 18:42 
>log.0000000666
>-rw-------    1 ldap     ldap     10485720 2005-03-28 19:02 
>log.0000000667
>-rw-------    1 ldap     ldap     10485755 2005-03-28 19:24 
>log.0000000668
>-rw-------    1 ldap     ldap     10485661 2005-03-28 19:50 
>log.0000000669
>-rw-------    1 ldap     ldap     10485756 2005-03-28 20:16 
>log.0000000670
>-rw-------    1 ldap     ldap     10485723 2005-03-28 20:21 
>log.0000000671
>-rw-------    1 ldap     ldap     10485133 2005-03-28 20:35 
>log.0000000672
>-rw-------    1 ldap     ldap     10485752 2005-03-28 21:04 
>log.0000000673
>-rw-------    1 ldap     ldap     10485757 2005-03-28 21:32 
>log.0000000674
>-rw-------    1 ldap     ldap     10485744 2005-03-28 22:03 
>log.0000000675
>-rw-------    1 ldap     ldap     10485721 2005-03-28 22:18 
>log.0000000676
>-rw-------    1 ldap     ldap     10485395 2005-03-28 22:23 
>log.0000000677
>-rw-------    1 ldap     ldap     10485730 2005-03-28 22:48 
>log.0000000678
>-rw-------    1 ldap     ldap     10485489 2005-03-28 23:31 
>log.0000000679
>-rw-------    1 ldap     ldap     10485249 2005-03-29 00:15 
>log.0000000680
>-rw-------    1 ldap     ldap     10485760 2005-03-29 00:20 
>log.0000000681
>-rw-------    1 ldap     ldap     10485712 2005-03-29 00:25 
>log.0000000682
>-rw-------    1 ldap     ldap     10474961 2005-03-29 00:34 
>log.0000000683
>-rw-------    1 ldap     ldap     10475937 2005-03-29 02:06 
>log.0000000684
>-rw-------    1 ldap     ldap     10485756 2005-03-29 02:19 
>log.0000000685
>-rw-------    1 ldap     ldap     10485730 2005-03-29 02:27 
>log.0000000686
>-rw-------    1 ldap     ldap     10485702 2005-03-29 04:18 
>log.0000000687
>-rw-------    1 ldap     ldap     10485719 2005-03-29 04:23 
>log.0000000688
>-rw-------    1 ldap     ldap     10485730 2005-03-29 05:15 
>log.0000000689
>-rw-------    1 ldap     ldap     10485369 2005-03-29 06:17 
>log.0000000690
>-rw-------    1 ldap     ldap     10485453 2005-03-29 06:21 
>log.0000000691
>-rw-------    1 ldap     ldap     10485747 2005-03-29 06:44 
>log.0000000692
>-rw-------    1 ldap     ldap     10485709 2005-03-29 08:15 
>log.0000000693
>-rw-------    1 ldap     ldap     10485714 2005-03-29 08:20 
>log.0000000694
>-rw-------    1 ldap     ldap     10485716 2005-03-29 08:34 
>log.0000000695
>-rw-------    1 ldap     ldap     10485511 2005-03-29 09:25 
>log.0000000696
>-rw-------    1 ldap     ldap     10485690 2005-03-29 10:10 
>log.0000000697
>-rw-------    1 ldap     ldap     10485732 2005-03-29 10:19 
>log.0000000698
>-rw-------    1 ldap     ldap     10485751 2005-03-29 10:28 
>log.0000000699
>-rw-------    1 ldap     ldap     10485744 2005-03-29 11:03 
>log.0000000700
>-rw-------    1 ldap     ldap     10485558 2005-03-29 11:33 
>log.0000000701
>-rw-------    1 ldap     ldap     10484140 2005-03-29 12:01 
>log.0000000702
>-rw-------    1 ldap     ldap     10485706 2005-03-29 12:17 
>log.0000000703
>-rw-------    1 ldap     ldap     10485538 2005-03-29 12:41 
>log.0000000704
>-rw-------    1 ldap     ldap      6426750 2005-03-29 12:58 
>log.0000000705
>-rw-------    1 ldap     ldap      2707456 2005-03-29 12:58 
>objectClass.bdb
>-rw-------    1 ldap     ldap      8921088 2005-03-29 12:58 uid.bdb
>
>
>USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
>ldap      5385  0.0  2.6 275556 54184 ?      S    Mar27   0:00 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      5386  0.0  2.6 275556 54184 ?      S    Mar27   0:00 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      5387  0.1  2.6 275556 54184 ?      S    Mar27   4:28 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      5389  0.9  2.6 275556 54184 ?      S    Mar27  25:40 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      5391  0.9  2.6 275556 54184 ?      S    Mar27  24:01 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      5397  0.9  2.6 275556 54184 ?      S    Mar27  24:59 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      5423  0.9  2.6 275556 54184 ?      S    Mar27  25:40 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      5526  0.9  2.6 275556 54184 ?      S    Mar27  26:11 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      5642  0.9  2.6 275556 54184 ?      S    Mar27  24:40 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      5998  0.9  2.6 275556 54184 ?      S    Mar27  25:25 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      5999  0.9  2.6 275556 54184 ?      S    Mar27  25:26 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      6000  0.9  2.6 275556 54184 ?      S    Mar27  24:58 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      7433  0.9  2.6 275556 54184 ?      S    Mar27  23:48 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      7434  0.9  2.6 275556 54184 ?      S    Mar27  24:20 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      7436  0.9  2.6 275556 54184 ?      S    Mar27  25:32 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      7440  0.9  2.6 275556 54184 ?      S    Mar27  25:32 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      7441  1.0  2.6 275556 54184 ?      S    Mar27  27:36 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      7442  0.9  2.6 275556 54184 ?      S    Mar27  24:00 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap      7444  1.0  2.6 275556 54184 ?      S    Mar27  27:03 
>/opt/openldap/libexec/slapd -h ldap:/// -u ldap -g ldap -l local6
>ldap     12119  0.0  0.0  2476  936 pts/0    S    12:58   0:00 
>su - ldap
>ldap     12120  0.0  0.0  3064 1816 pts/0    S    12:58   0:00 -bash
>ldap     12232  0.0  0.0  2860  856 pts/0    R    13:01   0:00 ps uxw
>