[Date Prev][Date Next] [Chronological] [Thread] [Top]
Re: (ITS#5332) multimaster replication crash

To: openldap-its@OpenLDAP.org
Subject: Re: (ITS#5332) multimaster replication crash
From: Marcel.Ritter@rrze.uni-erlangen.de
Date: Mon, 4 Feb 2008 13:22:19 GMT
This is a multi-part message in MIME format.
--------------030505030300050004060200
Content-Type: text/plain; charset=ISO-8859-15
Content-Transfer-Encoding: 7bit

Attached the ./configure output (if you really need config.log - which
is quite big please let me know).

There's not much to say about the config: I'm simply using the config from
tests/scripts/test050-syncrepl-multimaster and a modified version of this
script (the only change is, to add quite a lot of users to the ldap database
at the very end)

# compile openldap
./configure
make

# prepare tests
cd tests
# replace original test
cp /download/test050-syncrepl-multimaster scripts/

# create input ldifs
mkdir scripts/data/ldifs/
cd scripts/data/ldifs
# this will create several ldif files, each containing 1000 users
/download/usersldif2.py | split -l 20000 -
cd -

# now run modified test:
./run test050-syncrepl-multimaster

# ... now wait and see ...
...
Start gdb for:
PIDS: 13801 13814 13824
Added: xaa in 76.78 s - OK (# of slapds: 3)
Added: xab in 129.22 s - OK (# of slapds: 3)
./scripts/test050-syncrepl-multimaster: line 621: 13824 Segmenation
fault  $SLAPD -F ./slapd.d -h $URI3 -d $LVL $TIMING >> $LOG3 2>&1  (wd:
~/packages/BUILD/openldap-2.4.7/tests/testrun/con2)
Added: xac in 176.26 s - OK (# of slapds: 2)
At least one slapd died ... test failed
./scripts/test050-syncrepl-multimaster: line 638: kill: (13824) - No
such process


As far as I can see there are 2 possible outcomes:

(At least) one of the slapd processes crashes (see above), or it get's
caught in an endless loop (using up 100% CPU usage).

Bye,
   Marcel

-- 
----
Dipl.-Inf. Marcel Ritter
Linux/Novell
Regionales Rechenzentrum Erlangen
Tel: 09131 / 85-27808
E-Mail: Marcel.Ritter@rrze.uni-erlangen.de
----
Unix _IS_ user friendly... It's just selective about who its friends are.



--------------030505030300050004060200
Content-Type: text/x-log;
 name="configure.log"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="configure.log"

Configuring OpenLDAP 2.4.7-Release ...
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking configure arguments... done
checking for cc... cc
checking for ar... ar
checking for style of include used by make... GNU
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables... 
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether cc accepts -g... yes
checking for cc option to accept ANSI C... none needed
checking dependency style of cc... none
checking for a sed that does not truncate output... /usr/bin/sed
checking for egrep... grep -E
checking for ld used by cc... /usr/x86_64-suse-linux/bin/ld
checking if the linker (/usr/x86_64-suse-linux/bin/ld) is GNU ld... yes
checking for /usr/x86_64-suse-linux/bin/ld option to reload object files... -r
checking for BSD-compatible nm... /usr/bin/nm -B
checking whether ln -s works... yes
checking how to recognise dependent libraries... pass_all
checking how to run the C preprocessor... cc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking dlfcn.h usability... yes
checking dlfcn.h presence... yes
checking for dlfcn.h... yes
checking the maximum length of command line arguments... 32768
checking command to parse /usr/bin/nm -B output from cc object... ok
checking for objdir... .libs
checking for ranlib... ranlib
checking for strip... strip
checking if cc static flag  works... yes
checking if cc supports -fno-rtti -fno-exceptions... no
checking for cc option to produce PIC... -fPIC
checking if cc PIC flag -fPIC works... yes
checking if cc supports -c -o file.o... yes
checking whether the cc linker (/usr/x86_64-suse-linux/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking for shl_load... no
checking for shl_load in -ldld... no
checking for dlopen... no
checking for dlopen in -ldl... yes
checking whether a program can dlopen itself... yes
checking whether a statically linked program can dlopen itself... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
configure: creating libtool
checking how to run the C preprocessor... cc -E
checking for be_app in -lbe... no
checking whether we are using the GNU C compiler... (cached) yes
checking whether cc accepts -g... (cached) yes
checking for cc option to accept ANSI C... (cached) none needed
checking dependency style of cc... (cached) none
checking for cc depend flag... -M
checking for afopen in -ls... no
checking for EBCDIC... no
checking for ANSI C header files... yes
checking for dirent.h that defines DIR... yes
checking for library containing opendir... none required
checking for sys/wait.h that is POSIX.1 compatible... yes
checking whether termios.h defines TIOCGWINSZ... no
checking whether sys/ioctl.h defines TIOCGWINSZ... yes
checking arpa/inet.h usability... yes
checking arpa/inet.h presence... yes
checking for arpa/inet.h... yes
checking arpa/nameser.h usability... yes
checking arpa/nameser.h presence... yes
checking for arpa/nameser.h... yes
checking assert.h usability... yes
checking assert.h presence... yes
checking for assert.h... yes
checking bits/types.h usability... yes
checking bits/types.h presence... yes
checking for bits/types.h... yes
checking conio.h usability... no
checking conio.h presence... no
checking for conio.h... no
checking crypt.h usability... yes
checking crypt.h presence... yes
checking for crypt.h... yes
checking direct.h usability... no
checking direct.h presence... no
checking for direct.h... no
checking errno.h usability... yes
checking errno.h presence... yes
checking for errno.h... yes
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking filio.h usability... no
checking filio.h presence... no
checking for filio.h... no
checking getopt.h usability... yes
checking getopt.h presence... yes
checking for getopt.h... yes
checking grp.h usability... yes
checking grp.h presence... yes
checking for grp.h... yes
checking io.h usability... no
checking io.h presence... no
checking for io.h... no
checking libutil.h usability... no
checking libutil.h presence... no
checking for libutil.h... no
checking limits.h usability... yes
checking limits.h presence... yes
checking for limits.h... yes
checking locale.h usability... yes
checking locale.h presence... yes
checking for locale.h... yes
checking malloc.h usability... yes
checking malloc.h presence... yes
checking for malloc.h... yes
checking for memory.h... (cached) yes
checking psap.h usability... no
checking psap.h presence... no
checking for psap.h... no
checking pwd.h usability... yes
checking pwd.h presence... yes
checking for pwd.h... yes
checking process.h usability... no
checking process.h presence... no
checking for process.h... no
checking sgtty.h usability... yes
checking sgtty.h presence... yes
checking for sgtty.h... yes
checking shadow.h usability... yes
checking shadow.h presence... yes
checking for shadow.h... yes
checking stddef.h usability... yes
checking stddef.h presence... yes
checking for stddef.h... yes
checking for string.h... (cached) yes
checking for strings.h... (cached) yes
checking sysexits.h usability... yes
checking sysexits.h presence... yes
checking for sysexits.h... yes
checking sys/file.h usability... yes
checking sys/file.h presence... yes
checking for sys/file.h... yes
checking sys/filio.h usability... no
checking sys/filio.h presence... no
checking for sys/filio.h... no
checking sys/fstyp.h usability... no
checking sys/fstyp.h presence... no
checking for sys/fstyp.h... no
checking sys/errno.h usability... yes
checking sys/errno.h presence... yes
checking for sys/errno.h... yes
checking sys/ioctl.h usability... yes
checking sys/ioctl.h presence... yes
checking for sys/ioctl.h... yes
checking sys/param.h usability... yes
checking sys/param.h presence... yes
checking for sys/param.h... yes
checking sys/privgrp.h usability... no
checking sys/privgrp.h presence... no
checking for sys/privgrp.h... no
checking sys/resource.h usability... yes
checking sys/resource.h presence... yes
checking for sys/resource.h... yes
checking sys/select.h usability... yes
checking sys/select.h presence... yes
checking for sys/select.h... yes
checking sys/socket.h usability... yes
checking sys/socket.h presence... yes
checking for sys/socket.h... yes
checking for sys/stat.h... (cached) yes
checking sys/syslog.h usability... yes
checking sys/syslog.h presence... yes
checking for sys/syslog.h... yes
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking for sys/types.h... (cached) yes
checking sys/uio.h usability... yes
checking sys/uio.h presence... yes
checking for sys/uio.h... yes
checking sys/vmount.h usability... no
checking sys/vmount.h presence... no
checking for sys/vmount.h... no
checking syslog.h usability... yes
checking syslog.h presence... yes
checking for syslog.h... yes
checking termios.h usability... yes
checking termios.h presence... yes
checking for termios.h... yes
checking for unistd.h... (cached) yes
checking utime.h usability... yes
checking utime.h presence... yes
checking for utime.h... yes
checking for resolv.h... yes
checking for netinet/tcp.h... yes
checking for sys/ucred.h... no
checking for sigaction... yes
checking for sigset... yes
checking for socket... yes
checking for select... yes
checking for sys/select.h... (cached) yes
checking for sys/socket.h... (cached) yes
checking types of arguments for select... int,fd_set *,struct timeval *
checking for poll... yes
checking poll.h usability... yes
checking poll.h presence... yes
checking for poll.h... yes
checking sys/poll.h usability... yes
checking sys/poll.h presence... yes
checking for sys/poll.h... yes
checking sys/epoll.h usability... yes
checking sys/epoll.h presence... yes
checking for sys/epoll.h... yes
checking for epoll system call... yes
checking sys/devpoll.h usability... no
checking sys/devpoll.h presence... no
checking for sys/devpoll.h... no
checking declaration of sys_errlist... yes
checking for strerror... yes
checking for strerror_r... yes
checking non-posix strerror_r... no
checking for regex.h... yes
checking for library containing regfree... none required
checking for compatible POSIX regex... yes
checking sys/uuid.h usability... no
checking sys/uuid.h presence... no
checking for sys/uuid.h... no
checking uuid/uuid.h usability... yes
checking uuid/uuid.h presence... yes
checking for uuid/uuid.h... yes
checking for library containing uuid_generate... -luuid
checking for resolver link (default)... no
checking for resolver link (-lresolv)... yes
checking for hstrerror... yes
checking for getaddrinfo... yes
checking for getnameinfo... yes
checking for gai_strerror... yes
checking for inet_ntop... yes
checking INET6_ADDRSTRLEN... yes
checking struct sockaddr_storage... yes
checking sys/un.h usability... yes
checking sys/un.h presence... yes
checking for sys/un.h... yes
checking openssl/ssl.h usability... yes
checking openssl/ssl.h presence... yes
checking for openssl/ssl.h... yes
checking for SSL_library_init in -lssl... yes
checking OpenSSL library version (CRL checking capability)... yes
checking for _beginthread... no
checking pthread.h usability... yes
checking pthread.h presence... yes
checking for pthread.h... yes
checking POSIX thread version... 10
checking for LinuxThreads pthread.h... no
checking for GNU Pth pthread.h... no
checking sched.h usability... yes
checking sched.h presence... yes
checking for sched.h... yes
checking for pthread_create in default libraries... no
checking for pthread link with -kthread... no
checking for pthread link with -pthread... yes
checking for sched_yield... yes
checking for pthread_yield... yes
checking for thr_yield... no
checking for pthread_kill... yes
checking for pthread_rwlock_destroy with <pthread.h>... yes
checking for pthread_detach with <pthread.h>... yes
checking for pthread_setconcurrency... yes
checking for pthread_getconcurrency... yes
checking for thr_setconcurrency... no
checking for thr_getconcurrency... no
checking for pthread_kill_other_threads_np... no
checking for LinuxThreads implementation... no
checking for LinuxThreads consistency... no
checking if pthread_create() works... yes
checking if select yields when using pthreads... yes
checking for thread specific errno... yes
checking for thread specific h_errno... yes
checking for ctime_r... yes
checking for gethostbyname_r... yes
checking for gethostbyaddr_r... yes
checking number of arguments of ctime_r... 2
checking number of arguments of gethostbyname_r... 6
checking number of arguments of gethostbyaddr_r... 8
checking db.h usability... yes
checking db.h presence... yes
checking for db.h... yes
checking for Berkeley DB major version... 4
checking for Berkeley DB minor version... 5
checking for Berkeley DB link (-ldb-4.5)... yes
checking for Berkeley DB version match... yes
checking for Berkeley DB thread support... yes
checking Berkeley DB version for BDB/HDB backends... yes
checking for openlog... yes
checking unicode/utypes.h usability... no
checking unicode/utypes.h presence... no
checking for unicode/utypes.h... no
configure: WARNING: ICU not available
checking sasl/sasl.h usability... yes
checking sasl/sasl.h presence... yes
checking for sasl/sasl.h... yes
checking sasl.h usability... no
checking sasl.h presence... no
checking for sasl.h... no
checking for sasl_client_init in -lsasl2... yes
checking Cyrus SASL library version... yes
checking for sasl_version... yes
checking fetch(3) library... no
checking for setproctitle... no
checking for setproctitle in -lutil... no
checking for mode_t... yes
checking for off_t... yes
checking for pid_t... yes
checking for ssize_t... yes
checking for caddr_t... yes
checking for size_t... yes
checking for long long... yes
checking for ptrdiff_t... yes
checking for socklen_t... yes
checking the type of arg 3 to accept()... socklen_t *
checking return type of signal handlers... void
checking for sig_atomic_t... yes
checking for uid_t in sys/types.h... yes
checking whether time.h and sys/time.h may both be included... yes
checking whether struct tm is in sys/time.h or time.h... time.h
checking for struct stat.st_blksize... yes
checking for struct passwd.pw_gecos... yes
checking for struct passwd.pw_passwd... yes
checking if toupper() requires islower()... no
checking for an ANSI C-conforming const... yes
checking if compiler understands volatile... yes
checking whether byte ordering is bigendian... no
checking for short... yes
checking size of short... 2
checking for int... yes
checking size of int... 4
checking for long... yes
checking size of long... 8
checking for long long... (cached) yes
checking size of long long... 8
checking for wchar_t... yes
checking size of wchar_t... 4
checking for working memcmp... yes
checking for strftime... yes
checking for inet_aton()... yes
checking for _spawnlp... no
checking for _snprintf... no
checking for _vsnprintf... no
checking for vprintf... yes
checking for _doprnt... no
checking for snprintf... yes
checking for vsnprintf... yes
checking for bcopy... yes
checking for closesocket... no
checking for chroot... yes
checking for endgrent... yes
checking for endpwent... yes
checking for fcntl... yes
checking for flock... yes
checking for fstat... yes
checking for getdtablesize... yes
checking for getgrgid... yes
checking for gethostname... yes
checking for getpass... yes
checking for getpassphrase... no
checking for getpwuid... yes
checking for getpwnam... yes
checking for getspnam... yes
checking for gettimeofday... yes
checking for initgroups... yes
checking for inet_ntoa_b... no
checking for ioctl... yes
checking for lockf... yes
checking for memcpy... yes
checking for memmove... yes
checking for memrchr... yes
checking for mkstemp... yes
checking for mktemp... yes
checking for pipe... yes
checking for read... yes
checking for recv... yes
checking for recvfrom... yes
checking for setpwfile... no
checking for setgid... yes
checking for setegid... yes
checking for setsid... yes
checking for setuid... yes
checking for seteuid... yes
checking for signal... yes
checking for strdup... yes
checking for strpbrk... yes
checking for strrchr... yes
checking for strsep... yes
checking for strstr... yes
checking for strtol... yes
checking for strtoul... yes
checking for strtoq... yes
checking for strtouq... yes
checking for strtoll... yes
checking for strspn... yes
checking for sysconf... yes
checking for waitpid... yes
checking for wait4... yes
checking for write... yes
checking for send... yes
checking for sendmsg... yes
checking for sendto... yes
checking for getopt... yes
checking for getpeereid... no
checking for getpeerucred... no
checking for struct msghdr.msg_accrightslen... no
checking for struct msghdr.msg_control... yes
checking for struct stat.st_fstype... no
checking for struct stat.st_vfstype... no
configure: creating ./config.status
config.status: creating Makefile
config.status: creating doc/Makefile
config.status: creating doc/man/Makefile
config.status: creating doc/man/man1/Makefile
config.status: creating doc/man/man3/Makefile
config.status: creating doc/man/man5/Makefile
config.status: creating doc/man/man8/Makefile
config.status: creating clients/Makefile
config.status: creating clients/tools/Makefile
config.status: creating include/Makefile
config.status: creating libraries/Makefile
config.status: creating libraries/liblber/Makefile
config.status: creating libraries/libldap/Makefile
config.status: creating libraries/libldap_r/Makefile
config.status: creating libraries/liblunicode/Makefile
config.status: creating libraries/liblutil/Makefile
config.status: creating libraries/librewrite/Makefile
config.status: creating servers/Makefile
config.status: creating servers/slapd/Makefile
config.status: creating servers/slapd/back-bdb/Makefile
config.status: creating servers/slapd/back-dnssrv/Makefile
config.status: creating servers/slapd/back-hdb/Makefile
config.status: creating servers/slapd/back-ldap/Makefile
config.status: creating servers/slapd/back-ldif/Makefile
config.status: creating servers/slapd/back-meta/Makefile
config.status: creating servers/slapd/back-monitor/Makefile
config.status: creating servers/slapd/back-null/Makefile
config.status: creating servers/slapd/back-passwd/Makefile
config.status: creating servers/slapd/back-perl/Makefile
config.status: creating servers/slapd/back-relay/Makefile
config.status: creating servers/slapd/back-shell/Makefile
config.status: creating servers/slapd/back-sql/Makefile
config.status: creating servers/slapd/shell-backends/Makefile
config.status: creating servers/slapd/slapi/Makefile
config.status: creating servers/slapd/overlays/Makefile
config.status: creating tests/Makefile
config.status: creating tests/run
config.status: creating tests/progs/Makefile
config.status: creating include/portable.h
config.status: creating include/ldap_features.h
config.status: creating include/lber_types.h
config.status: executing depfiles commands
config.status: executing default commands
Making servers/slapd/backends.c
    Add config ...
    Add ldif ...
    Add monitor ...
    Add bdb ...
    Add hdb ...
    Add relay ...
Making servers/slapd/overlays/statover.c
    Add seqmod ...
    Add syncprov ...
Please run "make depend" to build dependencies

--------------030505030300050004060200
Content-Type: text/x-python;
 name="usersldif2.py"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="usersldif2.py"

#!/usr/bin/env python

import string
import sys

uids=[]

for lineno in xrange(0,100000):

  print """dn: uid=test%(lineno)s,ou=People,dc=example,dc=com
objectClass: top
objectClass: posixAccount
objectClass: shadowAccount
objectClass: inetOrgPerson
cn: User %(lineno)s
gecos: User %(lineno)s
gidNumber: %(lineno)s
homeDirectory: /home/test%(lineno)s
loginShell: /bin/bash
shadowInactive: -1
shadowLastChange: 12570
shadowMax: 99999
shadowMin: 0
shadowWarning: 7
sn: test%(lineno)s
uid: test%(lineno)s
uidNumber: %(lineno)s
userPassword: {crypt}WNu26PDD.aYC.
""" % { 'lineno': lineno }

--------------030505030300050004060200
Content-Type: text/plain;
 name="test050-syncrepl-multimaster"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="test050-syncrepl-multimaster"

#! /bin/sh
# $OpenLDAP: pkg/ldap/tests/scripts/test050-syncrepl-multimaster,v 1.7 2008/01/07 23:20:17 kurt Exp $
## This work is part of OpenLDAP Software <http://www.openldap.org/>.
##
## Copyright 1998-2008 The OpenLDAP Foundation.
## All rights reserved.
##
## Redistribution and use in source and binary forms, with or without
## modification, are permitted only as authorized by the OpenLDAP
## Public License.
##
## A copy of this license is available in the file LICENSE in the
## top-level directory of the distribution or, alternatively, at
## <http://www.OpenLDAP.org/license.html>.

echo "running defines.sh"
. $SRCDIR/scripts/defines.sh

if test $SYNCPROV = syncprovno; then 
	echo "Syncrepl provider overlay not available, test skipped"
	exit 0
fi 

PRODIR=$TESTDIR/pro
CONDIR=$TESTDIR/con
CONDIR2=$TESTDIR/con2
DBPRO=$PRODIR/db
DBCON=$CONDIR/db
DBCON2=$CONDIR2/db
CFPRO=$PRODIR/slapd.d
CFCON=$CONDIR/slapd.d
CFCON2=$CONDIR2/slapd.d

mkdir -p $TESTDIR $PRODIR $CONDIR $CONDIR2 $DBPRO $DBCON $DBCON2 $CFPRO $CFCON $CFCON2

$SLAPPASSWD -g -n >$CONFIGPWF

#
# Test replication of dynamic config:
# - start producer
# - start consumer
# - start consumer2
# - configure over ldap
# - populate over ldap
# - configure syncrepl over ldap
# - retrieve database over ldap and compare against expected results
#

echo "Initializing server configurations..."
$SLAPADD -F $CFCON2 -n 0 <<EOF
dn: cn=config
objectClass: olcGlobal
cn: config
olcServerID: 3

dn: olcDatabase={0}config,cn=config
objectClass: olcDatabaseConfig
olcDatabase: {0}config
olcRootPW:< file://$CONFIGPWF
EOF

$SLAPADD -F $CFCON -n 0 <<EOF
dn: cn=config
objectClass: olcGlobal
cn: config
olcServerID: 2

dn: olcDatabase={0}config,cn=config
objectClass: olcDatabaseConfig
olcDatabase: {0}config
olcRootPW:< file://$CONFIGPWF
EOF

$SLAPADD -F $CFPRO -n 0 <<EOF
dn: cn=config
objectClass: olcGlobal
cn: config
olcServerID: 1

dn: olcDatabase={0}config,cn=config
objectClass: olcDatabaseConfig
olcDatabase: {0}config
olcRootPW:< file://$CONFIGPWF
EOF

echo "Starting producer slapd on TCP/IP port $PORT1..."
cd $PRODIR
$SLAPD -F slapd.d -h $URI1 -d $LVL $TIMING > $LOG1 2>&1 &
PID=$!
if test $WAIT != 0 ; then
    echo PID $PID
    read foo
fi
KILLPIDS="$PID"
cd $TESTWD

sleep 1

echo "Using ldapsearch to check that producer slapd is running..."
for i in 0 1 2 3 4 5; do
	$LDAPSEARCH -s base -b "" -H $URI1 \
		'objectclass=*' > /dev/null 2>&1
	RC=$?
	if test $RC = 0 ; then
		break
	fi
	echo "Waiting 5 seconds for slapd to start..."
	sleep 5
done

if test $RC != 0 ; then
	echo "ldapsearch failed ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi

echo "Inserting syncprov overlay on producer..."
if [ "$SYNCPROV" = syncprovmod ]; then
	$LDAPADD -D cn=config -H $URI1 -y $CONFIGPWF <<EOF > $TESTOUT 2>&1
dn: cn=module,cn=config
objectClass: olcModuleList
cn: module
olcModulePath: ../../../servers/slapd/overlays
olcModuleLoad: syncprov.la
EOF
	RC=$?
	if test $RC != 0 ; then
		echo "ldapadd failed for moduleLoad ($RC)!"
		test $KILLSERVERS != no && kill -HUP $KILLPIDS
		exit $RC
	fi
fi
#
# Note that we configure a timeout here; it's possible for both
# servers to attempt to bind to each other while a modify to
# cn=config is in progress. When the modify pauses the thread pool
# neither server will progress. The timeout will drop the syncrepl
# attempt and allow the modifies to complete.
#
read CONFIGPW < $CONFIGPWF
$LDAPMODIFY -D cn=config -H $URI1 -y $CONFIGPWF <<EOF >> $TESTOUT 2>&1
dn: cn=config
changetype: modify
replace: olcServerID
olcServerID: 1 $URI1
olcServerID: 2 $URI2
olcServerID: 3 $URI3

dn: olcOverlay=syncprov,olcDatabase={0}config,cn=config
changetype: add
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: syncprov

dn: olcDatabase={0}config,cn=config
changetype: modify
add: olcSyncRepl
olcSyncRepl: rid=001 provider=$URI1 binddn="cn=config" bindmethod=simple
  credentials=$CONFIGPW searchbase="cn=config" type=refreshAndPersist
  retry="5 5 300 5" timeout=1
olcSyncRepl: rid=002 provider=$URI2 binddn="cn=config" bindmethod=simple
  credentials=$CONFIGPW searchbase="cn=config" type=refreshAndPersist
  retry="5 5 300 5" timeout=1
olcSyncRepl: rid=003 provider=$URI3 binddn="cn=config" bindmethod=simple
  credentials=$CONFIGPW searchbase="cn=config" type=refreshAndPersist
  retry="5 5 300 5" timeout=1
-
add: olcMirrorMode
olcMirrorMode: TRUE
EOF
RC=$?
if test $RC != 0 ; then
	echo "ldapmodify failed for syncrepl config ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi

echo "Starting consumer slapd on TCP/IP port $PORT2..."
cd $CONDIR
$SLAPD -F ./slapd.d -h $URI2 -d $LVL $TIMING > $LOG2 2>&1 &
SLAVEPID=$!
if test $WAIT != 0 ; then
    echo SLAVEPID $SLAVEPID
    read foo
fi
KILLPIDS="$KILLPIDS $SLAVEPID"
cd $TESTWD

sleep 1

echo "Using ldapsearch to check that consumer slapd is running..."
for i in 0 1 2 3 4 5; do
	$LDAPSEARCH -s base -b "" -H $URI2 \
		'objectclass=*' > /dev/null 2>&1
	RC=$?
	if test $RC = 0 ; then
		break
	fi
	echo "Waiting 5 seconds for slapd to start..."
	sleep 5
done

if test $RC != 0 ; then
	echo "ldapsearch failed ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi

echo "Configuring syncrepl on consumer..."
$LDAPMODIFY -D cn=config -H $URI2 -y $CONFIGPWF <<EOF >>$TESTOUT 2>&1
dn: olcDatabase={0}config,cn=config
changetype: modify
add: olcSyncRepl
olcSyncRepl: rid=001 provider=$URI1 binddn="cn=config" bindmethod=simple
  credentials=$CONFIGPW searchbase="cn=config" type=refreshAndPersist
  retry="5 5 300 5" timeout=1
olcSyncRepl: rid=002 provider=$URI2 binddn="cn=config" bindmethod=simple
  credentials=$CONFIGPW searchbase="cn=config" type=refreshAndPersist
  retry="5 5 300 5" timeout=1
olcSyncRepl: rid=003 provider=$URI3 binddn="cn=config" bindmethod=simple
  credentials=$CONFIGPW searchbase="cn=config" type=refreshAndPersist
  retry="5 5 300 5" timeout=1
-
add: olcMirrorMode
olcMirrorMode: TRUE
EOF

echo "Starting consumer2 slapd on TCP/IP port $PORT3..."
cd $CONDIR2
$SLAPD -F ./slapd.d -h $URI3 -d $LVL $TIMING > $LOG3 2>&1 &
SLAVE2PID=$!
if test $WAIT != 0 ; then
    echo SLAVE2PID $SLAVE2PID
    read foo
fi
KILLPIDS="$KILLPIDS $SLAVE2PID"
cd $TESTWD

sleep 1

echo "Using ldapsearch to check that consumer2 slapd is running..."
for i in 0 1 2 3 4 5; do
	$LDAPSEARCH -s base -b "" -H $URI3 \
		'objectclass=*' > /dev/null 2>&1
	RC=$?
	if test $RC = 0 ; then
		break
	fi
	echo "Waiting 5 seconds for slapd to start..."
	sleep 5
done

if test $RC != 0 ; then
	echo "ldapsearch failed ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi

echo "Configuring syncrepl on consumer2..."
$LDAPMODIFY -D cn=config -H $URI3 -y $CONFIGPWF <<EOF >>$TESTOUT 2>&1
dn: olcDatabase={0}config,cn=config
changetype: modify
add: olcSyncRepl
olcSyncRepl: rid=001 provider=$URI1 binddn="cn=config" bindmethod=simple
  credentials=$CONFIGPW searchbase="cn=config" type=refreshAndPersist
  retry="5 5 300 5" timeout=1
olcSyncRepl: rid=002 provider=$URI2 binddn="cn=config" bindmethod=simple
  credentials=$CONFIGPW searchbase="cn=config" type=refreshAndPersist
  retry="5 5 300 5" timeout=1
olcSyncRepl: rid=003 provider=$URI3 binddn="cn=config" bindmethod=simple
  credentials=$CONFIGPW searchbase="cn=config" type=refreshAndPersist
  retry="5 5 300 5" timeout=1
-
add: olcMirrorMode
olcMirrorMode: TRUE
EOF

echo "Adding schema and databases on producer..."
$LDAPADD -D cn=config -H $URI1 -y $CONFIGPWF <<EOF >>$TESTOUT 2>&1
include: file://$ABS_SCHEMADIR/core.ldif

include: file://$ABS_SCHEMADIR/cosine.ldif

include: file://$ABS_SCHEMADIR/inetorgperson.ldif

include: file://$ABS_SCHEMADIR/openldap.ldif

include: file://$ABS_SCHEMADIR/nis.ldif
EOF
RC=$?
if test $RC != 0 ; then
	echo "ldapadd failed for schema config ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi

if [ "$BACKENDTYPE" = mod ]; then
	$LDAPADD -D cn=config -H $URI1 -y $CONFIGPWF <<EOF >>$TESTOUT 2>&1
dn: cn=module,cn=config
objectClass: olcModuleList
cn: module
olcModulePath: ../../../servers/slapd/back-$BACKEND
olcModuleLoad: back_$BACKEND.la
EOF
	RC=$?
	if test $RC != 0 ; then
		echo "ldapadd failed for backend config ($RC)!"
		test $KILLSERVERS != no && kill -HUP $KILLPIDS
		exit $RC
	fi
fi

$LDAPADD -D cn=config -H $URI1 -y $CONFIGPWF <<EOF >>$TESTOUT 2>&1
dn: olcDatabase={1}$BACKEND,cn=config
objectClass: olcDatabaseConfig
objectClass: olc${BACKEND}Config
olcDatabase: {1}$BACKEND
olcSuffix: $BASEDN
olcDbDirectory: ./db
olcRootDN: $MANAGERDN
olcRootPW: $PASSWD
olcSyncRepl: rid=004 provider=$URI1 binddn="$MANAGERDN" bindmethod=simple
  credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly
  interval=00:00:00:10 retry="5 5 300 5" timeout=1
olcSyncRepl: rid=005 provider=$URI2 binddn="$MANAGERDN" bindmethod=simple
  credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly
  interval=00:00:00:10 retry="5 5 300 5" timeout=1
olcSyncRepl: rid=006 provider=$URI3 binddn="$MANAGERDN" bindmethod=simple
  credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly
  interval=00:00:00:10 retry="5 5 300 5" timeout=1
olcMirrorMode: TRUE

dn: olcOverlay=syncprov,olcDatabase={1}${BACKEND},cn=config
changetype: add
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: syncprov
EOF
RC=$?
if test $RC != 0 ; then
	echo "ldapadd failed for database config ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi

echo "Using ldapadd to populate producer..."
$LDAPADD -D "$MANAGERDN" -H $URI1 -w $PASSWD -f $LDIFORDERED \
	>> $TESTOUT 2>&1
RC=$?
if test $RC != 0 ; then
	echo "ldapadd failed for database config ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi

SLEEP=20
echo "Waiting $SLEEP seconds for syncrepl to receive changes..."
sleep $SLEEP

echo "Using ldapsearch to check that syncrepl received database changes..."
RC=32
for i in 0 1 2 3 4 5; do
	RESULT=`$LDAPSEARCH -H $URI2 \
		-s base -b "cn=Ursula Hampster,ou=Alumni Association,ou=People,dc=example,dc=com" \
		'(objectClass=*)' 2>&1 | awk '/^dn:/ {print "OK"}'`
	if test "x$RESULT" = "xOK" ; then
		RC=0
		break
	fi
	echo "Waiting 5 seconds for syncrepl to receive changes..."
	sleep 5
done

if test $RC != 0 ; then
	echo "ldapsearch failed ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi

echo "Using ldapsearch to check that syncrepl received database changes on consumer2..."
RC=32
for i in 0 1 2 3 4 5; do
	RESULT=`$LDAPSEARCH -H $URI3 \
		-s base -b "cn=Ursula Hampster,ou=Alumni Association,ou=People,dc=example,dc=com" \
		'(objectClass=*)' 2>&1 | awk '/^dn:/ {print "OK"}'`
	if test "x$RESULT" = "xOK" ; then
		RC=0
		break
	fi
	echo "Waiting 5 seconds for syncrepl to receive changes..."
	sleep 5
done

if test $RC != 0 ; then
	echo "ldapsearch failed ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi

echo "Using ldapsearch to read config from the producer..."
$LDAPSEARCH -b cn=config -D cn=config -H $URI1 -y $CONFIGPWF  \
	'objectclass=*' > $MASTEROUT 2>&1
RC=$?

if test $RC != 0 ; then
	echo "ldapsearch failed at producer ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi

echo "Using ldapsearch to read config from the consumer..."
$LDAPSEARCH -b cn=config -D cn=config -H $URI2 -y $CONFIGPWF \
	'objectclass=*' > $SLAVEOUT 2>&1
RC=$?

if test $RC != 0 ; then
	echo "ldapsearch failed at consumer ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi

echo "Using ldapsearch to read config from consumer2..."
$LDAPSEARCH -b cn=config -D cn=config -H $URI3 -y $CONFIGPWF \
	'objectclass=*' > $SLAVE2OUT 2>&1
RC=$?

if test $RC != 0 ; then
	echo "ldapsearch failed at consumer2 ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi

echo "Filtering producer results..."
. $LDIFFILTER < $MASTEROUT > $MASTERFLT
echo "Filtering consumer results..."
. $LDIFFILTER < $SLAVEOUT > $SLAVEFLT
echo "Filtering consumer2 results..."
. $LDIFFILTER < $SLAVE2OUT > $SLAVE2FLT

echo "Comparing retrieved configs from producer and consumer..."
$CMP $MASTERFLT $SLAVEFLT > $CMPOUT

if test $? != 0 ; then
	echo "test failed - producer and consumer configs differ"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit 1
fi

echo "Comparing retrieved configs from producer and consumer2..."
$CMP $MASTERFLT $SLAVE2FLT > $CMPOUT

if test $? != 0 ; then
	echo "test failed - producer and consumer2 configs differ"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit 1
fi

echo "Using ldapsearch to read all the entries from the producer..."
$LDAPSEARCH -S "" -b "$BASEDN" -D "$MANAGERDN" -H $URI1 -w $PASSWD  \
	'objectclass=*' > $MASTEROUT 2>&1
RC=$?

if test $RC != 0 ; then
	echo "ldapsearch failed at producer ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi

echo "Using ldapsearch to read all the entries from the consumer..."
$LDAPSEARCH -S "" -b "$BASEDN" -D "$MANAGERDN" -H $URI2 -w $PASSWD  \
	'objectclass=*' > $SLAVEOUT 2>&1
RC=$?

if test $RC != 0 ; then
	echo "ldapsearch failed at consumer ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi

echo "Using ldapsearch to read all the entries from the consumer2..."
$LDAPSEARCH -S "" -b "$BASEDN" -D "$MANAGERDN" -H $URI3 -w $PASSWD  \
	'objectclass=*' > $SLAVE2OUT 2>&1
RC=$?

if test $RC != 0 ; then
	echo "ldapsearch failed at consumer2 ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi


test $KILLSERVERS != no && kill -HUP $KILLPIDS

echo "Filtering producer results..."
. $LDIFFILTER < $MASTEROUT > $MASTERFLT
echo "Filtering consumer results..."
. $LDIFFILTER < $SLAVEOUT > $SLAVEFLT
echo "Filtering consumer2 results..."
. $LDIFFILTER < $SLAVE2OUT > $SLAVE2FLT

echo "Comparing retrieved entries from producer and consumer..."
$CMP $MASTERFLT $SLAVEFLT > $CMPOUT

if test $? != 0 ; then
	echo "test failed - producer and consumer databases differ"
	exit 1
fi

echo "Comparing retrieved entries from producer and consumer2..."
$CMP $MASTERFLT $SLAVE2FLT > $CMPOUT

if test $? != 0 ; then
	echo "test failed - producer and consumer2 databases differ"
	exit 1
fi

test $KILLSERVERS != no && wait

echo "Restarting servers..."
echo "Starting producer slapd on TCP/IP port $PORT1..."
cd $PRODIR
echo "======================= RESTART =======================" >> $LOG1
$SLAPD -F ./slapd.d -h $URI1 -d $LVL $TIMING >> $LOG1 2>&1 &
PID=$!
if test $WAIT != 0 ; then
    echo PID $PID
    read foo
fi
KILLPIDS="$PID"
cd $TESTWD
echo "Using ldapsearch to check that producer slapd is running..."
for i in 0 1 2 3 4 5; do
	$LDAPSEARCH -s base -b "" -H $URI1 \
		'objectclass=*' > /dev/null 2>&1
	RC=$?
	if test $RC = 0 ; then
		break
	fi
	echo "Waiting 5 seconds for slapd to start..."
	sleep 5
done

if test $RC != 0 ; then
	echo "ldapsearch failed ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi
#exit 0
echo "Starting consumer slapd on TCP/IP port $PORT2..."
cd $CONDIR
echo "======================= RESTART =======================" >> $LOG2
$SLAPD -F ./slapd.d -h $URI2 -d $LVL $TIMING >> $LOG2 2>&1 &
SLAVEPID=$!
if test $WAIT != 0 ; then
    echo SLAVEPID $SLAVEPID
    read foo
fi
KILLPIDS="$KILLPIDS $SLAVEPID"
cd $TESTWD

sleep 1

echo "Using ldapsearch to check that consumer slapd is running..."
for i in 0 1 2 3 4 5; do
	$LDAPSEARCH -s base -b "" -H $URI2 \
		'objectclass=*' > /dev/null 2>&1
	RC=$?
	if test $RC = 0 ; then
		break
	fi
	echo "Waiting 5 seconds for slapd to start..."
	sleep 5
done

if test $RC != 0 ; then
	echo "ldapsearch failed ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi

echo "Starting consumer2 slapd on TCP/IP port $PORT3..."
cd $CONDIR2
echo "======================= RESTART =======================" >> $LOG3
$SLAPD -F ./slapd.d -h $URI3 -d $LVL $TIMING >> $LOG3 2>&1 &
SLAVE2PID=$!
if test $WAIT != 0 ; then
    echo SLAVE2PID $SLAVE2PID
    read foo
fi
KILLPIDS="$KILLPIDS $SLAVE2PID"
cd $TESTWD

sleep 1

echo "Using ldapsearch to check that consumer2 slapd is running..."
for i in 0 1 2 3 4 5; do
	$LDAPSEARCH -s base -b "" -H $URI3 \
		'objectclass=*' > /dev/null 2>&1
	RC=$?
	if test $RC = 0 ; then
		break
	fi
	echo "Waiting 5 seconds for slapd to start..."
	sleep 5
done

if test $RC != 0 ; then
	echo "ldapsearch failed ($RC)!"
	test $KILLSERVERS != no && kill -HUP $KILLPIDS
	exit $RC
fi

# Insert modifications and more tests here.
SLEEP=10
echo "Waiting $SLEEP seconds for servers to resync..."
sleep $SLEEP

echo "Start gdb for:"
echo "PIDS: $KILLPIDS"

for LDIFORDERED in $PWD/data/ldifs/*; do
  /usr/bin/time -f "%e" -o time -- $LDAPADD -D "$MANAGERDN" -H $URI1 -w $PASSWD -f $LDIFORDERED >> ldif.log 2>&1
  RC=$?
  SLAPDS=$(pidof slapd | wc -w)
  TOOK=$(cat time)
  if [ "x$RC" = "x0" ]; then
     echo "Added: $(basename $LDIFORDERED) in $TOOK s - OK (# of slapds: $SLAPDS)"
  else
     echo "Added: $(basename $LDIFORDERED) in $TOOK s - Failed (# of slapds: $SLAPDS)"
  fi
  sleep 1
  if [ "x$SLAPDS" != "x3" ]; then
    echo "At least one slapd died ... test failed"
    break
  fi
done

test $KILLSERVERS != no && kill -HUP $KILLPIDS

echo ">>>>> Test succeeded"

test $KILLSERVERS != no && wait

exit 0

--------------030505030300050004060200--
Prev by Date: Re: (ITS#5343) cn=config crash when deleting attrs
Next by Date: Re: (ITS#5332) multimaster replication crash
Index(es):
- Chronological
- Thread