User Authentication Hangs.. I'm losing my sanity!

[Rich West <Rich.West@wesmo.com>]

OK.. I'm not sure if this is the right list or not, but I thought I 
would give it a shot.

Installed: openldap-2027-8 (from RH), nss_ldap 215, pam_ldap 167

The problem I am having is that on a newly installed RH 9 system, if I 
use nss_ldap and/or pam_ldap for user authentication, and, hence, add 
ldap to the nsswitch.conf file for shadow/groups/passwd/hosts, no one, 
not even root, can log in to the system.  SSH sits and waits forever.  
From the console, the system hangs and eventually times out, bringing 
me back to the login screen (non-graphic, runlevel 3).

Now, the history is that I have (err.. had) this all running happily on 
a RH8 system (well, sorta RH8.. It was upgraded from 61 to 6.2 to 7.1 to 
7.2 to 7.3 to 8.. I figured it was time to do everything "clean" 
again).  On my original machine, with the same ldap database and server 
version running, I can happily set the the nsswitch.conf file 
accordingly, set up the /etc/pam.d/system-auth and sshd files, and do 
all of the getent's I want and the server happily works.

I'm replicating everything on the new machine, and, well, it has been 
disasterous.  I can get valid results from getent passwd/shadow/groups, 
but not hosts (totally weird).  IF I have ldap in /etc/nsswitch.conf for 
passwd/shadow/groups, then no one can log in.. not even via the console.

Now, if I do the nsswitch.conf file change while logged in as root, and 
I /bin/su - someuser, it hangs.. If I ssh localhost, it hangs.  If I 
strace everything, it all seems to go about 90% of the way there, and 
then drop in to a suspended state.

I turned up the logging on slapd for a few minutes (wow, it was noisy at 
debug level 9. :) and grabbed the relevant output from an attempt at 
/bin/su'ing as a user serviced by ldap.

I'm completely baffled (8hrs in to trying to resolve this)...  Any help 
would be wonderful..

-Rich

Jan 19 23:45:14 cranium slapd: ====> cache_return_entry_r( 89 ): 
returned (0)
Jan 19 23:45:14 cranium slapd: ldbm_search: no candidates
Jan 19 23:45:14 cranium slapd: send_ldap_search_result 0::
Jan 19 23:45:14 cranium slapd: send_ldap_response: msgid=6 tag=101 err=0
Jan 19 23:45:14 cranium slapd: ber_flush: 14 bytes to sd 17
Jan 19 23:45:14 cranium slapd: daemon: select: listen=6 active_threads=1 
tvp=NULL
Jan 19 23:45:14 cranium slapd: daemon: activity on 1 descriptors
Jan 19 23:45:14 cranium slapd: daemon: new connection on 15
Jan 19 23:45:14 cranium slapd: daemon: added 15r
Jan 19 23:45:14 cranium slapd: daemon: activity on:
Jan 19 23:45:14 cranium slapd: daemon: select: listen=6 active_threads=0 
tvp=NULL
Jan 19 23:45:14 cranium slapd: daemon: activity on 1 descriptors
Jan 19 23:45:14 cranium slapd: daemon: activity on: 15r
Jan 19 23:45:14 cranium slapd: daemon: read activity on 15
Jan 19 23:45:14 cranium slapd: connection_get(15): got connid=14
Jan 19 23:45:14 cranium slapd: connection_read(15): checking for input 
on id=14
Jan 19 23:45:14 cranium slapd: ber_get_next
Jan 19 23:45:14 cranium slapd: ber_get_next: tag 0x30 len 46 contents:
Jan 19 23:45:14 cranium slapd: ber_get_next
Jan 19 23:45:14 cranium slapd: ber_get_next on fd 15 failed errno=11 
(Resource temporarily unavailable)
Jan 19 23:45:14 cranium slapd: do_bind
Jan 19 23:45:14 cranium slapd: ber_scanf fmt ({iat) ber:
Jan 19 23:45:14 cranium slapd: ber_scanf fmt (o}) ber:

Last bits of the strace from a /bin/su...

getdents64(4, /* 4 entries */, 1024)    = 96
stat64("/dev/pts/0", {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), 
...}) = 0
close(4)                                = 0
access("/var/run/utmpx", F_OK)          = -1 ENOENT (No such file or 
directory)
open("/var/run/utmp", O_RDWR)           = -1 EACCES (Permission denied)
open("/var/run/utmp", O_RDONLY)         = 4
fcntl64(4, F_GETFD)                     = 0
fcntl64(4, F_SETFD, FD_CLOEXEC)         = 0
_llseek(4, 0, [0], SEEK_SET)            = 0
alarm(0)                                = 0
rt_sigaction(SIGALRM, {0x405fa600, [], SA_RESTORER, 0x400895a8}, 
{SIG_DFL}, 8) = 0
alarm(1)                                = 0
fcntl64(4, F_SETLKW, {type=F_RDLCK, whence=SEEK_SET, start=0, len=0}) = 0
read(4, "\10\0\0\0\25\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
384) = 384
read(4, "\2\0\0\0\0\0\0\0~\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
384) = 384
read(4, "\1\0\0\0003N\0\0~\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
384) = 384
read(4, "\10\0\0\0=\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
384) = 384
read(4, "\6\0\0\0\204\22\0\0tty1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
384) = 384
read(4, "\6\0\0\0\30\r\0\0tty2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
384) = 384
read(4, "\6\0\0\0\224\5\0\0tty3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
384) = 384
read(4, "\6\0\0\0\225\5\0\0tty4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
384) = 384
read(4, "\6\0\0\0\226\5\0\0tty5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
384) = 384
read(4, "\6\0\0\0\227\5\0\0tty6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
384) = 384
read(4, "\7\0\0\0\307\22\0\0pts/0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
384) = 384
fcntl64(4, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
alarm(0)                                = 1
rt_sigaction(SIGALRM, {SIG_DFL}, NULL, 8) = 0
close(4)                                = 0
time([1074574870])                      = 1074574870
open("/etc/localtime", O_RDONLY)        = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=1267, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0x40027000
read(4, "TZif\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\4\0\0\0\4\0"..., 
4096) = 1267
close(4)                                = 0
munmap(0x40027000, 4096)                = 0
getpid()                                = 6007
rt_sigprocmask(SIG_SETMASK, NULL, [RTMIN], 8) = 0
rt_sigsuspend([]