sasl_encode type conflict in 64 bit builds (ITS#3212)

[jeff.lewis29@verizon.net]

Full_Name: Jeff Lewis
Version: 2.1.17
OS: SunOS 5.8
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (192.127.94.7)


I'm an open source novice and have run into a problem building the OpenLDAP
clients on SunOS.  This very well could be a cockpit error on my part and I'd be
delighted to learn that it's my error.

I think I've stumbled upon a type conflict that's keeping the ldap client tools
from working.  I just built the clients on my SunOS platform with is a big
endian 64 bit machine.  When I run ldapsearch (and ldapwhoami...haven't tried
the others but I suspect they'll do something similar), I get the following
results:

  $ ./ldapsearch -b "" -s base -h tnt80 -U djl -Y DIGEST-MD5
  SASL/DIGEST-MD5 authentication started
  Please enter your password:
  SASL username: djl
  SASL SSF: 128
  SASL installing layers
  # extended LDIF
  #
  # LDAPv3
  # base <> with scope base
  # filter: (objectclass=*)
  # requesting: ALL
  #

  ldapsearch: ldap_search_ext: Can't contact LDAP server (81)
  $

I did a network trace and found that the bind was successful, but the very next
write/send to the server carrying the search request had "run away" and was
sending a massive amount of data.  After some study, I found my way into
sb_sasl_write in the libraries/libldap/cyrus.c source module.  In it is a call
to sasl_encode that is failing for me.  What's happening is the sasl_encode()
call wants a pointer to an "unsigned int" and sb_sasl_write passes a pointer to
a ber_len_t cast as "unsigned *".  The ber_len_t type wants to be "unsigned
long" which is 8 bytes wide on my 64 bit machine.  The bottom line is
sasl_encode stores 4 bytes of unsigned data into an 8 byte area, and since my
machine is big endian, the resulting value is 2^^32 times bigger than it should
be and the next write tries to send a huge amount of data.  If my machine were
small endian, I wouldn't have noticed a problem.

I figured that if there were a sasl_encode() call in the OpenLDAP source,
there's probably a sasl_decode() call as well.  I found it in sb_sasl_read and
sure enough, it suffers from the same problem.

Knowing that I'm not running the latest OpenLDAP package, I downloaded the
latest stable release (2.2.13, I think), configured it and examined the source. 
I did not build the package.  But from desk checking the code and type
definitions, it appears that this little problem of mine is still there.

For now, I've added some code to copy the lengths into and out of an unsigned
temp variable and passed a pointer to the temp to sasl_encode/decode.  The code
is much happier and produces good results for me.

Here's the original call to sasl_encode:

        ret = sasl_encode( p->sasl_context, buf, len,
                (SASL_CONST char **)&p->buf_out.buf_base,
                (unsigned *)&p->buf_out.buf_size );

Here's my kludged call:

        temp = p->buf_out.buf_size;
        ret = sasl_encode( p->sasl_context, buf, len,
                (SASL_CONST char **)&p->buf_out.buf_base,
                (unsigned *)&temp );
        p->buf_out.buf_size = temp;

I did something similar to the sasl_decode call in sb_sasl_read.

If you're interested in the diffs:

  $ diff cyrus.orig cyrus.c
  227a228
  >       unsigned                temp;
  292a294
  >       temp = p->buf_in.buf_end;
  296c298,299
  <               (unsigned *)&p->buf_in.buf_end );
  ---
  >               (unsigned *)&temp );
  >         p->buf_in.buf_end = temp;
  321a325
  >       unsigned                temp;
  350a355
  >         temp = p->buf_out.buf_size;
  353c358,359
  <               (unsigned *)&p->buf_out.buf_size );
  ---
  >               (unsigned *)&temp );
  >         p->buf_out.buf_size = temp;
  $


My ldapsearch is much much better after the changes, but I doubt seriously you'd
want these particular changes in your code.  They're a bit of a kludge and are
completely wasteful on 32 bit platforms.  But after making the changes above,
here's what ldapsearch does for me (correctly, I presume):

  $ ./ldapsearch -b "" -s base -h tnt80 -U djl -Y DIGEST-MD5
  SASL/DIGEST-MD5 authentication started
  Please enter your password:
  SASL username: djl
  SASL SSF: 128
  SASL installing layers
  # extended LDIF
  #
  # LDAPv3
  # base <> with scope base
  # filter: (objectclass=*)
  # requesting: ALL
  #

  #
  dn:
  objectClass: top
  objectClass: OpenLDAProotDSE

  # search result
  search: 3
  result: 0 Success

  # numResponses: 2
  # numEntries: 1
  $

I've searched the archives and did not see anything that caused me to think
someone else was having a similar problem.  But then, being an open source
novice, I could have very easily missed something important.  :-)  I do hope
this is enough information for you to go on.  If not, email me and I'll fill in
any missing blanks.  Like I said, I'd be very happy to learn that the error is
mine.