Sounds like you're on the right track with the server. But I see no note of using ldap.conf or .ldaprc to set TLS_CACERT directive for your client. See ldap.conf(5).