It seems that this special configuration is not possible.
Trying to set the key will always result in
TLS: could not use key file `xyz'.
TLS: error:02001002:system library:fopen:No such file or directory
bss_file.c:398
TLS: error:20074002:BIO routines:FILE_CTRL:system lib bss_file.c:400
TLS: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system
lib ssl_rsa.c:648
The ldap code has to be adjusted to use a key or certificate from a
configured pkcs#11 keystore.
Is there another way to accomplish that?