Re: String conversions UTF8 <-> ISO-8859-1

["Kurt D. Zeilenga" <Kurt@OpenLDAP.org>]

At 12:08 AM 5/23/2003, Hallvard B Furuseth wrote:
>Kurt D. Zeilenga writes:
>>At 04:52 AM 5/15/2003, Hallvard B Furuseth wrote:
>>>At 30 Apr I wrote:
>>>
>>>> However, how about keeping charset handling in a separate library,
>>>> but provide it as callbacks from the LDAP library?  (...)
>> 
>> Personally, I rather not add callbacks here.  It is likely
>> more efficiently and effectively to do such transcoding directly
>> upon returned values by the application then via callbacks.
>
>Plenty of experience with both LDAP and X.500 shows that if the
>library doesn't offer charset conversion, very often it doesn't
>get done.

Well, as we've already discussed, the library does not have the
application-specific knowledge necessarily to reliably determine
when to do conversion and what conversion to do.

I assumed that when you suggested adding callbacks, that the
purpose of this callback would be to provide a mechanism for
library to provide values with enough protocol information
so that the application could determine if and what kind of
conversion was appropriate and then do it.  The problem with
this approach is that the application may been the whole
PDU to determine whether or not if and what kind of
conversion should take place.

Now, you might have envisioned some other sort of callback.
But, I find it hard to think of useful callback mechanism.
The key here is that to do determine when and what conversion
to do, one needs both knowledge of protocol context as well
as application-specific context.  I don't see how the library
can ever know the application-specific context.  And its
also may not be fully aware of protocol context (due to
various kinds of extensions which can alter character
encodings).

So, it seems to me, the best place to do conversion in LDAP
is in the application.

>There are just too many places to do it to bother, 

And each of these places may require different conversion,
dependent on various protocol and application-specific
factors.

>So I'd really like OpenLDAP to provide charset conversion.  From message <HBF.20030430vsve@bombur.uio.no>, the options then seem to
>be callbacks or to provide another charset-aware library which
>completely duplicates the LDAP API, but which does charset conversion.
>That seems very cumbersome, so I prefer callbacks.

It's my opinion that no reasonable way for the library to
determine in which places conversion is needed and,
if so, what kind of conversion.  Excessing the necessary
protocol information in a callback so the application is
certainly possible, but seems kind of pointless.  It seems
far more reasonable just to hand the content of the PDU to
the application and let it, with its application-specific
knowledge, apply necessary conversions.

Kurt