Re: String conversions UTF8 <-> ISO-8859-1

[Hallvard B Furuseth <h.b.furuseth@usit.uio.no>]

Kurt D. Zeilenga writes:
>At 06:32 AM 5/23/2003, Hallvard B Furuseth wrote:
>>> It seems far more reasonable just to hand the content of the PDU to
>>> the application and let it, with its application-specific knowledge,
>>> apply necessary conversions.
>>
>> The thing is, then everyone have to add a lot of code to do this. 
> 
> Well, to some degree, yes.  But they should have to write
> their own conversion functions.

I don't see why.  That it will again be a lot of duplication of effort,
since everyone will have to write code to do the same things.  Load the
list of attribute types and whatever to be translated, choose whether or
not to translate based on request/response type and attribute name.
Maybe parse search filters and assemble them again, I don't know how
much of such things you want people to do themselves.

> And I'm actually fine with providing them some common LDAP-specific
> tools.  But I do have a few concerns.
> 
> The first is simply that application be able to control when and
> where the conversion takes place

I don't understand what you mean by this.  Whatever interface we
provide, the interface will decide that.  All the application will
decide is whether or not to convert the specific pieces of data.

> AND the application be given
> enough (protocol context) information to do this.

I still think my callback suggestion does that.  If not, what exactly do
you propose?  A collection of calls just like the LDAP API calls, that
are wrappers around the LDAP API calls but do conversion, like I was
talking about before?  In that case, when and how does the user specify
what to convert?  If not, what kind of API do you think could do this?

> The second is that the mechanism we provide be general.

Actually, I don't care so much about that.  As I said, I just want
it to be usable in most cases.  However,

> That is, able to applied to API input and well as output,

Of course.

> able to be applied to "core" protocol fields as well as extension
> protocol fields, etc..

If you mean extra Ber elements inserted in an LDAPMessage, that would
have to be implemented at the same time as that element is implemented -
whether it is done with callbacks or via some other API.

If you mean converting text in controls,

> For example, I'm going to implement a read-back control (...)

I imagined the code would call something like
  (*callback)(LDAP_CONVERT_CONTROL, "control OID", "control value");
for each control in the request/response that hs a control value.

It could do the same for extendedRequests.


The alternative would be to implement specific controls and extended
requests so that libldap would parse them and call the conversion
function with the decoded text values in the request or control.  That
would be nicer to use, since the user wouldn't have to write code to
decode the control or extended request himself, but it should not be
done unless libldap operates with the decoded control/request anyway.
I don't know how that works now?

I'm not too keen on that solution, though.  Because if the user
implements handling of the the original undecoded control/request, and
we later add support for it so he instead receives a decoded variant of
it, his code would stop working.  Unless we _also_ call the conversion
function with the undecoded control/request if it did not convert
anything in the decoded variant.

> And lastly, I rather not get into implementing transcoding
> libraries.  I rather leave this to icu, iconv, and others.

Sure, I thought that was clear.  The code would be in 2 parts:
1) general code to install and call the callbacks in libldap,
2) a utility library which would install such a callback, read the
   config file with types to convert, and call iconv or whatever to do
   the work.

> Also, I rather leave the choice of which transcoding library
> to use up to the application developer (not the builder of
> the LDAP libraries).

Oh dear.  Well, OK...  Don't call iconv or some other library directly
in step (2) above.  Instead, call some pointers to functions that must
be configured at run-time.  These can be set to very simple wrapper
functions written by the user which call iconv, icu, or some other
library.  I'll have to look at the interfaces to the most common
conversion libraries to make an API which they all can be plugged into.

I want to add charset conversion to ldapsearch & co, though.  That
conversion library would have to be chosen by the builder.  But the
wrapper functions (in previous paragraph) for that library could be in
clients/tools/.  Or step 1 and 2 above could be moved into libldap,
since they now do not link any conversion library, and a small library
which calls a default conversion library chosen by the builder could
optionally be installed in addition.  That way the user will have a
default installed conversion library all ready, but he can still use his
own choice by adding a few trivial lines of code if he wants that.

> Now, I do want to dampen your desire to do work improving the
> tool set.

I hope you meant a 'not' inside there:-)

> Improvements are needed.  However, I am a bit leery
> of callback approaches to this problem... maybe because our
> past experience with the U-Mich LDAP transcoding callback
> was quite bad.  Maybe we've learned enough from that
> experience to actually now do it right.  But then again,
> one has to wonder whether the approach is workable at all.

Well, the U-M callbacks were definitely in the wrong place.  I think my
solution will work.  If you disagree, I'd like to hear if you have any
alteriative in mind and how it would do better.

-- 
Hallvard