[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: commit: ldap/servers/slapd/back-perl SampleLDAP.pm

Hello Hallvard,

Am 19.06.2007 um 17:22 schrieb Hallvard B Furuseth:
Dagobert Michelsen writes: 
- each Perl backend database has a separate Perl interpreter
- each connection gets an interpreter cloned from the specific
   backend. Interpreters are recycled in a pool for performance
- concurrent requests to different interpreters can run
   concurrently in separate threads

Is there a max number of Perl interpreters per backend? 

I suppose you mean "per database". In my understanding there
are different backends coded in slapd/back-* where the back-perl
I am working on is one backend which can be instantiated in
slapd.conf. There may be more databases configured for the
Perl backend.

There is exactly one interpreter in use for each active connection
and each configured back-perl database because allocation is done
on connection_init.

The maximum number of interpreters is the maximum count of
connections (dtblsize) multiplied with the number of back-perl
databases.

What happens if there are more long-lived connections than that? 

As this is the maximum this can never be the case.

Does a connection release its Perl interpreter to the pool if
a few requests on the connection use the Perl database and then
the rest of the requests use a non-Perl database?


Perl interpreters are allocated when a connection to OpenLDAP
is made and released to the pool when the connection is terminated.

This method has some advantages:
- As access to the Perl interpreter must be exclusive there is
  no contention because no two operations can be active for one
  connection at the same time
- Data from a connection is automatically available in all method
  invocations for this connection as the calls are all directed to
  the same Perl interpreter

The disadvange is that there is no data sharing between connections.
Implementing data sharing with this model is quite complex and
involves attaching (Perl-)magic to shared variables which implements
mutex'ed access between interpreters.

Currently Perl interpreters are never freed once they are released.
Releasing of more than, say, 5 idle interpreters may be implemented
as an optimization. I look into that if I have a fully working
prototype.

Are you using threads created by Perl, or ldap_pvt_thread_create(),
or are you somehow giving the interpreters short-lived tasks via
ldap_pvt_thread_pool_submit()?  There are some slapd features which
can only be used if you do the latter, since that creates a
"thread context" which holds thread-specific variables.


I don't do any explicit thread management. All threads have
been set up by OpenLDAP when the backend entrypoints are called,
so I assume that this is done right :-)

- argument passing is done in a style similar to Net::LDAP,
   parsing is no longer needed as attributes and values are
   separately stored


You might not even need to do that.  Could wrap an Entry in a class
which translates slapd attributes to Perl lists on demand.


This is possible, however not trivial: The C data structures
would be implemented as blessed scalars containing the C pointer.
Access method could then convert the data on demand. However,
the C pointer must be some kind of "weak" pointer which is
only valid in the context of this method invocation. Otherwise
the scalar with the C reference could be stored in the interpreter
and deferenced later in another method entry where the C data
has already been freed. This all is a bit complex and can be
done later without API change if proven useful.

As the method usually does something with the parameters the data
is needed anyway, so savings here should be minimal if not negative.
Another thing is access to seldom used data like connection
parameters or operation details where transformation to your
proposed structure would indeed be useful.

- switching to old-backend-compatible-mode is done when
   no connection_init method is found


That sounds like a hack, one might not be interested in defining a
connection_init method anyway.


In my current implementation it is mandatory to implement
connection_init. The returned value must be a reference to
an object on which the methods for the operations are called
(like search, compare, etc.)

If you have a usecase for an application without connection_init
I would like to hear it.

The new backend is not 100% compatible to the old backend
due to the lack of data sharing. This is a complex issue
but maybe solvable with the help of the mod_perl code.
Alternatively the code could fall back to a single
interpreter when compatibility mode is on.


Unless the incompatibilities are quite small:

I think it would be a bad idea to automatically invoke an "almost-
compatible" mode.  Then a number of old Perl programs can break
mysteriously, and maybe munge whatever data they are maintaining
before it is discovered.

I suggest you make the new database fail to configure if provided an old
Perl program and vice versa. 

Ok. I see the importance now and make the changes accordingly.
To summarize:
- Exit with fault if old Perl-module is used with new backend
- Provide compatibility-module which must be enabled explicitly
- Exit with fault if new Perl-module is used with old backend

I also suggest you do not let compatibility get in the way of good
design decisions for the new code.  If good design leads to bad
compatibility, we could always put the old back-perl in
contrib/back-oldperl or something.


Including the old backend is something I really would like to
avoid as this means adapting autoconf-stuff and be confusing.

It is still not possible to do all things from Perl which
can be done in C. Exposing more datastructures and functions
to Perl may even make it possible to write overlays in Perl
in some distant future.

Both are possible, in particular if you provide Perl types as wrappers
around the basic C types instead of translating them to arrays or hashes
before delivering them to Perl. (If you've translated them it's a bit
of a problem to call a number of C functions because you've lost the
original data structure which you would pass _to_ the C function. 

This is true but otherwise you need "weak" references or induce
the possibility of subtle bugs with dangling pointers. If this
feature is provided it must be as safe as possible in my opinion.

Or the user has made updates in Perl structures which are not reflected in
the C structures they came from. I looked into doing this with Python
once, but punted since the Python C API #includes a header with Python
autoconf variables which conflict with slapd's autoconf variables. 

Do you have some design notes of this left which you can send me?

When I have something which is usable for testing I'll post.


Best regards

  -- Dagobert


BTW: For my personal use I set up OpenGrok for browsing the
     OpenLDAP sources which I really enjoy. Feel free to have
     a look at http://openldap.baltic-online.de

--
Dagobert Michelsen (Leiter IT)          Baltic Online Computer GmbH
Firmensitz:   Alter Markt 1-2, 24103 Kiel, Telefon: +49 431 54003-0
Geschäftsführer:        Erik Cickovskis, Amtsgericht Kiel, HRB 3756
"Of course computer servers don't need thrust, since they generally
don't go anywhere."  -- Comment in TR on new HP server turbine fans