FTree backend module (ITS#1499)

[A.J.Martin@qmul.ac.uk]

Full_Name: Alex Martin
Version: 2.0.18 
OS: Linux  
URL: ftp://ftp.openldap.org/incoming/Alex-Martin-back-ftree-1.0.0pre7-011218.tar.gz
Submission from: (NULL) (138.37.51.209)


The following files contain my FTree module + patches 
 
Alex-Martin-back-ftree-1.0.0pre7-011218.tar.gz
Alex-Martin-ldap-ftree-011218.patch


Here is its README:



                        FTree slapd module
                        ==================


Introduction
============

The FTree slapd module is designed to implement a LDAP DIT using a flexible
memory-based directory-like tree structure. Access to LDAP entries stored in
the
tree is protected by read-write mutexes in order to facilitate efficient 
threaded operation. The tree structure is initiated from the slapd
configuration
file. A basic sample configuration could be:


        database                ftree
        suffix                  "ou=myOrg"
        FtRootDn                ou=myOrg
        FtRootObjectClass       Top


The lines starting with "FtRoot" are internally converted into LDIF format and 
used to initialize the root of the DIT. In this case the root would be set to 
be:

        dn: ou=myOrg
        objectclase: Top


Once initialized, additional entries may added and deleted from backend using
normal LDAP operations.


Entry Types
===========

Although a basic memory-based backend provides useful functionality, a key 
design  feature of the FTree module is the implementation of entries in the 
memory-based structure as polymorphic types. The type information of entries is
defined by the value(s) of the ObjectClass attribute and the sub-classed 
functionality is loaded into the server by means of additional modules (shared 
libraries). When an entry is added to the server its type information is 
determined and compared with those registered by any loaded modules. If it is 
found to match it is associated with the corresponding function table. When a 
type match is not found, basic default behaviour is assumed.       

This method is used to implement the "special" LDAP entry types  Alias,
Referral
and also DynamicObject ( rfc2589 ). The functionality to support these types
may
 be loaded into the server by the addition of the lines:    

        moduleload      back_ftree_alias.la
        moduleload      back_ftree_referral.la
        moduleload      back_ftree_dynamic.la


to the slapd configuration file. 



External Entries
================

The concept of LDAP entries with overloaded functionality is expanded further 
to allow additional user-defined types. The intention here is to provide an 
API which can facilitate the addition of more modular functionality. 

One use of this concept is the integration of external information sources
e.g. scripts, files and other LDAP servers into the DIT. At a basic level
this can be thought of as a flexible version of some of the existing 
OpenLDAP backends i.e. back-shell, back-ldap. However, unlike the existing
backends, multiple entries corresponding to these external info sources        

may be freely added and deleted from a server. The schema defining these 
types are contained in the file 
%SYSCONFIGDIR%/schema/openldap-ftree.schema.  

This functionality is loaded via the modules:


        moduleload      back_ftree_external.la
        moduleload      back_ftree_exec.la
        moduleload      back_ftree_ldap.la   

External is an abstract base class whose methods are implemented by the Exec
and
Ldap classes.Examples of their use are provided in the files exec-example.ldif
and ldap-example.ldif in the demos directory %SYSCONFDIR%/demos:  


	dn: in=loadaverage1, dc=localhost, dc=localdomain, ou=myOrg  
	objectClass: exec
	objectClass: external
	command: loadaverage
	arguments: in=loadaverage1,dc=localhost,dc=localdomain,ou=myOrg

This entry specifies the external info source is a command "loadaverage"
with arguments "in=loadaverage1,dc=localhost,dc=localdomain,ou=myOrg". 
For security reasons, the path used to locate commands is defined by a class 
variable which is set by the "execpath" variable in the configuation file. 
The API used when forking the command is the same as that used in the standard
back-shell backend module.

A similar configuration specifying another LDAP server as an info source
is:

	dn: dc=localhost1, dc=localdomain, ou=myOrg
	objectClass: ldap
	objectClass: external
	ref: ldap://localhost:2171/dc=localhost,dc=localdomain,ou=myOrg

Here "ref" specifies the URL of the external server using the same 
syntax as a referral.



Caches
======

An additional feature provided by the FTree backend is the ability to 
setup a cache of an external info source.  The basic functionality is 
provided by abstract classes "Cache"  and "ThreadedCache".  "CacheSearches"
implements a simple non-threaded model where search results are cached. 
Several threaded cache models are provided "CachePeriodically", where a thread
periodically updates the cache from its source and "CacheFromStream", where  
the info source pushes updates into the cache. In addition there is a threaded 
cache model "FlushPeriodically" in which the memory contents are flushed to a 
backing store. All of this core functionality is loaded via:

	moduleload      back_ftree_cache.la 

"FlushPeriodically" may be used to provide a simple persistent store 
for the contents of the server. A root configuration:

        ldifpath        %SYSCONFDIR%/demos 
        ...
        ... 
	FtRootDn           ou=myOrg
	FtRootObjectClass  Ldif
	FtRootObjectClass  FlushPeriodically
	FtRootObjectClass  ThreadedCache
	FtRootObjectClass  Cache
	FtRootObjectClass  ThreadedCache
	FtRootFile         exec-example.ldif
	FtRootUpdatePeriod 60 
      
Maps the root of the server to the contents of the ldif file exec-example.ldif
residing in the directory %SYSCONFDIR%/demos. The ldif file 
will be read when the server starts and the current in-memory contents of the 
server will be flushed backed to this file every 60 secs.

Other Cache Examples are provided in the exec-example.ldif file:

	dn: in=loadaverage2, dc=localhost, dc=localdomain, ou=myOrg
	objectClass: exec
	objectClass: cachesearches
	objectClass: caches
	command: loadaverage
	arguments: in=loadaverage2,dc=localhost,dc=localdomain,ou=myOrg
	validFor: 60

Specifies a simple cache in which the results of any search handled  
by the "loadaverage" script are cached and assumed to be valid for 60 seconds.


	dn: in=loadaverage3, dc=localhost, dc=localdomain, ou=myOrg
	objectClass: exec
	objectClass: cacheperiodically
	objectClass: threadedcache
	objectClass: cache
	command: loadaverage
	arguments: in=loadaverage3,dc=localhost,dc=localdomain,ou=myOrg
	updatePeriod: 60

Specifies a threaded cache should be setup. A thread is started which 
periodically updates the cache by running "loadaverage" every 60 seconds.

	dn: in=loadaverage4, dc=localhost, dc=localdomain, ou=myOrg
	objectClass: exec
	objectClass: cachefromstream
	objectClass: threadedcache
	objectClass: cache
	command: loadaverage-stream
	arguments: in=loadaverage4,dc=localhost,dc=localdomain,ou=myOrg
	retryAfter: 60

Also specifies a threaded cache should be setup. However, in this case, 
the command forked, "loadaverage-stream", is assumed to continue running 
and stream updates back to the server. Individual update records are 
indicated by a terminating "RESULT" string. If the command exits the 
server thread will attempt to restart it after 60 seconds.

The same cache methods are also implemented when another LDAP server forms  
the information source (see  ldap-example.ldif ). In order to facilitate
the streaming of LDAP updates between servers the Persistent Search
(draft-ietf-ldapext-psearch-03.txt) draft standard has been adapted.
This may also be utilised by clients, and a simple example ( ldap_psearch )   
is provided.