All, IBM has implemented the Netscape Directory Server Plug-in API, for its own Directory Server implementation over three years ago. IBM would like to contribute this implementation to the OpenLDAP. I am the coordinator and full time senior software developer assigned to this task. I have authored the attach design document stating the characteristics of this implementation and the required steps to integrate the package, seamlessly with the OpenLDAP. We welcome any comments and feedbacks. Steve Omrani (See attached file: design_doc.html)Title: design_doc
Phase One
Contributed by IBM Corporation
by Steve Omrani
Table of Contents
This document describes the design, implementation and the necessary changes to the internal data structures and algorithms of the OpenLDAP Directory Server. These changes allow the Netscape SLAPI implementation contributed by the IBM Corporation to be integrated with the OpenLDAP Directory Server. This implementation, provides the Application Programming Interface (API) and the library (libslapi. so | sl | dll | a) that facilitate the development and integration of various plug-ins for the OpenLDAP Directory Server. The plug-ins can be used to extend the functionality of the OpenLDAP Directory Server in many ways, e.g., plug-ins can be used to develop new database backends, or to validate date before storage in the Directory Server (Per-Operation) or to notify users after the data has changed in the Directory Server (Post-Operation).
OpenLDAP Directory Server currently supports a unique form of plug-ins (modules). These plug-ins, however, can only be of the backend/database type and must be compiled as a component of the entire source tree package (some special flags must be specified when configure utility is run). On the other hand, Netscape SLAPI, offers seven different types of plug-ins that can be developed and installed at customer sites without the need for the complete source tree.
SLAPI is not a replacement for the current OpenLDAP API. It is an additional API. Both APIs and hence, the plug-ins developed based on the two APIs will coexist.
It is intended that SLAPI be an optional feature that by default is not include in the final package. Customers requiring this feature must specify a special flag when the "configure" utility is run (of course we can change the sense of the default if necessary).
The IBM implementation of SLAPI is both a subset and a superset of the Netscape API. It is a subset because it does not implement all the APIs and it does not support all the parameters. It is a superset because it offers additional set of parameters and APIs and a new plug-in type. Furthermore, the implementation includes a set of pre developed plug-ins such as DB2 backend/database, ChangeLog, Audit and EventNotification that could be useful to OpenLDAP users.
This contribution is ported in three phases and this document will be updated and reissued with the details for each phase. In phase one (this issue of the document) the SLAPI API proper (libslapi.*) and the necessary code to support Pre-Operation, Post-Operation and the extended-Opeartion plug-ins will be ported. In phase two, support for the IBM relational database (DB2) as backend/database using this API will be added. Finally, in phase three, a set of IBM developed plug-ins such as Audit, ChanageLog and EventNotification will be ported.
For a complete list of Netscape defined parameters and description of each parameter, please refer to "Netscape Directory Server Plug-in Programmer Guide", Chapter 15, Parameter Reference.
The IBM Supported parameters are listed according to plug-in types.
SLAPI_PLUGIN_PRE_BIND_FN
SLAPI_PLUGIN_PRE_UNBIND_FN
SLAPI_PLUGIN_PRE_ADD_FN
SLAPI_PLUGIN_PRE_DELETE_FN
SLAPI_PLUGIN_PRE_SEARCH_FN
SLAPI_PLUGIN_PRE_COMPARE_FN
SLAPI_PLUGIN_PRE_MODIFY_FN
SLAPI_PLUGIN_PRE_MODRDN_FN
SLAPI_PLUGIN_PRE_ABANDON_FN
SLAPI_PLUGIN_POST_BIND_FN
SLAPI_PLUGIN_POST_UNBIND_FN
SLAPI_PLUGIN_POST_ADD_FN
SLAPI_PLUGIN_POST_DELETE_FN
SLAPI_PLUGIN_POST_SEARCH_FN
SLAPI_PLUGIN_POST_COMPARE_FN
SLAPI_PLUGIN_POST_MODIFY_FN
SLAPI_PLUGIN_POST_MODRDN_FN
SLAPI_PLUGIN_POST_ABANDON_FN
Parameters for Database Backend Support
SLAPI_PLUGIN_DB_BIND_FN
SLAPI_PLUGIN_DB_UNBIND_FN
SLAPI_PLUGIN_DB_ADD_FN
SLAPI_PLUGIN_DB_DELETE_FN
SLAPI_PLUGIN_DB_SEARCH_FN
SLAPI_PLUGIN_DB_COMPARE_FN
SLAPI_PLUGIN_DB_MODIFY_FN
SLAPI_PLUGIN_DB_MODRDN_FN
SLAPI_PLUGIN_DB_ABANDON_FN
SLAPI_PLUGIN_DB_INIT_FN
-- IBM Special
SLAPI_PLUGIN_DB_CONFIG_FN
SLAPI_PLUGIN_DB_CLOSE_FN
Parameters for Extended Operation
SLAPI_PLUGIN_EXT_OP_FN
SLAPI_PLUGIN_EXT_OP_OIDLIST
Parameters for Threads Operation
SLAPI_PLUGIN_DB_THREAD_INITIALIZE_FN
-- IBM Special
SLAPI_PLUGIN_DB_THREAD_TERMINATE_FN
-- IBM Special
Parameters for Schema Operation
SLAPI_PLUGIN_DB_SCHEMA_MODIFY_ATTRTYPE_FN
-- IBM Special
SLAPI_PLUGIN_DB_SCHEMA_MODIFY_OBJCLASS_FN
-- IBM Special
Parameters for Replication Operation
SLAPI_PLUGIN_DB_REGISTER_SERVICE_FN
-- IBM Special
SLAPI_PLUGIN_DB_INSERT_REPL_ENTRIES_FN
-- IBM Special
SLAPI_PLUGIN_DB_GET_REPL_ENTRIES_FN
-- IBM Special
SLAPI_PLUGIN_DB_REPLICA_DONE_FN
-- IBM Special
SLAPI_PLUGIN_DB_INIT_REPL_LIST_FN
-- IBM Special
For a complete list of Netscape defined frontend APIs available to plug-ins and full description of each API, please refer to "Netscape Directory Server Plug-in Programmer Guide", Chapter 14, Function Reference.
The IBM supported APIs are listed according to subsystems/functions, rather than plug-in types.
char *slapi_ch_malloc(unsigned long size)
void slapi_ch_free(void *ptr)
char *slapi_ch_calloc(unsigned long nelem, unsigned
long size)
char *slapi_ch_realloc(char *block, unsigned long
size)
char *slapi_ch_strdup(char *s)
APIs for Slapi_PBlock Management
Slapi_PBlock *slapi_pblock_new()
void slapi_pblock_destroy(Slapi_PBlock *pb)
int slapi_pblock_get(Slapi_PBlock *pb, int arg,
void *value)
int slapi_pblock_set(Slapi_PBlock *pb, int arg,
void *value)
APIs for Sending Result Back to Client
slapi_send_ldap_result(Slapi_PBlock *pb, int err,
char *matched, char *test,
int nentries, struct berval **urls)
slapi_send_ldap_search_entry(Slapi_PBlock *pb, Slapi_Entry
*e,
LDAPControl **ectrls, char **attrs, int attrsonly)
APIs for Management of LDAP Objects
char *slapi_dn_normalize(char *dn)
char *slapi_dn_normalize_case(char *dn)
int slapi_dn_issuffix(char *dn, char *suffix)
char *slapi_dn_ignore_case( char *dn )
int slapi_dn_isroot( Slapi_PBlock *pb, char *dn
)
int slapi_attr_get_values(Slapi_Attr *attr, strucy
berval ***vals)
Slapi_Entry *slapi_entry_dup(Slapi_Entry *e)
int slapi_entry_delete(Slapi_Entry *e)
int slapi_entry_attr_find(Slapi_Entry *e, char *type,
Slapi_Attr**vals)
char slapi_entry_get_dn(Slapi_Entry *e)
void slapi_entry_set_dn(Slapi_Entry *e, char *dn)
Slapi_Entry *slapi_str2entry(char **s, int flag)
void slapi_filter_free(Slapi_Filter *f, int recurse)
Slapi_Filter *slapi_str2filter(char *str)
int slapi_filter_get_choice(Slapi_Filter *f)
int slapi_filter_get_ava(Slapi_Filter *f, char **type,
struct berval **bval)
Slapi_Filter *slapi_filter_list_first(Slapi_Filter
*f)
Slapi_Filter *slapi_filter_list_next(Slapi_Filter
*f, SlapI_Filter *fprev)
char *slapi_entry2str(Slapi_Entry *e, int *len)
int slapi_log_error(int severity, char *subsystem,
char *frmt, ...)
void slapi_entry_free(Slapi_PBlock *e)
int slapi_pw_find(struct beval **vals, struct beval
*v)
Slapi_Entry *slapi_entry_alloc()
int slapi_entry_attr_merge(Slapi_Entry *e, char
*type, sruct berval **vals)
APIs for Management of SASL Mechanisms and Controls
void slapi_register_supported_control(char *controloid,
unsigned long controlops)
int slapi_get_supported_controls(char ***ctrloidsp,
unsigned long **ctrlopsp)
void slapi_register_supported_saslmechnisms(char
*mechanism)
char **slapi_get_supported_saslmechanisms()
int slapi_control_present(LDAP **controls, char
*oid struct berval **val, int *iscritical)
slapi_get_supported_extended_ops(void)
void slapi_printmessage(int catid, int level, int
num, ... )
char *slapi_get_hostname()
void slapi_broadcast_be(int funcType, Slapi_PBlock
*pPB)
int slapi_is_connection_ssl(Slapi_PBlock *pPB, int
*isSSL)
int slapi_get_client_port(Slapi_PBlock *pPB, int
*fromPort)
int slapi_get_num_be(char *type)
unsigned long slapi_timer_current_time()
unsigned long slapi_timer_get_time(char *label)
void slapi_timer_elapsed_time(char *label,unsigned
long start)
slapi_send_ldap_extended_response(Connection *conn,
Operation *op,
int errornum, char *respName, struct berval *response)
APIs for Management of DataBase/Backend Operations
Slapi_PBlock *slapi_search_internal( char *base,
int scope, char *filter,
LDAPControl **controls, char **attrs, int attrsonly )
Slapi_PBlock *slapi_modify_internal(char *dn, LDAPMod
**modes,
LDAPControl **controls, int log_change)
Slapi_PBlock *slapi_add_entry_internal(Slapi_Entry
*e, LDAPControl **controls,
int log_change)
Slapi_PBlock *slapi_add_internal(char *dn,
LDAPMod **attrs,
LDAPControl **controls, int log_change)
Slapi_PBlock *slapi_delete_internal(char *dn, LDAPControl
**controls, int log_change)
Slapi_PBlock *slapi_modrdn_internal(char *olddn,
char *newrdn, int delolddn,
LDAPControl **controls, int log_chnage)
char **slapi_get_supported_extended_ops(void)
Mapping of SLAPI APIs to OpenLDAP APIs
This section describes the mapping from SLAPI APIs to OpenLDAP APIs. Some of the mapping are simple and one-to-one, while others require more than one OpenLDAP functions.
The notation "this -> that" implies that the SLAPI API is essentially a wrapper for the OpenLDAP API and the notation "this -> No direct mapping" means low level code has to be written in order to provide the required SLAPI functionality.
Mapping for Memory Management APIs
slapi_ch_malloc -> ch_malloc
slapi_ch_calloc -> ch_calloc
slapi_ch_realloc -> ch_realloc
slapi_ch_strdup -> ch_strdup
slapi_ch_free -> ch_free
NOTE: The prototype for slapi_ch_free, per Netscape documentation is "void slapi_ch_free(void **ptr)". However, IBM implementation requires "void slapi_ch_free(void *ptr)".
Mapping for Slapi_PBlock Management APIs
slapi_pblock_new -> No direct mapping
slapi_pblock_destroy -> No direct mapping
slapi_pblock_get -> No direct mapping
slapi_pblock_set -> No direct mapping
Mapping for Sending Result Back to Client APIs
-> send_ldap_sasl
-- if sasl in progress is indicated
slapi_send_ldap_result -> send_ldap_extended
-- if there is a response OID
-> send_ldap_result
-- otherwise
slapi_send_ldap_search_entry -> send_search_entry
Mapping for Management of LDAP Objects APIs
slapi_dn_normalize -> dn_normalize
slapi_dn_normalize_case -> dn_normalize
slapi_dn_issuffix -> dn_issffix
slapi_dn_ignore_case -> dn_normalize
slapi_dn_isroot ->be_isroot
slapi_attr_get_values -> No direct mapping
Slapi_Entry *slapi_entry_dup -> No direct mapping
slapi_entry_delete -> attr_delete
slapi_entry_attr_find -> attr_find
char slapi_entry_get_dn -> No direct mapping
slapi_entry_set_dn -> No direct mapping
slapi_str2entry -> str2entry
slapi_filter_free -> filter_free
slapi_str2filter -> str2filter
slapi_filter_get_choice -> No direct mapping
slapi_filter_get_ava -> get_ava
slapi_filter_list_first -> No direct mapping
slapi_filter_list_next -> No direct mapping
slapi_entry2str -> entry2str
slapi_log_error -> No direct mapping
slapi_entry_free -> entry_free
slapi_pw_find -> no direct mapping
slapi_entry_alloc -> No direct mapping
slapi_entry_attr_merge -> attr_merge
Mapping for Management of SASL Mechanisms and Controls APIs
slapi_register_supported_control -> No direct mapping
slapi_get_supported_controls -> No direct mapping
slapi_control_present -> No direct mapping
slapi_register_supported_saslmechnisms -> No direct
mapping
slapi_get_supported_saslmechanisms -> No direct
mapping
slapi_search_internal -> No direct mapping
slapi_modify_internal -> No direct mapping
slapi_add_entry_internal -> No direct mapping
slapi_add_internal -> No direct mapping
slapi_delete_internal -> No direct mapping
slapi_modrdn_internal -> No direct mapping
slapi_get_supported_extended_ops -> No direct mapping
A new data structure (Slapi_PBlock) is introduced. The purpose of the data structure is to maintain relationships between SLAPI parameters ( IBM Supported Parameters) and user supplied plug-in function pointers. The structure is actually a C++ class encapsulating two array elements (int [], and void *[]) and the set of functions that manipulate these elements ( APIs for Slapi_PBlock Management). Form the C++ advance features point of view, the class is rudimentary (no inheritance, no virtual function, no overloading, ...) and if it deemed necessary (based on port difficulties) I will convert it to regular C structure.
Both OpenLDAP and IBM Directory Servers are based on the University of Michigan original implementation of LDAP protocol (http:/www.umich.edu/~dirsvcs/ldap) and hence used the same data structures. However, over the years, these structures diverged to the point where synchronization is impractical. For example, the U-M Backend data structure has 27 primitive data members, while the IBM implementation has 49 primitive and constructed data members and OpenLDAP split the Backend structure into two structures; Backend and BackendInfo for a total of 58 primitive and constructed members. Furthermore, the data members added by IBM and OpenLDAP are non intersecting. Fortunately, for purpose of this program, such synchronization seems unnecessary.
For phase one, only one data member (void *) is added to the Backend, Operation and Connection structures. These data members point to the same instant of Slapi_PBlock object. Sharing this object eliminates the need for changing signature of the OpenLDAP front-end functions that do not receive a pointer to an object of the Backend type.
Implementation of LDAP from U-Mich used the terms "backend" and "database" somewhat interchangeably (and confusingly). While the data structure managing the information about a database is called "backend", there is no directive or keyword in slapd.conf that specify the start of a backend. In addition, read_config function (config.c) does not parse the directive "backend" while reading slad.conf configuration file. However, when read_config encounters the keyword "database", it calls the function "new_backend!" (OpenLDAP changed the name of this function to backend_db_init which is somewhat more meaningful) to create a database of that particular type. IBM implementation of LDAP did not change the U-Mich model. On the other hand, OpenLDAP makes a distinction between "backend" and "database". In general, one or more "databases" are associated with a backend type and common information about the databases within a backend are contained in the BackendInfo structure and shared by all the databases of that particular type. It should also be noted that in OpenLDAP the name of the backend and the database must be the same, i.e., ldbm.
Netscape plug-ins can be classified as database plug-in type and non database plug-in type. The non database plug-in types must be associated with a database, while the database plug-in types need an associated backend object.
This section describes the general Architecture of the IBM implementation of SLAPI framework and the details of the supported set of plug-in types (Phase One plug-ins only). We will give a brief description of the new files that we will introduce and changes we will make to the existing files.
A new directive of the form
plugin <plugin_type> <lib_path> <init_function> [arguments]
will be added to slapd.conf file following a database directive. This positional relationship (the plug-in directives follow the database directive) associates one or more plug-ins with a particular database. For Phase One, the <plugin_type> is one of peroperation, postoperation, and extendedop keywords that specifies the type of the plug-in. The <lib_path> is the absolute path to the user supplied shared library that implements the plug-in. The <init_function> is a function from the shared library that must be executed once the library is loaded. Note that the OpenLDAP "modulelaod" plug-in mechanism does not require the initialization function to be specified, however, it requires the function to be named "init_module". SLAPI API is more flexible, it allows arbitrary names for the initialization function, but the function name must be specified. Finally, the optional [arguments], if specified, are made available to all functions of the plug-in in argc, argv forms.
It should be noted that Netscape defines two formats for the plugin directive. We may add support for the second format at some future time.
The function read_config (config.c) will be enhanced to parse the above directive. Parsing of this directive is similar to the parsing of any other slapd.conf directive. Once all checks are done, if the plug-in is of type preoperation, or postoperation the function "newPlugin" is called to create a plug-in of that particular type, then the function "insertPlugin" is called to associate this newly created plug-in with its database. For the extendedop type, four functions are called in the following order; "newPlugin", "newExtendedOp", "insertPlugin", and "load_extop". This last function is an OpenLDAP function that adds this extended operation object to the OpenLDAP "supp_ext_list" list.
The code for the SLAPI interface consists of three source files, four header files and and one Makefile.in file, all are located in the new server/slapd/slapi directory (I could move it one level up to server/slapi if necessary). During compilation, typical OpenLDAP libraries will be generated. And during installation, the library "libslapi" (static an dynamic versions) will be placed in /${prefix}/lib and one of the header files, "slapi-plugin.h" is place in /${prefix}/include.
The two files slapi_pblock.[cpp | h] implement the APIs for Slapi_PBlock Management that are available to end users. There are two versions of the Slapi_PBlock structure; one version uses an "avl tree" for the storage of parameters and user supplied function pointers. The other, uses simple arrays of "int" an "void *". The choice of the structures is a compilation time issue.
The plugin.[cpp | h ] files implement a set of support functions that in general are of no interest to end users. The function
Slapi_PBlock *newPlugin(int type, char *path, char *initfunct, int argc, char *argv[])
is called from read_config to create a new plug-in of the particular type when read_config encounters the plugin directive. This function loads the shared library specified by path and executes the initialization function specified by initfunct. The function makes argc and argv available to the initialization function and all the plug-in functions by adding them to Slapi_PBlock object.
If the directive is of the type "extendedop", read_config calls
int newExtendedOp(Backend *pBE, ExtendedOp **opList, Slapi_PBlock *pPB)
to extract the OIDs and the function pointer from the newly created Slapi_PBlook object and construct an ExtendedOp object for each OID and link them to the global opList.
Finally, read_config calls
int insertPlugin(Backend *be, Slapi_PBlock *pPB)
to associate the plug-in with its backend/database.
It should be noted one backend/database may be associated with N plug-ins. There is one Slapi_PBlock object for each plug-in, forming a link list with the head of the list located in the Backend object.
Finally these files implement the function
int doPluginFNs(Backend *be, int funcType, Slapi_PBlock * pPB)
that allows communication between the OpenLDAP frontend functions (do_xxx) and all preoperation and postoperation plug-ins.
The remaining supported APIs, APIs for Memory Management , APIs for sending Result Back to Client , APIs for Management of LDAP Objects , APIs for Management of SASLMechanism and Control and IBM Special API are all implemented in slapi_utils.[cpp | h] files. With the exception of the IBM Special API and the slapi_ch_free() signature, all other APIs have exactly the same syntax and semantic as define by the Netscape Directory Server Programmer's Guide.
The following is a brief description of the IBM Special API. Complete
man pages will be provided when the port is complete.
The peroperation / postoperation plug-ins can be defined for the following LDAP requests: add, bind, compare, delete, modify, modrdn, search and unbind. Consequently, OpenLDAP files add.c, bind.c, compare.c, delete.c, modify.c, modrdn.c, search.c and unbind.c need to be modified.
The two types of plug-ins may be associated with an LDAP request simultaneously or separately. They are totally independent although require the same set up and proceesing.
The design patterns for the code fragment that need to be added to each file is uniform for all LDAP requests. For preoperation, just before the backend function (*be->be_xxx) is called, a test is made to determine if there are one or more plug-ins associated with this backend. For the postoperation the test is made just after the (*be->be_xxx) function call returns. In either case, the necessary parameters are pushed (set) onto the Slapi_PBlock object and the function "doPluginFNs" is called. The "doPluginFNs" identifies the set of user supplied plug-in functions, and indirectly call each one and return the results to its caller.
The "Extended Operations" introduced via the SLAPI API, are internally maintained in a separate list, however, they are also linked to the "supp_ext_list" list so that the OpenLDAP "get_supported_extop" and the SLAPI "slapi_get_supported_extended_ops" report the complete list of supported extensions regardless of how they are introduced.
The code for processing the "Extended Operation" consists of two parts: a small code fragment is added to the OpenLDAP server/slapd/extended.c file that determines if this particular extended operation was introduced via the SLAPI API. If so, it makes the necessary set up (adds the OID, user data, the connection object and the operation object to the Slapi_PBlock object) and indirectly calls the user supplied function (the second part) to do the actual processing. Otherwise it passes control to OpenLDAP proper code.
The development environment is the LINUX 6.2 from Red Hat. However, the code will be compiled and minimally tested on other supported platforms.