RE: back-sql improvements (was: Slapd frontend performance issues)

["Eric T. Blue" <openldap@star-cs.com>]

Dmitry & Tomas,
	I've spent a little time over the last couple weeks, and have a barebones
OCI backend working.  The performance looks very good thus far(~ 500
searches / sec), but I'd still like to do the connection pool.  Currently, I
initiate 1 persistent OCI connection on db_init and subsequently perform
searches using the same cursor/handle (obviously locking via a mutex).  For
some reason, and it may be my lack of understanding of the underlying
OpenLDAP code, utilizing multiple OCI connections doesn't seem to contribute
towards speed in any way.  Here's the basic setup:

The speed is fairly fast, since there is only support for equality searches
right now.  Basically, a filter of uid=test translates to "select col1,col2
.. colz from table where (uid='username')".


============= SNIPPETS START HERE =============

// back-oci.h

#define CONNECT_POOL_SIZE		10

typedef struct {

 int conn_inuse;
 int conn_ok;
 Lda_Def lda;
 Cda_Def cda;
 ub4     hda[HDA_SIZE/(sizeof(ub4))];

}backoci_db_conn;

backoci_db_conn *dbc[CONNECT_POOL_SIZE];
ldap_pvt_thread_mutex_t dbconn_mutex;


// init.c

I do a loop to init the cursor, data area, etc.

sprintf(connect_string,"%s/%s@%s",si->dbuser,si->dbpasswd,si->dbname);

for (i=0; i <= CONNECT_POOL_SIZE; i++) {

        dbc[i]=(backoci_db_conn*)ch_calloc(1,sizeof(backoci_db_conn));
        dbc[i]->conn_inuse = 0;

        /* Authenticate to Oracle */
        if (olog(&dbc[i]->lda, (ub1 *)dbc[i]->hda, connect_string, -1,
0, -1, (text *) 0, -1, (ub4)OCI_LM_DEF)) {
                Debug(LDAP_DEBUG_TRACE,"backoci_db_open(): Oracle connection
failed, exiting\n",0,0,0);
                return 1;
        }


        /* Open a cursor, exit on error (unrecoverable). */
        if (oopen(&dbc[i]->cda, &dbc[i]->lda, (text *) 0, -1, -1, (text *)
0, -1)) {
                Debug(LDAP_DEBUG_TRACE,"backoci_db_open(): Oracle - can't
open cursor\n",0,0,0);
                ologof(&dbc[i]->lda);
                return 1;
        }
}

ldap_pvt_thread_mutex_init(&dbconn_mutex);


// search.c

for (i=0; i <= CONNECT_POOL_SIZE; i++) {
        if (dbc[i]->conn_inuse == 0) {
                freeconn = i;
                break;
        }
}

dbc[freeconn]->conn_inuse = 1;
ldap_pvt_thread_mutex_lock(&dbconn_mutex);

...... DO OCI specific fcns and return results to client

dbc[freeconn]->conn_inuse = 0;
ldap_pvt_thread_mutex_unlock(&dbconn_mutex);

============= SNIPPETS STOP HERE =============

Is this a problem with thread serialization, or the mutex?  My logic may be
slightly flawed here, but I'm sure the solution is simple.  I'd appreciate
any input or suggestions.













-----Original Message-----
From: owner-openldap-devel@OpenLDAP.org
[mailto:owner-openldap-devel@OpenLDAP.org]On Behalf Of Arredondo, Tomas
Sent: Tuesday, April 10, 2001 10:05 AM
To: 'mitya@seismic.ru'
Cc: openldap-devel@OpenLDAP.org
Subject: RE: back-sql improvements (was: Slapd frontend performance issues)


Hi Dmitry,

Comments inline.

Regards,
Tomas

-----Original Message-----
From: Dmitry Kovalev [mailto:mitya@seismic.geol.msu.ru]
Sent: Tuesday, April 10, 2001 6:52 AM
To: Arredondo, Tomas
Cc: openldap-devel@openldap.org
Subject: back-sql improvements (was: Slapd frontend performance issues)


Hello Tomas!

As this thread eventually spinned down to back-sql, I thought it would be
reasonable to rename it...
I think two separate threads are appropriate - one about back-sql - specific
issues, and another - about slapd frontend profiling.

Also, if you have no objections - let us transfer posts that have more than
one
recipient to the list.
As I understand, it is what Kurt suggested - use -devel list for
development.
Kurt, if you think that too much back-sql specifics is bad for this list - I
could arrange a temporal list for this, and post only results to
openldap-devel.


"Arredondo, Tomas" wrote:

> I think that in an optimal environment it should exceed 100 reads/second.

did you test the "packaged" back-sql, or you made some optimizations? what
was
the data you tested on?

[TA] The OCI data access library has been tested to go about 100 reads/sec
with a different application.  I have not integrated this with back-sql or
OpenLDAP and so I could
not tell you how fast an OCI version of back-sql could be.  I would think it
should be pretty
close unless there is a massive amount of processing going on in back-sql.
>
> There are two pieces to the code that I could help with.  One is a set of
> routines that can make db calls to read and write data using connections.
>
> The other is a set of connection related classes that can manage
connection
> handles from
>
> OCI and other formats such as ODBC concurrently.
>
> These connection classes are things we can call 'dbo', 'sets' and
> 'connections'.

[description skipped]

as I undestand, it could be used
1) to use direct APIs (lightweight wrapper)
2) to "aggregate" data from different RDBMSes in single slapd

[TA] Right, the 'dbo', 'sets' and 'connections' are a potential way to
abstract different connections
to different RDBMSes in a single slapd.  These are currently used with OCI
but I tested them
previously with ODBC (a couple of years ago).  OCI and ODBC are from a
connection point of
view very similar so a lightweight wrapper could use this abstraction to so
that depending on the
connection type the underlying direct API would change but to the caller it
would be transparent
except for the connection type used to create the 'key' (OCI, ODBC,...).

Am I right? It is very interesting, anyway - as soon as we manage to
summarize
what each of the contributers can offer, we can develop a little "plan" of
incorporating new code in appropriate order...

[TA] Sounds great.   I'm flexible, any plan is better than none! :-)

WBW, Dmitry