LDIF parser performance (was: write performance)

[Michael Ströder <michael@stroeder.com>]

Howard,

(Cc:-ed openldap-devel@OpenLDAP.org in opposite to our off-line
conversation).

Howard Chu wrote:
> I'm wondering if it's worth the effort
> to rewrite the client's LDIF parser as I did for slapadd -q.

As I said I cannot test on the machine where I did the original tests.
But I tried to test OpenLDAP's LDIF parser with -n.

$ time ldapadd -f test.ldif -n
[..]
real    1m30.402s
user    1m29.090s
sys     0m0.376s

Now I have a small Python script which uses the module 'ldif' from
python-ldap for reading in the LDIF file. I've implemented module 'ldif'
in pure Python but off course the string module in the underlying Python
standard lib is implemented in C. And the Python runtime environment
does all the ugly memory management. :-)

$ time python count_members.py < test.ldif
[..]
real    0m19.145s
user    0m18.349s
sys     0m0.568s

I re-ran the tests twice, so test.ldif should have been in the
filesystem cache.

How does that sound to you? I'm not sure what different actions
ldapadd -n does in comparison to my simple script. But at least
count_members.py also reads the complete entries into a tuple containing
the DN as string and the entry as so-called dictionary.

Ciao, Michael.