Re: write performance (was Re: LDIF parser performance)

[Howard Chu <hyc@symas.com>]

Howard Chu wrote:
> Back on the server side of things...
>
> Using ldapadd to load my test database (380836 entries, 533MB LDIF, 
> 1.2GB id2entry size) takes well over an hour using synchronous writes.
>
> With DB_TXN_WRITE_NOSYNC and a 512 MB BDB cache, plus periodic 
> checkpoints, it takes 39:21.55 minutes. With the patch in add.c:1.247, 
> and the duplicate check in slap_mods_check replaced by a quicksort, it 
> takes 20:23.24 minutes.
>
> With a 1GB BDB cache it takes 14:06.48 minutes. With the above and the 
> new ldapadd client code, it takes 12:49.74 minutes.

With synchronous writes and the original code in HEAD, 1GB BDB cache, it 
took 1:33:08.74 to ldapadd the database. With the patched add.c and the 
quicksort it took 1:14:52.11. As a point of reference, it took only 
2:42.64 for slapadd -q.

Unfortunately the bulk of the ldapadd time is really determined by 
BerkeleyDB. With DB_TXN_NOSYNC and transaction logs written to a tmpfs, 
the original code in HEAD completes the ldapadd in 18:54.56. With the 
add.c patch that goes down to 13:35.82. With the quicksort in 
slap_mods_check that drops down further to 7:19.52.

--
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  OpenLDAP Core Team            http://www.openldap.org/project/