[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Improved handling for large number of databases / Access newly opened database from another transaction



On 27/05/16 07:37, "Howard Chu" <hyc@symas.com> wrote:

>Jürg Bircher wrote:
>> Hello
>>
>> Improved handling for large number of databases
>> ===============================================
>
>> If interested let me know how to contribute.
>
>Looks interesting, yes. I assume you have profiled the code before and after 
>the suggested changes, please provide your profiling results.
>
>Please read the Developer Guidelines.
>  http://www.openldap.org/devel/contributing.html
>
> > Access newly opened database from another transaction
> > =======================================================
>
>Sounds like an oddball case. Applications should open all their DBIs from a 
>single thread and not start any other threads/transactions until all setup is 
>completed.

Yes it is a simpler way to open all the databases at startup. However if the environment contains many databases which are not necessarily opened during the life time of the application it is an advantage to open them lazily. Any additional open database generates more overhead especially when using begin and abort.


lmdb improved (renew, reset)

[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 10] open databases in [0.31975] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [10 of 10] open databases in [0.20350] seconds

[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 100] open databases in [0.25845] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [100 of 100] open databases in [0.29663] seconds

[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 1000] open databases in [0.28590] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1000 of 1000] open databases in [0.42897] seconds

[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 10000] open databases in [0.30004] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [10000 of 10000] open databases in [1.68870] seconds

lmdb improved (begin, commit)

[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 10] open databases in [0.36538] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [10 of 10] open databases in [0.35923] seconds

[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 100] open databases in [0.34858] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [100 of 100] open databases in [0.39294] seconds

[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 1000] open databases in [0.40222] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1000 of 1000] open databases in [0.54752] seconds

[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 10000] open databases in [0.78595] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [10000 of 10000] open databases in [2.32414] seconds


lmdb original (renew, reset)

[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 10] open databases in [0.18597] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [10 of 10] open databases in [0.21572] seconds

[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 100] open databases in [0.24173] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [100 of 100] open databases in [0.46497] seconds

[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 1000] open databases in [0.27127] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1000 of 1000] open databases in [4.18579] seconds

[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 10000] open databases in [0.30128] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [10000 of 10000] open databases in [49.35048] seconds

lmdb original (begin, commit)


[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 10] open databases in [0.45135] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [10 of 10] open databases in [0.38990] seconds

[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 100] open databases in [0.47890] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [100 of 100] open databases in [0.69917] seconds

[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 1000] open databases in [1.84908] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1000 of 1000] open databases in [5.88098] seconds

[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 10000] open databases in [22.12491] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [10000 of 10000] open databases in [74.53854] seconds





>
>> Hope it is useful!
>
>Thanks.
>
>-- 
>   -- Howard Chu
>   CTO, Symas Corp.           http://www.symas.com
>   Director, Highland Sun     http://highlandsun.com/hyc/
>   Chief Architect, OpenLDAP  http://www.openldap.org/project/