Re: data import very slow for data migration

[Dieter KlÃnter <dieter@dkluenter.de>]

Am Tue, 25 Dec 2012 21:27:39 +0530
schrieb anil beniwal <beni.anil@gmail.com>:

> Hi
> 
> We are having 4 million users to migrate, all data exported from
> oracle to multiple ldif files.
> Imported 1 million till now, took almost 28 hours. and openldap-data
> dir of about 28G.
> openldap version 2.4.33 bdb version 5.1.29  RHEL 6.3 RAM 8G 4 cpu ,
> system is a VM.
> 
> Currently running slapadd output
> + /apps/openldap/sbin/slapadd -q -c -w -f
> /apps/openldap/etc/openldap/slapd.conf -l /root/User9.ldif
> bdb_monitor_db_open: monitoring disabled; configure monitor database
> to enable
> .                       2.27% eta 21h31m elapsed          29m57s
> spd   1.6 k/s str2entry: invalid value for attributeType
> postalAddress #0 (syntax 1.3.6.1.4.1.1466.115.121.1.41)
> slapadd: could not parse entry (line=394416)
> *                       2.81% eta 19h59m elapsed          34m40s spd
> 10.1 k/s

1. There are too many errors like above.

> Its seems to be taking weeks go import whole data.

It takes about 2 - 4 hours in order to slapadd 4 mio.entires, depending
on file system and disk type.
> 
> is there  any tool or any other approach which we can use to make it
> fast,Or we are going with wrong configuration.
> Or we have to switch to ODS or RHDS

There is no necessity for other tools, just modify the ldif file.

[...]
> DB_CONFIG
> 
> set_cachesize   0       4294967295      0

increase cachesize to at least 4GB that is
set_cachesize 4 0 1

[...]
> checkpoint 128 15

I would set checkpointing to 0 15
[...]

> concurrency 100
> index entryCSN eq
> index entryUUID eq
> index
> mail,uid,postalCode,smail,channelType,channelValue,answer,behavName,objectclass,tokenID,type
> eq
> index givenName,sn,city,question,behavValue,cn,extName sub
> index displayName approx
> # Replication Configuration
> overlay syncprov
> syncprov-checkpoint 100 10
> syncprov-sessionlog 100
> 
> serverid        1
> 
> syncrepl        rid=111
>                 provider=ldap://s01.com
>                 binddn="cn=Manager,dc=example,dc=com"
>                 bindmethod=simple
>                 starttls=yes
>                 tls_reqcert=allow
>         schemachecking=off
>                 credentials=G00gle#
>                 searchbase="dc=example,dc=com"
>                 type=refreshAndPersist
>                 retry="5 5 300 +"
>                 interval=00:00:00:10
> 
> syncrepl        rid=222
>                 provider=ldap://m04.com
>                 binddn="cn=Manager,dc=example,dc=com"
>                 bindmethod=simple
>                 starttls=yes
>                 tls_reqcert=allow
>         schemachecking=off
>                 credentials=G00gle#
>                 searchbase="dc=example,dc=com"
>                 type=refreshAndPersist
>                 retry="5 5 300 +"
>                 interval=00:00:00:10
> 
> ######
> 
> mirrormode TRUE
> 
> directory    /apps/openldap/var/openldap-data
> 
> overlay unique
> unique_attributes mail
> 
> overlay ppolicy
> ppolicy_default "cn=default,ou=pwdPolicy,dc=example,dc=com"
> ppolicy_use_lockout

Please not that overlay declarations follow all database declarations,
modify slapd.conf accordingly.

-Dieter

-- 
Dieter KlÃnter | Systemberatung
http://dkluenter.de
GPG Key ID:DA147B05
53Â37'09,95"N
10Â08'02,42"E