Re: data import very slow for data migration

Hi

Please ignore me previous mail.

1. There are too many errors like above.

Ans:-We are having some 48 files of 500 M each and no of such errors on each files on average are 15-20

2 increase cachesize to at least 4GB that is
set_cachesize 4 0 1
Ans:- i have done this in DB_CONFIG file
set_cachesize 0 4294967295 0 it showing in db_stats as 3GB 1023M 1023K

Please let me know if its wrong and how and where i should modify this.

3.checkpoint 128 15

I would set checkpointing to 0 15

Ans:- Its for only data import or after data import also.

4. Please not that overlay declarations follow all database declarations,
modify slapd.conf accordingly.

Ans:- We are using static config file for all configuration, i didn't get how it matters declaration of overlays, can i specify just after schema declaration.

Thanks a lot, it gives me some better insight.

On Wed, Dec 26, 2012 at 9:58 AM, anil beniwal <beni.anil@gmail.com> wrote:

Hi

I didn't get it.

Can you please elaborate it a bit. It would be great help for me.

On Wed, Dec 26, 2012 at 2:37 AM, Dieter Klünter <dieter@dkluenter.de> wrote:

Am Tue, 25 Dec 2012 21:27:39 +0530
schrieb anil beniwal <beni.anil@gmail.com>:

> Hi
>
> We are having 4 million users to migrate, all data exported from
> oracle to multiple ldif files.
> Imported 1 million till now, took almost 28 hours. and openldap-data
> dir of about 28G.
> openldap version 2.4.33 bdb version 5.1.29 RHEL 6.3 RAM 8G 4 cpu ,
> system is a VM.
>
> Currently running slapadd output
> + /apps/openldap/sbin/slapadd -q -c -w -f
> /apps/openldap/etc/openldap/slapd.conf -l /root/User9.ldif
> bdb_monitor_db_open: monitoring disabled; configure monitor database
> to enable
> . 2.27% eta 21h31m elapsed 29m57s
> spd 1.6 k/s str2entry: invalid value for attributeType
> postalAddress #0 (syntax 1.3.6.1.4.1.1466.115.121.1.41)
> slapadd: could not parse entry (line=394416)
> * 2.81% eta 19h59m elapsed 34m40s spd
> 10.1 k/s

1. There are too many errors like above.

> Its seems to be taking weeks go import whole data.

It takes about 2 - 4 hours in order to slapadd 4 mio.entires, depending
on file system and disk type.

>
> is there any tool or any other approach which we can use to make it
> fast,Or we are going with wrong configuration.
> Or we have to switch to ODS or RHDS

There is no necessity for other tools, just modify the ldif file.

[...]

> DB_CONFIG
>
> set_cachesize 0 4294967295 0

increase cachesize to at least 4GB that is
set_cachesize 4 0 1

[...]
> checkpoint 128 15

I would set checkpointing to 0 15
[...]

> concurrency 100
> index entryCSN eq
> index entryUUID eq
> index
> mail,uid,postalCode,smail,channelType,channelValue,answer,behavName,objectclass,tokenID,type
> eq
> index givenName,sn,city,question,behavValue,cn,extName sub
> index displayName approx
> # Replication Configuration
> overlay syncprov
> syncprov-checkpoint 100 10
> syncprov-sessionlog 100
>
> serverid 1
>
> syncrepl rid=111
> provider=ldap://s01.com
> binddn="cn=Manager,dc=example,dc=com"
> bindmethod=simple
> starttls=yes
> tls_reqcert=allow
> schemachecking=off
> credentials=G00gle#
> searchbase="dc=example,dc=com"
> type=refreshAndPersist
> retry="5 5 300 +"
> interval=00:00:00:10
>
> syncrepl rid=222
> provider=ldap://m04.com
> binddn="cn=Manager,dc=example,dc=com"
> bindmethod=simple
> starttls=yes
> tls_reqcert=allow
> schemachecking=off
> credentials=G00gle#
> searchbase="dc=example,dc=com"
> type=refreshAndPersist
> retry="5 5 300 +"
> interval=00:00:00:10
>
> ######
>
> mirrormode TRUE
>
> directory /apps/openldap/var/openldap-data
>
> overlay unique
> unique_attributes mail
>
> overlay ppolicy
> ppolicy_default "cn=default,ou=pwdPolicy,dc=example,dc=com"
> ppolicy_use_lockout

Please not that overlay declarations follow all database declarations,
modify slapd.conf accordingly.

-Dieter

--
Dieter Klünter | Systemberatung
http://dkluenter.de
GPG Key ID:DA147B05
53°37'09,95"N
10°08'02,42"E

--

Thanks&Regards
Anil Beniwal
+919891695048

Thanks&Regards
Anil Beniwal
+919891695048