Re: Another observation about the beta release ...

[Ed Carp <erc@pobox.com>]

Anthony Brock (abrock@georgefox.edu) writes:

> While attempting to import some of our current data, we have noticed some 
> significant resource requirements by the beta software.  Basically, as the 
> import progresses, the server steadily starts to slow from one student per 
> second until student records are being add once every couple minutes.  This 
> is usually around 700 to 800 records into the import.
> 
> As it progresses, we also noticed available memory on the machine dropping 
> from over 200 meg until we only have 5 meg available with over 100 meg of 
> swap space in use.  The server then dumped it's core and exited.
> 
> I haven't had time to look into it, but this seems a bit extreme for a 
> simple import, especially when the source LDAP database (we used a perl 
> script on a different machine to read from a remote LDAP server and import 
> to the beta) backend only occupies about 10 meg of disk space.

Sounds like a database issue to me.  This often occurs because as each new
record is added, the (steadily growing) index must be searched to avoid adding
duplicate records.  Hence, the slowdown.

If this were an exposed database, I'd recommend removing the indices, adding
the records, then re-adding the indices.  That would speed things up quite a
bit.
--
Ed Carp, N7EKG  	erc@pobox.com		940/367-2744 cell phone
			http://www.pobox.com/~erc