Re: Large delay before transmitting search result

[Hallvard B Furuseth <h.b.furuseth@usit.uio.no>]

On Mon, 12 Mar 2012 18:30:11 +0100, Emmanuel LÃcharny wrote:
> Le 3/12/12 6:04 PM, Hallvard Breien Furuseth a Ãcrit :
> > For now I suggest we always use 5 bytes when the length exceeds some
> > threshold.  This is valid in BER and in LDAP's somewhat restricted 
> > BER,
> > but not valid DER.  (...)

Whoops, that's a bit more work than it sounds like.  OpenLDAP uses
LBER_USE_DER flag to request minimal lengths etc.  We shouldn't
break DER for liblber users who actually want DER.  So we need a
new flag LBER_USE_LDAPBER, and then walk through ~90 uses of
der_alloc() and LBER_USE_DER to see what we mean where.

> > Next would be two-pass generation of sequences/sets.  First use 
> > 5-byte
> > lengts and remember their positions, then shorten them.  That'll
> > eliminate all but one memmove.  This needs some new data structures 
> > to
> > track the length positions though.

Some ber_printf formats like 'V' take a sequence/set of strings,
but just outputs them one at a time.  liblber could easly do two
passes over these.  Maybe keep the lengths at the end of the ber
buffer, then use them in 2nd pass.

> > Maybe it'd be easier to let the caller do more of the work.  Call
> > liblber twice, first just so it can find the lengths and then again 
> > with
> > the same data so it can fill that out.
>
> This is what we do at Apache Directory. Works well for big data, cost
> a bit more time for smaller ones. The idea is to allocate the exact
> amount of memory needed to store the full result.

Sounds nice.  And we've been dreaming of rewriting the libraries
for years.  And all its callers, that's the trouble.  But sooner
or later we'll have to do it.

slapd does have entry_partsize() and entry_encode() for this for
internally stored entries.  We're just not using it for messages.

> OTOH, creating such a buffer this is an atrocious solution when you
> have to transfer Mb of data, as you will just suck a hell lot of
> memory on the server before the sockect can swallow all of the data.
>
> At some point, you'd like to transfer the entry as a set of buffers,
> with huge data being read directly on the disk, instead of transiting
> by the memory.
>
> Not an obvious optimization...

It wouldn't have helped that much either before back-mdb, since
most of slapd operates on full entries.  With back-mdb we don't
actually have to "read" from disk the parts of an entry we do
not use, which finally makes such optimizations non-hairtearing.

--
Hallvard