Issues with pcache overlay purging entries

Hello,

Seeing strange issues when using the pcache overlay as a caching method on some servers. More specifically, not seeing results getting purged from the cache even though the TTL has expired. I have been able to reproduce on the following:

Ubuntu 16.04 running slapd 2.4.42+dfsg-2ubuntu3.1
Ubuntu 14.04 running slapd 2.4.31-1+nmu2ubuntu8

Here are the steps that I am taking in order to reproduce the issue. First, I'm running a query similar to this:

ldapsearch -x -LLL -h localhost -b 'dc=foo,dc=com' '(isphoneoperator=TRUE)' uid | grep "uid:" | grep someuser

When I set the "isphoneoperator" attribute to TRUE the user shows up and everything is great. If I then go and set the attribute to FALSE and restart slapd *before* the TTL of the query expires on it's own naturally then that user remains in the cache forever and the only way to correct the issue is to stop slapd, delete all the caching databases and start slapd again.

There are a few things I've noticed in the logs that I don't quite understand. When starting slapd for the first time with a fresh set of cache databases the logs look like the ones below and will repeat every time the TTL expires:

Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=1
Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=2
Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=3
Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=4
Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=5
Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=6
Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=7
Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=8
Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=9

However, when I restart slapd the logs change to look like this:

Jun 1 20:44:55 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0]
Jun 1 20:45:25 vagrant slapd[11817]: ENTRY ADDED/MERGED, CACHED ENTRIES=0
Jun 1 20:45:25 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0]
Jun 1 20:45:55 vagrant slapd[11817]: ENTRY ADDED/MERGED, CACHED ENTRIES=0
Jun 1 20:45:55 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0]
Jun 1 20:46:26 vagrant slapd[11817]: ENTRY ADDED/MERGED, CACHED ENTRIES=0
Jun 1 20:46:26 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0]
Jun 1 20:46:56 vagrant slapd[11817]: ENTRY ADDED/MERGED, CACHED ENTRIES=0
Jun 1 20:46:56 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0]
Jun 1 20:47:26 vagrant slapd[11817]: ENTRY ADDED/MERGED, CACHED ENTRIES=0
Jun 1 20:47:26 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0]

I find this to be pretty odd and I'm not sure I quite understand the reason for this behavior. Also, when I run slapd in the foreground with (-d -1 for example) I notice some differences there as well. For example, with a fresh set of cache databases I'll see messages like:

57503573 ==> hdb_delete: uid=someuser,ou=users,dc=foo,dc=com

However, when I restart slapd I never see these again.

I have noticed that if I set pcachePersist to TRUE both my issue above where users are never purged and this logging inconsistency go away and everything works great. I guess I don't really understand what the pcachePersist parameter does. I've read the man pages for slapo-pcache(5) and I also found the following mailing list post and it is all a bit confusing to me:

http://www.openldap.org/cgi-bin/wilma_hiliter/openldap-technical/201007/msg00226.html

I've looked for fixed bugs or any other posts that might indicate what is going on here but haven't turned up anything relevant that might be affecting the versions of slapd I'm running.

I guess I'm wondering a few things:

1. Have a stumbled upon a bug?
2. Is this logging inconsistency normal?
3. What does pcachePersist do?
4. Is there a way to forcefully clear the cache other than deleting the database files or do you have to wait for the TTL to expire?

Thanks,
Andrew