PSPNG: Failure when group is deleted before Group-Add changelog entry is processed

When PSPNG handles a Group-Add changelog event, it fetches the current Group or PITGroup information from the group registry. However, if the group has been deleted before the changelog entry is processed, PSPNG still tries to create it which is both unnecessary and impossible (since a PITGroup is probably missing properties that are important for group creation). At this point, PSPNG is stuck on this event forever.

PSPNG should ignore a Group-Add event if the group has been deleted from grouper.

From group-users list Jan 31, 2017.

Hi There,

I have a pilot instance of Grouper running, with PSPNG provisioning group memberships to active directory for an academic department.

I’m having an issue with the ChangeLog consumer tripping over groups that were created, then deleted or renamed.
When the consumer gets to the transaction in the log, the result is:

2017-01-23 11:02:04,488: [DefaultQuartzScheduler_Worker-4] INFO LdapGroupProvisioner.createGroup(251) - - Creating LDAP group for GrouperGroup: null

(which makes sense to me, since the group no longer exists).
However, this throws a wrench in the JEXL evaluation:

2017-01-23 11:02:04,495: [DefaultQuartzScheduler_Worker-4] ERROR Provisioner.evaluateJexlExpression(523) - - Jexl Expression dn: cn=${grouperUtil.extensionFromName(name)}
sAMAccountName: ${grouperUtil.extensionFromName(name)}
cn: ${grouperUtil.extensionFromName(name)}
objectclass: group could not be evaluated for subject 'null/null' and group 'null/null' which used variableMap '{userSearchBaseDn=OU=people,DC=…, provisionerType=LdapGroupProvisioner, groupCreationBaseDn=OU=Grouper,OU=…, utils=edu.internet2.middleware.grouper.pspng.PspJexlUtils@193221e, provisionerName=pspng_nexus, groupSearchBaseDn=OU=Grouper,…'
2017-01-23 11:02:04,497: [DefaultQuartzScheduler_Worker-4] ERROR ChangeLogHelper.processRecords(255) - - Error
java.lang.NullPointerException
at java.io.StringReader.<init>(StringReader.java:50)
at edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner.createGroup(LdapGroupProvisioner.java:258)
at edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner.createGroup(LdapGroupProvisioner.java:54)
at edu.internet2.middleware.grouper.pspng.Provisioner.provisionItem(Provisioner.java:887)
at edu.internet2.middleware.grouper.pspng.Provisioner.provisionBatchOfItems(Provisioner.java:1299)
at edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim.processChangeLogEntries(PspChangelogConsumerShim.java:71)
at edu.internet2.middleware.grouper.changeLog.ChangeLogHelper.processRecords(ChangeLogHelper.java:245)
at edu.internet2.middleware.grouper.app.loader.GrouperLoaderType$4.runJob(GrouperLoaderType.java:629)
at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.runJob(GrouperLoaderJob.java:416)
at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.execute(GrouperLoaderJob.java:318)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

Then:
2017-01-23 11:02:04,501: [DefaultQuartzScheduler_Worker-4] ERROR ChangeLogHelper.processRecords(286) - - Did not get all the way through the batch! -1 != 2884196

Which puts us in a state where the same transactions are tried again and again without moving forward. I tried setting “retryOnError” to false, but that did not appear to change anything. I suspect that setting may only affect “catchable” errors? In any case, even if we get past this transaction, there are a number right after representing memberships being added to the group that could also be tried, and will likely fail…

Have I missed a configuration that will allow the PSPNG to skip over these types of entries, or at the very least, move on from this error?
I am on the latest patches of both the API, and PSPNG, and willing to experiment to get this going.

Thanks,
Sean.