pspng full sync trigger configurability

Chris Hyzer 3 days ago
i could picture a PSPNG setting that says only provision subjects from grouper that have a certain subject attribute and value... that could work :slightly_smiling_face:

Tamara 2 days ago
In Auckland we were looking at how to configure PSPNG so we only listen/publish the events we care about rather than have it listen to all activity.

Chris Hyzer 1 day ago
it has to go through each event to know if it cares about it... since the config is based on attributes. know what i mean?

Chris Hyzer 1 day ago
we are working on improving the speed of that though... we have some ideas

Tamara 1 day ago
Sounds good. I guess going through them to decide what to publish vs publishing it all is where we are scratching our heads. To also reduce the volumes going out to AD

Chris Hyzer 1 day ago
if you can be more explicit as to what you mean i would appreciate it

Tamara 1 day ago
:+1::skin-tone-3:
I’ll get @Wenlai Wang to describe what we are seeing / wanting to do when it comes to PSPNG as she’s better equipped than me. (Sunday morning here. So it’ll be this time tomorrow)

Carey Black 22 hours ago
( Warning: commentary from the cheap seats.)
I have kicked this idea around in my head a few times and the only idea I have really come up with is that the TempToChangeLog job would need to do the sorting ( per CLC ) as it is creating the events and produce independent queue's of event ID's to be processed per CLC.
If the CLC could register a set of JEXL expressions that the TempToChangeLog process would "pre-run" against all events it is writing to the ChangeLog, then that could work. But I have guessed that the overhead of the N JEXL expressions for the TempToChangeLog would likely just move the bottle neck to that job and likely would slow everything down. ( Unless it was done as a "post" task that was spun out to a thread per CLC. ) Maybe.....
AND if each CLC did have it's own table/queue, then log events could be added to those queues and contention for tables would avoided too.

Wenlai Wang 27 minutes ago
In Auckland, we are not interested in some of the changelog events that trigger incremental/full sync to AD, i.e. we only interest in add/delete membership for incremental sync, and add "provisiont_to" attribute with value "pspng_activedirectory" for single group sync. We don't want to invoke fullSync for all groups at all because of the bad performance.

Wenlai Wang 23 minutes ago
Currently in ChangelogHandlingConfig, changelogTypesThatAreHandledViaFullSync and changelogTypesThatAreHandledIncrementally are hardcoded, it would be useful if these two fields can be configured in properties file.

Wenlai Wang 9 minutes ago
For add attribute value changelog event, the groupInfo is not set, so it will invoke fullSync everything instead of single group sync.

Chris Hyzer 7 minutes ago
@black.123 yes we dont want to slow everything down, so the thought is the temp->changeLog is minimal and quick. each changelog makes its own mind up. plus each change log consumer doesnt have its own queue, just a pointer to the full queue...