Uploaded image for project: 'Grouper'
  1. Grouper
  2. GRP-1695

hard code built in ignore attribute defs and names

    XMLWordPrintable

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 2.3.1, 2.4.0, 2.3.0.patch
    • API
    • None

    Description

      From: Shilen Patel shilen@duke.edu
      Sent: Friday, February 16, 2018 2:06 PM
      To: Hyzer, Chris <mchyzer@isc.upenn.edu>
      Cc: grouper-core@internet2.edu
      Subject: Re: [grouper-core] RE: [grouper-users] StaleObjectStateException thrown by loader jobs after patch

      Yeah I can’t imagine anybody would want to remove those builtins.

      • Shilen

      From: <grouper-core-request@internet2.edu> on behalf of "Hyzer, Chris" <mchyzer@isc.upenn.edu>
      Date: Friday, February 16, 2018 at 1:37 PM
      Cc: "grouper-core@internet2.edu" <grouper-core@internet2.edu>
      Subject: [grouper-core] RE: [grouper-users] StaleObjectStateException thrown by loader jobs after patch

      Core-team

      Im kinda thinking that the builtins should be hard coded or configured somewhere else.

      i.e. if someone overrides, they cant easily upgrade. Know what I mean? I think its ok for Grouper just to mandate we aren’t change logging certain things…

      Thoughts?

      Thanks
      Chris

      From: Shilen Patel shilen@duke.edu
      Sent: Friday, February 16, 2018 1:33 PM
      To: Scott Koranda <skoranda@gmail.com>; Hyzer, Chris <mchyzer@isc.upenn.edu>
      Cc: grouper-users <grouper-users@internet2.edu>
      Subject: Re: [grouper-users] StaleObjectStateException thrown by loader jobs after patch

      Just to make sure, you do have the latest grouper.base.properties with this config:

      grouper.attribute.namesOfAttributeDefNamesToIgnoreAuditsChangeLogPit.elConfig = $$grouper.attribute.rootStem$$:attestation:attestationCalculatedDaysLeft,$$grouper.attribute.rootStem$$:attestation:attestationLastEmailedDate,$$grouper.attribute.rootStem$$:loaderMetadata:grouperLoaderMetadataLastFullMillisSince1970,$$grouper.attribute.rootStem$$:loaderMetadata:grouperLoaderMetadataLastIncrementalMillisSince1970,$$grouper.attribute.rootStem$$:loaderMetadata:grouperLoaderMetadataLastSummary

      (making sure it includes all the loaderMetadata attributes)

      And you haven’t overwritten this config in grouper.properties. Just making sure.

      Thanks!

      • Shilen

      On 2/16/18, 1:16 PM, "grouper-users-request@internet2.edu on behalf of Scott Koranda" <grouper-users-request@internet2.edu on behalf of skoranda@gmail.com> wrote:

      > How many groups are loaded? Millions?

      This morning I see this:

      SQL> SELECT COUNT FROM GROUPER_GROUPS;
      62221

      SQL> SELECT COUNT FROM GROUPER_CHANGE_LOG_ENTRY_TEMP;
      26791591

      SQL> SELECT COUNT FROM GROUPER_CHANGE_LOG_ENTRY_TEMP WHERE STRING04 NOT LIKE 'etc:attribute:loaderMetadata:%';
      0

      SQL> SELECT COUNT FROM GROUPER_CHANGE_LOG_ENTRY;
      6856091

      There were no entries in either the change log or the change log temp
      before patching.

      I do not understand how so many entries were put into
      GROUPER_CHANGE_LOG_ENTRY_TEMP by the new functionality.

      I also drastically underestimated that amount of time it will take for
      this deployment to process 27 million rows in the change log. It is
      clear it will not be able to do it in a reasonable time (less than a few
      weeks) so I am going to TRUNCATE the tables now instead.

      > > Maybe I am not, however, understanding how those 27 million rows in
      > > GROUPER_CHANGE_LOG_TEMP come about. Are they only created when the
      > > "real" loader jobs run?
      >
      > Hmmmm. I deleted two of the new attributes, and change the incorrect value in a list of groups with two groups. I ran the loader. I got two change log entries per loaded group. One for etc:attribute:loaderMetadata:grouperLoaderMetadataLastSummary and one for etc:attribute:loaderMetadata:grouperLoaderMetadataLastFullMillisSince1970
      > I believe you already have assignments on loaded groups for grouperLoaderMetadataLoaded and grouperLoaderMetadataGroupId.
      >
      > Not saying this is better, but before the patch, you could create those attributes, and run a GSH script to assign them to loaded groups, maybe a sleeping a second after a dozen assignments. Up to you.

      I will consider it as an option. Thanks.

      Another issue with these patches: every time a loader job runs it
      generates thousands of log entries at the WARN level with a message like

      2018-02-16 13:04:55,826: [DefaultQuartzScheduler_Worker-9] WARN
      GrouperLoaderType.updateLoaderMetadataForGroupsNoLongerInLoader(2081) -

      • Non-fatal error removing metadata on group:
        my:group:name:here

      This is filling up the Grouper logs.

      Is this expected?

      Thanks,

      Scott K

      Attachments

        Activity

          People

            chris.hyzer@at.internet2.edu Chris Hyzer (upenn.edu)
            chris.hyzer@at.internet2.edu Chris Hyzer (upenn.edu)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Smart Checklist