Uploaded image for project: 'Grouper'
  1. Grouper
  2. GRP-2065

Large groups suffer slow perf in loader

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 2.4.0.patch, 2.5.0, 2.4.1
    • grouperLoader
    • None
    • TIER packaging

    Description

      From grouper slack channel:
      am observing a curious/troublesome behavior. I have groups > 250K members. loaders jobs (even when there is only 1) when processing such a large group, the “finding new members to remove” message in the log is presented every few seconds in the beginning and slowly speeds up. And then when we get to be about the last 20% or so, it quickly speeds up to presenting the log message about 10-15 per second. No matter what the data, and how many other concurrent (even 1) loader group syncs running - this seems to be consistent. Loader message is below. Does anyone else observe this behavior?
      daemon_1_4aacfa21eaad | 2019-03-17T13:03:12.270331977Z grouper-api;grouper_error.log;grouper_dev;daemon;2019-03-17 09:03:12,269: [pool-1-thread-2726] INFO GrouperLoaderType.syncOneGroupMembership(2886) - - basis:eduPerson:Aff:affiliate processed 113000 records, finding new members to remove, 113000 of 362139 subjects

      Chris Hyzer says it appears to be a problem in GrouperLoaderResultSet.find()

      Attachments

        Activity

          People

            chris.hyzer@at.internet2.edu Chris Hyzer (upenn.edu)
            gettes@ufl.edu Michael Gettes
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Smart Checklist