Uploaded image for project: 'Grouper'
  1. Grouper
  2. GRP-812

[psp] memory issue with bulk sync

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.1
    • Component/s: provisioning
    • Labels:
      None

      Description

      It seems difficult to bulk provision a large number of new groups to LDAP. If you're loading course terms and each term has 50,000 groups/500,000 memberships, you can easily run into memory issues. Even if you give 3GB of memory to the bulk sync process, it gets to the point of doing constant full GCs and eventually throws an out of memory exception. It seems that's the case because the bulk sync process keeps all the diffs in memory. I wonder if it would make sense to either process the diffs immediately (rather than all at the end) so they don't have to be kept in memory. Or write the diffs to a file (again so everything doesn't have to be kept in memory).

        Smart Checklist

          Attachments

            Activity

              People

              Assignee:
              tom.zeller.2@at.internet2.edu Tom Zeller
              Reporter:
              shilen.patel@at.internet2.edu Shilen Patel (duke.edu)
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: