Uploaded image for project: 'Grouper'
  1. Grouper
  2. GRP-812

[psp] memory issue with bulk sync

    XMLWordPrintable

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.1.0
    • 2.1.1
    • provisioning
    • None

    Description

      It seems difficult to bulk provision a large number of new groups to LDAP. If you're loading course terms and each term has 50,000 groups/500,000 memberships, you can easily run into memory issues. Even if you give 3GB of memory to the bulk sync process, it gets to the point of doing constant full GCs and eventually throws an out of memory exception. It seems that's the case because the bulk sync process keeps all the diffs in memory. I wonder if it would make sense to either process the diffs immediately (rather than all at the end) so they don't have to be kept in memory. Or write the diffs to a file (again so everything doesn't have to be kept in memory).

      Attachments

        Activity

          People

            tom.zeller.2@at.internet2.edu Tom Zeller
            shilen.patel@at.internet2.edu Shilen Patel (duke.edu)
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Smart Checklist