Uploaded image for project: 'Grouper'
  1. Grouper
  2. GRP-2310

Bulk Subject Lookup: Use grouper_members data first

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Fixed
    • Minor
    • 2.4.0.patch, 2.4.1
    • None
    • subject API
    • None

    Description

      SubjectFinder.findByIds(IdList, SubjectSource) and SubjectResolvers loop over the given IDs and do individual lookups. At Georgia Tech, we had a loader job with about 1M subjects and it took multiple hours to look up the subjects one by one.

      While the Ldap Subject Source Adapter could be optimized to do bulk or other efficient ldap lookups, it seemed that the first method should be to do bulk SQL look ups of matching members in the grouper_members table.

      Doing Member lookups changed the performance from multiple hours to <5 minutes.

      Attachments

        Activity

          People

            shilen.patel@at.internet2.edu Shilen Patel (duke.edu)
            bert.beelindgren@at.internet2.edu Bert Bee-Lindgren (gatech.edu)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: