Uploaded image for project: 'Grouper'
  1. Grouper
  2. GRP-2311

Grouper Loader and SQL FetchSize

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Fixed
    • Minor
    • 2.5.44
    • 2.4.0.patch
    • grouperLoader
    • None

    Description

      By default, Oracle's jdbc driver makes very small fetches of data as ResultSets are fetched. This led to incredibly slow performance reading larger loader queries at Georgia Tech where it took hours to cursor through a 3M row result across our AWS link. (The problem of these small fetches are worst with network latency)

      The solution to this is JDBC's Statement's setFetchSize(N) method (defaults to 10, at least with Oracle).

      While this is a general problem for Grouper's different connections to databases, this is mostly a problem for Loading and, probably, Subject Sources and not the Grouper Database because Grouper has to have very little network latency to its main database.

      Grouper should have a db.<dbname>.fetchSize property that it then uses for its database queries. There might be a hibernate property for cases when Grouper has too much latency to its database.

      Attachments

        Activity

          People

            bert.beelindgren@at.internet2.edu Bert Bee-Lindgren (gatech.edu)
            bert.beelindgren@at.internet2.edu Bert Bee-Lindgren (gatech.edu)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: