It should be noted that two different loader jobs could have created the stem in question. It appears that two different loader jobs were running in two different containers at the same time, each possibly trying to find/create the stem in question. Could there be a cache consistency issue of some sort? Daemon host: bc013f1c2c1a Likely job failure: 2021-05-14 06:48:07.818 Log ID: 8a80908b7967005b01796a7b72550b29 id | job_name | status | started_time | ended_time | millis | millis_get_data | millis_load_data | job_type | job_schedule_type | job_description | job_message | host | group_uuid | job_schedule_quartz_cron | job_schedule_interval_seconds | last_updated | unresolvable_subject_count | insert_count | update_count | delete_count | total_count | parent_job_name | parent_job_id | and_group_names | job_schedule_priority | context_id ----------------------------------+------------------------------------------------------------------------------------------+--------+-------------------------+-------------------------+--------+-----------------+------------------+----------------+-------------------+-----------------+-------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------------------+--------------+----------------------------------+--------------------------+-------------------------------+-------------------------+----------------------------+--------------+--------------+--------------+-------------+-----------------+---------------+-----------------+-----------------------+------------ 8a80908b7967005b01796a7b72550b29 | SQL_GROUP_LIST__Loaders:roster:roster_term_students_r3__b2a49c5d444d4ef7a0190197a3f48cba | ERROR | 2021-05-14 06:45:45.684 | 2021-05-14 06:48:07.817 | 142143 | 4003 | 138108 | SQL_GROUP_LIST | CRON | | .... edu.internet2.middleware.grouper.internal.dao.GrouperDAOException: Problem find stem by name: 'basis:people:roster:class:term-2218:phys:222', Problem in HibernateSession: HibernateSession (3a64957a): notNew, notReadonly, READ_WRITE_NEW, activeTransaction, session (3adfc7ee), Exception in uniqueResult: (class edu.internet2.middleware.grouper.Stem), ByHqlStatic, query: 'from Stem as ns where ns.nameDb = :name or ns.alternateNameDb = :name', cacheable: true, cacheRegion: edu.internet2.middleware.grouper.internal.dao.hib3.Hib3StemDAO.FindByName, tx type: null, options: QueryOptions: secondLevelCache: false, , tx type: nullBind var[0]: 'Param (class java.lang.String): 'name'->'basis:people:roster:class:term-2218:phys:222', , Problem in HibernateSession: HibernateSession (787ba893): new, notReadonly, READ_WRITE_NEW, notActiveTransaction, session (3adfc7ee) at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3StemDAO.findByName(Hib3StemDAO.java:2159) at edu.internet2.middleware.grouper.StemFinder.findByName(StemFinder.java:259) at edu.internet2.middleware.grouper.Stem._createStemAndParentStemsIfNotExist(Stem.java:3909) at edu.internet2.middleware.grouper.StemSave$1$1.callback(StemSave.java:435) at edu.internet2.middleware.grouper.GrouperSession.callbackGrouperSession(GrouperSession.java:976) at edu.internet2.middleware.grouper.StemSave$1.callback(StemSave.java:587) at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3TransactionDAO$1.callback(Hib3TransactionDAO.java:66) at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:703) at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3TransactionDAO.transactionCallback(Hib3TransactionDAO.java:56) at edu.internet2.middleware.grouper.hibernate.GrouperTransaction.callbackGrouperTransaction(GrouperTransaction.java:87) at edu.internet2.middleware.grouper.hibernate.GrouperTransaction.callbackGrouperTransaction(GrouperTransaction.java:106) at edu.internet2.middleware.grouper.StemSave.save(StemSave.java:358) at edu.internet2.middleware.grouper.app.loader.GrouperLoaderType.syncFolderList(GrouperLoaderType.java:1974) at edu.internet2.middleware.grouper.app.loader.GrouperLoaderType.syncGroupList(GrouperLoaderType.java:1857) at edu.internet2.middleware.grouper.app.loader.GrouperLoaderType$5.runJob(GrouperLoaderType.java:575) at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.runJob(GrouperLoaderJob.java:465) at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.execute(GrouperLoaderJob.java:345) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) Caused by: edu.internet2.middleware.grouper.internal.dao.GrouperDAOException: Problem in HibernateSession: HibernateSession (3a64957a): notNew, notReadonly, READ_WRITE_NEW, activeTransaction, session (3adfc7ee), Exception in uniqueResult: (class edu.internet2.middleware.grouper.Stem), ByHqlStatic, query: 'from Stem as ns where ns.nameDb = :name or ns.alternateNameDb = :name', cacheable: true, cacheRegion: edu.internet2.middleware.grouper.internal.dao.hib3.Hib3StemDAO.FindByName, tx type: null, options: QueryOptions: secondLevelCache: false, , tx type: nullBind var[0]: 'Param (class java.lang.String): 'name'->'basis:people:roster:class:term-2218:phys:222', at edu.internet2.middleware.grouper.hibernate.HibernateSession._internal_hibernateSessionCatch(HibernateSession.java:591) at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:713) at edu.internet2.middleware.grouper.hibernate.ByHqlStatic.uniqueResult(ByHqlStatic.java:356) at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3StemDAO.findByName(Hib3StemDAO.java:2150) ... 18 more Caused by: org.hibernate.exception.GenericJDBCException: could not extract ResultSet at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQL Folder basis:people:roster:class:term-2218:phys:222 is in the audit log as having been created 2021/05/14 06:48. Container b26ef5fc23ab added it. Entry id 5d6e02055e7d48f78fd02f5602322cbf Added stem: basis:people:roster:class:term-2218:phys:222 IP address 10.0.16.6 This is likely the job that added the stem, but I can't prove it at this time: grouper_v2_5_33=# select * from grouper_loader_log where id = '8a8090867966ffe701796a7d7f310ae9'; id | job_name | status | started_time | ended_time | millis | millis_get_data | millis_load_data | job_type | job_schedule_type | job_description | job_message | host | group_uuid | job_schedule_quartz_cron | job_schedule_interval_seconds | last_updated | unre solvable_subject_count | insert_count | update_count | delete_count | total_count | parent_job_name | parent_job_id | and_group_names | job_schedule_priority | context_id ----------------------------------+------------------------------+---------+-------------------------+-------------------------+--------+-----------------+------------------+-----------+-------------------+-----------------+-------------+--------------+------------+--------------------------+-------------------------------+-------------------------+----- -----------------------+--------------+--------------+--------------+-------------+-----------------+---------------+-----------------+-----------------------+------------ 8a8090867966ffe701796a7d7f310ae9 | OTHER_JOB_incrementalLoader1 | SUCCESS | 2021-05-14 06:48:00.048 | 2021-05-14 06:48:00.083 | 35 | | | OTHER_JOB | | | | b26ef5fc23ab | | | | 2021-05-14 06:48:00.084 | 0 | 0 | 0 | 0 | 0 | | | | | (1 row) grouper_v2_5_33=# select * from grouper_audit_entry where id = '5d6e02055e7d48f78fd02f5602322cbf'; act_as_member_id | audit_type_id | context_id | created_on | description | env_name | grouper_engine | grouper_version | hibernate_version_number | id | int01 | int02 | int03 | int04 | int05 | last_updated | logged_in_member_id | serve r_host | string01 | string02 | string03 | string04 | string05 | string06 | string07 | string08 | user_ip_address | duration_microseconds | query_count | server_user_name ------------------+----------------------------------+----------------------------------+---------------+----------------------------------------------------------+----------+----------------+-----------------+--------------------------+----------------------------------+-------+-------+-------+-------+-------+---------------+---------------------+------ --------+----------------------------------+----------------------------------------------+----------------------------------+----------------------------------------------+----------+----------+----------+----------+-----------------+-----------------------+-------------+------------------ | b5239ecd9a094c3ead17ad06b8fae898 | 07b2db9e9f3440cd83139eee01a0819a | 1620989287001 | Added stem: basis:people:roster:class:term-2218:phys:222 | dev | grouperLoader | 2.5.50 | 0 | 5d6e02055e7d48f78fd02f5602322cbf | | | | | | 1620989287002 | | b26ef 5fc23ab | 5444cc3f89344d428e634be956bf2404 | basis:people:roster:class:term-2218:phys:222 | e26318f86761489fa4607493e3f39264 | basis:people:roster:class:term-2218:phys:222 | | | | | 10.0.16.6 | 219407 | 17 | tomcat