Uploaded image for project: 'Grouper'
  1. Grouper
  2. GRP-1278

deadlock in grouper on upgrade (and maybe other times)

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.3.0
    • 2.3.1, 2.3.0.patch
    • None
    • None

    Description

      the upgrade sql script is done, but it never returns, thinks the script is still running

      Looks like the script is finishing but there is a deadlock

      private static GrouperCache<Class, Method[]> declaredMethodsCache() {
      if (declaredMethodsCache == null) {
      synchronized(GrouperStartup.class) {
      if (declaredMethodsCache == null) {

      I can fix this and make a patch…

      Thanks
      Chris

      [appadmin@i2midev1 grouper_v2_3]$ /opt/java8/bin/jstack 406 | less
      2016-04-19 06:19:11
      Full thread dump Java HotSpot(TM) 64-Bit Server VM (16.3-b01 mixed mode):

      "Attach Listener" daemon prio=10 tid=0x0000000062373800 nid=0x787 runnable [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "pool-1-thread-1" daemon prio=10 tid=0x0000000061c57800 nid=0x67c waiting for monitor entry [0x00002b19aaec3000]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at edu.internet2.middleware.grouper.util.GrouperUtil.declaredMethodsCache(GrouperUtil.java:2574)

      • waiting to lock <0x00002b19a0b8eb60> (a java.lang.Class for edu.internet2.middleware.grouper.misc.GrouperStartup)
        at edu.internet2.middleware.grouper.util.GrouperUtil.retrieveDeclaredMethods(GrouperUtil.java:3876)
        at edu.internet2.middleware.grouper.util.GrouperUtil.getterHelper(GrouperUtil.java:5055)
        at edu.internet2.middleware.grouper.util.GrouperUtil.getter(GrouperUtil.java:5041)
        at edu.internet2.middleware.grouper.util.GrouperUtil.propertyValue(GrouperUtil.java:7493)
        at edu.internet2.middleware.grouper.hibernate.HibUtilsMapping.primaryKeyCurrentValue(HibUtilsMapping.java:81)
        at edu.internet2.middleware.grouper.hibernate.HibUtilsMapping.isInsert(HibUtilsMapping.java:50)
        at edu.internet2.middleware.grouper.hibernate.ByObject.saveOrUpdate(ByObject.java:373)
        at edu.internet2.middleware.grouper.hibernate.ByObjectStatic$5.callback(ByObjectStatic.java:376)
        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:700)
        at edu.internet2.middleware.grouper.hibernate.ByObjectStatic.saveOrUpdate(ByObjectStatic.java:363)
        at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3ChangeLogTypeDAO.saveOrUpdate(Hib3ChangeLogTypeDAO.java:51)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeFinder.internal_findOrReplaceChangeLogType(ChangeLogTypeFinder.java:176)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeFinder.internal_updateBuiltinTypesOnce(ChangeLogTypeFinder.java:150)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeFinder.internal_updateKnownTypes(ChangeLogTypeFinder.java:127)
      • locked <0x00002b19a39453d8> (a java.lang.Class for edu.internet2.middleware.grouper.changeLog.ChangeLogTypeFinder)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeFinder.find(ChangeLogTypeFinder.java:75)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeBuiltin.getChangeLogType(ChangeLogTypeBuiltin.java:513)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeBuiltin.getChangeLogCategory(ChangeLogTypeBuiltin.java:530)
        at edu.internet2.middleware.grouper.Member$2$1.callback(Member.java:4392)
        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:700)
        at edu.internet2.middleware.grouper.Member$2.call(Member.java:4374)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

      "org.hibernate.cache.spi.UpdateTimestampsCache.data" daemon prio=10 tid=0x000000006099d800 nid=0x67b waiting on condition [0x00002b19aadc3000]
      java.lang.Thread.State: TIMED_WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)

      • parking to wait for <0x00002b19829a1250> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

      "net.sf.ehcache.CacheManager@118eb00c" daemon prio=10 tid=0x00000000609a5000 nid=0x67a in Object.wait() [0x00002b19aacc2000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00002b1982962e58> (a java.util.TaskQueue)
        at java.lang.Object.wait(Object.java:485)
        at java.util.TimerThread.mainLoop(Timer.java:483)
      • locked <0x00002b1982962e58> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

      "C3P0PooledConnectionPoolManager[identityToken->1lim7by9gcj52daxdu3tt|62e59e40]HelperThread#2" daemon prio=10 tid=0x0000000060e7b000 nid=0x678 in Object.wait() [0x00002b19aabc1000]
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00002b19822ab640> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)
        at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
      • locked <0x00002b19822ab640> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)

      "C3P0PooledConnectionPoolManager[identityToken->1lim7by9gcj52daxdu3tt|62e59e40]HelperThread#1" daemon prio=10 tid=0x0000000060909000 nid=0x677 in Object.wait() [0x00002b19aaac0000]
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00002b19822ab640> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)
        at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
      • locked <0x00002b19822ab640> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)

      "C3P0PooledConnectionPoolManager[identityToken->1lim7by9gcj52daxdu3tt|62e59e40]HelperThread#0" daemon prio=10 tid=0x0000000060127000 nid=0x676 in Object.wait() [0x00002b19aa9bf000]
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00002b19822ab640> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)
        at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
      • locked <0x00002b19822ab640> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)

      "C3P0PooledConnectionPoolManager[identityToken->1lim7by9gcj52daxdu3tt|62e59e40]-AdminTaskTimer" daemon prio=10 tid=0x00000000607cf800 nid=0x675 in Object.wait() [0x00002b19aa7b7000]
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00002b19822ab8f8> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
      • locked <0x00002b19822ab8f8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

      "org.hibernate.cache.spi.UpdateTimestampsCache.data" daemon prio=10 tid=0x0000000060609800 nid=0x672 waiting on condition [0x00002b19aa6b6000]
      java.lang.Thread.State: TIMED_WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)

      • parking to wait for <0x00002b19819dbb00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

      "net.sf.ehcache.CacheManager@1f758500" daemon prio=10 tid=0x0000000060ac5800 nid=0x671 in Object.wait() [0x00002b19aa5b5000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00002b19819dbd50> (a java.util.TaskQueue)
        at java.lang.Object.wait(Object.java:485)
        at java.util.TimerThread.mainLoop(Timer.java:483)
      • locked <0x00002b19819dbd50> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

      "Abandoned connection cleanup thread" daemon prio=10 tid=0x00000000607e0000 nid=0x1a0 in Object.wait() [0x00002b19aa49e000]
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00002b19815817d8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
      • locked <0x00002b19815817d8> (a java.lang.ref.ReferenceQueue$Lock)
        at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)

      "Low Memory Detector" daemon prio=10 tid=0x000000006003e000 nid=0x19e runnable [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "CompilerThread1" daemon prio=10 tid=0x000000006003b800 nid=0x19d waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "CompilerThread0" daemon prio=10 tid=0x0000000060037000 nid=0x19c waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "Signal Dispatcher" daemon prio=10 tid=0x0000000060035000 nid=0x19b runnable [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "Finalizer" daemon prio=10 tid=0x0000000060010800 nid=0x19a in Object.wait() [0x00002b19a61ca000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00002b1981210450> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
      • locked <0x00002b1981210450> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

      "Reference Handler" daemon prio=10 tid=0x000000006000e800 nid=0x199 in Object.wait() [0x00002b19a60c9000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00002b1981210530> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
      • locked <0x00002b1981210530> (a java.lang.ref.Reference$Lock)

      "main" prio=10 tid=0x000000005ffb6800 nid=0x197 waiting for monitor entry [0x00002b196e0d3000]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeFinder.internal_updateKnownTypes(ChangeLogTypeFinder.java:115)

      • waiting to lock <0x00002b19a39453d8> (a java.lang.Class for edu.internet2.middleware.grouper.changeLog.ChangeLogTypeFinder)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeFinder.find(ChangeLogTypeFinder.java:75)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeBuiltin.getChangeLogType(ChangeLogTypeBuiltin.java:513)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeBuiltin.getChangeLogCategory(ChangeLogTypeBuiltin.java:530)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogEntry.<init>(ChangeLogEntry.java:395)
        at edu.internet2.middleware.grouper.Stem.onPreSave(Stem.java:3603)
        at edu.internet2.middleware.grouper.hibernate.ByObject.save(ByObject.java:208)
        at edu.internet2.middleware.grouper.hibernate.ByObjectStatic$7.callback(ByObjectStatic.java:494)
        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:700)
        at edu.internet2.middleware.grouper.hibernate.ByObjectStatic.save(ByObjectStatic.java:481)
        at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3StemDAO.createChildStem(Hib3StemDAO.java:220)
        at edu.internet2.middleware.grouper.Stem$9.callback(Stem.java:2790)
        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:700)
        at edu.internet2.middleware.grouper.Stem.internal_addChildStem(Stem.java:2747)
      • locked <0x00002b197288d1f8> (a java.lang.String)
        at edu.internet2.middleware.grouper.Stem.addChildStem(Stem.java:598)
        at edu.internet2.middleware.grouper.StemSave$1$1.callback(StemSave.java:351)
        at edu.internet2.middleware.grouper.GrouperSession.callbackGrouperSession(GrouperSession.java:974)
        at edu.internet2.middleware.grouper.StemSave$1.callback(StemSave.java:252)
        at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3TransactionDAO$1.callback(Hib3TransactionDAO.java:66)
        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:700)
        at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3TransactionDAO.transactionCallback(Hib3TransactionDAO.java:56)
        at edu.internet2.middleware.grouper.hibernate.GrouperTransaction.callbackGrouperTransaction(GrouperTransaction.java:87)
        at edu.internet2.middleware.grouper.hibernate.GrouperTransaction.callbackGrouperTransaction(GrouperTransaction.java:106)
        at edu.internet2.middleware.grouper.StemSave.save(StemSave.java:246)
        at edu.internet2.middleware.grouper.misc.GrouperCheckConfig.checkAttributes(GrouperCheckConfig.java:1777)
        at edu.internet2.middleware.grouper.misc.GrouperCheckConfig.checkConfig(GrouperCheckConfig.java:437)
        at edu.internet2.middleware.grouper.misc.GrouperStartup.startup(GrouperStartup.java:256)
      • locked <0x00002b19a0b8eb60> (a java.lang.Class for edu.internet2.middleware.grouper.misc.GrouperStartup)
        at edu.internet2.middleware.grouper.registry.RegistryInitializeSchema.main(RegistryInitializeSchema.java:188)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at edu.internet2.middleware.grouper.app.gsh.GrouperShell.handleSpecialCase(GrouperShell.java:204)
        at edu.internet2.middleware.grouper.app.gsh.GrouperShell.main(GrouperShell.java:144)
        at edu.internet2.middleware.grouper.app.gsh.GrouperShellWrapper.main(GrouperShellWrapper.java:31)

      "VM Thread" prio=10 tid=0x000000006000a000 nid=0x198 runnable

      "VM Periodic Task Thread" prio=10 tid=0x0000000060041000 nid=0x19f waiting on condition

      JNI global references: 1054

      Found one Java-level deadlock:
      =============================
      "pool-1-thread-1":
      waiting to lock monitor 0x0000000060899aa8 (object 0x00002b19a0b8eb60, a java.lang.Class),
      which is held by "main"
      "main":
      waiting to lock monitor 0x0000000060898c38 (object 0x00002b19a39453d8, a java.lang.Class),
      which is held by "pool-1-thread-1"

      Java stack information for the threads listed above:
      ===================================================
      "pool-1-thread-1":
      at edu.internet2.middleware.grouper.util.GrouperUtil.declaredMethodsCache(GrouperUtil.java:2574)

      • waiting to lock <0x00002b19a0b8eb60> (a java.lang.Class for edu.internet2.middleware.grouper.misc.GrouperStartup)
        at edu.internet2.middleware.grouper.util.GrouperUtil.retrieveDeclaredMethods(GrouperUtil.java:3876)
        at edu.internet2.middleware.grouper.util.GrouperUtil.getterHelper(GrouperUtil.java:5055)
        at edu.internet2.middleware.grouper.util.GrouperUtil.getter(GrouperUtil.java:5041)
        at edu.internet2.middleware.grouper.util.GrouperUtil.propertyValue(GrouperUtil.java:7493)
        at edu.internet2.middleware.grouper.hibernate.HibUtilsMapping.primaryKeyCurrentValue(HibUtilsMapping.java:81)
        at edu.internet2.middleware.grouper.hibernate.HibUtilsMapping.isInsert(HibUtilsMapping.java:50)
        at edu.internet2.middleware.grouper.hibernate.ByObject.saveOrUpdate(ByObject.java:373)
        at edu.internet2.middleware.grouper.hibernate.ByObjectStatic$5.callback(ByObjectStatic.java:376)
        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:700)
        at edu.internet2.middleware.grouper.hibernate.ByObjectStatic.saveOrUpdate(ByObjectStatic.java:363)
        at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3ChangeLogTypeDAO.saveOrUpdate(Hib3ChangeLogTypeDAO.java:51)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeFinder.internal_findOrReplaceChangeLogType(ChangeLogTypeFinder.java:176)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeFinder.internal_updateBuiltinTypesOnce(ChangeLogTypeFinder.java:150)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeFinder.internal_updateKnownTypes(ChangeLogTypeFinder.java:127)
      • locked <0x00002b19a39453d8> (a java.lang.Class for edu.internet2.middleware.grouper.changeLog.ChangeLogTypeFinder)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeFinder.find(ChangeLogTypeFinder.java:75)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeBuiltin.getChangeLogType(ChangeLogTypeBuiltin.java:513)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeBuiltin.getChangeLogCategory(ChangeLogTypeBuiltin.java:530)
        at edu.internet2.middleware.grouper.Member$2$1.callback(Member.java:4392)
        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:700)
        at edu.internet2.middleware.grouper.Member$2.call(Member.java:4374)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
        "main":
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeFinder.internal_updateKnownTypes(ChangeLogTypeFinder.java:115)
      • waiting to lock <0x00002b19a39453d8> (a java.lang.Class for edu.internet2.middleware.grouper.changeLog.ChangeLogTypeFinder)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeFinder.find(ChangeLogTypeFinder.java:75)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeBuiltin.getChangeLogType(ChangeLogTypeBuiltin.java:513)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogTypeBuiltin.getChangeLogCategory(ChangeLogTypeBuiltin.java:530)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogEntry.<init>(ChangeLogEntry.java:395)
        at edu.internet2.middleware.grouper.Stem.onPreSave(Stem.java:3603)
        at edu.internet2.middleware.grouper.hibernate.ByObject.save(ByObject.java:208)
        at edu.internet2.middleware.grouper.hibernate.ByObjectStatic$7.callback(ByObjectStatic.java:494)
        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:700)
        at edu.internet2.middleware.grouper.hibernate.ByObjectStatic.save(ByObjectStatic.java:481)
        at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3StemDAO.createChildStem(Hib3StemDAO.java:220)
        at edu.internet2.middleware.grouper.Stem$9.callback(Stem.java:2790)
        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:700)
        at edu.internet2.middleware.grouper.Stem.internal_addChildStem(Stem.java:2747)
      • locked <0x00002b197288d1f8> (a java.lang.String)
        at edu.internet2.middleware.grouper.Stem.addChildStem(Stem.java:598)
        at edu.internet2.middleware.grouper.StemSave$1$1.callback(StemSave.java:351)
        at edu.internet2.middleware.grouper.GrouperSession.callbackGrouperSession(GrouperSession.java:974)
        at edu.internet2.middleware.grouper.StemSave$1.callback(StemSave.java:252)
        at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3TransactionDAO$1.callback(Hib3TransactionDAO.java:66)
        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:700)
        at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3TransactionDAO.transactionCallback(Hib3TransactionDAO.java:56)
        at edu.internet2.middleware.grouper.hibernate.GrouperTransaction.callbackGrouperTransaction(GrouperTransaction.java:87)
        at edu.internet2.middleware.grouper.hibernate.GrouperTransaction.callbackGrouperTransaction(GrouperTransaction.java:106)
        at edu.internet2.middleware.grouper.StemSave.save(StemSave.java:246)
        at edu.internet2.middleware.grouper.misc.GrouperCheckConfig.checkAttributes(GrouperCheckConfig.java:1777)
        at edu.internet2.middleware.grouper.misc.GrouperCheckConfig.checkConfig(GrouperCheckConfig.java:437)
        at edu.internet2.middleware.grouper.misc.GrouperStartup.startup(GrouperStartup.java:256)
      • locked <0x00002b19a0b8eb60> (a java.lang.Class for edu.internet2.middleware.grouper.misc.GrouperStartup)
        at edu.internet2.middleware.grouper.registry.RegistryInitializeSchema.main(RegistryInitializeSchema.java:188)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at edu.internet2.middleware.grouper.app.gsh.GrouperShell.handleSpecialCase(GrouperShell.java:204)
        at edu.internet2.middleware.grouper.app.gsh.GrouperShell.main(GrouperShell.java:144)
        at edu.internet2.middleware.grouper.app.gsh.GrouperShellWrapper.main(GrouperShellWrapper.java:31)

      Found 1 deadlock.

      [appadmin@i2midev1 grouper_v2_3]$

      From: Hyzer, Chris
      Sent: Monday, April 18, 2016 2:10 PM
      To: 'Hyzer, Chris' <mchyzer@isc.upenn.edu>; 'grouper-core@internet2.edu' <grouper-core@internet2.edu>
      Subject: RE: upgrade issues

      If you put this in log4j.properties you get logging on sql running… wonder if we should default to that or prompt to put it in the installer? Hmmm

      log4j.logger.org.apache.tools.ant = DEBUG

      I will set this and run the script again in mysql and oracle and see what happens

      Thanks
      Chris

      From: Hyzer, Chris
      Sent: Sunday, April 17, 2016 2:30 PM
      To: Hyzer, Chris <mchyzer@isc.upenn.edu>; grouper-core@internet2.edu
      Subject: RE: upgrade issues

      Btw, the oracle one was 2.1 to 2.3. The mysql demo server was 2.2 to 2.3 which doesn’t seem like it should be controversial. From 2.1 there was more going on for sure… hmmmm

      From: grouper-core-request@internet2.edu grouper-core-request@internet2.edu On Behalf Of Hyzer, Chris
      Sent: Sunday, April 17, 2016 2:24 PM
      To: grouper-core@internet2.edu
      Subject: [grouper-core] upgrade issues

      I tried an upgrade of the demo server and it just hangs. Don’t know why. Maybe we need some more debug info about running scripts? I have no idea how far it went before hanging. See output below. I tried to upgrade our test env at penn and ran out of resources

                      1. oracle error
                        java.sql.SQLException: ORA-30032: the suspended (resumable) statement has timed out
                        ORA-01652: unable to extend temp segment by 1024 in tablespace SYSTEM
                      2. end oracle error

      stderr: Grouper ddl object type 'Grouper' has dbVersion: 29 and java version: 30
      Grouper database schema DDL requires updates
      (should run script manually and carefully, in sections, verify data before drop statements, backup/export important data before starting, follow change log on confluence, dont run exact same script in multiple envs - generate a new one for each env),
      script file is:
      /opt/tomcats/tomcat_f_gsh/webapps/grouper_v2_3/WEB-INF/ddlScripts/grouperDdl_20160416_16_22_08_050.sql
      Note: this script was not executed due to option passed in
      To run script via gsh, carefully review it, then run this:
      gsh -registry -runsqlfile /opt/tomcats/tomcat_f_gsh/webapps/grouper_v2_3/WEB-INF/ddlScripts/grouperDdl_20160416_16_22_08_050.sql

      Review the script(s) above if there are any, do you want the upgrader to run it to upgrade the DDL for you (t|f)? [t]:

      ##################################
      Upgrading database with command: /opt/tomcats/tomcat_f_gsh/webapps/grouper_v2_3/WEB-INF/bin/gsh.sh -registry -noprompt -runsqlfile /opt/tomcats/tomcat_f_gsh/webapps/grouper_v2_3/WEB-INF/ddlScripts/grouperDdl_20160416_16_22_08_050.sql

      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ........................................
      ......................[appadmin@i2midev1 grouper_2_3_upgrade]$

      Attachments

        Activity

          People

            chris.hyzer@at.internet2.edu Chris Hyzer (upenn.edu)
            chris.hyzer@at.internet2.edu Chris Hyzer (upenn.edu)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Smart Checklist