Quantcast
Channel: High Availability (Clustering) forum
Viewing all articles
Browse latest Browse all 2306
↧

Why automatic failover doen't happened on from Node 2 - node 1 ,2-Node cluster , sql / windows server 2016

December 20, 2019, 7:29 am
$
0
0

Hello Team,

System OS : Windows Server 2016 Standard​
​Physical server ​/on-premises / 2-node cluster /Production
​Error(s) Observed : drive went offline ​

Time of occurrence ? Saturday 12-14- 7:48 CST​

Storage : IBM SVC Storage​

Needs an  RCA why the resources did not failover between the clustered nodesNode 2 and Node 1 on the cluster

Note :Time in question is 12/14/2019 7:38 PM CST or 12/15/2019 1:38 UTC.

Node 2 was the active node with ownership of the cluster resources.  The removal of the drive occurred, SQL Services terminated, and the cluster resources did not automatically failover to Node 1 (which resides in a different Data Center location and would have been unaffected by the scheduled SAN upgrade work)

10:18 PM time period is our manual recovery efforts to fail the resources over to Node 1

Cluster Logs of Node 2  :

:2019/12/15-01:37:58.706 INFO  [GUM] Node 2: executing request locally, gumId:133616, my action: /dm/update, # of updates: 1
000013e0.00001570::2019/12/15-01:38:26.289 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] SQL Server component 'io_subsystem' health state has been changed from 'clean' to 'warning' at 2019-12-14 19:38:26.267
000013e0.00001570::2019/12/15-01:38:46.291 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] SQL Server component 'query_processing' health state has been changed from 'clean' to 'warning' at 2019-12-14 19:38:46.270
000013e0.00001570::2019/12/15-01:38:46.291 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] SQL Server component 'io_subsystem' health state has been changed from 'warning' to 'clean' at 2019-12-14 19:38:46.270
00000e9c.00002a18::2019/12/15-01:38:54.009 INFO  [STM]: Got device removal notification
000023fc.00004498::2019/12/15-01:38:54.009 INFO  [RES] Physical Disk: PNP: \\?\MPIO#Disk&Ven_IBM&Prod_2145&Rev_0000#1&7f6ac24&0&3630303530373638303139313031323141303030303030303030303030443632#{53f56307-b6bf-11d0-94f2-00a0c91efb8b} disk disappeared
000023fc.00004498::2019/12/15-01:38:54.038 INFO  [RES] Physical Disk: PNP: \\?\STORAGE#Volume#{de55b03d-c907-11e9-b804-806e6f6e6963}#0000000000100000#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b} volume disappeared
000023fc.00004498::2019/12/15-01:38:54.038 INFO  [RES] Physical Disk: PnpRemoveVolume: Removing volume \\?\STORAGE#Volume#{de55b03d-c907-11e9-b804-806e6f6e6963}#0000000000100000#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
000023fc.00004498::2019/12/15-01:38:54.038 INFO  [RES] Physical Disk <DB Engine (E)>: Incrementing resource refcount to 2.
000023fc.00004498::2019/12/15-01:38:54.038 INFO  [RES] Physical Disk <DB Engine (E)>: Incrementing resource refcount to 3.
000023fc.00004498::2019/12/15-01:38:54.038 INFO  [RES] Physical Disk: PNPDEBUG: CM_Unregister_Notification handle 000002653C855C60
000023fc.00004498::2019/12/15-01:38:54.038 INFO  [RES] Physical Disk <DB Engine (E)>: Decrementing resource refcount to 2.
000023fc.000047dc::2019/12/15-01:38:54.065 INFO  [RES] Physical Disk <DB Engine (E)>: Incrementing resource refcount to 3.
000023fc.000047dc::2019/12/15-01:38:54.065 ERR   [RES] Physical Disk: Failed to open device \Device\Harddisk4\Partition0, status 0xc0000034
000023fc.000047dc::2019/12/15-01:38:54.065 ERR   [RES] Physical Disk: PnpUpdateDiskConfigThread: Failed to open device \Device\Harddisk4\Partition0, status 2
000023fc.000047dc::2019/12/15-01:38:54.065 INFO  [RES] Physical Disk <DB Engine (E)>: Decrementing resource refcount to 2.
000023fc.00004498::2019/12/15-01:38:54.065 INFO  [RES] Physical Disk <DB Engine (E)>: Decrementing resource refcount to 1.
000023fc.00003198::2019/12/15-01:38:54.087 ERR   [RHS] Error 2 from ResourceControl 16777617 for resource DB Engine (E).
000013e0.00001570::2019/12/15-01:38:54.313 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] No more diagnostics results
000013e0.00001570::2019/12/15-01:38:54.313 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] Discard the pending result sets
000013e0.00001570::2019/12/15-01:38:54.342 ERR   [RES] SQL Server <SQL Server>: [sqsrvres] ODBC Error: [24000] [Microsoft][SQL Server Native Client 11.0]Invalid cursor state (0)
000013e0.00001570::2019/12/15-01:38:54.342 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] Diagnostics is stopped
000013e0.00001618::2019/12/15-01:38:54.342 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] Online worker helper is stopped
000013e0.00001570::2019/12/15-01:38:54.342 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] Disconnect from SQL Server
000013e0.00001570::2019/12/15-01:38:55.501 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] SQL Server was down
000013e0.0000452c::2019/12/15-01:38:57.048 ERR   [RES] SQL Server <SQL Server>: [sqsrvres] Failure detected, SQL Server is down
000013e0.0000452c::2019/12/15-01:38:57.048 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] IsAlive returns FALSE
000013e0.0000452c::2019/12/15-01:38:57.048 WARN  [RHS] Resource SQL Server IsAlive has indicated failure.
00000e9c.00003fd0::2019/12/15-01:38:57.048 INFO  [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'SQL Server', gen(0) result 1/0.
000013e0.0000452c::2019/12/15-01:38:57.048 INFO  [RHS-WER] Scheduling WER ERROR report in 10.000. ReportId d03519cb-a7e4-4e81-8c3e-3a1021532002;
00000e9c.00003fd0::2019/12/15-01:38:57.048 INFO  [RCM] Res SQL Server: Online -> ProcessingFailure( StateUnknown )
00000e9c.00003fd0::2019/12/15-01:38:57.048 INFO  [RCM] TransitionToState(SQL Server) Online-->ProcessingFailure.
00000e9c.00003fd0::2019/12/15-01:38:57.048 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (SQL Server (MSSQLSERVER), Online --> Pending)
00000e9c.00003fd0::2019/12/15-01:38:57.048 ERR   [RCM] rcm::RcmResource::HandleFailure: (SQL Server)
00000e9c.00003fd0::2019/12/15-01:38:57.049 INFO  [RCM] resource SQL Server: failure count: 0, restartAction: 0 persistentState: 1.
00000e9c.00003fd0::2019/12/15-01:38:57.049 INFO  [RCM] Will queue immediate restart (500 milliseconds) of SQL Server after terminate is complete.

000023fc.0000377c::2019/12/15-01:38:58.387 INFO  [RES] Physical Disk <DB Engine (E)>: Decrementing resource refcount to 1.
00001f24.00004788::2019/12/15-01:38:58.688 INFO  [RES] Generic Service <GxClusPlugIn (shpt19db12) (Instance001)>: GenSvcTerminate : calling SCM (didStop=1)
00001f24.00004788::2019/12/15-01:38:58.689 INFO  [RES] Generic Service <GxClusPlugIn (shpt19db12) (Instance001)>: GenSvcTerminate: retrying...
00000e9c.00001dcc::2019/12/15-01:38:58.829 INFO  [CM] mscs::RegCheckpoint::RegisterForKeyChangeNotifications: Registering for key change notifications for key SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQLServer and resource SQL Network Name (SHPT19DB12).
00000e9c.00001dcc::2019/12/15-01:38:58.886 INFO  [GUM] Node 2: executing request locally, gumId:133617, my action: /dm/update, # of updates: 1
00001f24.00004788::2019/12/15-01:38:59.001 INFO  [RES] Generic Service <GxClusPlugIn (shpt19db12) (Instance001)>: GenSvcTerminate : calling SCM (didStop=1)
00001f24.00004788::2019/12/15-01:38:59.001 INFO  [RES] Generic Service <GxClusPlugIn (shpt19db12) (Instance001)>: GenSvcTerminate: retrying...
00000e9c.00003f64::2019/12/15-01:38:59.252 INFO  [RCM] HandleMonitorReply: TERMINATERESOURCE for 'SQL Server Agent', gen(0) result 0/0.
00000e9c.00003f64::2019/12/15-01:38:59.252 INFO  [RCM] Res SQL Server Agent: [Terminating to OfflineDueToProvider] -> OfflineDueToProvider( StateUnknown )
00000e9c.00003f64::2019/12/15-01:38:59.252 INFO  [RCM] TransitionToState(SQL Server Agent) [Terminating to OfflineDueToProvider]-->OfflineDueToProvider.
00000e9c.00003f64::2019/12/15-01:38:59.252 INFO  [RCM] Res SQL Server: [WaitingToTerminate to OfflineDueToProvider] -> Terminating( OfflineDueToProvider )
00000e9c.00003f64::2019/12/15-01:38:59.252 INFO  [RCM] TransitionToState(SQL Server) [WaitingToTerminate to OfflineDueToProvider]-->[Terminating to OfflineDueToProvider].
000013e0.0000452c::2019/12/15-01:38:59.252 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] Request to terminate SQL Server
000013e0.0000452c::2019/12/15-01:38:59.252 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] Stop service MSSQLSERVER immediately
000013e0.00001570::2019/12/15-01:38:59.252 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] Extended Event logging is stopped
000013e0.0000452c::2019/12/15-01:38:59.252 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] Service was stopped successfully
000013e0.00001570::2019/12/15-01:38:59.254 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] Extended Event target state:
000013e0.00001570::2019/12/15-01:38:59.254 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] Extended Event session summary: dropped buffers = 0, dropped events = 0
000013e0.00001570::2019/12/15-01:38:59.272 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] Online worker is stopped
000013e0.0000452c::2019/12/15-01:38:59.273 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] Terminate handling is completed
000013e0.0000452c::2019/12/15-01:38:59.273 INFO  [RES] SQL Server <SQL Server>: [sqsrvres] SQL Server resource state is changed from 'ClusterResourceOnline' to 'ClusterResourceFailed'
000013e0.0000452c::2019/12/15-01:38:59.273 WARN  [RHS] returning ResourceExitStateTerminate.
00000e9c.00003fd0::2019/12/15-01:38:59.273 INFO  [RCM] HandleMonitorReply: TERMINATERESOURCE for 'SQL Server', gen(1) result 0/0.
00000e9c.00003fd0::2019/12/15-01:38:59.273 INFO  [RCM] Res SQL Server: [Terminating to OfflineDueToProvider] -> OfflineDueToProvider( StateUnknown )
00000e9c.00003fd0::2019/12/15-01:38:59.273 INFO  [RCM] TransitionToState(SQL Server) [Terminating to OfflineDueToProvider]-->OfflineDueToProvider.
00000e9c.00003fd0::2019/12/15-01:38:59.273 INFO  [RCM] DB Engine (E) not yet ready to terminate; dependent GxClusPlugIn (shpt19db12) (Instance001) still terminating.
00001f24.00004788::2019/12/15-01:38:59.314 INFO  [RES] Generic Service <GxClusPlugIn (shpt19db12) (Instance001)>: GenSvcTerminate : calling SCM (didStop=1)
00001f24.00004788::2019/12/15-01:38:59.314 INFO  [RES] Generic Service <GxClusPlugIn (shpt19db12) (Instance001)>: GenSvcTerminate: retrying...019/12/15-01:39:03.690 INFO  [RCM] Res GxClusPlugIn (shpt19db12) (Instance001): [Terminating to OfflineDueToProvider] -> OfflineDueToProvider( StateUnknown )

000013e0.000041d8::2019/12/15-01:39:07.247 INFO  [RHS-WER] Capture C:\Windows\Cluster\Reports\CLUSWER_RHS_ERROR_89020180-cdcb-4b69-b97a-62abf26ec554_9.evtx completed.
000013e0.000041d8::2019/12/15-01:39:07.247 INFO  [RHS-WER] Capturing log using query <QueryList><Query Id="0"><Select Path="Microsoft-Windows-ClusterAwareUpdating/Admin">*[System[TimeCreated[timediff(@SystemTime) &lt;= 86400000]]]</Select></Query></QueryList> to C:\Windows\Cluster\Reports\CLUSWER_RHS_ERROR_89020180-cdcb-4b69-b97a-62abf26ec554_10.evtx.
000013e0.000041d8::2019/12/15-01:39:07.271 INFO  [RHS-WER] Capture C:\Windows\Cluster\Reports\CLUSWER_RHS_ERROR_89020180-cdcb-4b69-b97a-62abf26ec554_10.evtx completed.

2019/12/15-01:52:40.681 ERR   [RES] Physical Disk: Failed to open device \Device\Harddisk4\ClusterPartition1, status 0xc0000034
000023fc.00003198::2019/12/15-01:52:40.681 ERR   [RES] Physical Disk: HarddiskpIsPartitionHidden: failed to open device \Device\Harddisk4\ClusterPartition1, status 2
000023fc.00003198::2019/12/15-01:52:40.681 ERR   [RES] Physical Disk: HardDiskpGetDiskInfo: GetVolumeInformation failed for \\?\GLOBALROOT\Device\Harddisk4\ClusterPartition1\, status 3
000023fc.00003198::2019/12/15-01:52:40.681 ERR   [RES] Physical Disk: HardDiskpGetDiskInfo: failed to get partition size for \\?\GLOBALROOT\Device\Harddisk4\ClusterPartition1\, status 3
000023fc.00004498::2019/12/15-01:52:40.685 INFO  [RES] Physical Disk: HardDiskpQueryDiskFromStm: ClusterStmFindDisk returned device='\\?\MPIO#Disk&Ven_IBM&Prod_2145&Rev_0000#1&7f6ac24&0&3630303530373638303139313031323141303030303030303030303030443632#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}'
000023fc.00004498::2019/12/15-01:52:40.685 ERR   [RES] Physical Disk: Failed to open device \Device\Harddisk4\ClusterPartition1, status 0xc0000034
000023fc.00004498::2019/12/15-01:52:40.685 ERR   [RES] Physical Disk: HarddiskpIsPartitionHidden: failed to open device \Device\Harddisk4\ClusterPartition1, status 2
000023fc.00004498::2019/12/15-01:52:40.685 ERR   [RES] Physical Disk: HardDiskpGetDiskInfo: GetVolumeInformation failed for \\?\GLOBALROOT\Device\Harddisk4\ClusterPartition1\, status 3
000023fc.00004498::2019/12/15-01:52:40.686 ERR   [RES] Physical Disk: HardDiskpGetDiskInfo: failed to get partition size for \\?\GLOBALROOT\Device\Harddisk4\ClusterPartition1\, status 3

2019/12/15-01:52:43.451 ERR   [RES] Physical Disk: Failed to open device \Device\Harddisk4\ClusterPartition1, status 0xc0000034
000023fc.000015b4::2019/12/15-01:52:43.451 ERR   [RES] Physical Disk: HarddiskpIsPartitionHidden: failed to open device \Device\Harddisk4\ClusterPartition1, status 2
000023fc.000015b4::2019/12/15-01:52:43.451 ERR   [RES] Physical Disk: HardDiskpGetDiskInfo: GetVolumeInformation failed for \\?\GLOBALROOT\Device\Harddisk4\ClusterPartition1\, status 3
000023fc.000015b4::2019/12/15-01:52:43.452 ERR   [RES] Physical Disk: HardDiskpGetDiskInfo: failed to get partition size for \\?\GLOBALROOT\Device\Harddisk4\ClusterPartition1\, status 3
2019/12/15-02:07:43.953 WARN  [RCM] ResourceTypeChaseTheOwnerLoop::DoCall: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
00000e9c.00003f64::2019/12/15-02:07:43.966 WARN  [RCM] ResourceTypeChaseTheOwnerLoop::DoCall: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
000023fc.00002804::2019/12/15-02:07:43.985 INFO  [RES] Physical Disk: HardDiskpQueryDiskFromStm: ClusterStmFindDisk returned device='\\?\MPIO#Disk&Ven_IBM&Prod_2145&Rev_0000#1&7f6ac24&0&3630303530373638303139313031323141303030303030303030303030443632#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}'
000023fc.00002804::2019/12/15-02:07:43.986 ERR   [RES] Physical Disk: Failed to open device \Device\Harddisk4\ClusterPartition1, status 0xc0000034

019/12/15-02:22:40.260 INFO  [RES] Network Name <Cluster Name>: Getting Read only private properties
00002140.00002d30::2019/12/15-02:22:40.438 INFO  [RES] Network Name <SHPT19DB12DTC>: Getting Read only private properties
00002140.00002d30::2019/12/15-02:22:40.483 INFO  [RES] Network Name <SQL Network Name (SHPT19DB12)>: Getting Read only private properties

2019/12/15-02:22:40.614 INFO  [RES] Physical Disk: HardDiskpQueryDiskFromStm: ClusterStmFindDisk returned device='\\?\MPIO#Disk&Ven_IBM&Prod_2145&Rev_0000#1&7f6ac24&0&3630303530373638303139313031323141303030303030303030303030443632#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}'
000023fc.000021dc::2019/12/15-02:22:40.614 ERR   [RES] Physical Disk: Failed to open device \Device\Harddisk4\ClusterPartition1, status 0xc0000034
000023fc.000021dc::2019/12/15-02:22:40.614 ERR   [RES] Physical Disk: HarddiskpIsPartitionHidden: failed to open device \Device\Harddisk4\ClusterPartition1, status 2
000023fc.000021dc::2019/12/15-02:22:40.614 ERR   [RES] Physical Disk: HardDiskpGetDiskInfo: GetVolumeInformation failed for \\?\GLOBALROOT\Device\Harddisk4\ClusterPartition1\, status 3
000023fc.000021dc::2019/12/15-02:22:40.614 ERR   [RES] Physical Disk: HardDiskpGetDiskInfo: failed to get partition size for \\?\GLOBALROOT\Device\Harddisk4\ClusterPartition1\, status 3
000023fc.00003198::2019/12/15-02:22:40.619 INFO  [RES] Physical Disk: HardDiskpQueryDiskFromStm: ClusterStmFindDisk returned device='\\?\MPIO#Disk&Ven_IBM&Prod_2145&Rev_0000#1&7f6ac24&0&3630303530373638303139313031323141303030303030303030303030443632#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}'
000023fc.00003198::2019/12/15-02:22:40.619 ERR   [RES] Physical Disk: Failed to open device \Device\Harddisk4\ClusterPartition1, status 0xc00000342019/12/15-03:22:42.522 ERR   [RES] Physical Disk: Failed to open device \Device\Harddisk4\ClusterPartition1, status 0xc0000034
000023fc.000044cc::2019/12/15-03:22:42.522 ERR   [RES] Physical Disk: HarddiskpIsPartitionHidden: failed to open device \Device\Harddisk4\ClusterPartition1, status 2
000023fc.000044cc::2019/12/15-03:22:42.523 ERR   [RES] Physical Disk: HardDiskpGetDiskInfo: GetVolumeInformation failed for \\?\GLOBALROOT\Device\Harddisk4\ClusterPartition1\, status 3
000023fc.000044cc::2019/12/15-03:22:42.523 ERR   [RES] Physical Disk: HardDiskpGetDiskInfo: failed to get partition size for \\?\GLOBALROOT\Device\Harddisk4\ClusterPartition1\, status 3

Logs after manual failover happened : 

000023fc.00002804::2019/12/15-04:07:44.556 ERR   [RES] Physical Disk: HardDiskpGetDiskInfo: failed to get partition size for \\?\GLOBALROOT\Device\Harddisk4\ClusterPartition1\, status 3
00000e9c.00001238::2019/12/15-04:18:39.550 INFO  [CS] PreShutdown notification.
00000e9c.00001238::2019/12/15-04:18:39.550 INFO  [CS] Service Stopping...
00000e9c.00001238::2019/12/15-04:18:39.550 INFO  [CORE] Node quorum state is 'Successfully formed or joined a cluster'. Form/join status with other nodes is as follows:
00000e9c.00001238::2019/12/15-04:18:39.550 INFO  [NODE] Node 2: Farthest reported progress joining with node SHPT19DB01 (id 1) is: Join Succeeded at time 2019/11/22-05:13:06.752: status 0
00000e9c.000011f0::2019/12/15-04:18:39.550 INFO  [CORE] Graceful shutdown reported by node SHPT19DB02, reason ServiceStopReason::SystemPreShutdown (payload 4)
00000e9c.00001238::2019/12/15-04:18:39.677 INFO  [DCM] UnregisterSwProvider(exit): HrError(0x00000000)
00000e9c.00001238::2019/12/15-04:18:39.678 INFO  [GUM] Node 2: Processing RequestLock 2:32966
00000e9c.00004160::2019/12/15-04:18:39.679 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 133620)
00000e9c.00001238::2019/12/15-04:18:39.679 INFO  [GUM] Node 2: executing request locally, gumId:133621, my action: qm/set-node-weight, # of updates: 1
00000e9c.00001238::2019/12/15-04:18:39.679 WARN  [QUORUM] Node 2: weight adjustment not performed. Cannot go below weight count 3 in a hybrid configuration with 2+ nodes
00000e9c.00001238::2019/12/15-04:18:39.680 INFO  [DM] An empty single transaction is cancelled 27:27:146656+1::0
00000e9c.00001238::2019/12/15-04:18:39.683 INFO  [GUM] Node 2: executing request locally, gumId:133622, my action: /dm/update, # of updates: 1
00000e9c.00001fec::2019/12/15-04:18:39.687 INFO  [RCM] rcm::RcmAgent::Unload: [RCM] Shutdown lock acquired, proceeding, initPhase:false StopReason:ServiceStopReason::SystemPreShutdown
00000e9c.00001fec::2019/12/15-04:18:39.687 INFO  [RCM] rcm::RcmAgent::Unload: [RCM] Director node is shutting down, choose new director
00000e9c.00001fec::2019/12/15-04:18:39.687 INFO  [RCM] Sending causal to set new director node 1
00000e9c.00001fec::2019/12/15-04:18:39.687 INFO  [RCM] Director is set to node 1
00000e9c.00001fec::2019/12/15-04:18:39.688 INFO  [RCM-ClusterInit] Cluster initialization done using older create cluster mechanism
00000e9c.00001fec::2019/12/15-04:18:39.688 INFO  [RCM-ClusterInit] Not queuing work item to initialize cluster core as it has already been initialized.
00000e9c.00001fec::2019/12/15-04:18:39.688 INFO  [RCM] rcm::DrainMgr::DrainNode: [DrainMgr] Initiating DrainNode
00000e9c.00001f18::2019/12/15-04:18:39.696 INFO  [RCM] Waiting controls thread detected Rcm shutting down, exiting
00000e9c.00001f18::2019/12/15-04:18:39.696 INFO  [RCM] Waiting controls processing thread is shutting down and draining any remaining controls it was monitoring.
00000e9c.00001fec::2019/12/15-04:18:39.726 INFO  [RCM] rcm::DrainMgr::ValidateDrainNode: [DrainMgr] Node state:Up. DrainStatus:0. bServiceShutdown:1
00000e9c.00001fec::2019/12/15-04:18:39.726 INFO  [RCM] rcm::DrainMgr::StartDrain: [DrainMgr] Starting Drain.
00000e9c.00001fec::2019/12/15-04:18:39.726 INFO  [RCM] rcm::DrainMgr::UpdateDrainStatus: [DrainMgr] Changing DrainStatus for node "2" to 1
00000e9c.00001fec::2019/12/15-04:18:39.726 INFO  [GUM] Node 2: executing request locally, gumId:133623, my action: /dm/update, # of updates: 1
00000e9c.00001fec::2019/12/15-04:18:39.743 INFO  [RCM] rcm::DrainMgr::UpdateDrainTarget: [DrainMgr] Changing DrainTarget for node to "-1"
00000e9c.00001fec::2019/12/15-04:18:39.743 INFO  [RCM] rcm::DrainMgr::StartDrain: [DrainMgr] Doing Drain without setting to paused state
00000e9c.00001fec::2019/12/15-04:18:39.752 INFO  [RCM] rcm::RcmGroup::GetGroupMoveContext: (Cluster Group)
00000e9c.00001fec::2019/12/15-04:18:39.752 INFO  [RCM] rcm::RcmGroup::GetGroupMoveContext: (Cluster Group)
00000e9c.00001fec::2019/12/15-04:18:39.752 INFO  [RCM] rcm::DrainMgr::MoveGroupForDrain: [DrainMgr] Initiating MoveGroupForDrain for group:Cluster Group with Flags:0, move count:0
00000e9c.00001fec::2019/12/15-04:18:39.752 INFO  [RCM-plcmt] This node is not director, node 1 is.  Asking others for placement...
00000e9c.00001fec::2019/12/15-04:18:39.752 INFO  [RCM-plcmt] asking node 1 placement decision, attempt 1
00000e9c.00001f18::2019/12/15-04:18:39.755 ERR   [API] s_ApiGetNotifyAsync: (5073)' because of 'Shutting down.'
00000e9c.00001fec::2019/12/15-04:18:39.763 INFO  [RCM-plcmt] done waiting...
00000e9c.00001fec::2019/12/15-04:18:39.763 INFO  [RCM-plcmt] Node 1 replied to placement request g=Cluster Group tgt=1 wait=false
00000e9c.00001fec::2019/12/15-04:18:39.763 INFO  MTimer(GetPlacementFromDirector): [Start to Multitimer_destroyed : 16 ms]
00000e9c.00001fec::2019/12/15-04:18:39.763 INFO  MTimer(GetPlacementFromDirector): [Total: 16 ms ( 0 s )]
00000e9c.00001fec::2019/12/15-04:18:39.763 INFO  [RCM] placement manager picked node 1 for drain move of group Cluster Group
00000e9c.00001fec::2019/12/15-04:18:39.763 INFO  [RCM] rcm::DrainMgr::MoveGroupForDrain: [DrainMgr] calling queueGroup:Cluster Group with flags:4 operationType:1 bBounceBack:0
00000e9c.00001fec::2019/12/15-04:18:39.763 INFO  [RCM] rcm::RcmGroup::QueueGroup: (Cluster Group, 1, 4)
00000e9c.00001fec::2019/12/15-04:18:39.763 INFO  [RCM] Cluster Group: Added Flags 4 to StatusInformation. New StatusInformation 4
00000e9c.00001fec::2019/12/15-04:18:39.763 INFO  [RCM] rcm::QueuedMovesHolder::AddGroup: Adding group Cluster Group)
00000e9c.000034f0::2019/12/15-04:18:39.763 ERR   [API] s_ApiGetNotifyAsync: (5073)' because of 'Shutting down.'
00000e9c.00001f18::2019/12/15-04:18:39.764 ERR   [API] s_ApiGetNotifyAsync: (5073)' because of 'Shutting down.'
00002140.00002f9c::2019/12/15-04:18:39.765 ERR   [RES] Network Name <Cluster Name>: Error: 0x800706BA in FileServerDefaultShareImpl::ResyncDisks@352.
00002140.00002f9c::2019/12/15-04:18:39.765 ERR   [RES] Network Name <Cluster Name>: Error: 0x800706BA in FileServerDefaultShareImpl::NotificationThread@528.
00000e9c.00001fec::2019/12/15-04:18:39.767 INFO  [RCM] rcm::DrainMgr::MoveGroupForDrain: [DrainMgr] returning from MoveGroupForDrain for group:Cluster Group with status:0
00000e9c.00001fec::2019/12/15-04:18:39.767 INFO  [RCM] rcm::RcmGroup::GetGroupMoveContext: (SHPT19DB12DTC)
00000e9c.00001fec::2019/12/15-04:18:39.767 INFO  [RCM] rcm::RcmGroup::GetGroupMoveContext: (SHPT19DB12DTC)
00000e9c.00001fec::2019/12/15-04:18:39.767 INFO  [RCM] rcm::DrainMgr::MoveGroupForDrain: [DrainMgr] Initiating MoveGroupForDrain for group:SHPT19DB12DTC with Flags:0, move count:0
00000e9c.00001f18::2019/12/15-04:18:39.767 INFO  [RCM-plcmt] This node is not director, node 1 is.  Asking others for placement...
00000e9c.00001fec::2019/12/15-04:18:39.767 INFO  [RCM-plcmt] This node is not director, node 1 is.  Asking others for placement...
00000e9c.00001f18::2019/12/15-04:18:39.767 INFO  [RCM-plcmt] asking node 1 placement decision, attempt 1
00000e9c.00001fec::2019/12/15-04:18:39.767 INFO  [RCM-plcmt] asking node 1 placement decision, attempt 1
00000e9c.00001f18::2019/12/15-04:18:39.770 INFO  [RCM-plcmt] done waiting...
00000e9c.00001f18::2019/12/15-04:18:39.770 INFO  [RCM-plcmt] Node 1 replied to placement request g=Cluster Group tgt=1 wait=false
00000e9c.00001f18::2019/12/15-04:18:39.770 INFO  MTimer(GetPlacementFromDirector): [Total: 0 ms ( 0 s )]
00000e9c.00001f18::2019/12/15-04:18:39.770 INFO  [RCM] placement manager picked node 1 for MoveType::Drain move of group Cluster Group
00000e9c.000034f0::2019/12/15-04:18:39.770 INFO  [RCM] rcm::RcmGroup::DoQueuedMove: (Cluster Group)
00000e9c.000034f0::2019/12/15-04:18:39.770 INFO  [RCM] rcm::RcmGroup::Move: (Cluster Group, 1, MoveType::Drain)
00000e9c.000034f0::2019/12/15-04:18:39.770 INFO  [RCM] rcm::RcmGroup::Move: Bringing group 'Cluster Group' offline first...
00000e9c.000034f0::2019/12/15-04:18:39.770 INFO  [RCM] Res Cluster Name: Online -> WaitingToGoOffline( StateUnknown )
00000e9c.000034f0::2019/12/15-04:18:39.770 INFO  [RCM] TransitionToState(Cluster Name) Online-->WaitingToGoOffline.
00000e9c.000034f0::2019/12/15-04:18:39.771 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (Cluster Group, Online --> Pending)
00000e9c.000034f0::2019/12/15-04:18:39.771 INFO  [RCM] Res Cluster Name: WaitingToGoOffline -> OfflineCallIssued( StateUnknown )
00000e9c.000034f0::2019/12/15-04:18:39.771 INFO  [RCM] TransitionToState(Cluster Name) WaitingToGoOffline-->OfflineCallIssued.
00000e9c.000034f0::2019/12/15-04:18:39.771 INFO  [RCM] Res Cluster IP Address: Online -> WaitingToGoOffline( StateUnknown )
00000e9c.000034f0::2019/12/15-04:18:39.771 INFO  [RCM] TransitionToState(Cluster IP Address) Online-->WaitingToGoOffline.
00000e9c.000034f0::2019/12/15-04:18:39.771 INFO  [RCM] 'Cluster IP Address' cannot go offline yet; Dependent Resource 'Cluster Name' is in state OfflineCallIssued.
00000e9c.000034f0::2019/12/15-04:18:39.771 INFO  [RCM] Res Cluster IP Address: WaitingToGoOffline -> WaitingToGoOffline( StateUnknown )
00000e9c.000034f0::2019/12/15-04:18:39.771 INFO  [RCM] rcm::RcmGroup::Offline: deferring offline of quorum resource 'File Share Witness' until all other resources are offline.
00000e9c.00004128::2019/12/15-04:18:39.771 ERR   [API] s_ApiGetNotifyAsync: (5073)' because of 'Shutting down.'
00002140.000017f0::2019/12/15-04:18:39.771 INFO  [RES] Network Name <Cluster Name>: Offline called for resource Cluster Name
00002140.000045c4::2019/12/15-04:18:39.771 INFO  [RES] Network Name <Cluster Name>: Entering Offline thread
00000e9c.00003f20::2019/12/15-04:18:39.771 INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'Cluster Name', gen(0) result 997/0.
00000e9c.00003f20::2019/12/15-04:18:39.771 INFO  [RCM] Res Cluster Name: OfflineCallIssued -> OfflinePending( StateUnknown )
00000e9c.00003f20::2019/12/15-04:18:39.771 INFO  [RCM] TransitionToState(Cluster Name) OfflineCallIssued-->OfflinePending.
00002140.000045c4::2019/12/15-04:18:39.771 INFO  [RES] Network Name <Cluster Name>: Performing actual offline of resource.
00002140.000045c4::2019/12/15-04:18:39.771 INFO  [RES] Network Name: Agent: Closing (1b9cd52e-c9ce-445b-a6cb-30a3b0f86d69,AdminShare)
00002140.000017f0::2019/12/15-04:18:39.771 INFO  [RES] Network Name: Agent: OnClose (1b9cd52e-c9ce-445b-a6cb-30a3b0f86d69,AdminShare)
00002140.000017f0::2019/12/15-04:18:39.771 INFO  [RES] Network Name <Cluster Name>: AdminShare: Canceling work, state: Closing/Idle
00002140.000017f0::2019/12/15-04:18:39.771 INFO  [RES] Network Name <Cluster Name>: AdminShare: OnCloseBase, previous state: Initialized/Idle
00000e9c.00001fec::2019/12/15-04:18:39.773 INFO  [RCM-plcmt] done waiting...
00000e9c.00001fec::2019/12/15-04:18:39.773 INFO  [RCM-plcmt] Node 1 replied to placement request g=SHPT19DB12DTC tgt=1 wait=false
00000e9c.00001fec::2019/12/15-04:18:39.773 INFO  MTimer(GetPlacementFromDirector): [Total: 0 ms ( 0 s )]
00000e9c.00001fec::2019/12/15-04:18:39.773 INFO  [RCM] placement manager picked node 1 for drain move of group SHPT19DB12DTC
00000e9c.00001fec::2019/12/15-04:18:39.773 INFO  [RCM] rcm::DrainMgr::MoveGroupForDrain: [DrainMgr] calling queueGroup:SHPT19DB12DTC with flags:4 operationType:1 bBounceBack:0
00000e9c.00001fec::2019/12/15-04:18:39.773 INFO  [RCM] rcm::RcmGroup::QueueGroup: (SHPT19DB12DTC, 1, 4)
00000e9c.00001fec::2019/12/15-04:18:39.773 INFO  [RCM] SHPT19DB12DTC: Added Flags 4 to StatusInformation. New StatusInformation 4
00000e9c.00001fec::2019/12/15-04:18:39.773 INFO  [RCM] rcm::QueuedMovesHolder::AddGroup: Adding group SHPT19DB12DTC)
00000e9c.00001fec::2019/12/15-04:18:39.773 INFO  [RCM] rcm::DrainMgr::MoveGroupForDrain: [DrainMgr] returning from MoveGroupForDrain for group:SHPT19DB12DTC with status:0
00000e9c.00001fec::2019/12/15-04:18:39.773 INFO  [RCM] rcm::RcmGroup::GetGroupMoveContext: (SQL Server (MSSQLSERVER))
00000e9c.00001fec::2019/12/15-04:18:39.773 INFO  [RCM] rcm::RcmGroup::GetGroupMoveContext: (SQL Server (MSSQLSERVER))
00000e9c.00001fec::2019/12/15-04:18:39.773 INFO  [RCM] rcm::DrainMgr::MoveGroupForDrain: [DrainMgr] Initiating MoveGroupForDrain for group:SQL Server (MSSQLSERVER) with Flags:0, move count:0
00000e9c.00001f18::2019/12/15-04:18:39.773 INFO  [RCM-plcmt] This node is not director, node 1 is.  Asking others for placement...
00000e9c.00001fec::2019/12/15-04:18:39.773 INFO  [RCM-plcmt] This node is not director, node 1 is.  Asking others for placement...
00000e9c.00001f18::2019/12/15-04:18:39.773 INFO  [RCM-plcmt] asking node 1 placement decision, attempt 1
00000e9c.00001fec::2019/12/15-04:18:39.773 INFO  [RCM-plcmt] asking node 1 placement decision, attempt 1
00000e9c.00003f20::2019/12/15-04:18:39.773 ERR   [API] s_ApiGetNotifyAsync: (5073)' because of 'Shutting down.'
00000e9c.00001f18::2019/12/15-04:18:39.776 INFO  [RCM-plcmt] done waiting...
00000e9c.00001f18::2019/12/15-04:18:39.776 INFO  [RCM-plcmt] Node 1 replied to placement request g=SHPT19DB12DTC tgt=1 wait=false
00000e9c.00001f18::2019/12/15-04:18:39.776 INFO  MTimer(GetPlacementFromDirector): [Start to Multitimer_destroyed : 16 ms]
00000e9c.00001f18::2019/12/15-04:18:39.776 INFO  MTimer(GetPlacementFromDirector): [Total: 16 ms ( 0 s )]
00000e9c.00001f18::2019/12/15-04:18:39.776 INFO  [RCM] placement manager picked node 1 for MoveType::Drain move of group SHPT19DB12DTC
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] rcm::RcmGroup::DoQueuedMove: (SHPT19DB12DTC)
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] rcm::RcmGroup::Move: (SHPT19DB12DTC, 1, MoveType::Drain)
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] rcm::RcmGroup::Move: Bringing group 'SHPT19DB12DTC' offline first...
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] Res SHPT19DB12DTC: Online -> WaitingToGoOffline( StateUnknown )
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] TransitionToState(SHPT19DB12DTC) Online-->WaitingToGoOffline.
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (SHPT19DB12DTC, Online --> Pending)
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] Bringing dependent resource 'New Distributed Transaction Coordinator' offline before provider resource 'SHPT19DB12DTC'.
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] Res New Distributed Transaction Coordinator: Online -> WaitingToGoOffline( StateUnknown )
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] TransitionToState(New Distributed Transaction Coordinator) Online-->WaitingToGoOffline.
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] Res New Distributed Transaction Coordinator: WaitingToGoOffline -> OfflineCallIssued( StateUnknown )
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] TransitionToState(New Distributed Transaction Coordinator) WaitingToGoOffline-->OfflineCallIssued.
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] 'SHPT19DB12DTC' cannot go offline yet; Dependent Resource 'New Distributed Transaction Coordinator' is in state OfflineCallIssued.
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] Res SHPT19DB12DTC: WaitingToGoOffline -> WaitingToGoOffline( StateUnknown )
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] Res MS DTC: Online -> WaitingToGoOffline( StateUnknown )
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] TransitionToState(MS DTC) Online-->WaitingToGoOffline.
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] 'MS DTC' cannot go offline yet; Dependent Resource 'New Distributed Transaction Coordinator' is in state OfflineCallIssued.
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] Res MS DTC: WaitingToGoOffline -> WaitingToGoOffline( StateUnknown )
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] Res IP Address 10.126.5.129: Online -> WaitingToGoOffline( StateUnknown )
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] TransitionToState(IP Address 10.126.5.129) Online-->WaitingToGoOffline.
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] Bringing dependent resource 'SHPT19DB12DTC' offline before provider resource 'IP Address 10.126.5.129'.
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] 'SHPT19DB12DTC' cannot go offline yet; Dependent Resource 'New Distributed Transaction Coordinator' is in state OfflineCallIssued.
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] Res SHPT19DB12DTC: WaitingToGoOffline -> WaitingToGoOffline( StateUnknown )
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] 'IP Address 10.126.5.129' cannot go offline yet; Dependent Resource 'SHPT19DB12DTC' is in state WaitingToGoOffline.
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] Res IP Address 10.126.5.129: WaitingToGoOffline -> WaitingToGoOffline( StateUnknown )
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] Res cmd Application: Online -> WaitingToGoOffline( StateUnknown )
00000e9c.00004128::2019/12/15-04:18:39.776 INFO  [RCM] TransitionToState(cmd Application) Online-->WaitingToGoOffline.
00000e9c.00004128::2019/12/15-04:18:39.777 INFO  [RCM] Bringing dependent resource 'MS DTC' offline before provider resource 'cmd Application'.
00000e9c.00004128::2019/12/15-04:18:39.777 INFO  [RCM] 'MS DTC' cannot go offline yet; Dependent Resource 'New Distributed Transaction Coordinator' is in state OfflineCallIssued.
00000e9c.00004128::2019/12/15-04:18:39.777 INFO  [RCM] Res MS DTC: WaitingToGoOffline -> WaitingToGoOffline( StateUnknown )
00000e9c.00004128::2019/12/15-04:18:39.777 INFO  [RCM] 'cmd Application' cannot go offline yet; Dependent Resource 'MS DTC' is in state WaitingToGoOffline.
00000e9c.00004128::2019/12/15-04:18:39.777 INFO  [RCM] Res cmd Application: WaitingToGoOffline -> WaitingToGoOffline( StateUnknown )
00001f2c.00003db8::2019/12/15-04:18:39.777 INFO  [RES] Distributed Transaction Coordinator <New Distributed Transaction Coordinator>: 12-14-2019 22:18:39:774 : [1f2c.3db8] 0x00000000 [TRACE_RESOURCE] [  TRACE_INFO] DtcOffline (com\complus\dtc\shared\mtxclu\src\dtcresource.cpp@790): Taking DTC resource offline
00001f2c.00003db8::2019/12/15-04:18:39.777 INFO  [RES] Distributed Transaction Coordinator <New Distributed Transaction Coordinator>: 12-14-2019 22:18:39:774 : [1f2c.3db8] 0x00000000 [TRACE_RESOURCE] [  TRACE_INFO] DtcOffline (com\complus\dtc\shared\mtxclu\src\dtcresource.cpp@798): DTC offline thread successfully started
00000e9c.000034f0::2019/12/15-04:18:39.777 INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'New Distributed Transaction Coordinator', gen(0) result 997/0.
00000e9c.000034f0::2019/12/15-04:18:39.777 INFO  [RCM] Res New Distributed Transaction Coordinator: OfflineCallIssued -> OfflinePending( StateUnknown )
00000e9c.000034f0::2019/12/15-04:18:39.777 INFO  [RCM] TransitionToState(New Distributed Transaction Coordinator) OfflineCallIssued-->OfflinePending.
00001f2c.000021d8::2019/12/15-04:18:39.777 INFO  [RES] Distributed Transaction Coordinator <New Distributed Transaction Coordinator>: 12-14-2019 22:18:39:774 : [1f2c.21d8] 0x00000000 [TRACE_RESOURCE] [  TRACE_INFO] DtcOfflineThread (com\complus\dtc\shared\mtxclu\src\dtcresource.cpp@822): DTC offline thread is running
00001f2c.000021d8::2019/12/15-04:18:39.777 INFO  [RES] Distributed Transaction Coordinator <New Distributed Transaction Coordinator>: 12-14-2019 22:18:39:774 : [1f2c.21d8] 0x00000000 [TRACE_RESOURCE] [  TRACE_INFO] DtcOfflineThread (com\complus\dtc\shared\mtxclu\src\dtcresource.cpp@837): Stopping service...
00002140.00000b98::2019/12/15-04:18:39.778 ERR   [RES] Network Name <SHPT19DB12DTC>: Error: 0x800706BA in FileServerDefaultShareImpl::ResyncDisks@352.
00002140.00000b98::2019/12/15-04:18:39.778 ERR   [RES] Network Name <SHPT19DB12DTC>: Error: 0x800706BA in FileServerDefaultShareImpl::NotificationThread@528.
00000e9c.00001fec::2019/12/15-04:18:39.781 INFO  [RCM-plcmt] done waiting...
00000e9c.00001fec::2019/12/15-04:18:39.781 INFO  [RCM-plcmt] Node 1 replied to placement request g=SQL Server (MSSQLSERVER) tgt=1 wait=false
00000e9c.00001fec::2019/12/15-04:18:39.781 INFO  MTimer(GetPlacementFromDirector): [Start to Multitimer_destroyed : 16 ms]
00000e9c.00001fec::2019/12/15-04:18:39.781 INFO  MTimer(GetPlacementFromDirector): [Total: 16 ms ( 0 s )]
00000e9c.00001fec::2019/12/15-04:18:39.781 INFO  [RCM] placement manager picked node 1 for drain move of group SQL Server (MSSQLSERVER)
00000e9c.00001fec::2019/12/15-04:18:39.781 INFO  [RCM] rcm::DrainMgr::MoveGroupForDrain: [DrainMgr] calling queueGroup:SQL Server (MSSQLSERVER) with flags:4 operationType:1 bBounceBack:0
00000e9c.00001fec::2019/12/15-04:18:39.781 INFO  [RCM] rcm::RcmGroup::QueueGroup: (SQL Server (MSSQLSERVER), 1, 4)
00000e9c.00001fec::2019/12/15-04:18:39.781 INFO  [RCM] SQL Server (MSSQLSERVER): Added Flags 4 to StatusInformation. New StatusInformation 4
00000e9c.00001fec::2019/12/15-04:18:39.781 INFO  [RCM] rcm::QueuedMovesHolder::AddGroup: Adding group SQL Server (MSSQLSERVER))
00000e9c.00001fec::2019/12/15-04:18:39.781 INFO  [RCM] rcm::DrainMgr::MoveGroupForDrain: [DrainMgr] returning from MoveGroupForDrain for group:SQL Server (MSSQLSERVER) with status:0
00000e9c.00001fec::2019/12/15-04:18:39.781 INFO  [RCM] rcm::RcmGroup::GetGroupMoveContext: (Available Storage)
00000e9c.00001fec::2019/12/15-04:18:39.781 INFO  [RCM] rcm::RcmGroup::GetGroupMoveContext: (Available Storage)
00000e9c.00001fec::2019/12/15-04:18:39.781 INFO  [RCM] rcm::DrainMgr::MoveGroupForDrain: [DrainMgr] Initiating MoveGroupForDrain for group:Available Storage with Flags:0, move count:0
00000e9c.00001fec::2019/12/15-04:18:39.781 INFO  [RCM-plcmt] This node is not director, node 1 is.  Asking others for placement...
00000e9c.00001fec::2019/12/15-04:18:39.781 INFO  [RCM-plcmt] asking node 1 placement decision, attempt 1
00000e9c.00001f18::2019/12/15-04:18:39.781 INFO  [RCM-plcmt] This node is not director, node 1 is.  Asking others for placement...
00000e9c.00001f18::2019/12/15-04:18:39.781 INFO  [RCM-plcmt] asking node 1 placement decision, attempt 1
00000e9c.000034f0::2019/12/15-04:18:39.781 ERR   [API] s_ApiGetNotifyAsync: (5073)' because of 'Shutting down.'
00000e9c.00002d8c::2019/12/15-04:18:39.782 WARN  [RCM] rcm::ChaseTheOwnerLoop::NoLockIsCallComplete: RCM is shutting down. Not chasing the owner on error 0.
00000e9c.00001fec::2019/12/15-04:18:39.783 INFO  [RCM-plcmt] done waiting...
00000e9c.00001fec::2019/12/15-04:18:39.783 INFO  [RCM-plcmt] Node 1 replied to placement request g=Available Storage tgt=1 wait=false
00000e9c.00001fec::2019/12/15-04:18:39.783 INFO  MTimer(GetPlacementFromDirector): [Total: 0 ms ( 0 s )]
00000e9c.00001fec::2019/12/15-04:18:39.783 INFO  [RCM] placement manager picked node 1 for drain move of group Available Storage
00000e9c.00001fec::2019/12/15-04:18:39.783 INFO  [RCM] rcm::DrainMgr::MoveGroupForDrain: [DrainMgr] calling groupMove:Available Storage with flags:0 operationType:1 bBounceBack:0
00000e9c.00001fec::2019/12/15-04:18:39.783 INFO  [RCM] rcm::RcmGroup::Move: (Available Storage, 1, MoveType::Drain)
00000e9c.00001fec::2019/12/15-04:18:39.783 INFO  [RCM] No need to bring group 'Available Storage' offline before move; state Offline, owner 2.
00000e9c.00001fec::2019/12/15-04:18:39.783 INFO  [RCM] rcm::DrainMgr::MoveGroupForDrain: [DrainMgr] returning from MoveGroupForDrain for group:Available Storage with status:0
00000e9c.00001fec::2019/12/15-04:18:39.783 INFO  [RCM] rcm::DrainMgr::StartDrain: [DrainMgr] StartDrain returned with Status:0
00000e9c.00001fec::2019/12/15-04:18:39.783 INFO  [RCM] rcm::DrainMgr::DrainNode: [DrainMgr] DrainNode returned with Status:0
00000e9c.00001fec::2019/12/15-04:18:39.783 INFO  [RCM] rcm::RcmAgent::DrainOnShutdown: [Shutdown] Returned from DrainNode. TimeElapsed:94
00000e9c.00002d8c::2019/12/15-04:18:39.783 INFO  [RCM] rcm::QueuedMovesHolder::RemoveGroup: (Available Storage) GroupBeingMoved: true AllowMoveCancel: false NotifyMoveFailure: false
00000e9c.00002d8c::2019/12/15-04:18:39.783 INFO  [GUM] Node 2: executing request locally, gumId:133624, my action: /rcm/gum/GroupMoveOperation, # of updates: 1
00000e9c.00002d8c::2019/12/15-04:18:39.783 INFO  [RCM] rcm::RcmGum::GroupMoveOperation(1)
00000e9c.00002d8c::2019/12/15-04:18:39.783 INFO  [RCM] move of group Available Storage from SHPT19DB02(2) to SHPT19DB01(1) of type MoveType::Drain is about to succeed, failoverCount=0, lastFailoverTime=1601/01/01-00:00:00.000 targeted=false
00000e9c.00002d8c::2019/12/15-04:18:39.783 INFO  [RCM] rcm::DrainMgr::QueueMoveEvent: [DrainMgr] Queued GroupMoveEvent for Group Available Storage
00000e9c.00002d8c::2019/12/15-04:18:39.783 INFO  MTimer(OneGroupMovetimer): [Total: 0 ms ( 0 s )]
00000e9c.00002d8c::2019/12/15-04:18:39.783 INFO  [RCM] processed 1 group moves, 0 failures
00000e9c.00002d8c::2019/12/15-04:18:39.783 INFO  [DM] An empty single transaction is cancelled 27:27:146658+1::0
00000e9c.00003f20::2019/12/15-04:18:39.783 INFO  [RCM] rcm::DrainMgr::ProcessGroupMoveEvent: [DrainMgr] Processing GroupMoveEvent for Group Available Storage
00000e9c.00003f20::2019/12/15-04:18:39.783 INFO  [RCM] rcm::DrainMgr::GroupOwnershipChangedWorker: [DrainMgr] Handling Group owner changed Group Available Storage src:2 dest:1 MoveType:MoveType::Drain.
00000e9c.00003f20::2019/12/15-04:18:39.783 INFO  [RCM] rcm::DrainMgr::GroupOwnershipChangedWorker: [DrainMgr] Move of group Available Storage completed from src to 1.
00000e9c.00003f20::2019/12/15-04:18:39.783 INFO  [RCM] rcm::DrainMgr::GroupMoveSucceeded: [DrainMgr] Group move succeeded for Available Storage
00000e9c.00001f18::2019/12/15-04:18:39.788 INFO  [RCM-plcmt] done waiting...
00000e9c.00001f18::2019/12/15-04:18:39.788 INFO  [RCM-plcmt] Node 1 replied to placement request g=SQL Server (MSSQLSERVER) tgt=1 wait=false
00000e9c.00001f18::2019/12/15-04:18:39.788 INFO  MTimer(GetPlacementFromDirector): [Total: 0 ms ( 0 s )]
00000e9c.00001f18::2019/12/15-04:18:39.788 INFO  [RCM] placement manager picked node 1 for MoveType::Drain move of group SQL Server (MSSQLSERVER)
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] rcm::RcmGroup::DoQueuedMove: (SQL Server (MSSQLSERVER))
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] rcm::RcmGroup::Move: (SQL Server (MSSQLSERVER), 1, MoveType::Drain)
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] rcm::RcmGroup::Move: Bringing group 'SQL Server (MSSQLSERVER)' offline first...
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] Res DB Backups (I): Online -> WaitingToGoOffline( StateUnknown )
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] TransitionToState(DB Backups (I)) Online-->WaitingToGoOffline.
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (SQL Server (MSSQLSERVER), Failed --> Pending)
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] Res DB Backups (I): WaitingToGoOffline -> OfflineCallIssued( StateUnknown )
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] TransitionToState(DB Backups (I)) WaitingToGoOffline-->OfflineCallIssued.
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] Res DB Temp (H): Online -> WaitingToGoOffline( StateUnknown )
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] TransitionToState(DB Temp (H)) Online-->WaitingToGoOffline.
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] Res DB Temp (H): WaitingToGoOffline -> OfflineCallIssued( StateUnknown )
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] TransitionToState(DB Temp (H)) WaitingToGoOffline-->OfflineCallIssued.
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] Res MMFO_SHPT19DB12: Online -> WaitingToGoOffline( StateUnknown )
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] TransitionToState(MMFO_SHPT19DB12) Online-->WaitingToGoOffline.
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] Bringing dependent resource 'DB Logs (G)' offline before provider resource 'MMFO_SHPT19DB12'.
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] Res DB Logs (G): Online -> WaitingToGoOffline( StateUnknown )
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] TransitionToState(DB Logs (G)) Online-->WaitingToGoOffline.
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] Bringing dependent resource 'DB Data (F)' offline before provider resource 'MMFO_SHPT19DB12'.
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] Res DB Data (F): Online -> WaitingToGoOffline( StateUnknown )
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] TransitionToState(DB Data (F)) Online-->WaitingToGoOffline.
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] Res DB Data (F): WaitingToGoOffline -> OfflineCallIssued( StateUnknown )
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] TransitionToState(DB Data (F)) WaitingToGoOffline-->OfflineCallIssued.
000023fc.00002804::2019/12/15-04:18:39.788 INFO  [RES] Physical Disk <DB Backups (I)>: Offline request.
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] Res DB Logs (G): WaitingToGoOffline -> OfflineCallIssued( StateUnknown )
00000e9c.000034f0::2019/12/15-04:18:39.788 INFO  [RCM] TransitionToState(DB Logs (G)) WaitingToGoOffline-->OfflineCallIssued.
00000e9c.000034f0::2019/12/15-04:18:39.789 INFO  [RCM] 'MMFO_SHPT19DB12' cannot go offline yet; Dependent Resource 'DB Temp (H)' is in state OfflineCallIssued.
00000e9c.000034f0::2019/12/15-04:18:39.789 INFO  [RCM] Res MMFO_SHPT19DB12: WaitingToGoOffline -> WaitingToGoOffline( StateUnknown )
000023fc.00004494::2019/12/15-04:18:39.789 INFO  [RES] Physical Disk <DB Temp (H)>: Offline request.
00000e9c.000034f0::2019/12/15-04:18:39.789 INFO  [RCM] Res SQL Network Name (SHPT19DB12): Online -> WaitingToGoOffline( StateUnknown )
00000e9c.000034f0::2019/12/15-04:18:39.789 INFO  [RCM] TransitionToState(SQL Network Name (SHPT19DB12)) Online-->WaitingToGoOffline.
00000e9c.000034f0::2019/12/15-04:18:39.789 INFO  [RCM] Res SQL Network Name (SHPT19DB12): WaitingToGoOffline -> OfflineCallIssued( StateUnknown )
00000e9c.000034f0::2019/12/15-04:18:39.789 INFO  [RCM] TransitionToState(SQL Network Name (SHPT19DB12)) 

Could you let me know why the automatic failover is not happened from node 2 - node 1 @ 7:38 PM 12/14/2019

Let me know if you need any details required.

Thanking you 

swathi

↧
Search
Viewing all articles
Browse latest Browse all 2306

Trending Articles

Practice Sheet of Right form of verbs for HSC Students

September 22, 2019, 11:40 pm

Download: FK ft Shenky – Nakuyewa ”Prod by: Shenky”

February 16, 2017, 4:24 pm

How to win at Markstrat (Markstrat Tips and Tricks) – Vodites

January 5, 2014, 10:34 pm

Ominde Commission Report and Recommendations – Ominde Report of 1964

March 16, 2015, 5:14 am

Bureau of Internal Revenue: Regional Offices (Directory)

January 9, 2014, 11:06 pm

GO 53 on Enhancement of Ex-gratia upto 5 Lakhs Toddy Tappers in Telangana

March 26, 2017, 11:23 pm

Cakewalk CA-2A Leveling Amplifier v2.0.1.97 WiN, v2.0.1.96 OSX Incl Keygen

October 17, 2016, 7:20 am

Mp3 Download: Mdu - Kunjenjenjena

December 7, 2017, 8:16 am

How the kill the job , when DTP request running for long hours.

July 26, 2013, 2:41 am

Microsoft Intune から展開しているアプリのアップデートについて

October 17, 2016, 4:11 am

18-year-old girl was beaten for half an hour by two Northampton men in 'an...

September 1, 2017, 10:00 pm

Car crash in Dunton Bassett leaves driver in critical condition

October 7, 2014, 7:51 am

Macky 2, Two Others In Road Accident

March 29, 2015, 5:34 am

Application log 00000000000000089514: Could not convert queue DLVST90CLNT

May 14, 2015, 11:27 pm

Detroit mafia: D’Anna Brothers agree to plea deal

April 21, 2016, 6:56 am

Delivery block field greyed out using VA02

January 26, 2016, 2:52 pm

Muloraki Au

June 22, 2016, 1:44 am

【個人撮影】スマホのプライベート映像♪「中に出さないで///」カラオケ屋での生ハメ撮りが流出ｗ【リベンジポルノ】＠PornHub

October 12, 2017, 2:23 pm

BREAKING NEWS: Diamond Platnumz Is Reported Dead After Ghastly Car Accident

February 9, 2018, 4:56 am

FIAT 500 B0111 B0112

July 5, 2018, 10:31 am
Search
<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>