cloud - Cloudstack:KVM:: Snapshot stuck in Creating Status forever.‏ -

i working on cloudstack infrastructure service build our organization framework. purpose, rhel 6.4 (kernel version 2.6.32 ), cloudstack 4.2.1 , kvm (version 0.12.1) being used.

i have been working around long figure out below mentioned issue unfortunately not.

issue: snapshot gets stuck in state "creating" forever.

impact: couldn't snapshot consequently , thus, instance cloudstack's ui not started/stopped.

please see below trace management server log think reason behind this.

me":"root-13","path":"ec5692cb-7758-4618-b589-09cb71e45dba","size":21474836480,"type":"root","storagepooltype":"networkfilesystem","storagepooluuid":"d41d4103-a11d-3160-8a3b-6469c574a93f","deviceid":0},{"id":19,"name":"data-13","path":"8017ac65-b52b-450b-937f-d92178de2f34","size":21474836480,"type":"datadisk","storagepooltype":"networkfilesystem","storagepooluuid":"d41d4103-a11d-3160-8a3b-6469c574a93f","deviceid":1}],"target":{"id":13,"snapshotname":"i-2-13-vm_vs_20140225120244","type":"disk","current":false,"description":"new"},"vmname":"i-2-13-vm","guestostype":"red hat enterprise linux 6.4 (64-bit)","wait":1800}}] }  2014-02-25 17:32:45,254 debug [agent.transport.request] (agentmanager-handler-11:null) seq 1-2016739514: processing: { ans: , mgmtid: 181122461670954, via: 1, ver: v1, flags: 10, [{"com.cloud.agent.api.unsupportedanswer":{"result":false,"details":"unsupported command issued:com.cloud.agent.api.createvmsnapshotcommand. sure got right type of server?","wait":0}}] }  2014-02-25 17:32:45,254 debug [agent.transport.request] (job-executor-2:job-143 = [ 8ebc7ede-829d-473b-b909-a397f4c83490 ]) seq 1-2016739514: received: { ans: , mgmtid: 181122461670954, via: 1, ver: v1, flags: 10, { unsupportedanswer } }  2014-02-25 17:32:45,254 warn [agent.manager.agentmanagerimpl] (job-executor-2:job-143 = [ 8ebc7ede-829d-473b-b909-a397f4c83490 ]) unsupported command: unsupported command issued:com.cloud.agent.api.createvmsnapshotcommand. sure got right type of server?  2014-02-25 17:32:45,254 error [vm.snapshot.vmsnapshotmanagerimpl] (job-executor-2:job-143 = [ 8ebc7ede-829d-473b-b909-a397f4c83490 ]) create vm snapshot i-2-13-vm_vs_20140225120244 failed vm: i-2-13-vm due com.cloud.agent.api.unsupportedanswer cannot cast com.cloud.agent.api.createvmsnapshotanswer  2014-02-25 17:32:45,265 debug [cloud.api.apiservlet] (catalina-exec-11:null) ===start=== 192.168.125.241 -- command=listostypes&response=json&sessionkey=bp1mluio7kmyb3krowtd1ipg0b0%3d&_=1393329472271  2014-02-25 17:32:45,324 debug [cloud.api.apiservlet] (catalina-exec-13:null) ===start=== 192.168.125.241 -- command=listtags&response=json&sessionkey=bp1mluio7kmyb3krowtd1ipg0b0%3d&resourceid=340c69ae-f3a0-4966-aa0b-c799e944404f&resourcetype=uservm&listall=true&_=1393329472305  2014-02-25 17:32:45,330 debug [cloud.api.apiservlet] (catalina-exec-13:null) ===end=== 192.168.125.241 -- command=listtags&response=json&sessionkey=bp1mluio7kmyb3krowtd1ipg0b0%3d&resourceid=340c69ae-f3a0-4966-aa0b-c799e944404f&resourcetype=uservm&listall=true&_=1393329472305  2014-02-25 17:32:45,373 debug [cloud.api.apiservlet] (catalina-exec-11:null) ===end=== 192.168.125.241 -- command=listostypes&response=json&sessionkey=bp1mluio7kmyb3krowtd1ipg0b0%3d&_=1393329472271  2014-02-25 17:32:45,419 error [cloud.async.asyncjobmanagerimpl] (job-executor-2:job-143 = [ 8ebc7ede-829d-473b-b909-a397f4c83490 ]) unexpected exception while executing org.apache.cloudstack.api.command.user.vmsnapshot.createvmsnapshotcmd  com.cloud.utils.exception.cloudruntimeexception: com.cloud.agent.api.unsupportedanswer cannot cast com.cloud.agent.api.createvmsnapshotanswer          @ com.cloud.vm.snapshot.vmsnapshotmanagerimpl.createvmsnapshotinternal(vmsnapshotmanagerimpl.java:406)          @ com.cloud.vm.snapshot.vmsnapshotmanagerimpl.creatvmsnapshot(vmsnapshotmanagerimpl.java:356)          @ com.cloud.utils.component.componentinstantiationpostprocessor$interceptordispatcher.intercept(componentinstantiationpostprocessor.java:125)          @ org.apache.cloudstack.api.command.user.vmsnapshot.createvmsnapshotcmd.execute(createvmsnapshotcmd.java:100)          @ com.cloud.api.apidispatcher.dispatch(apidispatcher.java:158)          @ com.cloud.async.asyncjobmanagerimpl$1.run(asyncjobmanagerimpl.java:531)          @ java.util.concurrent.executors$runnableadapter.call(executors.java:471)          @ java.util.concurrent.futuretask$sync.innerrun(futuretask.java:334)          @ java.util.concurrent.futuretask.run(futuretask.java:166)          @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1110)          @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:603)          @ java.lang.thread.run(thread.java:722)  2014-02-25 17:32:45,420 debug [cloud.async.asyncjobmanagerimpl] (job-executor-2:job-143 = [ 8ebc7ede-829d-473b-b909-a397f4c83490 ]) complete async job-143 = [ 8ebc7ede-829d-473b-b909-a397f4c83490 ], jobstatus: 2, resultcode: 530, result: error code: 530 error text: com.cloud.agent.api.unsupportedanswer cannot cast com.cloud.agent.api.createvmsnapshotanswer  2014-02-25 17:32:50,526 debug [storage.secondary.secondarystoragemanagerimpl] (secstorage-1:null) zone 1 ready launch secondary storage vm  2014-02-25 17:32:50,626 debug [cloud.consoleproxy.consoleproxymanagerimpl] (consoleproxy-1:null) zone 1 ready launch console proxy  2014-02-25 17:32:50,760 debug [agent.manager.agentmanagerimpl] (agentmanager-handler-14:null) seqa 3-1596: processing seq 3-1596: { cmd , mgmtid: -1, via: 3, ver: v1, flags: 11, [{"com.cloud.agent.api.consoleproxyloadreportcommand":{"_proxyvmid":2,"_loadinfo":"{\n \"connections\": []\n}","wait":0}}] }  2014-02-25 17:32:50,835 debug [agent.manager.agentmanagerimpl] (agentmanager-handler-14:null) seqa 3-1596: sending seq 3-1596: { ans: , mgmtid: 181122461670954, via: 3, ver: v1, flags: 100010, [{"com.cloud.agent.api.agentcontrolanswer":{"result":true,"wait":0}}] }  2014-02-25 17:32:51,461 debug [network.router.virtualnetworkappliancemanagerimpl] (routerstatusmonitor-1:null) found 0 routers update status.  2014-02-25 17:32:51,462 debug [network.router.virtualnetworkappliancemanagerimpl] (routerstatusmonitor-1:null) found 0 networks update rvr status.  2014-02-25 17:32:54,001 debug [agent.manager.agentmanagerimpl] (agentmanager-handler-13:null) ping 3  2014-02-25 17:32:54,082 debug [agent.manager.agentmanagerimpl] (agentmanager-handler-12:null) ping 2  2014-02-25 17:32:54,340 debug [agent.manager.agentmanagerimpl] (agentmanager-handler-15:null) ping 1  2014-02-25 17:32:55,800 debug [agent.manager.agentmanagerimpl] (agentmanager-handler-2:null) seqa 3-1598: processing seq 3-1598: { cmd , mgmtid: -1, via: 3, ver: v1, flags: 11, [{"com.cloud.agent.api.consoleproxyloadreportcommand":{"_proxyvmid":2,"_loadinfo":"{\n \"connections\": []\n}","wait":0}}] }  2014-02-25 17:32:55,945 debug [agent.manager.agentmanagerimpl] (agentmanager-handler-2:null) seqa 3-1598: sending seq 3-1598: { ans: , mgmtid: 181122461670954, via: 3, ver: v1, flags: 100010, [{"com.cloud.agent.api.agentcontrolanswer":{"result":true,"wait":0}}] }  2014-02-25 17:33:05,762 debug [agent.manager.agentmanagerimpl] (agentmanager-handler-1:null) seqa 3-1599: processing seq 3-1599: { cmd , mgmtid: -1, via: 3, ver: v1, flags: 11, [{"com.cloud.agent.api.consoleproxyloadreportcommand":{"_proxyvmid":2,"_loadinfo":"{\n \"connections\": []\n}","wait":0}}] }  2014-02-25 17:33:05,847 debug [agent.manager.agentmanagerimpl] (agentmanager-handler-1:null) seqa 3-1599: sending seq 3-1599: { ans: , mgmtid: 181122461670954, via: 3, ver: v1, flags: 100010, [{"com.cloud.agent.api.agentcontrolanswer":{"result":true,"wait":0}}] }  2014-02-25 17:33:12,091 debug [cloud.server.statscollector] (statscollector-1:null) vmstatscollector running...  2014-02-25 17:33:12,185 debug [agent.transport.request] (statscollector-1:null) seq 1-2016739515: received: { ans: , mgmtid: 181122461670954, via: 1, ver: v1, flags: 10, { getvmstatsanswer } }  2014-02-25 17:33:13,400 debug [cloud.server.statscollector] (statscollector-2:null) storagecollector running...  2014-02-25 17:33:13,460 debug [agent.transport.request] (statscollector-2:null) seq 2-1323958339: received: { ans: , mgmtid: 181122461670954, via: 2, ver: v1, flags: 10, { getstoragestatsanswer } }  2014-02-25 17:33:13,500 debug [agent.transport.request] (statscollector-2:null) seq 1-2016739516: received: { ans: , mgmtid: 181122461670954, via: 1, ver: v1, flags: 10, { getstoragestatsanswer } }  

please see below trace agent log written @ moment clicked on snapshot button in cloudstack's ui:

2014-02-25 17:27:09,129 warn [kvm.resource.libvirtcomputingresource] (agentrequest-handler-4:null) unsupported command 

in order restart instance again, manually delete row table vm_snapshots in mysql database corresponding stuck snapshot.

please suggest me fix proper explanation issue.

thanks in advance.

regards, rohit