apache storm - Zookeeper: Connection request from old client will be dropped if server is in r-o mode -
storm version: 0.82
zookeeper version: 3.4.5.
we have small storm cluster (1 nimbus , 3 supervisors), using 1 zookeeper instance that's co-located storm nimbus.
infrequently start getting following errors in zookeeper logs , our storm cluster comes standstill.
2014-04-05 13:27:32,885 [myid:] - info [nioservercxn.factory:0.0.0.0/0.0.0.0:2181:nioservercnxnfact ory@197] - accepted socket connection /10.0.1.183:56121 2014-04-05 13:27:32,886 [myid:] - warn [nioservercxn.factory:0.0.0.0/0.0.0.0:2181:zookeeperserver@7 93] - connection request old client /10.0.1.183:56121; dropped if server in r-o mode 2014-04-05 13:27:32,886 [myid:] - info [nioservercxn.factory:0.0.0.0/0.0.0.0:2181:zookeeperserver@8 32] - client attempting renew session 0x1452dd02834002e @ /10.0.1.183:56121 2014-04-05 13:27:32,886 [myid:] - info [nioservercxn.factory:0.0.0.0/0.0.0.0:2181:zookeeperserver@5 95] - established session 0x1452dd02834002e negotiated timeout 40000 client /10.0.1.183:561 21
on storm end start seeing following in supervisor , worker logs:
2014-04-05 11:37:29 connectionstatemanager [warn] there no connectionstatelisteners registered. 2014-04-05 11:37:29 cluster [warn] received event :disconnected::none: disconnected zookeeper. 2014-04-05 11:37:31 clientcnxn [warn] session 0x1452dd028340015 server null, unexpected error, losing socket connection , attempting reconnect java.net.connectexception: connection refused @ sun.nio.ch.socketchannelimpl.checkconnect(native method) @ sun.nio.ch.socketchannelimpl.finishconnect(socketchannelimpl.java:739) @ org.apache.zookeeper.clientcnxn$sendthread.run(clientcnxn.java:1119) 2014-04-05 11:37:42 curatorframeworkimpl [error] background operation retry gave org.apache.zookeeper.keeperexception$connectionlossexception: keepererrorcode = connectionloss @ org.apache.zookeeper.keeperexception.create(keeperexception.java:90) @ com.netflix.curator.framework.imps.curatorframeworkimpl.processbackgroundoperation(curat rframeworkimpl.java:380) @ com.netflix.curator.framework.imps.backgroundsyncimpl$1.processresult(backgroundsyncimpl java:49) @ org.apache.zookeeper.clientcnxn$eventthread.processevent(clientcnxn.java:617) @ org.apache.zookeeper.clientcnxn$eventthread.run(clientcnxn.java:506)
do need downgrade zookeeper 3.3.3 or there known issue/config we're missing?
we experienced several issues storm 0.9 , zookeeper 3.4.x, though not 1 describe.
storm mailing list reporting such incompatibility issues:
https://mail.google.com/mail/u/0/#search/label%3astorm+zookeeper+3.4/144313a45ba069b5 https://mail.google.com/mail/u/0/#search/label%3astorm+zookeeper+3.4/1447d95d10ce7582
this later 1 pointing storm pull request, should let use zk 3.4.x future versions of storm when released:
https://github.com/apache/incubator-storm/pull/29
until then, recommend downgrading zk 3.3.6 (you may install specific separate instance of zk storm if absolutely need zk 3.4.x system). clone storm code , merge pull request locally or compile latest version of trunk, that's bit adventurous , more tiresome waiting nice folks deliver new release :)
Comments
Post a Comment