[筆記] Hbase ZooKeeper Client Connection Errors

 

Hbase + Zookeeper 常常會遇到一狗票連線問題，當遇到這些問題是否要處理呢？又該怎麼處理?

首先先來看最常見的問題之一： zookeeper.ClientCnxn: Session 0x0 for server null

[INFO ][2013-05-07 18:19:25,205] zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-m1:2181 sessionTimeout=180000 watcher=hconnection  
[INFO ][2013-05-07 18:19:25,206] zookeeper.ClientCnxn: Opening socket connection to server /hadoop-m1:2181  
[INFO ][2013-05-07 18:19:25,207] client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.  
[INFO ][2013-05-07 18:19:25,208] zookeeper.RecoverableZooKeeper: The identifier of this process is  
[WARN ][2013-05-07 18:19:25,209] zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect  
 java.net.ConnectException: Connection refuce 
     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)  
     at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)  
     at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)  
[WARN ][2013-05-07 18:19:25,210] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master  

發生這個問題可能有好幾種原因，首先根據Hbase 的Trobuleshooting建議：

Error like this are either due to ZooKeeper being down, or unreachable due to network issues.

 所以錯誤排除步驟建議如下：

1. 確認ZooKeeper 是否還活著，可否正確連線，建議使用以下工具：
The utility Section 12.4.1.3, “zkcli” may help investigate ZooKeeper issues.

hbase zkcli -server host:port <cmd> <args>

(最奇怪的是我使用zkcli這個工具來測試...也會遇到這個問題!? 那我就真的無解了.....)

2.確認Client連線Hbase 設定檔是否正確

   
      
        hbase.rootdir  
        hdfs://hadoop-m1:9010/hbase  
      
      
        hbase.cluster.distributed  
        true  
      
      
      
        hbase.zookeeper.quorum  
        hadoop-m1:2181,hadoop-m2:2181,hadoop-m3:2181  
      
      
      
        hbase.zookeeper.property.dataDir  
        /tmp/zookeeper  
      
      
 

再來也有可能遇到：EndOfStreamException: Unable to read additional data from client ....

2012-11-13 17:28:05,302 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x13af9131eee0000, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:722)
2012-11-13 17:28:05,308 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:54810 which had sessionid 0x13af9131eee0000

這個問題很有可能是JDK版本問題，因為剛好在JIRA 看到這個Issue - ZOOKEEPER-1582
所以建議還是乖乖使用Oracle JDK6 不要冒然使用OpenJDK 67 和Oracle JDK7


另一個很常遇到的Warning 訊息是:SecurityException: java.lang.SecurityException: 無法定位登入配置 occurred when trying to find JAAS configuration

[INFO ][2013-05-07 18:19:27,342][main-SendThread()][org.apache.zookeeper.ClientCnxn$SendThread-startConnect] Opening socket connection to server /192.168.56.101:2181
[WARN ][2013-05-07 18:19:27,342][main-SendThread(sandbox:2181)][org.apache.zookeeper.client.ZooKeeperSaslClient-] SecurityException: java.lang.SecurityException: 無法定位登入配置 occurred when trying to find JAAS configuration.
[INFO ][2013-05-07 18:19:27,343][main-SendThread(sandbox:2181)][org.apache.zookeeper.client.ZooKeeperSaslClient-] Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.

不過目前尚無解決方法....還在嘗試中...Orz..後來發驗似乎的確還Bug - ZOOKEEPER-1554