进行在对hbase的一个表进行统计分析的时候， 平常半个小时报完的程序， 今天跑了一个小时，还是没有完成， 1、查看spark任务， 发现大量报错， 但是主要是如下错误
# 由于是spark读取hbase数据， 怀疑hbase有问题
Cannot receive any reply from null in 120 seconds. This timeout is controlled by spark.rpc.askTimeout
# 但是下面这个问题不是为什么， 资源分配都非常充足， 缓存数据丢失
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 2
2019-06-11 21:38:42,314 INFO [RS_LOG_REPLAY_OPS-regionserver/hadoop01:16020-0] wal.WALSplitter: Processed 0 edits across 0 regions; edits skipped=0; log file=hdfs://hadoop01:9000/hbase/WALs/hadoop02,16020,1560241726381-splitting/hadoop02%2C16020%2C1560241726381.1560241733565, length=8, corrupted=false, progress failed=false
2019-06-11 21:38:42,314 WARN [RS_LOG_REPLAY_OPS-regionserver/hadoop01:16020-0] regionserver.SplitLogWorker: log splitting of WALs/hadoop02,16020,1560241726381-splitting/hadoop02%2C16020%2C1560241726381.1560241733565 failed, **returning error**
org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either that the input has been truncated or that an embedded message misreported its own length.
at org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:86)
at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$StreamDecoder.refillBuffer(CodedInputStream.java:2771)
at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$StreamDecoder.readRawLittleEndian64(CodedInputStream.java:2667)
at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$StreamDecoder.readFixed64(CodedInputStream.java:2272)
at org.apache.hbase.thirdparty.com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:534)
at org.apache.hbase.thirdparty.com.google.protobuf.GeneratedMessageV3.parseUnknownField(GeneratedMessageV3.java:293)
at org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos$WALHeader.(WALProtos.java:220)
at org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos$WALHeader.(WALProtos.java:178)
at org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos$WALHeader$1.parsePartialFrom(WALProtos.java:1086)
at org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos$WALHeader$1.parsePartialFrom(WALProtos.java:1081)
at org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos$WALHeader$Builder.mergeFrom(WALProtos.java:793)
at org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos$WALHeader$Builder.mergeFrom(WALProtos.java:630)
at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:247)
at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:602)
at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:326)
at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:316)
at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:616)
at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:322)
at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:608)
at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readHeader(ProtobufLogReader.java:192)
at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:218)
at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:173)
at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:65)
at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:168)
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:321)
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:303)
at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:806)
at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:733)
at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:256)
at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:195)
at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:100)
at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
