第13课:Spark Streaming源码解读之Driver容错安全性
/* 王家林老师授课http://weibo.com/ilovepains 每天晚上20:00YY频道现场授课频道68917580*/
1、数据层面:ReceivedBlockTracker 是专门负责管理Spark Streaming运行的元数据。跟踪数据需维持状态。
2、调度层面:DStream和JobGenerator 框架调度的层面,从业务角度考虑的。 JobGenerator(作业生成,进展到什么地步了)。
ReceivedBlockTracker 注释写的好精彩!藏龙卧虎之地
/** * Class that keep track of all the received blocks, and allocate them to batches * when required. All actions taken by this class can be saved to a write ahead log * (if a checkpoint directory has been provided), so that the state of the tracker * (received blocks and block-to-batch allocations) can be recovered after driver failure. * * Note that when any instance of this class is created with a checkpoint directory, * it will try reading events f