您当前的位置: 首页 >  ar

段智华

暂无认证

  • 0浏览

    0关注

    1232博文

    0收益

  • 0浏览

    0点赞

    0打赏

    0留言

私信
关注
热门博文

第13课:Spark Streaming源码解读之Driver容错安全性

段智华 发布时间:2016-05-22 09:17:36 ,浏览量:0

第13课:Spark Streaming源码解读之Driver容错安全性

/* 王家林老师授课http://weibo.com/ilovepains  每天晚上20:00YY频道现场授课频道68917580*/

 

1、数据层面:ReceivedBlockTracker 是专门负责管理Spark Streaming运行的元数据。跟踪数据需维持状态。

2、调度层面:DStream和JobGenerator  框架调度的层面,从业务角度考虑的。 JobGenerator(作业生成,进展到什么地步了)。

 

ReceivedBlockTracker 注释写的好精彩!藏龙卧虎之地

/**  * Class that keep track of all the received blocks, and allocate them to batches  * when required. All actions taken by this class can be saved to a write ahead log  * (if a checkpoint directory has been provided), so that the state of the tracker  * (received blocks and block-to-batch allocations) can be recovered after driver failure.  *  * Note that when any instance of this class is created with a checkpoint directory,  * it will try reading events f

关注
打赏
1659361485
查看更多评论
立即登录/注册

微信扫码登录

0.0470s