Python 从零开始爬虫(二)：BeautifulSoup解析网页

嗨学编程发布时间：2019-04-20 14:53:23 ，浏览量：1

学了requests，了解了伪装技巧后，终于能爬到些比较正常的网页源码（html文档）了，但这离结果还差最后和是最重要的一步——筛选。这个过程就像在泥沙中淘金一样，没有合适的筛子，你就会把有价值的漏掉了，或者做了无用功把没用的也筛了出来。淘金者看土质，做筛子。对应到爬虫领域就是观察html，定制筛选器。

稍稍了解HTML

信息都在网页源码里，浏览器通过解析源码来加载我们所看到的东西，那我们是不是也应该学下如何看源码呢？——是的

但不要方，这不是html语法课，做爬虫的，只需了解一下html的原理和标签关系就行了，这跟认亲戚一样简单，你会看家族树的话根本不成问题。





    
      Harry Potter
      J K. Rowling 
      2005
      29.99
    




有时候会堆成一行，影响观察但不影响使用，丢到排版器排一下就好了
Harry PotterJ K. Rowling


    
        
        
        
            最近更新
            
                深拷贝和浅拷贝的区别（重点）
【Vue】走进Vue框架世界
【云服务器】项目部署—搭建网站—vue电商后台管理系统
【React介绍】 一文带你深入React
【React】React组件实例的三大属性之state，props，refs（你学废了吗）
【脚手架VueCLI】从零开始，创建一个VUE项目
【React】深入理解React组件生命周期----图文详解（含代码）
【React】DOM的Diffing算法是什么？以及DOM中key的作用----经典面试题
【React】1_使用React脚手架创建项目步骤--------详解(含项目结构说明)
【React】2_如何使用react脚手架写一个简单的页面？
            
        
        
        
            热门博客
            
                优秀的代码都是如何分层的？
Spring 最常用的 7 大类注解，史上最强整理！
别再用currentTimeMillis统计耗时了，太 Low，试试StopWatch吧！
HTTP 3.0彻底放弃TCP，TCP到底做错了什么？
为什么有些大公司技术弱爆了？
聊聊8 种架构模式，你经过几种？
同事写了一个责任链模式，bug无数...
那些让你起飞的计算机知识。。
3行代码写出8个接口，开挂了？
AI 加持实时互动｜ZegoAvatar ⾯部表情随动技术解析





        
        [ 申请 ]友情链接：
        
            ClashX教程
            绘画宝宝
            配音宝宝
        
    


    
        
            关于我们
            服务条款
            广告服务
            联系我们
            网站地图
            免责声明
            WAP
        
        技术支持：
            武汉快勤科技有限公司
            XML网站地图 
            备案号：鄂ICP备18027844号-9
            
        
    




    
        立即登录/注册
        
    
    
        
        微信扫码登录
    













	    基本
        文件
        流程
        错误
        SQL
        调试
    

		    
    
	请求信息 : 2025-05-10 23:11:10 HTTP/2.0 GET : /home/article/detail/id/226742.html
运行时间 : 0.1070s ( Load:0.0153s Init:0.0018s Exec:0.0712s Template:0.0187s )
吞吐率 : 9.35req/s
内存开销 : 1,934.84 kb
查询信息 : 18 queries 0 writes 
文件加载 : 36
缓存信息 : 5 gets 0 writes 
配置加载 : 132
会话信息 : SESSION_ID=k5qbq5tgjif9ri7q0r8jqi9oj7
    
    
        
    
	/www/wwwroot/www.chaojiit.com/index.php ( 1.33 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/ThinkPHP.php ( 4.71 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Think.class.php ( 12.32 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Storage.class.php ( 1.38 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Storage/Driver/File.class.php ( 3.56 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Mode/common.php ( 2.82 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Common/functions.php ( 51.07 KB )
/www/wwwroot/www.chaojiit.com/Application/Common/Common/function.php ( 6.52 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Hook.class.php ( 4.02 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/App.class.php ( 12.44 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Dispatcher.class.php ( 15.15 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Route.class.php ( 13.38 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Controller.class.php ( 10.95 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/View.class.php ( 7.96 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Behavior/BuildLiteBehavior.class.php ( 3.69 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Behavior/ParseTemplateBehavior.class.php ( 3.89 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Behavior/ContentReplaceBehavior.class.php ( 1.93 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Conf/convention.php ( 11.18 KB )
/www/wwwroot/www.chaojiit.com/Application/Common/Conf/config.php ( 1.81 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Lang/zh-cn.php ( 2.57 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Conf/debug.php ( 1.51 KB )
/www/wwwroot/www.chaojiit.com/Application/Home/Conf/config.php ( 0.05 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Behavior/ReadHtmlCacheBehavior.class.php ( 5.62 KB )
/www/wwwroot/www.chaojiit.com/Application/Home/Controller/ArticleController.class.php ( 6.10 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Model.class.php ( 67.27 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Db.class.php ( 5.70 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Db/Driver/Mysql.class.php ( 8.73 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Db/Driver.class.php ( 41.60 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Cache.class.php ( 3.84 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Cache/Driver/File.class.php ( 5.90 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Template.class.php ( 28.35 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Template/TagLib/Cx.class.php ( 22.62 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Template/TagLib.class.php ( 9.19 KB )
/www/wwwroot/www.chaojiit.com/Application/Runtime/Cache/Home/3c8a1a47a3534a7b1252c226abfc3928.php ( 15.70 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Behavior/WriteHtmlCacheBehavior.class.php ( 1.43 KB )
/www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Behavior/ShowPageTraceBehavior.class.php ( 5.27 KB )
    
    
        
    
	[ app_init ] --START--
Run Behavior\BuildLiteBehavior [ RunTime:0.000018s ]
[ app_init ] --END-- [ RunTime:0.000078s ]
[ app_begin ] --START--
Run Behavior\ReadHtmlCacheBehavior [ RunTime:0.000633s ]
[ app_begin ] --END-- [ RunTime:0.000679s ]
[ view_parse ] --START--
[ template_filter ] --START--
Run Behavior\ContentReplaceBehavior [ RunTime:0.000107s ]
[ template_filter ] --END-- [ RunTime:0.000168s ]
Run Behavior\ParseTemplateBehavior [ RunTime:0.014452s ]
[ view_parse ] --END-- [ RunTime:0.014509s ]
[ view_filter ] --START--
Run Behavior\WriteHtmlCacheBehavior [ RunTime:0.000401s ]
[ view_filter ] --END-- [ RunTime:0.000430s ]
[ app_end ] --START--
    
    
        
    
	[2] session_save_path(): open_basedir restriction in effect. File(/var/lib/php/session) is not within the allowed path(s): (/www/wwwroot/www.chaojiit.com/:/tmp/) /www/wwwroot/www.chaojiit.com/ThinkPHP/Common/functions.php 第 1239 行.
[8192] Array and string offset access syntax with curly braces is deprecated /www/wwwroot/www.chaojiit.com/ThinkPHP/Library/Think/Cache/Driver/File.class.php 第 59 行.
[8] Undefined variable: user /www/wwwroot/www.chaojiit.com/Application/Runtime/Cache/Home/3c8a1a47a3534a7b1252c226abfc3928.php 第 34 行.
[8] Undefined variable: user /www/wwwroot/www.chaojiit.com/Application/Runtime/Cache/Home/3c8a1a47a3534a7b1252c226abfc3928.php 第 139 行.
[8] Trying to access array offset on value of type null /www/wwwroot/www.chaojiit.com/Application/Runtime/Cache/Home/3c8a1a47a3534a7b1252c226abfc3928.php 第 139 行.
[8] Undefined variable: user /www/wwwroot/www.chaojiit.com/Application/Runtime/Cache/Home/3c8a1a47a3534a7b1252c226abfc3928.php 第 140 行.
[8] Trying to access array offset on value of type null /www/wwwroot/www.chaojiit.com/Application/Runtime/Cache/Home/3c8a1a47a3534a7b1252c226abfc3928.php 第 140 行.
[8] Undefined variable: pinglun_list /www/wwwroot/www.chaojiit.com/Application/Runtime/Cache/Home/3c8a1a47a3534a7b1252c226abfc3928.php 第 149 行.
[8] Undefined variable: top_list /www/wwwroot/www.chaojiit.com/Application/Runtime/Cache/Home/3c8a1a47a3534a7b1252c226abfc3928.php 第 184 行.
    
    
        
    
	SHOW COLUMNS FROM `configuration` [ RunTime:0.0009s ]
SELECT `value` FROM `configuration` WHERE `name` = 'site_name' LIMIT 1   [ RunTime:0.0005s ]
SHOW COLUMNS FROM `menu` [ RunTime:0.0010s ]
SELECT * FROM `menu` WHERE `fid` = 0 AND `status` = 1  [ RunTime:0.0005s ]
SELECT * FROM `menu` WHERE `fid` = 1 AND `status` = 1  [ RunTime:0.0005s ]
SELECT * FROM `menu` WHERE `fid` = 2 AND `status` = 1  [ RunTime:0.0004s ]
SELECT * FROM `menu` WHERE `fid` = 3 AND `status` = 1  [ RunTime:0.0004s ]
SELECT * FROM `menu` WHERE `fid` = 4 AND `status` = 1  [ RunTime:0.0003s ]
SHOW COLUMNS FROM `article` [ RunTime:0.0010s ]
SELECT * FROM `article` WHERE `id` = 226742 LIMIT 1   [ RunTime:0.0005s ]
SHOW COLUMNS FROM `bloger` [ RunTime:0.0010s ]
SELECT * FROM `bloger` WHERE `id` = 204 LIMIT 1   [ RunTime:0.0004s ]
SELECT COUNT(*) AS tp_count FROM `article` WHERE `bloger_id` = 204 LIMIT 1   [ RunTime:0.0008s ]
SHOW COLUMNS FROM `article_content` [ RunTime:0.0007s ]
SELECT `content` FROM `article_content` WHERE `article_id` = 226742 LIMIT 1   [ RunTime:0.0134s ]
SHOW COLUMNS FROM `article_cate` [ RunTime:0.0013s ]
SELECT `name` FROM `article_cate` WHERE `id` = 6 LIMIT 1   [ RunTime:0.0007s ]
SELECT * FROM `article` WHERE `bloger_id` = 204 ORDER BY view_count desc LIMIT 0,10   [ RunTime:0.0117s ]
    
    
        
    
	    
    
    



0.1070s