爬虫基础-塔卡库urllib和urllib30301

1、Python爬虫基础课教室讲师：塔卡 时间：2018年4月教学监督热线 教学监督热线 01了解fiddler抓包工具的基本使用知识点回顾本节知识点01. 掌握urllib模块02. 了解urllib3模块教学监督热线 urllib知识点一教学监督热线 urllib库中的4大模块01020304urllib.requests 请求模块urllib.parse 解析模块urllib.error 异常处理模块urllib.robotparse robots.txt解析模块urllib.request内容urlopen基本请求添加header头信息使用cookie设置代理发出请求: urllib.re

2、quest.urlopenfrom urllib import requesturl = rs = request.urlopen(url)print(rs)# print(rs.read()print(rs.read().decode(utf-8) # 读取网页源代码print(rs.getcode() # 状态码print(rs.headers) # 和info一样的,返回的头信息print()print(rs.geturl() # 返回请求的url例:使用Request，添加heardersfrom urllib import requesturl = headers = User-Ag

3、ent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36,req = request.Request(url,headers=headers) #添加headers头信息rs1 = request.urlopen(req)print(rs1)例:from urllib import parsex = parse.quote(潭州) # url 编码print(x)y = parse.unquote(x) # ur

4、l解码print(y)url3 = .format(x)rs = request.urlopen(url3)urllib.parse.quote/urllib.parse.unquote例:d1 = a:1,b:2x = parse.urlencode(d1) # 转成字符串print(x)url3 = y = /?wd=pythonz = parse.urljoin(url3,y)print(z)urllib.parse.urlencode/urllib.parse.urljoin例:urllib.parse需掌握的方法:urllib.parse.quote() # url 中文编码urll

5、ib.parse.unquote() # url中文解码urllib.parse.urlencode() # 转成&拼接的字符串urllib.parse.urljoin() # url拼接get/post方法请求urllib库，默认是get方式提交，urlopen的data参数默认为None，当data参数不为空的时候，urlopen提交方式为Post（request的get_method）,如果要使用其他方式必须Request(method=options)或req.get_method = lambda : options。urllib.request.Request(url, data=

6、None, headers=, method=None)get方法请求data = urllib.parse.urlencode(data) #对data进行转码，转化成str类型new_url = url+?+data # URL拼接result = urllib.request.urlopen(new_url) response = result.read() print(response.decode(utf8) url = 请求的地址/URL” data = 提交的数据post方法请求data = urllib.parse.urlencode(data) # 对data进行转码，转化成

7、str类型data = data.encode(utf-8) #进行编码,post请求只支持byte类型new_url = urllib.request.Request(url,data) #对url和参数进行包装response = urllib.request.urlopen(new_url) response = result.read() print(response.decode(utf8) url = 请求的地址/URL data = 提交的数据cookie的使用from http import cookiejarurl = # 创建cookiejar对象cookie = cook

8、iejar.CookieJar()# 使用HTTPCookieProcessor创建cookie处理器，并以它为参数构建opener对象opener = request.build_opener(request.HTTPCookieProcessor(cookie)# 把opner变成urlopenrequest.install_opener(opener)rs = request.urlopen(url)print(cookie)例:设置代理:url5 = # 测试ip地址的网站proxy = http:ip:port,https:ip:port# 创建代理处理器proxies = requ

9、est.ProxyHandler(proxy)# 创建opener对象opener = request.build_opener(proxies,request.HTTPHandler )request.install_opener(opener)data = request.urlopen(url5)print(data.read().decode()例:# error异常处理 URLError是HTTPError的父类 只有HTTPError有code属性url1 = url2 = try: request.urlopen(url1)except error.URLError as e:

10、if hasattr(e,code): print(HTTPError) elif hasattr(e,reason): print(URLError)error异常处理:例:urllib3知识点二教学监督热线 PoolManager:# PoolManager实例来生成请求,由该实例对象处理与线程池的连接以及线程安全的所有细节，不需要任何人为操作：http = urllib3.PoolManager()# 通过request()方法创建一个请求resp = http.request(get,url)html = resp.dataprint(html.decode()HTTPConnecti

11、onPool:url = http = urllib3.HTTPConnectionPool( )resp = http.request(get,url)print(resp.data.decode(gbk)设置代理:import urllib3proxy = urllib3.ProxyManager( , headers=connection: keep-alive)r = proxy.request(GET, )print(r.status)print(r.data)操作cookie:没有找到相关的操作，只能通过headers设置的方式实现http访问库的需求总结- 正确发出http、https请求-