conda SSL错误 SSLError 爬虫应用总结

1.conda SSL错误 SSLError("Can’t connect to HTTPS URL because the SSL module is not available.解决办法

使用conda环境来运行爬虫程序，因为无法使用电脑自带的SSL而出现无法连接上https的协议，因为https是基于SSL进行加密的。

OpenSSL下载地址

/products/Win32OpenSSL.html

即可正确运行。

2.文件操作：几个属性，'w’是可写，'a’是追加，'wb’是二进制可写。

3.格式化字符串,f" {i} “，这个i是自己给的值。

还可以” %d " %i

还可以" {}".format(i)

4.一个文件中的json的多行数据提取，使用readlines()，在使用json.jumps()来每行转换成一个列表操作！

import requestsimport jsonimport osheaders_ = {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36",}if os.path.exists("notebook1.json"):os.remove("notebook1.json")for i in range(2):url_ = f"/j/chart/top_list?type=24&interval_id=100%3A90&action=&start={i*20}&limit=20"response_ = requests.get(url=url_, headers=headers_)context = response_.textwith open("notebook1.json", "a") as f:f.write(context + '\n')# f.write('%s' % context + '\n')data = []f = open("notebook1.json", 'r')i = 1for line in f.readlines():data = []list = json.loads(line)print(i)for film in list:data.append(film["title"])print(data)i += 1f.close()

5.conda304reponse报错

清除下载缓存：conda clean -i

forge源出现问题：conda config --remove channels https://mirrors.tuna./anaconda/cloud/conda-forge

6.urllib2

import urllib2def download(url):return urllib2.urlopen(url).read()def download2(url):return urllib2.urlopen(url).readlines()def download3(url):response = urllib2.urlopen(url)while True:line = response.readline()if len(line):print lineelse:breakprint download3("")

7.selenium搭配正则表达式(regex)抓取信息的两种方式

# coding:utf-8import seleniumimport selenium.webdriverimport reurl = "/list/220600,000000,0000,00,9,99,%2B,2,1.html?lang=c&postchannel=0000&workyear=99&cotype=99&degreefrom=99&jobterm=99&companysize=99&ord_field=0&dibiaoid=0&line=&welfare="driver = selenium.webdriver.Firefox()driver.get(url)source = driver.page_sourceres = """<div class="rt">共798条职位</div>"""# pattern = "<div class=\"rt\">([\s\S]*?)</div>"pattern = u"共(\\d+)条职位"regex = pile(pattern, re.IGNORECASE)mylist = regex.findall(source)print mylist[0]# print re.findall("(\\d+)", mylist[0])[0]driver.close()