叮咚爬虫

yjsx86 大佬有话说 :

本帖最后由 yjsx86 于 2020-4-6 00:16 编辑

原帖: https://www.hostloc.com/thread-669605-1-1.html
只爬了完本的!
估计大多数人没有yaml库 pip install pyyaml
我草我刚才把我的账号密码写在代码里了, 虽然是测试用账号
import requests, yaml, os, logging

root = os.path.dirname(__file__)
logging.basicConfig(level=logging.INFO, format=’%(asctime)s – %(levelname)s – %(message)s’)
session = requests.session()

def login(u, p):
if u == "" or p == "":
   print("请填写: user pwd")
   exit(1)
url = f"https://www.meto3517.com/user/login?userName={u}&password={p}"
r = session.get(url).json()
if r["msg"] == "登錄成功":
   print("登陆成功")
else:
   print("登陆失败!")
   exit(1)

def completedBooks():
print("获取完本ID“`")
url = "https://www.meto3517.com/query/books?ranking=wjb&paged=true&size=100&page=1&type=cartoon"
r = session.get(url).json()
books = r["content"]["list"]
return books

def getBookChapters(id):
url = f"https://www.meto3517.com/query/book/directory?bookId={id}"
r = session.get(url).json()
return r["content"]

def writeYamlFile(obj, bookpath):
yamlpath = os.path.join(bookpath, "info.yaml")
with open(yamlpath, "w", encoding="utf-8") as f:
yaml.dump(obj, f, allow_unicode=True)

def getImgBinaryContent(imgurl):
r = session.get(imgurl)
return r.content

def downloadImgs(chapter, chapterpath):
url = f"https://www.meto3517.com/query/book/chapter?bookId={chapter[‘bookId’]}&chapterId={chapter[‘id’]}"
r = session.get(url).json()
imglist = r["content"]["imageList"]
for img in imglist:
   imgurl = img["url"]
   imgName = img["encodeName"].split("_")
   imgext = img["encodeName"].split(".")[-1]
   imgFullName = f"{imgName}.{imgext}"
   imgpath = os.path.join(chapterpath, imgFullName)
   with open(imgpath, "wb") as f:
         f.write(getImgBinaryContent(imgurl))
   logging.info(f’ -> {imgFullName} ok!’)

def downloadChapter(chapter, bookpath):
logging.info(f’ -> {chapter["title"]}’)
chapterpath = os.path.join(bookpath, chapter["title"])
os.mkdir(chapterpath)
downloadImgs(chapter, chapterpath)

def downloadBook(id):
url = f"https://www.meto3517.com/query/book?id={id}"
r = session.get(url).json()
logging.info(f’下载 -> {r["content"]["name"]}’)
obj = {}
obj["name"] = r["content"]["name"]
obj["author"] = r["content"]["author"]
obj["description"] = r["content"]["description"]
obj["tags"] = r["content"]["tags"]
bookpath = os.path.join(root, obj["name"])
os.mkdir(bookpath)
writeYamlFile(obj, bookpath)
chapters = getBookChapters(id)
for chapter in chapters:
downloadChapter(chapter, bookpath)

if __name__ == ‘__main__’:
# 用户名密码
user = ""
pwd = ""
login(user, pwd)
books = completedBooks()
for book in books:
downloadBook(book["id"])

flofire 大佬有话说 :

本帖最后由 flofire 于 2020-4-6 01:41 编辑

php多线程版、失败自动重试，看readme后运行。
希望帮到有需要的人

燕十三丶 大佬有话说 :

感谢分享拿走了

b66667777 大佬有话说 :

帮顶一下

jmmes 大佬有话说 :

等1健转存:lol

yjsx86 大佬有话说 :

燕十三丶大佬有话说 : 2020-4-5 21:23
感谢分享拿走了

w cao 密码我都写上了

ficefox 大佬有话说 :

大佬你太快了

egihosting99 大佬有话说 :

带走

Yrh 大佬有话说 :

拿走拿走，感谢大佬，今晚不睡了:lol:lol:lol

yjsx86 大佬有话说 :

ficefox 大佬有话说 : 2020-4-5 21:25
大佬你太快了

主要是代码一遍过, 基本没修改

lunatic 大佬有话说 :

大佬牛批，PY赛高