{"id":135505,"date":"2020-07-15T11:52:16","date_gmt":"2020-07-15T03:52:16","guid":{"rendered":"http:\/\/4563.org\/?p=135505"},"modified":"2020-07-15T11:52:16","modified_gmt":"2020-07-15T03:52:16","slug":"github-proxypool-%e7%88%ac%e8%99%ab%e4%bb%a3%e7%90%86-ip-%e6%b1%a0","status":"publish","type":"post","link":"http:\/\/4563.org\/?p=135505","title":{"rendered":"GitHub: ProxyPool \u722c\u866b\u4ee3\u7406 IP \u6c60"},"content":{"rendered":"<div>\n<div>\n<div>\n<h1>                  GitHub: ProxyPool \u722c\u866b\u4ee3\u7406 IP \u6c60               <\/h1>\n<p> <\/p>\n<div>\n<div> <span>\u8cc7\u6df1\u5927\u4f6c : zhijieju <\/span>  <span><i><\/i> 9<\/span> <\/div>\n<div> <\/div>\n<\/p><\/div>\n<\/p><\/div>\n<\/p><\/div>\n<div isfirst=\"1\"> <\/p>\n<p>hello\uff0c\u5c0f\u4f19\u4f34\u4eec\uff0c\u5927\u5bb6\u597d\uff0c\u4eca\u5929\u7ed9\u5927\u5bb6\u5206\u4eab\u7684\u5f00\u6e90\u9879\u76ee\u662f:<code>proxy_pool<\/code>\uff0c\u8fd9\u4e2a\u5f00\u6e90\u9879\u76ee\u662f\u6293\u53d6\u4e2a\u5927\u514d\u8d39\u4ee3\u7406 IP \u7f51\u7ad9\uff0c\u611f\u5174\u8da3\u7684\u722c\u866b\u53ef\u4ee5\u5c1d\u8bd5\u628a\u4ee3\u7801<code>clone<\/code>\u4e0b\u8f7d\u7136\u540e\u5c1d\u8bd5\u5e94\u7528\u4e00\u4e0b\uff0c\u52a0\u6cb9\uff01\uff01\uff01<\/p>\n<pre><code>______                        ______             _ | ___ _                      | ___            | | | |_\/ \/ __ __   __  _ __   _ | |_\/ \/___   ___  | | |  __\/|  _\/\/ _   \/ \/| | | ||  __\/\/ _  \/ _  | | | |   | | | (_) | &gt;  &lt;  |_| || |  | (_) | (_) || |___ _|   |_|  ___\/ \/_\/_ __  |_|   ___\/ ___\/ _____                        __ \/ \/                       \/___ \/ <\/code><\/pre>\n<h3>\u514d\u8d39\u4ee3\u7406\u6e90<\/h3>\n<p>\u76ee\u524d\u5b9e\u73b0\u7684\u91c7\u96c6\u514d\u8d39\u4ee3\u7406\u7f51\u7ad9\u6709(\u6392\u540d\u4e0d\u5206\u5148\u540e, \u4e0b\u9762\u4ec5\u662f\u5bf9\u5176\u53d1\u5e03\u7684\u514d\u8d39\u4ee3\u7406\u60c5\u51b5, \u4ed8\u8d39\u4ee3\u7406\u6d4b\u8bc4\u53ef\u4ee5\u53c2\u8003\u8fd9\u91cc):<\/p>\n<p><img decoding=\"async\" src=\"http:\/\/4563.org\/wp-content\/uploads\/2020\/07\/20200721_5f174796ac151.jpg\" alt=\"GitHub: ProxyPool \u722c\u866b\u4ee3\u7406 IP \u6c60\" \/><\/p>\n<p>\u5982\u679c\u8fd8\u6709\u5176\u4ed6\u597d\u7684\u514d\u8d39\u4ee3\u7406\u7f51\u7ad9, \u53ef\u4ee5\u5728\u63d0\u4ea4\u5728<code>issues<\/code>, \u4e0b\u6b21\u66f4\u65b0\u65f6\u4f1a\u8003\u8651\u5728\u9879\u76ee\u4e2d\u652f\u6301\u3002<\/p>\n<h3>\u8fd0\u884c\u9879\u76ee<\/h3>\n<h4>\u4e0b\u8f7d\u4ee3\u7801:<\/h4>\n<pre><code>git clone [email&#160;protected]:jhao104\/proxy_pool.git <\/code><\/pre>\n<p>or<\/p>\n<pre><code>https:\/\/github.com\/jhao104\/proxy_pool\/releases \u4e0b\u8f7d\u5bf9\u5e94 zip \u6587\u4ef6 <\/code><\/pre>\n<h4>\u5b89\u88c5\u4f9d\u8d56:<\/h4>\n<pre><code>pip install -r requirements.txt <\/code><\/pre>\n<h4>\u66f4\u65b0\u914d\u7f6e:<\/h4>\n<pre><code># setting.py \u4e3a\u9879\u76ee\u914d\u7f6e\u6587\u4ef6  # \u914d\u7f6e API \u670d\u52a1  HOST = \"0.0.0.0\"               # IP PORT = 5000                    # \u76d1\u542c\u7aef\u53e3   # \u914d\u7f6e\u6570\u636e\u5e93  DB_CONN = 'redis:\/\/@127.0.0.1:8888'   # \u914d\u7f6e ProxyFetcher  PROXY_FETCHER = [     \"freeProxy01\",      # \u8fd9\u91cc\u662f\u542f\u7528\u7684\u4ee3\u7406\u6293\u53d6\u65b9\u6cd5\u540d\uff0c\u6240\u6709 fetch \u65b9\u6cd5\u4f4d\u4e8e fetcher\/proxyFetcher.py     \"freeProxy02\",     # .... ] <\/code><\/pre>\n<h4>\u542f\u52a8\u9879\u76ee:<\/h4>\n<pre><code># \u5982\u679c\u5df2\u7ecf\u5177\u5907\u8fd0\u884c\u6761\u4ef6, \u53ef\u7528\u901a\u8fc7 proxyPool.py \u542f\u52a8\u3002 # \u7a0b\u5e8f\u5206\u4e3a: schedule \u8c03\u5ea6\u7a0b\u5e8f \u548c server Api \u670d\u52a1  # \u542f\u52a8\u8c03\u5ea6\u7a0b\u5e8f python proxyPool.py schedule  # \u542f\u52a8 webApi \u670d\u52a1 python proxyPool.py server <\/code><\/pre>\n<h4>Docker \u8fd0\u884c<\/h4>\n<pre><code>docker pull jhao104\/proxy_pool  docker run --env DB_CONN=redis:\/\/:[email&#160;protected]:port\/db -p 5010:5010 jhao104\/proxy_pool:2.1.0 <\/code><\/pre>\n<h3>\u4f7f\u7528<\/h3>\n<p><img decoding=\"async\" src=\"http:\/\/4563.org\/wp-content\/uploads\/2020\/07\/20200721_5f17479c2e2cf.jpg\" alt=\"GitHub: ProxyPool \u722c\u866b\u4ee3\u7406 IP \u6c60\" \/><\/p>\n<p>\u5982\u679c\u8981\u5728\u722c\u866b\u4ee3\u7801\u4e2d\u4f7f\u7528\u7684\u8bdd\uff0c \u53ef\u4ee5\u5c06\u6b64 api \u5c01\u88c5\u6210\u51fd\u6570\u76f4\u63a5\u4f7f\u7528\uff0c\u4f8b\u5982\uff1a<\/p>\n<pre><code>import requests  def get_proxy():     return requests.get(\"http:\/\/127.0.0.1:5010\/get\/\").json()  def delete_proxy(proxy):     requests.get(\"http:\/\/127.0.0.1:5010\/delete\/?proxy={}\".format(proxy))  # your spider code  def getHtml():     # ....     retry_count = 5     proxy = get_proxy().get(\"proxy\")     while retry_count &gt; 0:         try:             html = requests.get('http:\/\/www.example.com', proxies={\"http\": \"http:\/\/{}\".format(proxy)})             # \u4f7f\u7528\u4ee3\u7406\u8bbf\u95ee             return html         except Exception:             retry_count -= 1     # \u5220\u9664\u4ee3\u7406\u6c60\u4e2d\u4ee3\u7406     delete_proxy(proxy)     return None <\/code><\/pre>\n<h3>\u6269\u5c55\u4ee3\u7406<\/h3>\n<p>\u9879\u76ee\u9ed8\u8ba4\u5305\u542b\u51e0\u4e2a\u514d\u8d39\u7684\u4ee3\u7406\u83b7\u53d6\u6e90\uff0c\u4f46\u662f\u514d\u8d39\u7684\u6bd5\u7adf\u8d28\u91cf\u6709\u9650\uff0c\u6240\u4ee5\u5982\u679c\u76f4\u63a5\u8fd0\u884c\u53ef\u80fd\u62ff\u5230\u7684\u4ee3\u7406\u8d28\u91cf\u4e0d\u7406\u60f3\u3002\u6240\u4ee5\uff0c\u63d0\u4f9b\u4e86\u4ee3\u7406\u83b7\u53d6\u7684\u6269\u5c55\u65b9\u6cd5\u3002<\/p>\n<p>\u6dfb\u52a0\u4e00\u4e2a\u65b0\u7684\u4ee3\u7406\u6e90\u65b9\u6cd5\u5982\u4e0b: 1 \u3001\u9996\u5148\u5728 ProxyFetcher \u7c7b\u4e2d\u6dfb\u52a0\u81ea\u5b9a\u4e49\u7684\u83b7\u53d6\u4ee3\u7406\u7684\u9759\u6001\u65b9\u6cd5\uff0c \u8be5\u65b9\u6cd5\u9700\u8981\u4ee5\u751f\u6210\u5668(yield)\u5f62\u5f0f\u8fd4\u56de host:ip \u683c\u5f0f\u7684\u4ee3\u7406\uff0c\u4f8b\u5982:<\/p>\n<pre><code>class ProxyFetcher(object):     # ....      # \u81ea\u5b9a\u4e49\u4ee3\u7406\u6e90\u83b7\u53d6\u65b9\u6cd5     @staticmethod     def freeProxyCustom1():  # \u547d\u540d\u4e0d\u548c\u5df2\u6709\u91cd\u590d\u5373\u53ef          # \u901a\u8fc7\u67d0\u7f51\u7ad9\u6216\u8005\u67d0\u63a5\u53e3\u6216\u67d0\u6570\u636e\u5e93\u83b7\u53d6\u4ee3\u7406         # \u5047\u8bbe\u4f60\u5df2\u7ecf\u62ff\u5230\u4e86\u4e00\u4e2a\u4ee3\u7406\u5217\u8868         proxies = [\"x.x.x.x:3128\", \"x.x.x.x:80\"]         for proxy in proxies:             yield proxy         # \u786e\u4fdd\u6bcf\u4e2a proxy \u90fd\u662f host:ip \u6b63\u786e\u7684\u683c\u5f0f\u8fd4\u56de <\/code><\/pre>\n<p>2 \u3001\u6dfb\u52a0\u597d\u65b9\u6cd5\u540e\uff0c\u4fee\u6539 setting.py \u6587\u4ef6\u4e2d\u7684 PROXY_FETCHER \u9879\uff1a \u3000\u5728 PROXY_FETCHER \u4e0b\u6dfb\u52a0\u81ea\u5b9a\u4e49\u65b9\u6cd5\u7684\u540d\u5b57:<\/p>\n<pre><code>PROXY_FETCHER = [     \"freeProxy01\",         \"freeProxy02\",     # ....     \"freeProxyCustom1\"  #  # \u786e\u4fdd\u540d\u5b57\u548c\u4f60\u6dfb\u52a0\u65b9\u6cd5\u540d\u5b57\u4e00\u81f4 ] <\/code><\/pre>\n<p>schedule \u8fdb\u7a0b\u4f1a\u6bcf\u9694\u4e00\u6bb5\u65f6\u95f4\u6293\u53d6\u4e00\u6b21\u4ee3\u7406\uff0c\u4e0b\u6b21\u6293\u53d6\u65f6\u4f1a\u81ea\u52a8\u8bc6\u522b\u8c03\u7528\u4f60\u5b9a\u4e49\u7684\u65b9\u6cd5\u3002<\/p>\n<p>\u4ee5\u4e0a\u5c31\u662f\u8fd9\u4e2a\u5f00\u6e90\u9879\u76ee\u7684\u8be6\u7ec6\u4ecb\u7ecd\u4e86\uff0c\u611f\u5174\u8da3\u7684\u5c0f\u4f19\u4f34\u53ef\u4ee5\u5c1d\u8bd5\u5e94\u7528\u4e00\u4e0b\u3002<\/p>\n<p><strong>\u5f00\u6e90\u5730\u5740\uff1a https:\/\/github.com\/jhao104\/proxy_pool<\/strong><\/p>\n<p>\u4eca\u5929\u7684\u63a8\u8350\u4e0d\u77e5\u9053\u5927\u5bb6\u559c\u6b22\u5417\uff1f\u5982\u679c\u4f60\u4eec\u559c\u6b22\u8bdd\uff0c\u8bf7\u5728\u6587\u7ae0\u5e95\u90e8\u7559\u8a00\u6216\u70b9\u8d5e\uff0c\u4ee5\u8868\u793a\u5bf9\u6211\u7684\u652f\u6301\uff0c\u4f60\u4eec\u7684\u7559\u8a00\uff0c\u70b9\u8d5e\uff0c\u8f6c\u53d1\u5173\u6ce8\u662f\u6211\u6301\u7eed\u66f4\u65b0\u7684\u52a8\u529b\u54e6\uff01<\/p>\n<p>\u5173\u6ce8\u516c\u4f17\u53f7\u56de\u590d\uff1a&#8221;<code>1024<\/code>&#8220;\uff0c\u514d\u8d39\u9886\u53d6\u4e00\u5927\u6ce2\u5b66\u4e60\u8d44\u6e90\uff0c\u5148\u5230\u5148\u5f97\u54e6\uff01<\/p>\n<p> <img decoding=\"async\" src=\"http:\/\/4563.org\/wp-content\/uploads\/2020\/07\/20200721_5f1747a0c157b.jpg\" alt=\"GitHub: ProxyPool \u722c\u866b\u4ee3\u7406 IP \u6c60\" \/>  <\/div>\n<div> <b>\u5927\u4f6c\u6709\u8a71\u8aaa<\/b> (<span>0<\/span>)        <\/div>\n<div> <\/div>\n<\/p><\/div>\n<\/p><\/div>\n<ul>\n<li>\n","protected":false},"excerpt":{"rendered":"<p>GitHub: ProxyPool&hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[],"tags":[],"_links":{"self":[{"href":"http:\/\/4563.org\/index.php?rest_route=\/wp\/v2\/posts\/135505"}],"collection":[{"href":"http:\/\/4563.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/4563.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/4563.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/4563.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=135505"}],"version-history":[{"count":0,"href":"http:\/\/4563.org\/index.php?rest_route=\/wp\/v2\/posts\/135505\/revisions"}],"wp:attachment":[{"href":"http:\/\/4563.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=135505"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/4563.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=135505"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/4563.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=135505"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}