{"id":663,"date":"2020-08-07T09:30:05","date_gmt":"2020-08-07T00:30:05","guid":{"rendered":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/blog\/2020\/08\/07\/lambda-wget"},"modified":"2020-08-07T09:30:05","modified_gmt":"2020-08-07T00:30:05","slug":"lambda-wget","status":"publish","type":"post","link":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/blog\/2020\/08\/07\/lambda-wget\/","title":{"rendered":"Lambda\u4e0a\u3067wget\u30b3\u30de\u30f3\u30c9\u3092\u5b9f\u884c\u3067\u304d\u308b\u304b\uff1f"},"content":{"rendered":"<p>\u30a8\u30f3\u30b8\u30cb\u30a2\u306e\u5185\u5c71\u3067\u3059\u3002\u4eca\u56de\u306f\u3001Lambda\u4e0a\u3067wget\u30b3\u30de\u30f3\u30c9\u3092\u5165\u308c\u308b\u3068\u304d\u306b\u8272\u3005\u3068\u624b\u9593\u53d6\u3063\u305f\u306e\u3067\u3001\u305d\u306e\u77e5\u898b\u3092\u5171\u6709\u3055\u305b\u3066\u3044\u305f\u3060\u304d\u307e\u3059\u3002<\/p>\n<h2 id=\"outline__1\" id=\"lambda\u306b\u306fwget\u30b3\u30de\u30f3\u30c9\u304c\u306a\u3044\">Lambda\u306b\u306fwget\u30b3\u30de\u30f3\u30c9\u304c\u306a\u3044<\/h2>\n<p>Lambda\u306b\u306f\u3001\u30c7\u30d5\u30a9\u30eb\u30c8\u3067wget\u30b3\u30de\u30f3\u30c9\u306f\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3055\u308c\u3066\u3044\u307e\u305b\u3093\u3002\u305d\u306e\u305f\u3081\u3001Lambda\u4e0a\u306bwget\u30d0\u30a4\u30ca\u30ea\u30d5\u30a1\u30a4\u30eb\u3092\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n<p>\u3057\u304b\u3057\u3001\u3069\u3053\u304b\u306e\u30b5\u30fc\u30d0\u306b\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3055\u308c\u3066\u3044\u308bwget\u30d0\u30a4\u30ca\u30ea\u30d5\u30a1\u30a4\u30eb\u3092\u3001\u5358\u7d14\u306b\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9\u3057\u3066\u5b9f\u884c\u3059\u308b\u3068\u3001\u4ee5\u4e0b\u306e\u3088\u3046\u306a\u30a8\u30e9\u30fc\u304c\u51fa\u3066\u3057\u307e\u3044\u307e\u3059\u3002<\/p>\n<pre><code>\/var\/task\/handlers\/bin\/wget: \nerror while loading shared libraries: \nlibgnutls.so.28: cannot open shared object file:\n No such file or directory<\/code><\/pre>\n<p>\u3053\u306e\u30a8\u30e9\u30fc\u306f\u3001wget\u304c\u4f9d\u5b58\u3057\u3066\u3044\u308bShared Object\u304c\u5b58\u5728\u3057\u306a\u3044\u305f\u3081\u306b\u8d77\u3053\u308a\u307e\u3059\u3002\u305d\u306e\u305f\u3081\u3001\u95a2\u9023\u3059\u308bShared Object\u3082Lambda\u306b\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n<p>\u4eca\u56de\u306f\u3001\u4ee5\u4e0b\u306e\u3088\u3046\u306a\u624b\u9806\u3067\u3001wget\u30b3\u30de\u30f3\u30c9\u3092Lambda\u4e0a\u3067\u3001\u5b9f\u884c\u3067\u304d\u308b\u3088\u3046\u306b\u3057\u307e\u3057\u305f\u3002<\/p>\n<ul>\n<li>Docker\u30b3\u30f3\u30c6\u30ca\u4e0a\u3067wget\u3092\u30bd\u30fc\u30b9\u304b\u3089\u30d3\u30eb\u30c9<\/li>\n<li>Docker\u30b3\u30f3\u30c6\u30ca\u4e0a\u304b\u3089wget\u30d0\u30a4\u30ca\u30ea\u30d5\u30a1\u30a4\u30eb\u3092\u30b3\u30d4\u30fc\u3057\u3066\u3001Lambda\u306b\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9<\/li>\n<li>Docker\u30b3\u30f3\u30c6\u30ca\u4e0a\u304b\u3089wget\u304c\u4f9d\u5b58\u3057\u3066\u3044\u308bShared Object\u3092\u30b3\u30d4\u30fc\u3057\u3066\u3001Lambda Layer\u306b\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9<\/li>\n<li>Lambda\u95a2\u6570\u306bLambda Layer\u3092\u95a2\u9023\u4ed8\u3051\u308b<\/li>\n<\/ul>\n<p>\u4ee5\u4e0b\u3001\u305d\u308c\u305e\u308c\u306e\u8a73\u7d30\u3092\u7d39\u4ecb\u3057\u3066\u3044\u304d\u307e\u3059\u3002<\/p>\n<h2 id=\"outline__2\" id=\"docker\u30b3\u30f3\u30c6\u30ca\u4e0a\u3067wget\u3092\u30bd\u30fc\u30b9\u304b\u3089\u30d3\u30eb\u30c9\">Docker\u30b3\u30f3\u30c6\u30ca\u4e0a\u3067wget\u3092\u30bd\u30fc\u30b9\u304b\u3089\u30d3\u30eb\u30c9<\/h2>\n<p>\u4ee5\u4e0b\u306eDockerfile\u3092\u4f5c\u308a\u3001wget\u3092Docker\u30b3\u30f3\u30c6\u30ca\u4e0a\u3067\u30d3\u30eb\u30c9\u3057\u307e\u3059\u3002<\/p>\n<pre><code class=\"language-Dockerfile\">FROM lambci\/lambda:build-python3.8\n\nENV VERSION wget-1.18\n\nRUN yum install -y tar gzip gcc make gnutls-devel pkgconfig\n\nRUN curl -o ${VERSION}.tar.gz ftp.gnu.org\/pub\/gnu\/wget\/${VERSION}.tar.gz &amp;&amp; \n    tar xzvf ${VERSION}.tar.gz &amp;&amp; \n    cd ${VERSION} &amp;&amp; \n    .\/configure --with-gnu-ld  --disable-pcre --disable-threads --disable-ipv6 --disable-ntlm &amp;&amp; \n    make &amp;&amp; \n    mv src\/wget \/<\/code><\/pre>\n<p>\u4eca\u56de\u306fPython\u304b\u3089wget\u30b3\u30de\u30f3\u30c9\u3092\u5b9f\u884c\u3059\u308b\u5f62\u306b\u3057\u3066\u3044\u308b\u305f\u3081\u3001Docker\u30a4\u30e1\u30fc\u30b8\u306flambda:build-python\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>wget\u3092\u30bd\u30fc\u30b9\u304b\u3089\u30d3\u30eb\u30c9\u3057\u3066\u3044\u308b\u306e\u306f\u3001\u4e0d\u8981\u306a\u6a5f\u80fd\u306f\u30aa\u30d5\u306b\u3057\u3066\u3001\u4f9d\u5b58\u3059\u308bShared Object\u3092\u6700\u5c0f\u9650\u306b\u3059\u308b\u305f\u3081\u3067\u3059\u3002<\/p>\n<p>\u30d3\u30eb\u30c9\u306f\u3001\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u5b9f\u884c\u3057\u307e\u3059\u3002<\/p>\n<pre><code class=\"language-bash\">$ docker build -t wget:1 .<\/code><\/pre>\n<h2 id=\"outline__3\" id=\"docker\u30b3\u30f3\u30c6\u30ca\u4e0a\u304b\u3089wget\u30d0\u30a4\u30ca\u30ea\u30d5\u30a1\u30a4\u30eb\u3092\u30b3\u30d4\u30fc\u3057\u3066\u3001lambda\u306b\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9\">Docker\u30b3\u30f3\u30c6\u30ca\u4e0a\u304b\u3089wget\u30d0\u30a4\u30ca\u30ea\u30d5\u30a1\u30a4\u30eb\u3092\u30b3\u30d4\u30fc\u3057\u3066\u3001Lambda\u306b\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9<\/h2>\n<p>\u30d3\u30eb\u30c9\u304c\u7d42\u4e86\u3057\u305f\u3089\u3001\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u3001Lambda\u306b\u4e0a\u3052\u308b\u7528\u306e\u30c7\u30a3\u30ec\u30af\u30c8\u30ea\u3092\u4f5c\u6210\u3057\u3001\u30db\u30b9\u30c8\u5074\u306bwget\u30d0\u30a4\u30ca\u30ea\u30d5\u30a1\u30a4\u30eb\u3092\u30b3\u30d4\u30fc\u3057\u3066\u304d\u307e\u3059\u3002<\/p>\n<pre><code class=\"language-bash\">$ mkdir handlers\/bin\n$ docker cp $(docker run -d  wget:1 bash):\/wget handlers\/bin<\/code><\/pre>\n<p>\u307e\u305f\u3001wget\u30b3\u30de\u30f3\u30c9\u3092\u5b9f\u884c\u3059\u308bPython\u30b9\u30af\u30ea\u30d7\u30c8\u3092\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u5b9f\u88c5\u3057\u307e\u3057\u305f\u3002<\/p>\n<pre><code class=\"language-handlers\/main.py\">import subprocess\nimport tempfile\nimport os\n\nbasedir=os.path.dirname(__file__)\n\ndef lambda_handler(event, context):\n    with tempfile.TemporaryDirectory() as dir_name:\n        wget(dir_name, &#39;https:\/\/example.com\/&#39;)\n\n\ndef wget(dir_name, url: str) -&gt; str:\n    cmd = &#39;{}\/bin\/wget -k  -r 1 -P {} {}&#39;.format(basedir, dir_name, url)\n\n    result = subprocess.run(\n        cmd.split(&quot; &quot;),\n        stdout=subprocess.PIPE,\n        stderr=subprocess.STDOUT\n    )\n    print(result.stdout.decode())\n\n    return dir_name<\/code><\/pre>\n<pre><code class=\"language-bash\">$ zip -r main.zip handlers<\/code><\/pre>\n<p>main.zip\u306f\u3001Lambda\u306b\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9\u3057\u307e\u3059\u3002<\/p>\n<h2 id=\"outline__4\" id=\"docker\u30b3\u30f3\u30c6\u30ca\u4e0a\u304b\u3089wget\u304c\u4f9d\u5b58\u3057\u3066\u3044\u308bshared-object\u3092\u30b3\u30d4\u30fc\u3057\u3066\u3001lambda-layer\u306b\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9\">Docker\u30b3\u30f3\u30c6\u30ca\u4e0a\u304b\u3089wget\u304c\u4f9d\u5b58\u3057\u3066\u3044\u308bShared Object\u3092\u30b3\u30d4\u30fc\u3057\u3066\u3001Lambda Layer\u306b\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9<\/h2>\n<p>\u4eca\u56de\u306f\u4ee5\u4e0b\u306eShared Object\u304c\u5fc5\u8981\u3067\u3057\u305f\u3002<\/p>\n<ul>\n<li>\/lib64\/libgnutls.so.28 <\/li>\n<li>\/lib64\/libnettle.so.4<\/li>\n<li>\/lib64\/libhogweed.so.2 <\/li>\n<li>\/lib64\/libgmp.so.10 <\/li>\n<\/ul>\n<p>\u3053\u308c\u3089\u3059\u3079\u3066\u304c\u30b7\u30f3\u30dc\u30ea\u30c3\u30af\u30ea\u30f3\u30af\u306a\u306e\u3067\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044\u3002<\/p>\n<p>Lambda Layer\u7528\u306e\u30c7\u30a3\u30ec\u30af\u30c8\u30ea\u3092\u4f5c\u6210\u3057\u3001\u5fc5\u8981\u306aShared Object\u3092Docker\u30b3\u30f3\u30c6\u30ca\u304b\u3089\u30b3\u30d4\u30fc\u3057\u3066\u304d\u307e\u3059\u3002<\/p>\n<pre><code class=\"language-bash\">$ mkdir lib\n$ docker cp $(docker run -d  wget:1 bash):\/lib64\/libgnutls.so.28.43.3 lib\/libgnutls.so.28\n$ docker cp $(docker run -d  wget:1 bash):\/lib64\/libnettle.so.4.7 lib\/libnettle.so.4\n$ docker cp $(docker run -d  wget:1 bash):\/lib64\/libhogweed.so.2.5 lib\/libhogweed.so.2\n$ docker cp $(docker run -d  wget:1 bash):\/lib64\/libgmp.so.10.2.0 lib\/libgmp.so.10<\/code><\/pre>\n<p>Lambda\u306e\u5b9f\u884c\u74b0\u5883\u306b\u3088\u3063\u3066\u306f\u3001\u5fc5\u8981\u306b\u306a\u3063\u3066\u304f\u308bShared Object\u304c\u9055\u3046\u304b\u3082\u3057\u308c\u306a\u3044\u306e\u3067\u3001\u9069\u5b9c\u5fc5\u8981\u306a\u3082\u306e\u3092\u30b3\u30d4\u30fc\u3057\u3066\u304f\u3060\u3055\u3044\u3002<\/p>\n<p>\u30b3\u30d4\u30fc\u3057\u305f\u3082\u306e\u306f\u3001zip\u3067\u56fa\u3081\u307e\u3059\u3002<\/p>\n<pre><code class=\"language-bash\">$ zip layer.zip lib\/*<\/code><\/pre>\n<p>layer.zip\u3092\u30ec\u30a4\u30e4\u30fc\u306b\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9\u3057\u307e\u3059\u3002\u4eca\u56de\u306f\u3001Python3.8\u30e9\u30f3\u30bf\u30a4\u30e0\u3067\u5b9f\u884c\u3059\u308b\u60f3\u5b9a\u3067\u3059\u3002<\/p>\n<p><img src=\"\/images\/uchiko\/lambda-wget\/layer.png\" alt=\"\"><\/p>\n<h2 id=\"outline__5\" id=\"lambda\u95a2\u6570\u306blambda-layer\u3092\u95a2\u9023\u4ed8\u3051\u308b\">Lambda\u95a2\u6570\u306bLambda Layer\u3092\u95a2\u9023\u4ed8\u3051\u308b<\/h2>\n<p>\u4ee5\u4e0b\u306e\u753b\u9762\u3067\u3001\u30ec\u30a4\u30e4\u30fc\u3092Lambda\u95a2\u6570\u306b\u95a2\u9023\u4ed8\u3051\u307e\u3059\u3002<\/p>\n<p><img src=\"\/images\/uchiko\/lambda-wget\/lambda.png\" alt=\"\"><\/p>\n<h2 id=\"outline__6\" id=\"lambda\u3092\u5b9f\u884c\u3057\u3066\u3001wget\u30b3\u30de\u30f3\u30c9\u304c\u5b9f\u884c\u3067\u304d\u3066\u3044\u308b\u3053\u3068\u3092\u78ba\u8a8d\">Lambda\u3092\u5b9f\u884c\u3057\u3066\u3001wget\u30b3\u30de\u30f3\u30c9\u304c\u5b9f\u884c\u3067\u304d\u3066\u3044\u308b\u3053\u3068\u3092\u78ba\u8a8d<\/h2>\n<p>Lambda\u3092\u8a66\u3057\u306b\u5b9f\u884c\u3059\u308b\u3068\u3001\u4ee5\u4e0b\u306e\u3088\u3046\u306a\u30e1\u30c3\u30bb\u30fc\u30b8\u304c\u51fa\u529b\u3055\u308c\u3001wget\u30b3\u30de\u30f3\u30c9\u3092\u5b9f\u884c\u3067\u304d\u3066\u3044\u308b\u3053\u3068\u3092\u78ba\u8a8d\u3067\u304d\u308b\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n<pre><code class=\"language-bash\">--2020-08-07 04:56:56--  https:\/\/example.com\/\nResolving example.com... 93.184.216.34\nConnecting to example.com|93.184.216.34|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 1256 (1.2K) [text\/html]\nSaving to: \u2018\/tmp\/tmpnpvavroa\/example.com\/index.html\u2019\n     0K .                                                     100% 59.1M=0s<\/code><\/pre>\n<h2 id=\"outline__7\" id=\"\u304a\u308f\u308a\u306b\">\u304a\u308f\u308a\u306b<\/h2>\n<p>\u8272\u3005\u3068\u624b\u9593\u53d6\u308a\u307e\u3057\u305f\u304c\u3001Lambda\u4e0a\u3067wget\u30b3\u30de\u30f3\u30c9\u3092\u5b9f\u884c\u3067\u304d\u308b\u3088\u3046\u306b\u306a\u308a\u307e\u3057\u305f\u3002<br \/>\n\u53c2\u8003\u306b\u306a\u308c\u3070\u5e78\u3044\u3067\u3059\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u30a8\u30f3\u30b8\u30cb\u30a2\u306e\u5185\u5c71\u3067\u3059\u3002\u4eca\u56de\u306f\u3001Lambda\u4e0a\u3067wget\u30b3\u30de\u30f3\u30c9\u3092\u5165\u308c\u308b\u3068\u304d\u306b\u8272\u3005\u3068\u624b\u9593\u53d6\u3063\u305f\u306e\u3067\u3001\u305d\u306e\u77e5\u898b\u3092\u5171\u6709\u3055\u305b\u3066\u3044\u305f\u3060\u304d\u307e\u3059\u3002 Lambda\u306b\u306fwget\u30b3\u30de\u30f3\u30c9\u304c\u306a\u3044 Lambda\u306b\u306f\u3001\u30c7\u30d5\u30a9\u30eb\u30c8\u3067wget\u30b3\u30de\u30f3\u30c9\u306f\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3055\u308c\u3066\u3044\u307e\u305b\u3093\u3002\u305d\u306e\u305f\u3081\u3001Lambda\u4e0a\u306bwget\u30d0\u30a4\u30ca\u30ea\u30d5\u30a1\u30a4\u30eb\u3092\u30a2\u30c3\u30d7\u30ed\u30fc\u30c9\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002 \u3057\u304b\u3057\u3001\u3069\u3053\u304b\u306e\u30b5\u30fc\u30d0\u306b\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3055\u308c\u3066\u3044\u308bwget\u30d0\u30a4\u30ca\u30ea\u30d5 [&hellip;]<\/p>\n","protected":false},"author":5,"featured_media":684,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[13],"tags":[136],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/posts\/663"}],"collection":[{"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/comments?post=663"}],"version-history":[{"count":0,"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/posts\/663\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/"}],"wp:attachment":[{"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/media?parent=663"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/categories?post=663"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/tags?post=663"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}