{"id":3712,"date":"2022-11-25T11:01:13","date_gmt":"2022-11-25T02:01:13","guid":{"rendered":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/?p=3712"},"modified":"2022-11-25T11:01:13","modified_gmt":"2022-11-25T02:01:13","slug":"aws-glue-git-integration","status":"publish","type":"post","link":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/blog\/2022\/11\/25\/aws-glue-git-integration\/","title":{"rendered":"AWS Glue Git\u7d71\u5408\u306e\u6d3b\u7528"},"content":{"rendered":"<p>\u6700\u8fd1\u306f<strong>\u6012\u6d9b\u306e\u4eba\u751f\u30a4\u30d9\u30f3\u30c8<\/strong>\u304c\u91cd\u306a\u308a\u3001\u305d\u308d\u305d\u308d\u843d\u3061\u7740\u3044\u3066\u307b\u3057\u3044\u3068\u601d\u3046\u3084\u3063\u3055\u3093\u3067\u3054\u3056\u3044\u307e\u3059\u3002\u672c\u30d6\u30ed\u30b0\u6295\u7a3f\u3067\u306f\u3001<strong>AWS Glue\u306eGit\u7d71\u5408<\/strong>\u3092\u958b\u767a\u5de5\u7a0b\u3067\u6d3b\u7528\u3057\u307e\u3057\u305f\u306e\u3067\u3001\u305d\u306e\u65b9\u6cd5\u306b\u3064\u3044\u3066\u3054\u7d39\u4ecb\u3057\u307e\u3059\u3002\u5c1a\u3001\u3053\u3061\u3089\u306e\u984c\u6750\u306fAPN\u306e\u30e9\u30a4\u30c8\u30cb\u30f3\u30b0\u30c8\u30fc\u30af\u3067\u3082\u767a\u8868\u3055\u305b\u3066\u9802\u304d\u307e\u3057\u305f\u5185\u5bb9\u3068\u306a\u3063\u3066\u304a\u308a\u307e\u3059\u3002<\/p>\n<h2 id=\"outline__1\">AWS Glue Git\u7d71\u5408\u3068\u306f<\/h2>\n<p><code>AWS Glue Git\u7d71\u5408<\/code> \u306f 2022\u5e7410\u6708\u306b\u767a\u8868\u3055\u308c\u305fAWS Glue\u306e\u65b0\u6a5f\u80fd\u3067\u3001 <code>Glue Studio<\/code> \u3067\u4f5c\u6210\u3057\u305fGlue Job\u3092Git\u30ea\u30dd\u30b8\u30c8\u30ea\u3067\u7ba1\u7406\u3059\u308b\u6a5f\u80fd\u3067\u3059\u3002\u5bfe\u8c61\u306eGit\u30ea\u30dd\u30b8\u30c8\u30ea\u306f<code>AWS CodeCommit<\/code> \u3068 <code>GitHub<\/code> \u306b\u5bfe\u5fdc\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n<p><a href=\"\/images\/2022\/11\/aws_glue_git_001.png\"><img src=\"\/images\/2022\/11\/aws_glue_git_001.png\" alt=\"\" \/><\/a><\/p>\n<p><code>AWS Glue Git\u7d71\u5408<\/code> \u306b\u3064\u3044\u3066\u5177\u4f53\u7684\u306a\u5229\u7528\u65b9\u6cd5\u3092\u4ee5\u4e0b\u306b\u3054\u7d39\u4ecb\u3057\u307e\u3059\u3002\u4eca\u56de\u306f\u30ea\u30dd\u30b8\u30c8\u30ea\u7ba1\u7406\u306b <code>CodeCommit<\/code> \u3092\u5229\u7528\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n<h2 id=\"outline__2\">AWS Glue Job \u306e Git\u30ea\u30dd\u30b8\u30c8\u30ea\u3078\u306e\u30d7\u30c3\u30b7\u30e5<\/h2>\n<p>AWS Glue Job\u3092\u65e2\u306b\u4f5c\u6210\u6e08\u307f\u306e\u5834\u5408\u306f\u3001\u3059\u3050\u306bGit\u30ea\u30dd\u30b8\u30c8\u30ea\u306b\u30d7\u30c3\u30b7\u30e5\u53ef\u80fd\u3067\u3059\u3002\u4ee5\u4e0b\u753b\u50cf\u306e\u3088\u3046\u306bGit\u30ea\u30dd\u30b8\u30c8\u30ea\u3092\u6307\u5b9a\u3057\u307e\u3059\u3002<strong>Glue Job\u540d\u304c\u305d\u306e\u307e\u307e\u30ea\u30dd\u30b8\u30c8\u30ea\u306b\u30d7\u30c3\u30b7\u30e5\u3055\u308c\u308b\u30d5\u30a1\u30a4\u30eb\u540d\u306b\u306a\u308a\u307e\u3059\u3002<\/strong><\/p>\n<p><a href=\"\/images\/2022\/11\/aws_glue_git_002.png\"><img src=\"\/images\/2022\/11\/aws_glue_git_002.png\" alt=\"\" \/><\/a><br \/>\n<a href=\"\/images\/2022\/11\/aws_glue_git_003.png\"><img src=\"\/images\/2022\/11\/aws_glue_git_003.png\" alt=\"\" \/><\/a><\/p>\n<p>\u300cActions\u300d\u304b\u3089\u300cPush to repository\u300d\u3092\u9078\u629e\u3059\u308b\u3053\u3068\u3067\u3001Git\u30ea\u30dd\u30b8\u30c8\u30ea\u306b\u30d7\u30c3\u30b7\u30e5\u3055\u308c\u307e\u3059\u3002<\/p>\n<p><a href=\"\/images\/2022\/11\/aws_glue_git_004.png\"><img src=\"\/images\/2022\/11\/aws_glue_git_004.png\" alt=\"\" \/><\/a><\/p>\n<p>\u30d7\u30c3\u30b7\u30e5\u3055\u308c\u305f\u30d5\u30a1\u30a4\u30eb\u306f\u4ee5\u4e0b\u753b\u50cf\u306e\u3088\u3046\u306b\u306a\u3063\u3066\u304a\u308a\u307e\u3059\u3002 <code>json<\/code> \u30d5\u30a1\u30a4\u30eb\u306bGlue Job\u306e\u5404\u7a2e\u8a2d\u5b9a\u3084\u30d3\u30b8\u30e5\u30a2\u30eb\u30a8\u30c7\u30a3\u30bf\u3067\u306e\u8a18\u8ff0\u5185\u5bb9\u304c\u8a18\u8f09\u3055\u308c\u3066\u304a\u308a\u3001 <code>python<\/code> \u30d5\u30a1\u30a4\u30eb\u306b\u306f\u30d3\u30b8\u30e5\u30a2\u30eb\u30a8\u30c7\u30a3\u30bf\u3067\u306f\u306a\u304f\u30b9\u30af\u30ea\u30d7\u30c8\u30a8\u30c7\u30a3\u30bf\u3092\u5229\u7528\u3057\u3066\u3044\u308b\u5834\u5408\u306b\u30b9\u30af\u30ea\u30d7\u30c8\u5185\u5bb9\u304c\u4fdd\u5b58\u3055\u308c\u307e\u3059\u3002<\/p>\n<p><a href=\"\/images\/2022\/11\/aws_glue_git_005.png\"><img src=\"\/images\/2022\/11\/aws_glue_git_005.png\" alt=\"\" \/><\/a><\/p>\n<h2 id=\"outline__3\">AWS Glue Job\u306eGit\u30ea\u30dd\u30b8\u30c8\u30ea\u304b\u3089\u306e\u30d7\u30eb<\/h2>\n<p>Git\u30ea\u30dd\u30b8\u30c8\u30ea\u304b\u3089\u6700\u65b0\u306e\u5909\u66f4\u3092Glue Job\u306b\u53cd\u6620\u3057\u305f\u3044\u5834\u5408\u306f\u4ee5\u4e0b\u753b\u50cf\u306e\u3088\u3046\u306b\u300cActions\u300d\u2192\u300cPull from repository\u300d\u3092\u9078\u629e\u3059\u308b\u3053\u3068\u3067Glue Job\u306b\u53cd\u6620\u3055\u308c\u307e\u3059\u3002\u7c21\u5358\u3067\u3059\u306d\u3002<\/p>\n<p><a href=\"\/images\/2022\/11\/aws_glue_git_006.png\"><img src=\"\/images\/2022\/11\/aws_glue_git_006.png\" alt=\"\" \/><\/a><\/p>\n<h2 id=\"outline__4\">AWS\u30a2\u30ab\u30a6\u30f3\u30c8\u3092\u307e\u305f\u3044\u3060Glue Job\u30b9\u30af\u30ea\u30d7\u30c8\u79fb\u884c\u306e\u52d8\u6240<\/h2>\n<p>\u4eca\u56de\u3001\u4e00\u5ea6\u691c\u8a3c\u7528AWS\u30a2\u30ab\u30a6\u30f3\u30c8\u3067\u4f5c\u6210\u3057\u305fGlue Job\u3092\u7570\u306a\u308bAWS\u30a2\u30ab\u30a6\u30f3\u30c8\u306b\u79fb\u884c\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3057\u305f\u3002\u305d\u306e\u6642\u306b\u30de\u30cd\u30fc\u30b8\u30e1\u30f3\u30c8\u30b3\u30f3\u30bd\u30fc\u30eb\u3067\u4f5c\u6210\u3057\u3066\u3044\u305fGlue Job\u3092\u79fb\u884c\u3059\u308b\u969b\u306b\u3001\u3053\u306e <code>Git\u7d71\u5408<\/code> \u3092\u4fbf\u5229\u306b\u6d3b\u7528\u3044\u305f\u3057\u307e\u3057\u305f\u3002\u4fdd\u5b58\u3055\u308c\u305f <code>json<\/code> \u30d5\u30a1\u30a4\u30eb\u3084 <code>python<\/code> \u30d5\u30a1\u30a4\u30eb\u306b\u307b\u3093\u306e\u5c11\u3057\u5909\u66f4\u3092\u52a0\u3048\u308b\u3060\u3051\u3067\u79fb\u884c\u3067\u304d\u307e\u3059\u3002<\/p>\n<h3 id=\"outline__4_1\">\u5909\u66f4\u7b87\u6240\u306b\u3064\u3044\u3066<\/h3>\n<p>\u4ee5\u4e0b\u753b\u50cf\u306e\u3088\u3046\u306b\u3001 <code>AWS\u30a2\u30ab\u30a6\u30f3\u30c8ID<\/code> \u306f\u5168\u3066\u79fb\u884c\u5148\u306eID\u306b\u5909\u66f4\u3057\u307e\u3059\u3002AWS Glue Job\u304c\u5229\u7528\u3059\u308b\u30ed\u30fc\u30eb\u540d\u306b\u5909\u66f4\u304c\u3042\u308b\u5834\u5408\u3082\u4fee\u6b63\u3057\u307e\u3059\u3002<br \/>\n<a href=\"\/images\/2022\/11\/aws_glue_git_007.png\"><img src=\"\/images\/2022\/11\/aws_glue_git_007.png\" alt=\"\" \/><\/a><\/p>\n<p>\u3082\u3057Glue Job\u3067DB\u30b3\u30cd\u30af\u30b7\u30e7\u30f3\u304c\u3042\u308a\u3001\u30b3\u30cd\u30af\u30b7\u30e7\u30f3\u540d\u304c\u79fb\u884c\u5148\u3067\u7570\u306a\u308b\u540d\u79f0\u306e\u5834\u5408\u306f\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u5909\u66f4\u3057\u307e\u3059\u3002<\/p>\n<p><a href=\"\/images\/2022\/11\/aws_glue_git_008.png\"><img src=\"\/images\/2022\/11\/aws_glue_git_008.png\" alt=\"\" \/><\/a><\/p>\n<p>\u305d\u3057\u3066\u3001Glue\u30c7\u30fc\u30bf\u30ab\u30bf\u30ed\u30b0\u3068\u30c6\u30fc\u30d6\u30eb\u3082\u79fb\u884c\u5148\u3067\u7570\u306a\u308b\u540d\u79f0\u306e\u5834\u5408\u306f\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u5909\u66f4\u3057\u307e\u3059\u3002<\/p>\n<p><a href=\"\/images\/2022\/11\/aws_glue_git_009.png\"><img src=\"\/images\/2022\/11\/aws_glue_git_009.png\" alt=\"\" \/><\/a><\/p>\n<p>\u3042\u3068\u306f\u3001\u30de\u30cd\u30fc\u30b8\u30e1\u30f3\u30c8\u30b3\u30f3\u30bd\u30fc\u30eb\u3067\u65b0\u898fGlue Job\u3092\u4f5c\u6210\u6642\u306b\u3001<strong>\u30b8\u30e7\u30d6\u540d\u3060\u3051\u79fb\u884c\u5143\u306e\u540d\u524d\u3068\u4e00\u81f4\u3059\u308b\u3088\u3046\u306b\u5909\u66f4\u3057\u305f\u3042\u3068\u306b\u3001\u300cPull from repository\u300d\u3092\u9078\u629e\u3059\u308b\u3053\u3068\u3067\u7c21\u5358\u306b\u79fb\u884c\u3067\u304d\u307e\u3059\u3002<\/strong><\/p>\n<p><a href=\"\/images\/2022\/11\/aws_glue_git_010.png\"><img src=\"\/images\/2022\/11\/aws_glue_git_010.png\" alt=\"\" \/><\/a><\/p>\n<h2 id=\"outline__5\">\u4e00\u62ec\u3067Glub Job\u306b\u30b9\u30af\u30ea\u30d7\u30c8\u3092\u30d7\u30eb\u3057\u305f\u3044\u6642\u306f<\/h2>\n<p>\u5b9f\u306f\u4eca\u56deAWS\u30a2\u30ab\u30a6\u30f3\u30c8\u79fb\u884c\u306e\u5bfe\u8c61\u3068\u306a\u3063\u305fGlue Job\u306f\u4e00\u3064\u3067\u306f\u306a\u304f\u3001\u6570\u5341\u306eGlue Job\u3092\u79fb\u884c\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3057\u305f\u3002\u6570\u5341\u306eGlue Job\u3068\u306a\u308b\u3068\u30de\u30cd\u30fc\u30b8\u30e1\u30f3\u30c8\u30b3\u30f3\u30bd\u30fc\u30eb\u3067\u30dd\u30c1\u30dd\u30c1\u3068\u4e00\u3064\u305a\u3064\u79fb\u884c\u3059\u308b\u306e\u306f\u5927\u5909\u3067\u3059\u3002\u305d\u3093\u306a\u6642\u306fAWS CLI\u304c\u5229\u7528\u3067\u304d\u307e\u3059\u3002<\/p>\n<p>\u4ee5\u4e0b\u306e\u30b3\u30de\u30f3\u30c9\u306e\u3088\u3046\u306b\u3001 <code>update-job-from-source-control<\/code> \u30b3\u30de\u30f3\u30c9\u3092\u5229\u7528\u3059\u308b\u3053\u3068\u3067\u3001\uff11\u30b3\u30de\u30f3\u30c9\u3067Glue Job\u3092\u65b0\u3057\u3044AWS\u30a2\u30ab\u30a6\u30f3\u30c8\u306b\u79fb\u884c\u3067\u304d\u307e\u3059\u3002\u203b\u4e8b\u524d\u306bGit\u30ea\u30dd\u30b8\u30c8\u30ea\u304c\u79fb\u884c\u6e08\u307f\u306e\u524d\u63d0\u3068\u306a\u308a\u307e\u3059\u3002<\/p>\n<pre><code>aws glue update-job-from-source-control \\\n--job-name import_script \\\n--provider AWS_CODE_COMMIT \\\n--repository-name webiner-glue-job \\\n--branch-name master \\\n--folder src \\\n--region ap-northeast-1<\/code><\/pre>\n<h2 id=\"outline__6\">Glue Job\u306b\u306f\u3069\u306e\u30d0\u30fc\u30b8\u30e7\u30f3\u304c\u30d7\u30eb\u3055\u308c\u308b\u306e\u304b<\/h2>\n<p>\u6c17\u306b\u306a\u308b\u306e\u304c\u3001Glue Job\u306b\u30d7\u30eb\u3057\u305f\u5834\u5408\u306b\u3069\u306e\u30d0\u30fc\u30b8\u30e7\u30f3\u304c\u53cd\u6620\u3055\u308c\u308b\u306e\u304b\u3067\u3057\u305f\u3002\u3053\u3061\u3089\u306f\u3001\u6307\u5b9a\u3057\u305fGit\u30d6\u30e9\u30f3\u30c1\u306e\u6700\u65b0\u306e\u30b3\u30df\u30c3\u30c8ID\u306e\u72b6\u614b\u306b\u7d10\u3065\u304fGlue Job\u304c\u53cd\u6620\u3055\u308c\u308b\u3088\u3046\u3067\u3057\u305f\u3002<\/p>\n<p><a href=\"\/images\/2022\/11\/aws_glue_git_011.png\"><img src=\"\/images\/2022\/11\/aws_glue_git_011.png\" alt=\"\" \/><\/a><\/p>\n<h2 id=\"outline__7\">\u307e\u3068\u3081<\/h2>\n<p><code>AWS Glue Git\u7d71\u5408<\/code> \u3092\u5229\u7528\u3059\u308b\u3053\u3068\u3067\u3001<strong>Glue Job\u3092\u7c21\u5358\u306b\u30ea\u30dd\u30b8\u30c8\u30ea\u7ba1\u7406\u3059\u308b\u3053\u3068\u304c\u51fa\u6765\u307e\u3057\u305f<\/strong>\u3002\u5b9f\u969b\u306b\u30d7\u30c3\u30b7\u30e5\u3055\u308c\u305f\u30b9\u30af\u30ea\u30d7\u30c8\u3068\u30b8\u30e7\u30d6\u5b9a\u7fa9\u30d5\u30a1\u30a4\u30eb\u306f<strong>CloudFormation\u3068\u306f\u307e\u305f\u7570\u306a\u308b\u30d5\u30a9\u30fc\u30de\u30c3\u30c8\u306b\u306a\u3063\u3066\u304a\u308a<\/strong>\u3001\u30b8\u30e7\u30d6\u5b9a\u7fa9\u30d5\u30a1\u30a4\u30eb\u3092CloudFormation\u3084Terraform\u306a\u3069\u7570\u306a\u308bIaC\u306b\u79fb\u884c\u3057\u305f\u3044\u5834\u5408\u306b\u304a\u3044\u3066\u306f\u79fb\u884c\u65b9\u6cd5\u3092\u691c\u8a0e\u3059\u308b\u5fc5\u8981\u304c\u3042\u308b\u304b\u306a\u3068\u601d\u3044\u307e\u3057\u305f\u3002Glue Studio\u3067\u76f4\u63a5\u30b9\u30af\u30ea\u30d7\u30c8\u3092\u7de8\u96c6\u3057\u305f\u5834\u5408\u306f<strong>\u30de\u30cd\u30fc\u30b8\u30e1\u30f3\u30c8\u30b3\u30f3\u30bd\u30fc\u30eb\u306e\u30bf\u30a4\u30e0\u30a2\u30a6\u30c8\u5236\u9650<\/strong>\u306b\u6c17\u3092\u3064\u3051\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3057\u305f\u306e\u3067\u3001\u4e00\u5ea6Glue Studio\u3067\u4f5c\u6210\u3057\u305f\u30b9\u30af\u30ea\u30d7\u30c8\u306f\u3059\u3050\u306bGit\u7d71\u5408\u3067\u30ed\u30fc\u30ab\u30eb\u3067\u7de8\u96c6\u3067\u304d\u308b\u3088\u3046\u306b\u3059\u308b\u3053\u3068\u3067\u52b9\u7387\u7684\u306a\u958b\u767a\u304c\u51fa\u6765\u307e\u3057\u305f\u3002<\/p>\n<p>\u3054\u53c2\u8003\u306b\u306a\u308c\u3070\u5e78\u3044\u3067\u3059\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u6700\u8fd1\u306f\u6012\u6d9b\u306e\u4eba\u751f\u30a4\u30d9\u30f3\u30c8\u304c\u91cd\u306a\u308a\u3001\u305d\u308d\u305d\u308d\u843d\u3061\u7740\u3044\u3066\u307b\u3057\u3044\u3068\u601d\u3046\u3084\u3063\u3055\u3093\u3067\u3054\u3056\u3044\u307e\u3059\u3002\u672c\u30d6\u30ed\u30b0\u6295\u7a3f\u3067\u306f\u3001AWS Glue\u306eGit\u7d71\u5408\u3092\u958b\u767a\u5de5\u7a0b\u3067\u6d3b\u7528\u3057\u307e\u3057\u305f\u306e\u3067\u3001\u305d\u306e\u65b9\u6cd5\u306b\u3064\u3044\u3066\u3054\u7d39\u4ecb\u3057\u307e\u3059\u3002\u5c1a\u3001\u3053\u3061\u3089\u306e\u984c\u6750\u306fAPN\u306e\u30e9\u30a4\u30c8\u30cb\u30f3\u30b0\u30c8\u30fc\u30af\u3067\u3082\u767a\u8868\u3055\u305b\u3066\u9802\u304d\u307e\u3057\u305f\u5185\u5bb9\u3068\u306a\u3063\u3066\u304a\u308a\u307e\u3059\u3002 AWS Glue Git\u7d71\u5408\u3068\u306f AWS Glue Git\u7d71\u5408 \u306f 2022\u5e7410\u6708\u306b\u767a\u8868\u3055\u308c\u305fAWS Glue\u306e\u65b0 [&hellip;]<\/p>\n","protected":false},"author":8,"featured_media":3729,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[13],"tags":[32],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/posts\/3712"}],"collection":[{"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/comments?post=3712"}],"version-history":[{"count":9,"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/posts\/3712\/revisions"}],"predecessor-version":[{"id":3880,"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/posts\/3712\/revisions\/3880"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/media\/3729"}],"wp:attachment":[{"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/media?parent=3712"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/categories?post=3712"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/p-corporate-blog-cms.mmmcorp.co.jp\/wp-json\/wp\/v2\/tags?post=3712"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}