{"id":142679,"date":"2020-08-04T14:01:39","date_gmt":"2020-08-04T06:01:39","guid":{"rendered":"http:\/\/4563.org\/?p=142679"},"modified":"2020-08-04T14:01:39","modified_gmt":"2020-08-04T06:01:39","slug":"spark%e4%b9%8brdd","status":"publish","type":"post","link":"http:\/\/4563.org\/?p=142679","title":{"rendered":"Spark\u4e4bRDD"},"content":{"rendered":"

<\/a>RDD\u7684\u6982\u8ff0<\/h3>\n

<\/a>RDD\u662f\u4ec0\u4e48\uff1f<\/h4>\n

RDD\uff08Resilient Distributed Dataset\uff09\u53eb\u505a\u5f39\u6027\u5206\u5e03\u5f0f\u6570\u636e\u96c6\uff0c\u662fSpark\u4e2d\u6700\u57fa\u672c\u7684\u6570\u636e\u62bd\u8c61\uff0c\u4ee3\u8868\u4e00\u4e2a\u4e0d\u53ef\u53d8\u3001\u53ef\u5206\u533a\u3001\u91cc\u9762\u7684\u5143\u7d20\u53ef\u5e76\u884c\u8ba1\u7b97\u7684\u96c6\u5408\u3002<\/p>\n

<\/a>RDD\u7684\u4e3b\u8981\u5c5e\u6027\uff1f<\/h4>\n

RDD \u662fSpark \u4e2d\u6700\u57fa\u672c\u7684\u6570\u636e\u62bd\u8c61\uff0c\u662f\u4e00\u4e2a\u903b\u8f91\u6982\u5ff5\uff0c\u5b83\u53ef\u80fd\u5e76\u4e0d\u5bf9\u5e94\u6b21\u78c1\u76d8\u6216\u5185\u5b58\u4e2d\u7684\u7269\u7406\u6570\u636e\uff0c\u800c\u4ec5\u4ec5\u662f\u8bb0\u5f55\u4e86RDD\u7684\u7531\u6765, \u7236RDD\u662f\u8c01\uff0c\u4ee5\u53ca\u600e\u6837\u4ece\u7236RDD\u8ba1\u7b97\u800c\u6765\u3002<\/p>\n

spark \u6e90\u7801\u91cc\u9762\u5bf9 RDD \u7684\u63cf\u8ff0\uff1a<\/p>\n

\n\n\n
\n
1<\/span>
2<\/span>
3<\/span>
4<\/span>
5<\/span>
6<\/span>
<\/pre>\n<\/td>\n
\n
Internally, each RDD is characterized by five main properties:<\/span>
A list of partitions<\/span>
A function for computing each split<\/span>
A list of dependencies on other RDDs<\/span>
Optionally, a Partitioner for key-value RDDs (e.g. to say that the RDD is hash-partitioned)<\/span>
Optionally, a list of preferred locations to compute each split on (e.g. block locations for an HDFS file)<\/span>
<\/pre>\n<\/td>\n<\/tr>\n<\/table>\n<\/figure>\n
\u53ef\u4ee5\u77e5\u9053\uff0c\u6bcf\u4e2a RDD \u6709\u4ee5\u4e0b\u4e94\u90e8\u5206\u6784\u6210\uff1a<\/p>\n
<\/a>\u4e00\u7ec4\u5206\u7247\uff08Partition\uff09<\/h4>\n
\u6570\u636e\u5206\u533a\u5217\u8868\uff0c\u5373\u6570\u636e\u96c6\u7684\u57fa\u672c\u7ec4\u6210\u5355\u4f4d\u3002\u8fd9\u91cc\u8868\u793a\u4e00\u4e2ardd\u6709\u5f88\u591a\u5206\u533a\uff0c\u6bcf\u4e2a\u5206\u533a\u5185\u90e8\u5305\u542b\u4e86\u8be5rdd\u7684\u90e8\u5206\u6570\u636e\uff0cspark\u4e2d\u4efb\u52a1\u662f\u4ee5task\u7ebf\u7a0b\u65b9\u5f0f\u5b58\u5728\u3002\u7528\u6237\u53ef\u4ee5\u5728\u521b\u5efaRDD\u65f6\u6307\u5b9aRDD\u7684\u5206\u7247\u4e2a\u6570\uff0c\u5982\u679c\u6ca1\u6709\u6307\u5b9a\uff0c\u90a3\u4e48\u5c31\u4f1a\u91c7\u7528\u9ed8\u8ba4\u503c\u3002\u9ed8\u8ba4\u503c\u5c31\u662f\u7a0b\u5e8f\u6240\u5206\u914d\u5230\u7684CPU Core\u7684\u6570\u76ee\u3002<\/p>\n
<\/a>\u6bcf\u4e2aPartition\u7684\u8ba1\u7b97\u51fd\u6570<\/h4>\n
Spark\u4e2dRDD\u7684\u8ba1\u7b97\u662f\u4ee5\u5206\u7247\u4e3a\u5355\u4f4d\u7684\uff0c\u6bcf\u4e2aRDD\u90fd\u4f1a\u5b9e\u73b0compute\u51fd\u6570\u4ee5\u8fbe\u5230\u8fd9\u4e2a\u76ee\u7684\u3002compute\u51fd\u6570\u4f1a\u5bf9\u8fed\u4ee3\u5668\u8fdb\u884c\u590d\u5408\uff0c\u4e0d\u9700\u8981\u4fdd\u5b58\u6bcf\u6b21\u8ba1\u7b97\u7684\u7ed3\u679c\u3002<\/p>\n
<\/a>RDD\u4e4b\u95f4\u7684\u4f9d\u8d56\u5173\u7cfb<\/h4>\n
RDD\u7684\u6bcf\u6b21\u8f6c\u6362\u90fd\u4f1a\u751f\u6210\u4e00\u4e2a\u65b0\u7684RDD\uff0c\u6240\u4ee5RDD\u4e4b\u95f4\u5c31\u4f1a\u5f62\u6210\u7c7b\u4f3c\u4e8e\u6d41\u6c34\u7ebf\u4e00\u6837\u7684\u524d\u540e\u4f9d\u8d56\u5173\u7cfb\u3002\u5728\u90e8\u5206\u5206\u533a\u6570\u636e\u4e22\u5931\u65f6\uff0cSpark\u53ef\u4ee5\u901a\u8fc7\u8fd9\u4e2a\u4f9d\u8d56\u5173\u7cfb\u91cd\u65b0\u8ba1\u7b97\u4e22\u5931\u7684\u5206\u533a\u6570\u636e\uff0c\u800c\u4e0d\u662f\u5bf9RDD\u7684\u6240\u6709\u5206\u533a\u8fdb\u884c\u91cd\u65b0\u8ba1\u7b97\u3002<\/p>\n
<\/a>\uff08\u53ef\u9009\u7684\uff09\u5bf9\u4e8ekey-value\u7c7b\u578b\u7684RDD,\u5219\u5305\u542b\u4e00\u4e2aPartitioner\uff0c\u5373RDD\u7684\u5206\u7247\u51fd\u6570\u3002<\/h4>\n
\u5f53\u524dSpark\u4e2d\u5b9e\u73b0\u4e86\u4e24\u79cd\u7c7b\u578b\u7684\u5206\u7247\u51fd\u6570\uff0c\u4e00\u4e2a\u662f\u57fa\u4e8e\u54c8\u5e0c\u7684HashPartitioner\uff0c\u53e6\u5916\u4e00\u4e2a\u662f\u57fa\u4e8e\u8303\u56f4\u7684RangePartitioner\u3002\u53ea\u6709\u5bf9\u4e8e\u4e8ekey-value\u7684RDD\uff0c\u624d\u4f1a\u6709Partitioner\uff0c\u975ekey-value\u7684RDD\u7684Parititioner\u7684\u503c\u662fNone\u3002Partitioner\u51fd\u6570\u4e0d\u4f46\u51b3\u5b9a\u4e86RDD\u672c\u8eab\u7684\u5206\u7247\u6570\u91cf\uff0c\u4e5f\u51b3\u5b9a\u4e86parent RDD Shuffle\u8f93\u51fa\u65f6\u7684\u5206\u7247\u6570\u91cf\u3002<\/p>\n
<\/a>\uff08\u53ef\u9009\u7684\uff09\u8ba1\u7b97\u6bcf\u4e2aParition\u6240\u503e\u5411\u7684\u8282\u70b9\u4f4d\u7f6e\u3002\u5b58\u50a8\u5b58\u53d6\u6bcf\u4e2aPartition\u7684\u4f18\u5148\u4f4d\u7f6e\uff08preferred location\uff09\u3002<\/h4>\n
\u5bf9\u4e8e\u4e00\u4e2aHDFS\u6587\u4ef6\u6765\u8bf4\uff0c\u8fd9\u4e2a\u5217\u8868\u4fdd\u5b58\u7684\u5c31\u662f\u6bcf\u4e2aPartition\u6240\u5728\u7684\u5757\u7684\u4f4d\u7f6e\u3002\u6309\u7167\u201c\u79fb\u52a8\u6570\u636e\u4e0d\u5982\u79fb\u52a8\u8ba1\u7b97\u201d\u7684\u7406\u5ff5\uff0cSpark\u5728\u8fdb\u884c\u4efb\u52a1\u8c03\u5ea6\u7684\u65f6\u5019\uff0c\u4f1a\u5c3d\u53ef\u80fd\u5730\u5c06\u8ba1\u7b97\u4efb\u52a1\u5206\u914d\u5230\u5176\u6240\u8981\u5904\u7406\u6570\u636e\u5757\u7684\u5b58\u50a8\u4f4d\u7f6e\u3002<\/p>\n
RDD\u662f\u4e00\u4e2a\u5e94\u7528\u5c42\u9762\u7684\u903b\u8f91\u6982\u5ff5\u3002\u4e00\u4e2aRDD\u591a\u4e2a\u5206\u7247\u3002RDD\u5c31\u662f\u4e00\u4e2a\u5143\u6570\u636e\u8bb0\u5f55\u96c6\uff0c\u8bb0\u5f55\u4e86RDD\u5185\u5b58\u6240\u6709\u7684\u5173\u7cfb\u6570\u636e\u3002<\/p>\n
<\/a>\u57fa\u4e8espark\u7684\u5355\u8bcd\u7edf\u8ba1\u7a0b\u5e8f\u5256\u6790rdd\u7684\u4e94\u5927\u5c5e\u6027<\/h3>\n
<\/a>\u9700\u6c42<\/h4>\n
HDFS\u4e0a\u6709\u4e00\u4e2a\u5927\u5c0f\u4e3a300M\u7684\u6587\u4ef6\uff0c\u901a\u8fc7spark\u5b9e\u73b0\u6587\u4ef6\u5355\u8bcd\u7edf\u8ba1\uff0c\u6700\u540e\u628a\u7ed3\u679c\u6570\u636e\u4fdd\u5b58\u5230HDFS\u4e0a<\/p>\n
<\/a>\u4ee3\u7801<\/h4>\n
\n\n\n
\n
1<\/span>
<\/pre>\n<\/td>\n
\n
sc.textFile(\"\/words.txt\"<\/span>).flatMap(_.split(\" \"<\/span>)).map((_,1<\/span>)).reduceByKey(_+_).saveAsTextFile(\"\/out\"<\/span>)<\/span>
<\/pre>\n<\/td>\n<\/tr>\n<\/table>\n<\/figure>\n
<\/a>RDD\u4e94\u5927\u5c5e\u6027\u5256\u6790\u56fe<\/h4>\n
\"Spark\u4e4bRDD\"<\/p>\n
<\/a>Spark\u7f16\u7a0b\u63a5\u53e3<\/h3>\n
Spark \u7a0b\u5e8f\u8bbe\u8ba1\u6d41\u7a0b\u4e00\u822c\u5982\u4e0b\uff1a
\u6b65\u9aa4\u4e00\uff1a\u5b9e\u4f8b\u5316 sparkContent \u5bf9\u8c61\u3002sparkContent \u5c01\u88c5\u4e86\u7a0b\u5e8f\u8fd0\u884c\u7684\u4e0a\u4e0b\u6587\u73af\u5883\uff0c\u5305\u62ec\u914d\u7f6e\u4fe1\u606f\u3001\u6570\u636e\u5e93\u7ba1\u7406\u5668\u3001\u4efb\u52a1\u8c03\u5ea6\u7b49\u3002
\u6b65\u9aa4\u4e8c\uff1a\u6784\u9020 RDD\u3002\u53ef\u901a\u8fc7 sparkContent \u63d0\u4f9b\u7684\u51fd\u6570\u6784\u9020 RDD\uff0c\u5e38\u89c1\u7684 RDD \u6784\u9020\u65b9\u5f0f\u5206\u4e3a\uff1a\u5c06 Scala\u96c6\u5408\u8f6c\u6362\u4e3a RDD \u548c\u5c06 Hadoop \u6587\u4ef6\u8f6c\u6362\u4e3a RDD\u3002
\u6b65\u9aa4\u4e09\uff1a\u5728 RDD \u57fa\u7840\u4e0a\uff0c\u901a\u8fc7 Spark \u63d0\u4f9b\u7684 transformation \u7b97\u5b50\u5b8c\u6210\u6570\u636e\u5904\u7406\u6b65\u9aa4\u3002
\u6b65\u9aa4\u56db\uff1a\u901a\u8fc7 action \u7b97\u5b50\u5c06\u6700\u7ec8 RDD \u4f5c\u4e3a\u7ed3\u679c\u76f4\u63a5\u8fd4\u56de\u6216\u8005\u4fdd\u5b58\u5230\u6587\u4ef6\u4e2d\u3002<\/p>\n
Spark \u63d0\u4f9b\u4e86\u4e24\u5927\u7c7b\u7f16\u7a0b\u63a5\u53e3\uff0c\u5206\u522b\u4e3a RDD \u64cd\u4f5c\u7b26\u4ee5\u53ca\u5171\u4eab\u53d8\u91cf\u3002
\u5176\u4e2d RDD \u64cd\u4f5c\u7b26\u5305\u62ec transformation \u548c action \u4ee5\u53ca control API \u4e09\u7c7b\uff1b\u5171\u4eab\u53d8\u91cf\u5305\u62ec\u5e7f\u64ad\u53d8\u91cf\u548c\u7d2f\u52a0\u5668\u4e24\u79cd\u3002<\/p>\n
<\/a>\u521b\u5efa sparkContent \u5bf9\u8c61\uff0c\u5c01\u88c5\u4e86 Spark \u6267\u884c\u73af\u5883\u4fe1\u606f<\/h3>\n
<\/a>\u521b\u5efaconf ,\u5c01\u88c5\u4e86spark\u914d\u7f6e\u4fe1\u606f<\/h4>\n
\n\n\n
\n
1<\/span>
<\/pre>\n<\/td>\n
\n
val<\/span> sparkConf: SparkConf<\/span> = new<\/span> SparkConf<\/span>().setAppName(\"WordCountOnSpark\"<\/span>)<\/span>
<\/pre>\n<\/td>\n<\/tr>\n<\/table>\n<\/figure>\n
<\/a>\u521b\u5efa SparkContext\uff0c\u5c01\u88c5\u4e86\u8c03\u5ea6\u5668\u7b49\u4fe1\u606f<\/h4>\n
\n\n\n
\n
1<\/span>
<\/pre>\n<\/td>\n
\n
val<\/span> sc = new<\/span> SparkContext<\/span>(sparkConf)<\/span>
<\/pre>\n<\/td>\n<\/tr>\n<\/table>\n<\/figure>\n
<\/a>\u6784\u5efaRDD<\/h3>\n
<\/a>\u901a\u8fc7\u5df2\u7ecf\u5b58\u5728\u7684scala\u96c6\u5408\u53bb\u6784\u5efa<\/h4>\n
\n\n\n
\n
1<\/span>
2<\/span>
3<\/span>
<\/pre>\n<\/td>\n
\n
val rdd1=sc.parallelize(List(1,2,3,4,5))<\/span>
val rdd2=sc.parallelize(Array("hadoop","hive","spark"))<\/span>
val rdd3=sc.makeRDD(List(1,2,3,4))<\/span>
<\/pre>\n<\/td>\n<\/tr>\n<\/table>\n<\/figure>\n
<\/a>\u5c06\u6587\u672c\u6587\u4ef6\u8f6c\u6362\u4e3a RDD<\/h4>\n
\n\n\n
\n
1<\/span>
2<\/span>
3<\/span>
4<\/span>
5<\/span>
6<\/span>
<\/pre>\n<\/td>\n
\n
sc.textFile(\u201c/data\u201d, 1) <\/span>
sc.textFile(\u201c/data/file.txt\u201d, 1) <\/span>
sc.textFile(\u201c/data/*.txt\u201d, 1) <\/span>
sc.textFile(\u201chdfs://bigdata:9000/data/\u201d, 1) <\/span>
sc.sequenceFile(\u201c/data\u201d, 1) <\/span>
sc.wholeTextFiles(\u201c/data\u201d, 1)<\/span>
<\/pre>\n<\/td>\n<\/tr>\n<\/table>\n<\/figure>\n","protected":false},"excerpt":{"rendered":"
RDD\u7684\u6982\u8ff0 RDD\u662f\u4ec0\u4e48\uff1f RD…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[],"tags":[],"_links":{"self":[{"href":"http:\/\/4563.org\/index.php?rest_route=\/wp\/v2\/posts\/142679"}],"collection":[{"href":"http:\/\/4563.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/4563.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/4563.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/4563.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=142679"}],"version-history":[{"count":0,"href":"http:\/\/4563.org\/index.php?rest_route=\/wp\/v2\/posts\/142679\/revisions"}],"wp:attachment":[{"href":"http:\/\/4563.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=142679"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/4563.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=142679"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/4563.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=142679"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}