win7下使用Idea遠程連接spark執行spark pi，我自己的實驗
win7地址為192.168.0.2，ubuntu為虛擬機，地址為192.168.0.3
遠程連接spark
源代碼語言為：
package main.scala.sogou
/**
* Created by danger on 2016/9/16.
*/
import org.apache.spark.SparkContext._
import org.apache.spark.{SparkConf,SparkContext}
object RemoteDebug {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("Spark Pi").setMaster("spark://192.168.0.3:7077")
.setJars(List("D:\\scalasrc\\out\\artifacts\\scalasrc.jar"))
val spark = new SparkContext(conf)
val slices = if (args.length > 0) args(0).toInt else 2
val n = 100000 * slices
val count = spark.parallelize(1 to n, slices).map { i =>
val x = Math.random * 2 - 1
val y = Math.random * 2 - 1
if (x * x + y * y < 1) 1 else 0
}.reduce(_ + _)
println("Pi is roughly " + 4.0 * count / n)
spark.stop()
}
}
你需要修改的地方就是spark://192.168.0.3:7077
還有就是setJars的地址：D:\\scalasrc\\out\\artifacts\\scalasrc.jar另外我沒有Import spark的Math包
所以random用的是Math.random
在run的edit configuration界面中，配置參數
七十主要是mainclass哈
main.scala.sogou.RemoteDebug
單機run
遠程的服務器一陣狂賺，消停后，出現了
Process finished with exit code 0
呵呵，成功了
但結果呢？
網上找，原來在這里
16/09/16 09:40:57 INFO DAGScheduler: ResultStage 0 (reduce at RemoteDebug.scala:19) finished in 75.751 s
16/09/16 09:40:57 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
16/09/16 09:40:57 INFO DAGScheduler: Job 0 finished: reduce at RemoteDebug.scala:19, took 76.071948 s
Pi is roughly 3.1385
16/09/16 09:40:57 INFO SparkUI: Stopped Spark web UI at http://192.168.0.2:4040
16/09/16 09:40:57 INFO DAGScheduler: Stopping DAGScheduler
16/09/16 09:40:57 INFO SparkDeploySchedulerBackend: Shutting down all executors
16/09/16 09:40:57 INFO SparkDeploySchedulerBackend: Asking each executor to shut down
16/09/16 09:40:57 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/09/16 09:40:57 INFO MemoryStore: MemoryStore cleared
太帥了，成功了，參考http://blog.csdn.net/javastart/article/details/43372977
免責聲明：本站發布的內容（圖片、視頻和文字）以原創、轉載和分享為主，文章觀點不代表本網站立場，如果涉及侵權請聯系站長郵箱：is@yisu.com進行舉報，并提供相關證據，一經查實，將立刻刪除涉嫌侵權內容。
