运行spark官方的graphx 示例 ComprehensiveExample.scala报错解决

在Idea中，直接运行ComprehensiveExample.scala，报需要指定master异常。

修改源码：指定master为local模式，

 val spark = SparkSession

      .builder

      .appName(s"${this.getClass.getSimpleName}").master("local[2]")

      .getOrCreate()

继续运行，报如下错误：

Exception in thread "main" java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.mapred.FileInputFormat

	at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:312)

	at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:199)

	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)

	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)

	at scala.Option.getOrElse(Option.scala:121)

	at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)

	at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)

	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)

	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)

	at scala.Option.getOrElse(Option.scala:121)

	at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)

	at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)

	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)

	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)

	at scala.Option.getOrElse(Option.scala:121)

	at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)

	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2094)

	at org.apache.spark.rdd.RDD.count(RDD.scala:1158)

	at org.apache.spark.graphx.GraphLoader$.edgeListFile(GraphLoader.scala:94)

	at graphx.ComprehensiveExample$.main(ComprehensiveExample.scala:53)

	at graphx.ComprehensiveExample.main(ComprehensiveExample.scala)

	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

	at java.lang.reflect.Method.invoke(Method.java:498)

	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

com.google.common.base.Stopwatch 在guava jar包中，查了相关资料，说可能是guava包的版本问题，

我的测试程序使用的版本是18.0，而spark官方源码使用的是14.0.1，spark 2.2.1 使用依赖的hadoop 版本是2.6.5。

        <dependency>

            <groupId>com.google.guava</groupId>

            <artifactId>guava</artifactId>

            <version>14.0.1</version>

        </dependency>

修改后重新运行spark任务，问题解决。

巴特西

运行spark官方的graphx 示例 ComprehensiveExample.scala报错解决

运行spark官方的graphx 示例 ComprehensiveExample.scala报错解决

最新文章

热门文章