Caused by: java.lang.ClassNotFoundException: org.jets3t.service.ServiceException

Question 1

My code should access some file stored on S3 (this code works fine on one machine, while it fails on the other one; basically it fails when it gets executed locally (not on the cluster) from Intellij IDEA):

sc.hadoopConfiguration.set("fs.s3n.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "xxx")
sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey", "xxx")

val sqlContext = new SQLContext(sc)

var df = sqlContext.read.json("s3n://myPath/*.json")

1

2

3

4

5

6

7

8

sc.hadoopConfiguration.set("fs.s3n.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem")

sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "xxx")

sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey", "xxx")

val sqlContext = new SQLContext(sc)

var df = sqlContext.read.json("s3n://myPath/*.json")

I get the following error at the line var df = sqlContext.read.json("s3n://myPath/*.json"):

Caused by: java.lang.ClassNotFoundException: org.jets3t.service.ServiceException
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

1

2

3

4

5

6

Caused by: java.lang.ClassNotFoundException: org.jets3t.service.ServiceException

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

I read similar threads regarding this issue and it was mentioned that in case of using Spark 1.6.2, the solution is to use org.apache.hadoop hadoop-aws 2.6.0. In my case it has not solved the problem.

My pom.xml (an extract from it):

<properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>

        <java.version>1.8</java.version>
        <scala.version>2.10.6</scala.version>
        <spark.version>1.6.2</spark.version>
        <jackson.version>2.8.3</jackson.version>
    </properties>

<dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.10</artifactId>
            <!--<scope>provided</scope>-->
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka_2.10</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.10</artifactId>
            <!--<scope>provided</scope>-->
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_2.10</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.module</groupId>
            <artifactId>jackson-module-scala_2.10</artifactId>
            <version>${jackson.version}</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>${jackson.version}</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
            <version>${jackson.version}</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
            <version>${jackson.version}</version>
        </dependency>
        <dependency>
            <groupId>com.lambdaworks</groupId>
            <artifactId>jacks_2.10</artifactId>
            <version>2.3.3</version>
        </dependency>
        <dependency>
            <groupId>com.typesafe</groupId>
            <artifactId>config</artifactId>
            <version>1.3.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-aws</artifactId>
            <version>2.6.0</version>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-s3</artifactId>
            <version>1.11.53</version>
        </dependency>
        <dependency>
            <groupId>net.debasishg</groupId>
            <artifactId>redisclient_2.10</artifactId>
            <version>3.3</version>
        </dependency>
    </dependencies>

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

UTF-8

1.8

2.10.6

1.6.2

2.8.3

org.scala-lang

scala-library

${scala.version}

org.apache.spark

spark-streaming_2.10

${spark.version}

org.apache.spark

spark-streaming-kafka_2.10

${spark.version}

org.apache.spark

spark-sql_2.10

${spark.version}

org.apache.spark

spark-mllib_2.10

${spark.version}

com.fasterxml.jackson.module

jackson-module-scala_2.10

${jackson.version}

com.fasterxml.jackson.core

jackson-databind

${jackson.version}

com.fasterxml.jackson.core

jackson-annotations

${jackson.version}

com.fasterxml.jackson.core

jackson-core

${jackson.version}

com.lambdaworks

jacks_2.10

2.3.3

com.typesafe

config

1.3.1

org.apache.hadoop

hadoop-aws

2.6.0

com.amazonaws

aws-java-sdk-s3

1.11.53

net.debasishg

redisclient_2.10

3.3

Question 2

Adding the following in the dependency should solve the issue

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.6.0</version>
</dependency>

1

2

3

4

5

6

org.apache.hadoop

hadoop-client

2.6.0

I hope this helps

Caused by: java.lang.ClassNotFoundException: org.jets3t.service.ServiceException

Question:

Answer:

Leave a Reply Cancel reply