Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
Hi Paul,

Ah, I think I've seen this before.  $HADOOP_HOME/lib is being added to the Java classpath, but $HADOOP_HOME is not.  Since hadoop-common.jar is there, it isn't being put on the classpath. 

This might not be the perfect/permanent fix, but on line 174 of bin/geomesa, I'd change it from

hadoopDirs=("$HADOOP_LIB" "$HADOOP_COMMON" "$HADOOP_MAPREDUCE" "$HADOOP_YARN" "$HADOOP_HDFS")

to
hadoopDirs=("$HADOOP_LIB" "$HADOOP_COMMON" "$HADOOP_MAPREDUCE" "$HADOOP_YARN" "$HADOOP_HDFS" "$HADOOP_HOME")

That'd pick up the jars hanging out in the $HADOOP_HOME.  Other alternatives would include linking to them in $HADOOP_HOME/lib, etc.

Cheers,

Jim

On 2/1/2016 5:30 PM, Paul Levy wrote:
And geomesa classpath is picking up the hadoop jars:

/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.4.jar
/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar
/usr/lib/hadoop/lib/commons-lang-2.6.jar
/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar
/usr/lib/hadoop/lib/curator-framework-2.7.1.jar
/usr/lib/hadoop/lib/commons-math3-3.1.1.jar
/usr/lib/hadoop/lib/gson-2.2.4.jar
/usr/lib/hadoop/lib/commons-collections-3.2.2.jar
/usr/lib/hadoop/lib/jersey-server-1.9.jar
/usr/lib/hadoop/lib/commons-cli-1.2.jar
/usr/lib/hadoop/lib/jersey-json-1.9.jar
/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar
/usr/lib/hadoop/lib/commons-logging-1.1.3.jar
/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar
/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar
/usr/lib/hadoop/lib/commons-compress-1.4.1.jar
/usr/lib/hadoop/lib/jsch-0.1.42.jar
/usr/lib/hadoop/lib/commons-io-2.4.jar
/usr/lib/hadoop/lib/jsr305-3.0.0.jar
/usr/lib/hadoop/lib/mockito-all-1.8.5.jar
/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar
/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar
/usr/lib/hadoop/lib/zookeeper.jar
/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.4.jar
/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar
/usr/lib/hadoop/lib/jersey-core-1.9.jar
/usr/lib/hadoop/lib/htrace-core-3.0.4.jar
/usr/lib/hadoop/lib/hamcrest-core-1.3.jar
/usr/lib/hadoop/lib/netty-3.6.2.Final.jar
/usr/lib/hadoop/lib/log4j-1.2.17.jar
/usr/lib/hadoop/lib/paranamer-2.3.jar
/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar
/usr/lib/hadoop/lib/commons-configuration-1.6.jar
/usr/lib/hadoop/lib/asm-3.2.jar
/usr/lib/hadoop/lib/commons-codec-1.4.jar
/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar
/usr/lib/hadoop/lib/jets3t-0.9.0.jar
/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar
/usr/lib/hadoop/lib/avro.jar
/usr/lib/hadoop/lib/xmlenc-0.52.jar
/usr/lib/hadoop/lib/commons-digester-1.8.jar
/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar
/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar
/usr/lib/hadoop/lib/httpclient-4.2.5.jar
/usr/lib/hadoop/lib/commons-httpclient-3.1.jar
/usr/lib/hadoop/lib/guava-11.0.2.jar
/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar
/usr/lib/hadoop/lib/activation-1.1.jar
/usr/lib/hadoop/lib/stax-api-1.0-2.jar
/usr/lib/hadoop/lib/jettison-1.1.jar
/usr/lib/hadoop/lib/httpcore-4.2.5.jar
/usr/lib/hadoop/lib/xz-1.0.jar
/usr/lib/hadoop/lib/aws-java-sdk-1.7.14.jar
/usr/lib/hadoop/lib/jsp-api-2.1.jar
/usr/lib/hadoop/lib/junit-4.11.jar
/usr/lib/hadoop/lib/curator-client-2.7.1.jar
/usr/lib/hadoop/lib/servlet-api-2.5.jar
/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar
/usr/lib/hadoop/lib/logredactor-1.0.3.jar
/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar
/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar
/usr/lib/hadoop/lib/commons-net-3.1.jar
/usr/lib/hadoop/lib/commons-el-1.0.jar
/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar

--
Paul Levy : Pelican Mapping : 703-994-4125

On Mon, Feb 1, 2016 at 5:29 PM, Paul Levy <plevy@xxxxxxxxxxxxxxxxxx> wrote:
Yes, this is Cloudera.

For 1.2, it specifies hadoop 2,2 or better, does it need to be 2.2?  This happens to be 2.6.

HADOOP_HOME is set to /usr/lib/hadoop which does have a common:

bin                                     parquet-avro.jar
client                                  parquet-cascading.jar
client-0.20                             parquet-column.jar
cloudera                                parquet-common.jar
etc                                     parquet-encoding.jar
hadoop-annotations-2.6.0-cdh5.4.9.jar   parquet-format.jar
hadoop-annotations.jar                  parquet-format-javadoc.jar
hadoop-auth-2.6.0-cdh5.4.9.jar          parquet-format-sources.jar
hadoop-auth.jar                         parquet-generator.jar
hadoop-aws-2.6.0-cdh5.4.9.jar           parquet-hadoop-bundle.jar
hadoop-aws.jar                          parquet-hadoop.jar
hadoop-common-2.6.0-cdh5.4.9.jar        parquet-jackson.jar
hadoop-common-2.6.0-cdh5.4.9-tests.jar  parquet-pig-bundle.jar
hadoop-common.jar                       parquet-pig.jar
hadoop-common-tests.jar                 parquet-protobuf.jar
hadoop-nfs-2.6.0-cdh5.4.9.jar           parquet-scala_2.10.jar
hadoop-nfs.jar                          parquet-scrooge_2.10.jar
lib                                     parquet-test-hadoop2.jar
libexec                                 parquet-thrift.jar
LICENSE.txt                             parquet-tools.jar
NOTICE.txt                              sbin

Thank you


--
Paul Levy : Pelican Mapping : 703-994-4125
On Mon, Feb 1, 2016 at 5:23 PM, Jim Hughes <jnh5y@xxxxxxxx> wrote:
Paul,

A +1 to Jeff's suggestion.  There's also the 'geomesa classpath' command which will show the jars which are on the classpath.  If you don't see something like /opt/hadoop/share/hadoop/common/hadoop-common-2.2.0.jar, that'd be the problem.

Cheers,

Jim


On 02/01/2016 05:20 PM, Jeff Gold wrote:
Paul-
  I believe that is because your HADOOP_HOME isn't set correctly and
the Hadoop jars cannot be found (I experienced this recently).  Do you
happen to be using a Cloudera distribution of Hadoop?


-Jeff

On Mon, Feb 1, 2016 at 5:07 PM, Paul Levy <paul.levy@xxxxxxxxx> wrote:
Just finished building the 1.2 snapshot and wanted to test it out.  Any
ideas on this error running test-geomesa after editing it for my cluster
config:

Using GEOMESA_HOME =
/home/vagrant/geomesa/geomesa-dist/target/geomesa-1.2.0-SNAPSHOT/dist/tools/geomesa-tools-1.2.0-SNAPSHOT
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/io/Writable
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at
org.locationtech.geomesa.accumulo.data.AccumuloDataStoreFactory.getDisplayName(AccumuloDataStoreFactory.scala:101)
at
org.geotools.data.DataAccessFinder.getDataStore(DataAccessFinder.java:111)
at org.geotools.data.DataStoreFinder.getDataStore(DataStoreFinder.java:89)
at
org.locationtech.geomesa.tools.DataStoreHelper.getDataStore(DataStoreHelper.scala:36)
at
org.locationtech.geomesa.tools.FeatureCreator$.createFeature(FeatureCreator.scala:22)
at
org.locationtech.geomesa.tools.commands.CreateCommand.execute(CreateCommand.scala:19)
at org.locationtech.geomesa.tools.Runner$.main(Runner.scala:23)
at org.locationtech.geomesa.tools.Runner.main(Runner.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 20 more

Thanks for your time

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from
this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users




_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users

Back to the top