Send geomesa-users mailing list submissions to
geomesa-users@xxxxxxxxxxxxxxxx
To subscribe or unsubscribe via the World Wide Web, visit
https://www.locationtech.org/mailman/listinfo/geomesa-users
or, via email, send a message with subject or body 'help' to
geomesa-users-request@xxxxxxxxxxxxxxxx
You can reach the person managing the list at
geomesa-users-owner@xxxxxxxxxxxxxxxx
When replying, please edit your Subject line so it is more specific
than "Re: Contents of geomesa-users digest..."
Today's Topics:
1. Re: Hortonworks/Geomesa distributed ingestion, error.
(Andrew Morgan)
2. Re: Hortonworks/Geomesa distributed ingestion, error.
(Andrew Morgan)
3. Program using Accumulo backed DataStore won't exit (Bryan Moore)
4. Re: Program using Accumulo backed DataStore won't exit
(Jim Hughes)
----------------------------------------------------------------------
Message: 1
Date: Thu, 28 Jul 2016 15:37:36 +0100
From: Andrew Morgan <andrew@xxxxxxxxxxxx>
To: Jason Brown <jbrown@xxxxxxxx>
Cc: geomesa-users@xxxxxxxxxxxxxxxx
Subject: Re: [geomesa-users] Hortonworks/Geomesa distributed
ingestion, error.
Message-ID: <A560F774-8850-40B4-82E1-AFACF13FDF5E@xxxxxxxxxxxx>
Content-Type: text/plain; charset="utf-8"
I did try this, and I thought it would work.
In my shell scrip that launches the load I included these lines:
myHortonVersion=`hadoop version | grep "^This command" | sed " s/^.*hdp.// " | sed "s/.hadoop.hadoop-common.*//" | sed "s/^/-Dhdp.version=/" `
echo ?determined this is the local hortonworks version:"
echo ${myHortonVersion}
export JAVA_OPTS=${myHortonVersion}
echo "Using this Java Option: ?
echo ${JAVA_OPTS}
geomesa ingest \
-u ${accumulo_user} -p ${accumulo_pw} \
-i ${accumulo_instance_id_param} -z ${zookeeper_param} \
-c myGeomesa.gcam -s gcam-schema \
-C gcam-reader \
hdfs:///user/feeds/gdelt/datastore/GcamGeo/GCAM_201606*.csv
When I run it I still get the same error, pasted below:
[andrew@gzet loadscripts]$ . load_gcam.sh
using these java opts:
-Dhdp.version=2.4.0.0-169
Using this Java Option: -Dhdp.version=2.4.0.0-169
Using GEOMESA_HOME = /home/andrew/data/geo/geomesa-1.2.4/dist/tools/geomesa-tools-1.2.4
Creating schema gcam-schema
Running ingestion in distributed mode
Submitting job - please wait...
Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path
java.lang.IllegalArgumentException: Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path
when I inspect the error further I find the mapreduce.tar.gz that the code is looking for here:
/usr/hdp/2.4.0.0-169/hadoop/mapreduce.tar.gz
We see that we are looking at the wrong path, albeit with the right version embedded in it.
The way the version option is assembled from the JAVA_OPTS into the URI for the file it?s searching for needs adjusting.
Is there a way you can pass that in an an option too?
many thanks with this
Andrew
Andrew J Morgan
CEO, Bytesumo Limited
Tel: +44?(0)7970130767 <>
E-mail: andrew@xxxxxxxxxxxx <mailto:andrew@xxxxxxxxxxxx>
Bytesumo Limited - Registered Company in England and Wales 33 Brodrick Grove, London, SE2 0SR, UK. Company Number: 8505203
On 27 Jul 2016, at 23:17, Andrew Morgan <andrew@xxxxxxxxxxxx> wrote:
James - this is really useful, thanks.
I?m now working through creating valid ingestion configuration routines for my files.
I had some errors in my configurations and I?m tweaking things to fix them.
For now I?m quickly testing using some local test files.
When my converter works properly, which should be shortly, I?ll re-point it at the HDFS data and try a larger ingest of my back history, and report back in.
thanks again!
Andrew
Andrew J Morgan
CEO, Bytesumo Limited
Tel: +44?(0)7970130767 <>
E-mail: andrew@xxxxxxxxxxxx <mailto:andrew@xxxxxxxxxxxx>
Bytesumo Limited - Registered Company in England and Wales 33 Brodrick Grove, London, SE2 0SR, UK. Company Number: 8505203
On 27 Jul 2016, at 20:09, Jason Brown <jbrown@xxxxxxxx <mailto:jbrown@xxxxxxxx>> wrote:
Andrew,
Hi and welcome! We're glad you're up and running and got smoothly this far!
The fix is to set (or append to) an environment variable JAVA_OPTS with the key `-Dhdp.version`. Use hadoop version to get the hdp.version. For example:
$ hadoop version
Hadoop 2.7.1.2.4.2.0-258
Subversion git@xxxxxxxxxx:hortonworks/hadoop.git -r 13debf893a605e8a88df18a7d8d214f571e05289
Compiled by jenkins on 2016-04-25T05:46Z
Compiled with protoc 2.5.0
From source with checksum 2a2d95f05ec6c3ac547ed58cab713ac
This command was run using /usr/hdp/2.4.2.0-258/hadoop/hadoop-common-2.7.1.2.4.2.0-258.jar
We parse as Hadoop 2.7.1 and HDP version 2.4.2.0-258. Note the HDP version appears in both the hadoop version and the directory for hadoop-common.jar. So now set (or append) your JAVA_OPTS.
$ JAVA_OPTS="-Dhdp.version=2.4.2.0-258"
And try your ingest again. Let us know if you run into any additional issues.
-Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.locationtech.org/mhonarc/lists/geomesa-users/attachments/20160728/897dbfa8/attachment.html>
------------------------------
Message: 2
Date: Thu, 28 Jul 2016 16:19:36 +0100
From: Andrew Morgan <andrew@xxxxxxxxxxxx>
To: geomesa-users@xxxxxxxxxxxxxxxx
Subject: [geomesa-users] Re: Hortonworks/Geomesa distributed
ingestion, error.
Message-ID: <8E8F43F6-D42F-43FE-BBCB-4EC8C88EC467@xxxxxxxxxxxx>
Content-Type: text/plain; charset="utf-8"
I did try this, and I thought it would work, but I still have an issue.
In my shell script that launches the load I included these lines:
myHortonVersion=`hadoop version | grep "^This command" | sed " s/^.*hdp.// " | sed "s/.hadoop.hadoop-common.*//" | sed "s/^/-Dhdp.version=/" `
echo ?determined this is the local hortonworks version:"
echo ${myHortonVersion}
export JAVA_OPTS=${myHortonVersion}
echo "Using this Java Option: ?
echo ${JAVA_OPTS}
geomesa ingest \
-u ${accumulo_user} -p ${accumulo_pw} \
-i ${accumulo_instance_id_param} -z ${zookeeper_param} \
-c myGeomesa.gcam -s gcam-schema \
-C gcam-reader \
hdfs:///user/feeds/gdelt/datastore/GcamGeo/GCAM_201606*.csv <hdfs:///user/feeds/gdelt/datastore/GcamGeo/GCAM_201606*.csv>
When I run it I still get the same error, pasted below:
[andrew@gzet loadscripts]$ . load_gcam.sh
using these java opts:
-Dhdp.version=2.4.0.0-169
Using this Java Option: -Dhdp.version=2.4.0.0-169
Using GEOMESA_HOME = /home/andrew/data/geo/geomesa-1.2.4/dist/tools/geomesa-tools-1.2.4
Creating schema gcam-schema
Running ingestion in distributed mode
Submitting job - please wait...
Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path
java.lang.IllegalArgumentException: Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path
when I inspect the error further I find the mapreduce.tar.gz that the code is looking for here:
/usr/hdp/2.4.0.0-169/hadoop/mapreduce.tar.gz
We see that we are looking at the wrong path, albeit with the right version embedded in it.
The way the version option is assembled from the JAVA_OPTS into the URI for the file it?s searching for needs adjusting.
Is there a way to pass the URI path in as an an option too?
many thanks with this
Andrew
Andrew J Morgan
CEO, Bytesumo Limited
E-mail: andrew@xxxxxxxxxxxx <mailto:andrew@xxxxxxxxxxxx>
Bytesumo Limited - Registered Company in England and Wales 33 Brodrick Grove, London, SE2 0SR, UK. Company Number: 8505203
On 27 Jul 2016, at 23:17, Andrew Morgan <andrew@xxxxxxxxxxxx <mailto:andrew@xxxxxxxxxxxx>> wrote:
James - this is really useful, thanks.
I?m now working through creating valid ingestion configuration routines for my files.
I had some errors in my configurations and I?m tweaking things to fix them.
For now I?m quickly testing using some local test files.
When my converter works properly, which should be shortly, I?ll re-point it at the HDFS data and try a larger ingest of my back history, and report back in.
thanks again!
Andrew
Andrew J Morgan
CEO, Bytesumo Limited
Tel: +44?(0)7970130767 <>
E-mail: andrew@xxxxxxxxxxxx <mailto:andrew@xxxxxxxxxxxx>
Bytesumo Limited - Registered Company in England and Wales 33 Brodrick Grove, London, SE2 0SR, UK. Company Number: 8505203
On 27 Jul 2016, at 20:09, Jason Brown <jbrown@xxxxxxxx <mailto:jbrown@xxxxxxxx>> wrote:
Andrew,
Hi and welcome! We're glad you're up and running and got smoothly this far!
The fix is to set (or append to) an environment variable JAVA_OPTS with the key `-Dhdp.version`. Use hadoop version to get the hdp.version. For example:
$ hadoop version
Hadoop 2.7.1.2.4.2.0-258
Subversion git@xxxxxxxxxx:hortonworks/hadoop.git -r 13debf893a605e8a88df18a7d8d214f571e05289
Compiled by jenkins on 2016-04-25T05:46Z
Compiled with protoc 2.5.0
From source with checksum 2a2d95f05ec6c3ac547ed58cab713ac
This command was run using /usr/hdp/2.4.2.0-258/hadoop/hadoop-common-2.7.1.2.4.2.0-258.jar
We parse as Hadoop 2.7.1 and HDP version 2.4.2.0-258. Note the HDP version appears in both the hadoop version and the directory for hadoop-common.jar. So now set (or append) your JAVA_OPTS.
$ JAVA_OPTS="-Dhdp.version=2.4.2.0-258"
And try your ingest again. Let us know if you run into any additional issues.
-Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.locationtech.org/mhonarc/lists/geomesa-users/attachments/20160728/7054f183/attachment.html>
------------------------------
Message: 3
Date: Fri, 29 Jul 2016 11:15:34 -0400
From: Bryan Moore <bryan@xxxxxxxxxxxxxx>
To: geomesa-users@xxxxxxxxxxxxxxxx
Subject: [geomesa-users] Program using Accumulo backed DataStore won't
exit
Message-ID: <d052a8a1-5c88-675f-cd35-28bf31e4e77f@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
I've written a program using GeoMesa with an Accumulo backed DataStore
that works fine but won't exit. Below is a minimal program that
illustrates the problem. It prints the "Start" and "Finish" messages
but doesn't exit.
Have I done something wrong, not done something I need to do, or is this
a bug?
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import org.apache.accumulo.core.client.AccumuloException;
import org.apache.accumulo.core.client.AccumuloSecurityException;
import org.geotools.data.DataStoreFinder;
import org.locationtech.geomesa.accumulo.data.AccumuloDataStore;
public class Minimal {
public static void main(String[] args)
throws AccumuloException, AccumuloSecurityException,
IOException {
System.out.println("Start");
Map<String, String> dsConf = new HashMap<>();
dsConf.put("instanceId", "myinstancename");
dsConf.put("zookeepers", "localhost:2181");
dsConf.put("user", "myuserid");
dsConf.put("password", "mypassword");
dsConf.put("tableName", "mysearchtable");
dsConf.put("auths", "");
AccumuloDataStore dataStore = (AccumuloDataStore)
DataStoreFinder.getDataStore(dsConf);
dataStore.dispose();
System.out.println("Finish");
}
}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.locationtech.org/mhonarc/lists/geomesa-users/attachments/20160729/855e74d2/attachment.html>
------------------------------
Message: 4
Date: Fri, 29 Jul 2016 11:43:29 -0400
From: Jim Hughes <jnh5y@xxxxxxxx>
To: geomesa-users@xxxxxxxxxxxxxxxx
Subject: Re: [geomesa-users] Program using Accumulo backed DataStore
won't exit
Message-ID: <579B79A1.9010806@xxxxxxxx>
Content-Type: text/plain; charset="windows-1252"; Format="flowed"
Hi Bryan,
Which version of GeoMesa are you using? There is a known issue with
GeoMesa 1.2.3 where a thread for pre-computed stats writing is not
shutdown. We believe we addressed this in 1.2.4.
In terms of helping diagnose the problem, can you run jstack on the
hanging JVM and look for anything notable in the output?
Thanks,
Jim
On 07/29/2016 11:15 AM, Bryan Moore wrote:
I've written a program using GeoMesa with an Accumulo backed DataStore
that works fine but won't exit. Below is a minimal program that
illustrates the problem. It prints the "Start" and "Finish" messages
but doesn't exit.
Have I done something wrong, not done something I need to do, or is
this a bug?
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import org.apache.accumulo.core.client.AccumuloException;
import org.apache.accumulo.core.client.AccumuloSecurityException;
import org.geotools.data.DataStoreFinder;
import org.locationtech.geomesa.accumulo.data.AccumuloDataStore;
public class Minimal {
public static void main(String[] args)
throws AccumuloException, AccumuloSecurityException,
IOException {
System.out.println("Start");
Map<String, String> dsConf = new HashMap<>();
dsConf.put("instanceId", "myinstancename");
dsConf.put("zookeepers", "localhost:2181");
dsConf.put("user", "myuserid");
dsConf.put("password", "mypassword");
dsConf.put("tableName", "mysearchtable");
dsConf.put("auths", "");
AccumuloDataStore dataStore = (AccumuloDataStore)
DataStoreFinder.getDataStore(dsConf);
dataStore.dispose();
System.out.println("Finish");
}
}
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geomesa-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.locationtech.org/mhonarc/lists/geomesa-users/attachments/20160729/3bce0bce/attachment.html>
------------------------------
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geomesa-users
End of geomesa-users Digest, Vol 29, Issue 19
*********************************************
