[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [geomesa-users] Hortonworks/Geomesa distributed ingestion, error.
|
Hi Jason,
It's a good idea. I'm going to dig into this and drop you a line when
I test out some things and find out more.
A
Sent from my iPhone
> On 29 Jul 2016, at 17:26, Jason Brown <jbrown@xxxxxxxx> wrote:
>
> Andrew,
>
> Can you check the value of `mapreduce.application.classpath` in mapred-default.xml? If that's not set, I would use the output of `hadoop classpath` as a first guess.
>
> -Jason
>
>
>> On 07/29/2016 12:00 PM, geomesa-users-request@xxxxxxxxxxxxxxxx wrote:
>> Send geomesa-users mailing list submissions to
>> geomesa-users@xxxxxxxxxxxxxxxx
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> https://www.locationtech.org/mailman/listinfo/geomesa-users
>> or, via email, send a message with subject or body 'help' to
>> geomesa-users-request@xxxxxxxxxxxxxxxx
>>
>> You can reach the person managing the list at
>> geomesa-users-owner@xxxxxxxxxxxxxxxx
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of geomesa-users digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Re: Hortonworks/Geomesa distributed ingestion, error.
>> (Andrew Morgan)
>> 2. Re: Hortonworks/Geomesa distributed ingestion, error.
>> (Andrew Morgan)
>> 3. Program using Accumulo backed DataStore won't exit (Bryan Moore)
>> 4. Re: Program using Accumulo backed DataStore won't exit
>> (Jim Hughes)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Thu, 28 Jul 2016 15:37:36 +0100
>> From: Andrew Morgan <andrew@xxxxxxxxxxxx>
>> To: Jason Brown <jbrown@xxxxxxxx>
>> Cc: geomesa-users@xxxxxxxxxxxxxxxx
>> Subject: Re: [geomesa-users] Hortonworks/Geomesa distributed
>> ingestion, error.
>> Message-ID: <A560F774-8850-40B4-82E1-AFACF13FDF5E@xxxxxxxxxxxx>
>> Content-Type: text/plain; charset="utf-8"
>>
>>
>> I did try this, and I thought it would work.
>>
>> In my shell scrip that launches the load I included these lines:
>>
>>
>> myHortonVersion=`hadoop version | grep "^This command" | sed " s/^.*hdp.// " | sed "s/.hadoop.hadoop-common.*//" | sed "s/^/-Dhdp.version=/" `
>>
>> echo ?determined this is the local hortonworks version:"
>> echo ${myHortonVersion}
>>
>> export JAVA_OPTS=${myHortonVersion}
>> echo "Using this Java Option: ?
>> echo ${JAVA_OPTS}
>>
>>
>> geomesa ingest \
>> -u ${accumulo_user} -p ${accumulo_pw} \
>> -i ${accumulo_instance_id_param} -z ${zookeeper_param} \
>> -c myGeomesa.gcam -s gcam-schema \
>> -C gcam-reader \
>> hdfs:///user/feeds/gdelt/datastore/GcamGeo/GCAM_201606*.csv
>>
>>
>> When I run it I still get the same error, pasted below:
>>
>> [andrew@gzet loadscripts]$ . load_gcam.sh
>> using these java opts:
>> -Dhdp.version=2.4.0.0-169
>> Using this Java Option: -Dhdp.version=2.4.0.0-169
>> Using GEOMESA_HOME = /home/andrew/data/geo/geomesa-1.2.4/dist/tools/geomesa-tools-1.2.4
>> Creating schema gcam-schema
>> Running ingestion in distributed mode
>> Submitting job - please wait...
>> Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path
>> java.lang.IllegalArgumentException: Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path
>>
>>
>> when I inspect the error further I find the mapreduce.tar.gz that the code is looking for here:
>>
>> /usr/hdp/2.4.0.0-169/hadoop/mapreduce.tar.gz
>>
>>
>> We see that we are looking at the wrong path, albeit with the right version embedded in it.
>> The way the version option is assembled from the JAVA_OPTS into the URI for the file it?s searching for needs adjusting.
>>
>> Is there a way you can pass that in an an option too?
>>
>> many thanks with this
>> Andrew
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Andrew J Morgan
>> CEO, Bytesumo Limited
>> Tel: +44?(0)7970130767 <>
>> E-mail: andrew@xxxxxxxxxxxx <mailto:andrew@xxxxxxxxxxxx>
>>
>> Bytesumo Limited - Registered Company in England and Wales 33 Brodrick Grove, London, SE2 0SR, UK. Company Number: 8505203
>>
>>> On 27 Jul 2016, at 23:17, Andrew Morgan <andrew@xxxxxxxxxxxx> wrote:
>>>
>>> James - this is really useful, thanks.
>>>
>>> I?m now working through creating valid ingestion configuration routines for my files.
>>> I had some errors in my configurations and I?m tweaking things to fix them.
>>> For now I?m quickly testing using some local test files.
>>>
>>> When my converter works properly, which should be shortly, I?ll re-point it at the HDFS data and try a larger ingest of my back history, and report back in.
>>>
>>> thanks again!
>>> Andrew
>>>
>>>
>>>
>>>
>>> Andrew J Morgan
>>> CEO, Bytesumo Limited
>>> Tel: +44?(0)7970130767 <>
>>> E-mail: andrew@xxxxxxxxxxxx <mailto:andrew@xxxxxxxxxxxx>
>>>
>>> Bytesumo Limited - Registered Company in England and Wales 33 Brodrick Grove, London, SE2 0SR, UK. Company Number: 8505203
>>>
>>>> On 27 Jul 2016, at 20:09, Jason Brown <jbrown@xxxxxxxx <mailto:jbrown@xxxxxxxx>> wrote:
>>>>
>>>> Andrew,
>>>>
>>>> Hi and welcome! We're glad you're up and running and got smoothly this far!
>>>>
>>>> The fix is to set (or append to) an environment variable JAVA_OPTS with the key `-Dhdp.version`. Use hadoop version to get the hdp.version. For example:
>>>>
>>>> $ hadoop version
>>>>
>>>> Hadoop 2.7.1.2.4.2.0-258
>>>> Subversion git@xxxxxxxxxx:hortonworks/hadoop.git -r 13debf893a605e8a88df18a7d8d214f571e05289
>>>> Compiled by jenkins on 2016-04-25T05:46Z
>>>> Compiled with protoc 2.5.0
>>>> From source with checksum 2a2d95f05ec6c3ac547ed58cab713ac
>>>> This command was run using /usr/hdp/2.4.2.0-258/hadoop/hadoop-common-2.7.1.2.4.2.0-258.jar
>>>>
>>>> We parse as Hadoop 2.7.1 and HDP version 2.4.2.0-258. Note the HDP version appears in both the hadoop version and the directory for hadoop-common.jar. So now set (or append) your JAVA_OPTS.
>>>>
>>>> $ JAVA_OPTS="-Dhdp.version=2.4.2.0-258"
>>>>
>>>> And try your ingest again. Let us know if you run into any additional issues.
>>>>
>>>> -Jason
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <https://www.locationtech.org/mhonarc/lists/geomesa-users/attachments/20160728/897dbfa8/attachment.html>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Thu, 28 Jul 2016 16:19:36 +0100
>> From: Andrew Morgan <andrew@xxxxxxxxxxxx>
>> To: geomesa-users@xxxxxxxxxxxxxxxx
>> Subject: [geomesa-users] Re: Hortonworks/Geomesa distributed
>> ingestion, error.
>> Message-ID: <8E8F43F6-D42F-43FE-BBCB-4EC8C88EC467@xxxxxxxxxxxx>
>> Content-Type: text/plain; charset="utf-8"
>>
>> I did try this, and I thought it would work, but I still have an issue.
>>
>> In my shell script that launches the load I included these lines:
>>
>>
>> myHortonVersion=`hadoop version | grep "^This command" | sed " s/^.*hdp.// " | sed "s/.hadoop.hadoop-common.*//" | sed "s/^/-Dhdp.version=/" `
>>
>> echo ?determined this is the local hortonworks version:"
>> echo ${myHortonVersion}
>>
>> export JAVA_OPTS=${myHortonVersion}
>> echo "Using this Java Option: ?
>> echo ${JAVA_OPTS}
>>
>>
>> geomesa ingest \
>> -u ${accumulo_user} -p ${accumulo_pw} \
>> -i ${accumulo_instance_id_param} -z ${zookeeper_param} \
>> -c myGeomesa.gcam -s gcam-schema \
>> -C gcam-reader \
>> hdfs:///user/feeds/gdelt/datastore/GcamGeo/GCAM_201606*.csv <hdfs:///user/feeds/gdelt/datastore/GcamGeo/GCAM_201606*.csv>
>>
>>
>> When I run it I still get the same error, pasted below:
>>
>> [andrew@gzet loadscripts]$ . load_gcam.sh
>> using these java opts:
>> -Dhdp.version=2.4.0.0-169
>> Using this Java Option: -Dhdp.version=2.4.0.0-169
>> Using GEOMESA_HOME = /home/andrew/data/geo/geomesa-1.2.4/dist/tools/geomesa-tools-1.2.4
>> Creating schema gcam-schema
>> Running ingestion in distributed mode
>> Submitting job - please wait...
>> Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path
>> java.lang.IllegalArgumentException: Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path
>>
>>
>> when I inspect the error further I find the mapreduce.tar.gz that the code is looking for here:
>>
>> /usr/hdp/2.4.0.0-169/hadoop/mapreduce.tar.gz
>>
>>
>> We see that we are looking at the wrong path, albeit with the right version embedded in it.
>> The way the version option is assembled from the JAVA_OPTS into the URI for the file it?s searching for needs adjusting.
>>
>> Is there a way to pass the URI path in as an an option too?
>>
>> many thanks with this
>> Andrew
>>
>>
>> Andrew J Morgan
>> CEO, Bytesumo Limited
>> E-mail: andrew@xxxxxxxxxxxx <mailto:andrew@xxxxxxxxxxxx>
>>
>> Bytesumo Limited - Registered Company in England and Wales 33 Brodrick Grove, London, SE2 0SR, UK. Company Number: 8505203
>>>> On 27 Jul 2016, at 23:17, Andrew Morgan <andrew@xxxxxxxxxxxx <mailto:andrew@xxxxxxxxxxxx>> wrote:
>>>>
>>>> James - this is really useful, thanks.
>>>>
>>>> I?m now working through creating valid ingestion configuration routines for my files.
>>>> I had some errors in my configurations and I?m tweaking things to fix them.
>>>> For now I?m quickly testing using some local test files.
>>>>
>>>> When my converter works properly, which should be shortly, I?ll re-point it at the HDFS data and try a larger ingest of my back history, and report back in.
>>>>
>>>> thanks again!
>>>> Andrew
>>>>
>>>>
>>>>
>>>>
>>>> Andrew J Morgan
>>>> CEO, Bytesumo Limited
>>>> Tel: +44?(0)7970130767 <>
>>>> E-mail: andrew@xxxxxxxxxxxx <mailto:andrew@xxxxxxxxxxxx>
>>>>
>>>> Bytesumo Limited - Registered Company in England and Wales 33 Brodrick Grove, London, SE2 0SR, UK. Company Number: 8505203
>>>>
>>>>> On 27 Jul 2016, at 20:09, Jason Brown <jbrown@xxxxxxxx <mailto:jbrown@xxxxxxxx>> wrote:
>>>>>
>>>>> Andrew,
>>>>>
>>>>> Hi and welcome! We're glad you're up and running and got smoothly this far!
>>>>>
>>>>> The fix is to set (or append to) an environment variable JAVA_OPTS with the key `-Dhdp.version`. Use hadoop version to get the hdp.version. For example:
>>>>>
>>>>> $ hadoop version
>>>>>
>>>>> Hadoop 2.7.1.2.4.2.0-258
>>>>> Subversion git@xxxxxxxxxx:hortonworks/hadoop.git -r 13debf893a605e8a88df18a7d8d214f571e05289
>>>>> Compiled by jenkins on 2016-04-25T05:46Z
>>>>> Compiled with protoc 2.5.0
>>>>> From source with checksum 2a2d95f05ec6c3ac547ed58cab713ac
>>>>> This command was run using /usr/hdp/2.4.2.0-258/hadoop/hadoop-common-2.7.1.2.4.2.0-258.jar
>>>>>
>>>>> We parse as Hadoop 2.7.1 and HDP version 2.4.2.0-258. Note the HDP version appears in both the hadoop version and the directory for hadoop-common.jar. So now set (or append) your JAVA_OPTS.
>>>>>
>>>>> $ JAVA_OPTS="-Dhdp.version=2.4.2.0-258"
>>>>>
>>>>> And try your ingest again. Let us know if you run into any additional issues.
>>>>>
>>>>> -Jason
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <https://www.locationtech.org/mhonarc/lists/geomesa-users/attachments/20160728/7054f183/attachment.html>
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Fri, 29 Jul 2016 11:15:34 -0400
>> From: Bryan Moore <bryan@xxxxxxxxxxxxxx>
>> To: geomesa-users@xxxxxxxxxxxxxxxx
>> Subject: [geomesa-users] Program using Accumulo backed DataStore won't
>> exit
>> Message-ID: <d052a8a1-5c88-675f-cd35-28bf31e4e77f@xxxxxxxxxxxxxx>
>> Content-Type: text/plain; charset="utf-8"; Format="flowed"
>>
>> I've written a program using GeoMesa with an Accumulo backed DataStore
>> that works fine but won't exit. Below is a minimal program that
>> illustrates the problem. It prints the "Start" and "Finish" messages
>> but doesn't exit.
>>
>> Have I done something wrong, not done something I need to do, or is this
>> a bug?
>>
>>
>> import java.io.IOException;
>> import java.util.HashMap;
>> import java.util.Map;
>>
>> import org.apache.accumulo.core.client.AccumuloException;
>> import org.apache.accumulo.core.client.AccumuloSecurityException;
>> import org.geotools.data.DataStoreFinder;
>> import org.locationtech.geomesa.accumulo.data.AccumuloDataStore;
>>
>> public class Minimal {
>>
>> public static void main(String[] args)
>> throws AccumuloException, AccumuloSecurityException,
>> IOException {
>> System.out.println("Start");
>> Map<String, String> dsConf = new HashMap<>();
>> dsConf.put("instanceId", "myinstancename");
>> dsConf.put("zookeepers", "localhost:2181");
>> dsConf.put("user", "myuserid");
>> dsConf.put("password", "mypassword");
>> dsConf.put("tableName", "mysearchtable");
>> dsConf.put("auths", "");
>> AccumuloDataStore dataStore = (AccumuloDataStore)
>> DataStoreFinder.getDataStore(dsConf);
>> dataStore.dispose();
>> System.out.println("Finish");
>> }
>> }
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <https://www.locationtech.org/mhonarc/lists/geomesa-users/attachments/20160729/855e74d2/attachment.html>
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Fri, 29 Jul 2016 11:43:29 -0400
>> From: Jim Hughes <jnh5y@xxxxxxxx>
>> To: geomesa-users@xxxxxxxxxxxxxxxx
>> Subject: Re: [geomesa-users] Program using Accumulo backed DataStore
>> won't exit
>> Message-ID: <579B79A1.9010806@xxxxxxxx>
>> Content-Type: text/plain; charset="windows-1252"; Format="flowed"
>>
>> Hi Bryan,
>>
>> Which version of GeoMesa are you using? There is a known issue with
>> GeoMesa 1.2.3 where a thread for pre-computed stats writing is not
>> shutdown. We believe we addressed this in 1.2.4.
>>
>> In terms of helping diagnose the problem, can you run jstack on the
>> hanging JVM and look for anything notable in the output?
>>
>> Thanks,
>>
>> Jim
>>
>>> On 07/29/2016 11:15 AM, Bryan Moore wrote:
>>> I've written a program using GeoMesa with an Accumulo backed DataStore
>>> that works fine but won't exit. Below is a minimal program that
>>> illustrates the problem. It prints the "Start" and "Finish" messages
>>> but doesn't exit.
>>>
>>> Have I done something wrong, not done something I need to do, or is
>>> this a bug?
>>>
>>>
>>> import java.io.IOException;
>>> import java.util.HashMap;
>>> import java.util.Map;
>>>
>>> import org.apache.accumulo.core.client.AccumuloException;
>>> import org.apache.accumulo.core.client.AccumuloSecurityException;
>>> import org.geotools.data.DataStoreFinder;
>>> import org.locationtech.geomesa.accumulo.data.AccumuloDataStore;
>>>
>>> public class Minimal {
>>>
>>> public static void main(String[] args)
>>> throws AccumuloException, AccumuloSecurityException,
>>> IOException {
>>> System.out.println("Start");
>>> Map<String, String> dsConf = new HashMap<>();
>>> dsConf.put("instanceId", "myinstancename");
>>> dsConf.put("zookeepers", "localhost:2181");
>>> dsConf.put("user", "myuserid");
>>> dsConf.put("password", "mypassword");
>>> dsConf.put("tableName", "mysearchtable");
>>> dsConf.put("auths", "");
>>> AccumuloDataStore dataStore = (AccumuloDataStore)
>>> DataStoreFinder.getDataStore(dsConf);
>>> dataStore.dispose();
>>> System.out.println("Finish");
>>> }
>>> }
>>>
>>>
>>>
>>> _______________________________________________
>>> geomesa-users mailing list
>>> geomesa-users@xxxxxxxxxxxxxxxx
>>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>>> https://www.locationtech.org/mailman/listinfo/geomesa-users
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <https://www.locationtech.org/mhonarc/lists/geomesa-users/attachments/20160729/3bce0bce/attachment.html>
>>
>> ------------------------------
>>
>> _______________________________________________
>> geomesa-users mailing list
>> geomesa-users@xxxxxxxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://www.locationtech.org/mailman/listinfo/geomesa-users
>>
>> End of geomesa-users Digest, Vol 29, Issue 19
>> *********************************************
>
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://www.locationtech.org/mailman/listinfo/geomesa-users
