| James - this is really useful, thanks.
I’m now working through creating valid ingestion configuration routines for my files. I had some errors in my configurations and I’m tweaking things to fix them. For now I’m quickly testing using some local test files.
When my converter works properly, which should be shortly, I’ll re-point it at the HDFS data and try a larger ingest of my back history, and report back in.
thanks again! Andrew
Andrew J Morgan
CEO, Bytesumo Limited
Tel: +44 (0)7970130767
E-mail: andrew@xxxxxxxxxxxx
Bytesumo Limited - Registered Company in England and Wales 33 Brodrick Grove, London, SE2 0SR, UK. Company Number: 8505203
Andrew,
Hi and welcome! We're glad you're up and running and got smoothly this far!
The fix is to set (or append to) an environment variable JAVA_OPTS with the key `-Dhdp.version`. Use hadoop version to get the hdp.version. For example:
$ hadoop version
Hadoop 2.7.1.2.4.2.0-258
Subversion git@xxxxxxxxxx:hortonworks/hadoop.git -r 13debf893a605e8a88df18a7d8d214f571e05289
Compiled by jenkins on 2016-04-25T05:46Z
Compiled with protoc 2.5.0
From source with checksum 2a2d95f05ec6c3ac547ed58cab713ac
This command was run using /usr/hdp/2.4.2.0-258/hadoop/hadoop-common-2.7.1.2.4.2.0-258.jar
We parse as Hadoop 2.7.1 and HDP version 2.4.2.0-258. Note the HDP version appears in both the hadoop version and the directory for hadoop-common.jar. So now set (or append) your JAVA_OPTS.
$ JAVA_OPTS="-Dhdp.version=2.4.2.0-258"
And try your ingest again. Let us know if you run into any additional issues.
-Jason
|
