[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [geomesa-users] problem querying indexed data out with multi IN or = cql clause
|
Jim and Emilio,
Thanks for the quick responses, very appreciated. We came up with short term work arounds (re-ingest without the indexes or blow the GeoServer layers out which I didn't want to do) but may not need them!
It takes me a little to move code to the real system that had data sets that showed the issues but I will try to get the snapshot moved this week to test it out.
Regards,
Diane
------------------------------
Hi Diane,
I merged Emilio's PR earlier today, and made sure the Hudson snapshot build on master and deployed updated 1.2.2-SNAPSHOST artifacts to LocationTech's Nexus. All that means as an alternative to build GeoMesa locally, you can depend on 1.2.2-SNAPSHOT in a Maven project to try out the fix.
If you use the GeoMesa distribution bundle, you can download a snapshot
here:
https://repo.locationtech.org/content/repositories/geomesa-snapshots/org/locationtech/geomesa/geomesa-dist/1.2.2-SNAPSHOT/geomesa-dist-1.2.2-20160516.220910-22-bin.tar.gz
Thanks for the clear bug report!
Cheers,
Jim
On 05/16/2016 02:26 PM, Emilio Lahr-Vivaz wrote:
> Hi Diane,
>
> I believe the issue has already been fixed in the latest code. There
> were still some similar issues but they probably won't affect you -
> I've put up a fix but it's not merged yet. If you are comfortable
> building and deploying from source, the current master branch should
> work for you. If not, then we should have a release for you soon.
>
> Thanks,
>
> Emilio
>
> On 05/16/2016 11:45 AM, Emilio Lahr-Vivaz wrote:
>> Hi Diane,
>>
>> I've recreated this issue with a unit test - as you say, OR'd
>> attribute predicates AND'd with a geometry query fail to return any
>> records. We're going to be releasing version 1.2.2 in the next few
>> days - I'll make sure a fix gets included.
>>
>> Thanks,
>>
>> Emilio
>>
>> On 05/16/2016 10:29 AM, Diane Griffith wrote:
>>>
>>> We are on GeoMesa 1.2.0 on top of accumulo 1.6 with GeoServer 2.8.1.
>>> We started experiencing issues combining complex cql quleries
>>> through GeoServer for sure once we started applying more attribute
>>> indexes on our data.
>>>
>>> What seems to break down was when we did multiple IN or = cql
>>> queries through one geoserver call to the geomesa stack. Generic
>>> examples of what I?m talking about that could fail would be against
>>> a column defined as a String (I?ll call it Keyword) with index=full
>>> with the cql looking something like the following 2 examples:
>>>
>>> 1.(Keyword IN (?test, ?foo?))
>>>
>>> 2.((Keyword = ?test?) OR (Keyword = ?foo?))
>>>
>>> If each was queried separately through geoserver for the certain
>>> bounding box it works. But as soon as we combine each working
>>> IN/equal match together with the same bounding box it returns no
>>> matches via the GeoServer GetMap call.
>>>
>>> Also on the same dataset a query that could succeed was something like:
>>>
>>> ((Keyword LIKE ?foo%?) OR (Keyword IN (?test?))
>>>
>>> I knew I did combine other queries so I backed it off to a simple
>>> test dataset where I had not indexed any additional attribute
>>> columns. So when I did similar string tests like the 2 that failed
>>> above, they worked against a non-indexed string attribute column. I
>>> had no issues combining cql on admittedly non-indexed Double
>>> columns. Unclear if I indexed them if I would experience similar
>>> combined CQL issues.
>>>
>>> Is there a known issue doing a combined/multiple IN matches or =
>>> matches on an indexed string attribute column?
>>>
>>> Is there further things that anyone can suggest I look at to
>>> determine why a multi IN clause or = clause fails on an indexed
>>> string attribute column?
>>>
>>> Thanks,
>>>
>>> Diane Griffith
>>>
>>>
>>>
>>> _______________________________________________
>>> geomesa-users mailing list
>>> geomesa-users@xxxxxxxxxxxxxxxx
>>> To change your delivery options, retrieve your password, or
>>> unsubscribe from this list, visit
>>> https://www.locationtech.org/mailman/listinfo/geomesa-users
>>
>>
>>
>> _______________________________________________
>> geomesa-users mailing list
>> geomesa-users@xxxxxxxxxxxxxxxx
>> To change your delivery options, retrieve your password, or
>> unsubscribe from this list, visit
>> https://www.locationtech.org/mailman/listinfo/geomesa-users
>
>
>
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or
> unsubscribe from this list, visit
> https://www.locationtech.org/mailman/listinfo/geomesa-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.locationtech.org/mhonarc/lists/geomesa-users/attachments/20160516/0b0587d9/attachment.html>
------------------------------
Message: 3
Date: Tue, 17 May 2016 14:24:48 +0100
From: Jo Walsh <metazool@xxxxxxxxx>
To: geomesa-users@xxxxxxxxxxxxxxxx
Subject: [geomesa-users] CQLException on 'id' with Geomesa Geoserver
plugin
Message-ID:
<CAOdP5FyhnE=ZBbB-UZA44Y296O4ixPeSOyFK4+zKWLCvun9QaA@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset="utf-8"
dear all,
I'm evaluating Geomesa/Accumulo as a backend for Geoserver and so far, so reasonable, however I have come across an issue after ingesting a shapefile with a column named 'id' and then trying to access that data source as a WFS layer in Geoserver - Geomesa causes Geotools raises a CQLException on the "id" field. I thought it worth raising here to see if others have come across the problem, what insight others might have into the root causes, and whether this is in fact the right forum for the issue.
I found a couple of what seem to be related tickets in Geotools/Geoserver but they are both resolved:
https://osgeo-org.atlassian.net/browse/GEOT-2574
https://osgeo-org.atlassian.net/browse/GEOS-6634
And there's an older, also fixed geomesa issue referencing the ID filter in
ECQL: https://locationtech.org/mhonarc/lists/geomesa-users/msg00195.html
Here's the contents of geoserver.log where geomesa is triggering the error:
org.geoserver.wfs.WFSException: Error occurred getting features ...
Caused by: java.lang.IllegalArgumentException: Unable to parse expression id=id org.geotools.filter.text.cql2.CQLException: Encountered "id" at line 1, column 1.
Was expecting one of:
<STRING_LITERAL> ...
"true" ...
"false" ...
"point" ...
"linestring" ...
"polygon" ...
"multipoint" ...
"multilinestring" ...
"multipolygon" ...
"geometrycollection" ...
"envelope" ...
"(" ...
"[" ...
"-" ...
<INTEGER_LITERAL> ...
<FLOATING_LITERAL> ...
<DATE> ...
<DATE_TIME> ...
<IDENTIFIER> ...
Parsing : id.
at
org.geotools.process.vector.TransformProcess.toDefinition(TransformProcess.java:185)
at
org.locationtech.geomesa.accumulo.index.QueryPlanner$.setQueryTransforms(QueryPlanner.scala:324)
at
org.locationtech.geomesa.accumulo.index.QueryPlanner$.configureQuery(QueryPlanner.scala:228)
at
org.locationtech.geomesa.accumulo.data.AccumuloFeatureCollection.getSchema(AccumuloFeatureSource.scala:115)
at
org.locationtech.geomesa.accumulo.data.AccumuloFeatureCollection.getSchema(AccumuloFeatureSource.scala:105)
So it's happening here:
https://github.com/locationtech/geomesa/blob/master/geomesa-accumulo/geomesa-accumulo-datastore/src/main/scala/org/locationtech/geomesa/accumulo/index/QueryPlanner.scala#L325
https://github.com/geotools/geotools/blob/master/modules/library/cql/ECQL.md
states that ID is a predicate / reserved word in ECQL.
This is only happening if I ingest a shapefile with a column named 'id' - i can replace the column name, re-ingest and the same data appears fine via WFS.
cheers,
Jo
--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.locationtech.org/mhonarc/lists/geomesa-users/attachments/20160517/dcb0309f/attachment.html>
------------------------------
Message: 4
Date: Tue, 17 May 2016 10:50:24 -0400
From: Chris Eichelberger <cne1x@xxxxxxxx>
To: Geomesa User discussions <geomesa-users@xxxxxxxxxxxxxxxx>
Subject: Re: [geomesa-users] CQLException on 'id' with Geomesa
Geoserver plugin
Message-ID: <1463496624.29876.67.camel@xxxxxxxx>
Content-Type: text/plain; charset="UTF-8"
Jo,
Ah! ?This is an issue that's bitten us before, so 1) you're in good company; 2) this is exactly the right forum for your question.
It turns out that "id" is a reserved word in GeoTools: ?https://github.
com/geotools/geotools/blob/master/modules/library/cql/src/main/jjtree/E
CQLGrammar.jjt#L428
We do have an outstanding ticket for GeoMesa to warn users when ingesting using reserved words: ?https://geomesa.atlassian.net/browse/G
EOMESA-1028? We hope to make some headway on this as part of the upcoming 1.2.2 release of GeoMesa.
In the interim, there are good tools that can help you rename attributes in a Shapefile. ?For example, see this question about
ogr2ogr: ?http://gis.stackexchange.com/questions/58541/how-to-rename-fi
eld-names-in-a-shapefile-from-the-commandline??
Thanks!
Sincerely,
? -- Chris
On Tue, 2016-05-17 at 14:24 +0100, Jo Walsh wrote:
> dear all,
>
> I'm evaluating Geomesa/Accumulo as a backend for Geoserver and so far,
> so reasonable, however I have come across an issue after ingesting a
> shapefile with a column named 'id' and then trying to access that data
> source as a WFS layer in Geoserver - Geomesa causes Geotools raises a
> CQLException on the "id" field. I thought it worth raising here to see
> if others have come across the problem, what insight others might have
> into the root causes, and whether this is in fact the right forum for
> the issue.
>
> I found a couple of what seem to be related tickets in
> Geotools/Geoserver but they are both resolved:
> https://osgeo-org.atlassian.net/browse/GEOT-2574
> https://osgeo-org.atlassian.net/browse/GEOS-6634
>
> And there's an older, also fixed geomesa issue referencing the ID
> filter in ECQL: https://locationtech.org/mhonarc/lists/geomesa-users/
> msg00195.html?
>
> Here's the contents of geoserver.log where geomesa is triggering the
> error:
>
> org.geoserver.wfs.WFSException: Error occurred getting features ...
> Caused by: java.lang.IllegalArgumentException: Unable to parse
> expression id=id org.geotools.filter.text.cql2.CQLException:
> Encountered "id" at line 1, column 1.
> Was expecting one of:
> ? ? <STRING_LITERAL> ...
> ? ? "true" ...
> ? ? "false" ...
> ? ? "point" ...
> ? ? "linestring" ...
> ? ? "polygon" ...
> ? ? "multipoint" ...
> ? ? "multilinestring" ...
> ? ? "multipolygon" ...
> ? ? "geometrycollection" ...
> ? ? "envelope" ...
> ? ? "(" ...
> ? ? "[" ...
> ? ? "-" ...
> ? ? <INTEGER_LITERAL> ...
> ? ? <FLOATING_LITERAL> ...
> ? ? <DATE> ...
> ? ? <DATE_TIME> ...
> ? ? <IDENTIFIER> ...
> ? ? ?Parsing : id.
> ? ? ? ? at
> org.geotools.process.vector.TransformProcess.toDefinition(TransformPr
> ocess.java:185)
> ? ? ? ? at
> org.locationtech.geomesa.accumulo.index.QueryPlanner$.setQueryTransfo
> rms(QueryPlanner.scala:324)
> ? ? ? ? at
> org.locationtech.geomesa.accumulo.index.QueryPlanner$.configureQuery(
> QueryPlanner.scala:228)
> ? ? ? ? at
> org.locationtech.geomesa.accumulo.data.AccumuloFeatureCollection.getS
> chema(AccumuloFeatureSource.scala:115)
> ? ? ? ? at
> org.locationtech.geomesa.accumulo.data.AccumuloFeatureCollection.getS
> chema(AccumuloFeatureSource.scala:105)
>
> So it's happening here:
> https://github.com/locationtech/geomesa/blob/master/geomesa-accumulo/
> geomesa-accumulo-
> datastore/src/main/scala/org/locationtech/geomesa/accumulo/index/Quer
> yPlanner.scala#L325
>
> https://github.com/geotools/geotools/blob/master/modules/library/cql/
> ECQL.md
> states that ID is a predicate / reserved word in ECQL.
>
> This is only happening if I ingest a shapefile with a column named
> 'id' - i can replace the column name, re-ingest and the same data
> appears fine via WFS.
>
> cheers,
>
>
> Jo
> --
>
>
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or
> unsubscribe from this list, visit
> https://www.locationtech.org/mailman/listinfo/geomesa-users
------------------------------
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://www.locationtech.org/mailman/listinfo/geomesa-users
End of geomesa-users Digest, Vol 27, Issue 3
********************************************
