Re: [geomesa-users] Problem with Z3Idx queries (geomesa 1.1.0-rc2)

[Ben Southall <ben.southall@xxxxxxx>]

Hi Jim,

 

Bounding the temporal query on both sides does seem to work – thanks!

 

Unfortunately, I uncovered another issue regarding the polygon I’m querying. This query:

 

geom intersects POLYGON ((-170 -90, -170 90, 90 90, 90 -90, -170 -90))

 

generates a tweaked geometry of [[ geom intersects POLYGON ((-180 -90, -180 90, -170 90, -170 -90, -180 -90)) ] OR [ geom intersects POLYGON ((90 90, 180 90, 180 -90, 90 -90, 90 90)) ]]), which promptly makes GeomsToCover null, and the planner crashes.

 

However,

 

geom intersects POLYGON ((-170 -90, -170 90, 10 90, 10 -90, -170 -90))

 

works OK – the planner inserts a couple of extra co-ordinates at 0 degrees longitude.

 

Is this a known problem that’s been fixed in a later version?

 

Finally, regarding upgrading versions of Geomesa, is there anything between 1.1.0-rc2 and 1.2.0 that will have indexes compatible with 1.2.0 (so if I have to re-ingest, I can do it now and still be OK when 1.2.0 arrives)?

 

Thanks,

 

Ben

 

 

 

 

From: geomesa-users-bounces@xxxxxxxxxxxxxxxx [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Jim Hughes
Sent: Wednesday, January 27, 2016 5:50 PM
To: geomesa-users@xxxxxxxxxxxxxxxx
Subject: Re: [geomesa-users] Problem with Z3Idx queries (geomesa 1.1.0-rc2)

 

Hi Ben,

Thanks for the report.  I've got great news and some options for you.  First, the issue you are hitting has been reported as https://geomesa.atlassian.net/browse/GEOMESA-874 and is fixed in 1.1.0-rc4 and later. 

Upgrading should address this issue.  That said, we are pushing for a GeoMesa 1.2.0 release in the next month, and if I could get folks to do one update soon-ish, it'd be to that version.*

As a quick work around, since the bug deals with unbounded time ranges, we just need to put an upper temporal bound on the query.  It should be reasonable to say that we are querying for 2012 until now.  (I'm assuming you don't have future data or if you do that it could be bounded reasonably.)  If you try that with your filter, things should be up and working again.

Let us know if that doesn't do it for you!

Thanks,

Jim

* p.s.  For some small technical risk, you could try running client code with 1.1.0-rc4.  I *don't think* the GeoMesa Accumulo format changed, and if that's the case, you'd get the query planner fix without needing to worry with data re-ingest or upgrade.  Adding an upper bound to the query should be rock solid, but if that's impossible, this is another option which may work quickly.

On 1/27/2016 2:29 PM, Ben Southall wrote:

Hello,

 

I seem to be having some problems making spatio-temporal queries in geomesa 1.1.0-rc2; notably, it looks like something odd is happening to the temporal part of the query. Before I show the query planner output and the stack trace, a bit more info.  Here’s the table description from Accumulo:

 

~METADATA_ais_report attributes: []    mmsi:Long:index=full,latitude:Double,longitude:Double,observationTime:Date,currentHeading:Float,currentSpeed:Double,trueHeading:Integer,*geom:Point:srid=4326:index=full:index-value=true

~METADATA_ais_report dtgfield: []    observationTime

~METADATA_ais_report featureEncoding: []    kryo

~METADATA_ais_report schema: []    %~#s%2#r%#i%ais_report#cstr%0,3#gh%yyyyMMddHH#d::%~#s%3,2#gh::%~#s%#id

~METADATA_ais_report tables.idx.attr.name: []    nrl_global_ais_500k_attr_idx

~METADATA_ais_report tables.idx.st.name: []    nrl_global_ais_500k_st_idx

~METADATA_ais_report tables.queries.name: []    nrl_global_ais_500k_queries

~METADATA_ais_report tables.record.name: []    nrl_global_ais_500k_records

~METADATA_ais_report tables.sharing: []    true

~METADATA_ais_report tables.z3.name: []    nrl_global_ais_500k_ais_5freport_z3

~METADATA_ais_report version: []    5

~METADATA_ais_report visibilities: []

 

If I make just the spatial query, it works. If I make just the temporal query, it works. It looks like both of those queries are planned using the STIdxStrategy.

 

I’ve a small Accumulo cluster (3 nodes) with geomesa-distributed-runtime-1.1.0-rc.2.jar installed in the lib/ext directory of each node’s ACCUMULO_HOME.

 

It’s when I do a combined (“AND”) query that problems occur - here’s the output of the query planner, followed by the exception, which seems to indicate that the temporal part is causing problems:

 

--

 

Planning Query: feature type: ais_report, filter: [[ observationTime > Tue Jul 10 00:00:00 EDT 2012 ] AND [ geom intersects POLYGON ((-170 -80, -170 -79, -169 -79, -169 -80, -170 -80)) ]], [properties:  ALL ]

Original filter: [[ observationTime > Tue Jul 10 00:00:00 EDT 2012 ] AND [ geom intersects POLYGON ((-170 -80, -170 -79, -169 -79, -169 -80, -170 -80)) ]]

Rewritten filter: [[ observationTime > Tue Jul 10 00:00:00 EDT 2012 ] AND [ geom intersects POLYGON ((-170 -80, -170 -79, -169 -79, -169 -80, -170 -80)) ]]

Strategy: org.locationtech.geomesa.accumulo.index.Z3IdxStrategy

Transforms: None

Geometry filters: ArrayBuffer([ geom intersects POLYGON ((-170 -80, -170 -79, -169 -79, -169 -80, -170 -80)) ])

Temporal filters: ArrayBuffer([ observationTime > Tue Jul 10 00:00:00 EDT 2012 ])

Other filters: ArrayBuffer()

Tweaked geom filters are ArrayBuffer([ geom intersects POLYGON ((-170 -80, -170 -79, -169 -79, -169 -80, -170 -80)) ])

GeomsToCover: GEOMETRYCOLLECTION (POLYGON ((-170 -80, -170 -79, -169 -79, -169 -80, -170 -80)))

Interval:  2012-07-10T04:00:00.000Z/9999-12-31T23:59:59.000Z

Transforms: None

java.lang.ArithmeticException: Value cannot fit in an int: 253402300799

                at org.joda.time.field.FieldUtils.safeToInt(FieldUtils.java:190)

                at org.joda.time.field.BaseDurationField.getDifference(BaseDurationField.java:141)

                at org.joda.time.chrono.ZonedChronology$ZonedDurationField.getDifference(ZonedChronology.java:328)

                at org.joda.time.base.BaseSingleFieldPeriod.between(BaseSingleFieldPeriod.java:72)

                at org.joda.time.Seconds.secondsBetween(Seconds.java:101)

                at org.locationtech.geomesa.accumulo.data.tables.Z3Table$.secondsInCurrentWeek(Z3Table.scala:41)

                at org.locationtech.geomesa.accumulo.index.Z3IdxStrategy.getQueryPlans(Z3IdxStrategy.scala:134)

                at org.locationtech.geomesa.accumulo.index.QueryPlanner$$anonfun$4$$anonfun$5.apply(QueryPlanner.scala:130)

                at org.locationtech.geomesa.accumulo.index.QueryPlanner$$anonfun$4$$anonfun$5.apply(QueryPlanner.scala:126)

                at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)

                at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)

                at scala.collection.immutable.List.foreach(List.scala:318)

                at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)

                at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)

                at org.locationtech.geomesa.accumulo.index.QueryPlanner$$anonfun$4.apply(QueryPlanner.scala:126)

                at org.locationtech.geomesa.accumulo.index.QueryPlanner$$anonfun$4.apply(QueryPlanner.scala:113)

                at org.locationtech.geomesa.utils.stats.MethodProfiling$class.profile(MethodProfiling.scala:26)

                at org.locationtech.geomesa.accumulo.index.QueryPlanner.profile(QueryPlanner.scala:50)

                at org.locationtech.geomesa.accumulo.index.QueryPlanner.getStrategyPlans(QueryPlanner.scala:113)

                at org.locationtech.geomesa.accumulo.index.QueryPlanner.planQuery(QueryPlanner.scala:69)

                at org.locationtech.geomesa.accumulo.data.AccumuloDataStore.planQuery(AccumuloDataStore.scala:844)

                at org.locationtech.geomesa.accumulo.data.AccumuloDataStore.explainQuery(AccumuloDataStore.scala:836)

                at sri.mast.MastQuery.main(MastQuery.java:185)

 

--

 

Any help gratefully received!

 

Thanks,

 

Ben

_______________________________________________

geomesa-users mailing list

geomesa-users@xxxxxxxxxxxxxxxx

To change your delivery options, retrieve your password, or unsubscribe from this list, visit

http://www.locationtech.org/mailman/listinfo/geomesa-users