Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] WFS call for GeoMesa 1.2.2 and GeoServer 2.8.1 returns the wrong numberMatched value despite cql filter holding bbox and other filters provided in the request

Hi Diane,

The stats will automatically generate themselves, no need for you to do anything other than update. Make sure that you update your distributed accumulo jar before upgrading the geoserver jar though.

Thanks,

Emilio

On 06/24/2016 02:10 AM, Diane Griffith wrote:

Jim and Emilio,

 

Thanks for the quick responses.  I will probably be delayed in the trying the upgrade but I do plan to upgrade and test either this weekend or beginning of the week. 

 

I am interested in  better understanding how to run the job to compute the stats to see if I can work that path instead of re-ingest existing data.  And yes sounds like maybe we should look into trying to be able to run different versions on one cloud so thanks for that helpful pointer.

 

Regards,

Diane

 

From: geomesa-users-bounces@xxxxxxxxxxxxxxxx [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Jim Hughes
Sent: Thursday, June 23, 2016 7:11 PM
To: geomesa-users@xxxxxxxxxxxxxxxx
Subject: Re: [geomesa-users] WFS call for GeoMesa 1.2.2 and GeoServer 2.8.1 returns the wrong numberMatched value despite cql filter holding bbox and other filters provided in the request

 

Hi Diane,

Sorry the troubles; the good news is that the GeoMesa 1.2.3 release is out now.  Andrew Hulbert and I'll be making a more official announcement tomorrow with a few more details, etc.

Since we have better estimated counts in GeoMesa 1.2.3, we've set the geomesa.force.count default to false.  Emilio's PR does seem to fix the property handling, so you should be able to use the -Dgeomesa.force.count=true JAVA_OPTS to force an exact count. 

If you can try out the new release, that'd be great.  As a note, to benefit fully from the new pre-computed stats work, you may need to re-ingest your data or run a job to compute the stats.  I'll let Emilio help provide the details there.

In order to help with upgrading, I'd note that by using Accumulo namespaces, you could run multiple versions of GeoMesa on one cloud: http://www.geomesa.org/documentation/user/installation_and_configuration.html?#for-accumulo-1-6.  That might help with transitiions.

Cheers,

Jim

Details:

In GeoMesa, the code for getCount is here (1).  

1.  https://github.com/locationtech/geomesa/blob/master/geomesa-accumulo/geomesa-accumulo-datastore/src/main/scala/org/locationtech/geomesa/accumulo/data/AccumuloFeatureSource.scala#L68-L78

Line 71 is a little terse.  The 'getHints.isExactCount' calls out to (2); if the query hint is not set, then the QUERY_EXACT_COUNT property is checked (3). 

2.  https://github.com/locationtech/geomesa/blob/geomesa-1.2.3/geomesa-accumulo/geomesa-accumulo-datastore/src/main/scala/org/locationtech/geomesa/accumulo/index/index.scala#L103

3.  https://github.com/locationtech/geomesa/blob/geomesa-1.2.3/geomesa-accumulo/geomesa-accumulo-datastore/src/main/scala/org/locationtech/geomesa/accumulo/accumulo.scala#L53

On 06/23/2016 05:59 PM, Diane Griffith wrote:

So I am using WFS version 2.0.0 for my requests I send to pull the feature data.

 

The numberMatched is when I do &resultType=hits in the WFS  url,  it gives back xml versus json I think regardless of what I set as the outputFormat.  I only use that when I just want to see the count and not the actual features for debugging and validation testing.

 

When I use &outputFormat=json without &resultType=hits in the WFS url I get a totalFeatures field in the json response.

 

Regardless of the field name the value is consistent for WFS version 2.0.0:

 

GeoMesa 1.2.2 returns total number of features in the feature type specified so that means the total is not filtered at all by the cql_filter field passed (that contains a bbox and additional cql)

 

GeoMesa 1.2.0 does return the number of features that match the filter I provide in the cql_filter field (containing bbox and additional cql) for the feature type specified.

 

What class sets the totalFeatures return value.  I hadn't found that class yet to try and see if there was a patch  I could try to apply myself to fix this problem.

 

That or is there a patch I could apply for that system property if it does work?

 

We had hoped to allow users to beta test on our more operational system but that is set to geomesa 1.2.2 and this bug is more of a blocker to allow users in to test with.

 

Is there a target timeframe for the 1.2.3 release?  

 

Thanks,

Diane

 

From: Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx>
Reply-To: "geomesa-users@xxxxxxxxxxxxxxxx" <geomesa-users@xxxxxxxxxxxxxxxx>
Date: Thursday, June 23, 2016 4:15 PM
To: "geomesa-users@xxxxxxxxxxxxxxxx" <geomesa-users@xxxxxxxxxxxxxxxx>
Subject: Re: [geomesa-users] WFS call for GeoMesa 1.2.2 and GeoServer 2.8.1 returns the wrong numberMatched value despite cql filter holding bbox and other filters provided in the request

 

Diane, the WFS JSON testing I was doing didn't have a 'numberMatched' field - instead it has a 'totalFeatures' field. I assume they would be mapped to the same thing, but I couldn't figure out how to get that format back. I just clicked on the GML layer preview, and then added outputFormat=json to the URL.

Thanks,

Emilio

On 06/23/2016 03:32 PM, Emilio Lahr-Vivaz wrote:

Oops, I verified that the system property is not getting handled correctly. I'll open a bug for it, and we'll get it into 1.2.3.

Thanks,

Emilio

On 06/23/2016 03:14 PM, Emilio Lahr-Vivaz wrote:

What version of WFS are you using for the request? Yes, the system property should be set in the geoserver startup scripts, like: -Dgeomesa.force.count=true

On 06/23/2016 02:44 PM, Diane Griffith wrote:

For the system property route, do you mean set it on geoserver?  If so then that did not work.

 

For the second option I’m not directly querying the accumulo stack, I’m doing it via geoserver so I do not think I can add that hint.

 

What I do know is 1.2.0 dev system returns the correct number of matches with the WFS response.  The 1.2.2 based system is returning the total number of features regardless of the actual query.  I didn’t upgrade the 1.2.0 dev system but I bet as soon as I do it no longer returns a correct count.  When I say a different number the WFS request matched 3 records/features. 

 

On the 1.2.0 based system for the same data I had numberMatched=”3”  but for the 1.2.2 based system it had numberMatched=”1508503” (for resultType=hits)

 

Same data set, same query:

 

CQL_FILTER=(BBOX(pickupLocation, -73.82473468780518,40.82369327545166,-73.81924152374268,40.828328132629395, 'EPSG:4326') AND (Total_amount >=1 AND Total_amount <=5))

 

With outputFormat=json:

 

1.2.0 gave field "totalFeatures": 3

 

1.2.2 gave field “totalFeatures”: 1508503

 

Diane

From: geomesa-users-bounces@xxxxxxxxxxxxxxxx [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Emilio Lahr-Vivaz
Sent: Thursday, June 23, 2016 2:16 PM
To: geomesa-users@xxxxxxxxxxxxxxxx
Subject: Re: [geomesa-users] WFS call for GeoMesa 1.2.2 and GeoServer 2.8.1 returns the wrong numberMatched value despite cql filter holding bbox and other filters provided in the request

 

Hi Diane,

Most likely it is due to getCount being called on the feature collection being returned. Since GeoMesa is streaming data from accumulo, in order to calculate an exact count we essentially have to run the query twice. Because of this, we return a fake count, or in the upcoming 1.2.3 an estimated count.

Since sometimes the exact count is important, we have two ways to force the counting:

1. Set the system property  "geomesa.force.count" to "true" to force exact counting for all queries
2. For individual queries, set a query hint using the key org.locationtech.geomesa.accumulo.index.QueryHints.EXACT_COUNT and the value of Boolean.TRUE

I believe this behavior was the same in 1.2.0 though...

Could you try one of the above and see if it resolves your problem?

Thanks,

Emilio

On 06/23/2016 01:10 PM, Diane Griffith wrote:

We had upgraded to GeoMesa 1.2.2 in order to fix the ability to combine IN parameters in a cql call and get proper results.  But I did leave one environment set to GeoMesa 1.2.0.

 

What we noticed recently when finally hooking paging is that the WFS calls (version 2.0.0) we issued would return the correct number of results given the CQL_FILTER (that would hold a bbox as well as other cql filters) but the value in numberMatched was the total number of features for the feature type specified not the number of results that would match the CQL_FILTER.  So we would indicate there were more pages available for the user than actual number of results.

 

I also tried testing where I just sent a BBOX parameter instead of combining that with the cql_filter and it was behaving the same.  WFS responses hooked to GeoMesa 1.2.0 indicated the correct number but WFS responses hooked to GeoMesa 1.2.2 were reporting the total number of features for the specified feature type not the number of features that matched the bbox specified for that feature type. 

 

I looked at the queries logged in accumulo and they looked  the same on quick review.  I also looked at the call logged by GeoServer and it looked the same as well.

 

I was trying to find where in the code it set the numberMatched field for the WFS 2.0.0 response but I hadn’t found that yet.

 

I can upgrade the old environment but I believe once I do that it will behave the same and no longer give correct numberMatched values.

 

I compared JDK versions of the servers, geoserver versions, that they both have native JAI and it all matches.  Also both appear to use the same version of GeoTools 14.1.  The only difference I have found so far is GeoMesa version.  Has anyone noticed this issue yet?  Is there a fix for it?

 

Thanks,

Diane





_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geomesa-users

 




_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geomesa-users





_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geomesa-users





_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geomesa-users

 




_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geomesa-users

 



_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geomesa-users


Back to the top