Re: [geomesa-users] Accumulo tablet-servers dying when accessing table v

[Chris Eichelberger <cne1x@xxxxxxxx>]

Bob,

As this is a query for all locations and all time, how many records are
being exchanged?  Is your web server also running on this VM?
GeoServer's WFS service and Accumulo+GeoMesa should be using iterators,
but I wonder if some other down-stream process on the VM is trying to
hold all of the response features in memory at one time, forcing
something else to swap.  We *have* seen issues where the Accumulo server
starts swapping, Zookeeper starts timing out, and then everything
collapses.

Just a thought.

Sincerely,
  -- Chris


On Fri, 2014-12-19 at 16:43 +0000, Barnhart, Bob M. wrote:
> Thanks for responding Chris.
> 
> The query is identical to the ones we've used, except for the SERVER IP address.
> 
> I have examined the "tserver" logs and nothing really pops out except for a "failure during memory compaction". I'm not even positive that this was THE fatal error when the tserver terminates. 
> 
> This problem is really hard because it is not easily reproduced. Things work sometimes, but fail unpredictably. It may be exacerbated by the fact that we're running Zookeeper/Hadoop/Accumulo+GeoMesa/GeoServer and possibly Storm as well on an RHEL Server 6.5 VM with only 16GB of "ram".
> 
> -----Original Message-----
> From: geomesa-users-bounces@xxxxxxxxxxxxxxxx [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Chris Eichelberger
> Sent: Friday, December 19, 2014 8:18 AM
> To: geomesa-users@xxxxxxxxxxxxxxxx
> Subject: Re: [geomesa-users] Accumulo tablet-servers dying when accessing table via GeoServer using GeoMesa Plugin
> 
> Bob,
> 
> I have not seen this type of behavior before, but I recommend reading through the logs on one of the tablet-servers that fails.  
> 
> Out of curiosity, were there geographic, time, or attribute constraints on your query that you left out for brevity in your note?
> 
> Best of luck!
> 
> Sincerely,
>   -- Chris
> 
> 
> On Thu, 2014-12-18 at 23:39 +0000, Barnhart, Bob M. wrote:
> > Our web-application uses GoogleEarth technology and has the ability to 
> > request from Accumulo (via GeoServer using the GeoMesa Accumulo
> > plugin) geospatial feature-data that has been ingested using GeoMesa.
> > Once a user has zoomed into a particular area of the map, the 
> > web-application uses AJAX to issue an HTTP request (such as the
> > following) in order to bring back meta-data (in this case for
> > “Full-Motion-Videos”) for any “features” that lie within the visible 
> > map area:
> > 
> >  
> > 
> > http://SERVER:8080/geoserver/geomesa/ows?service=WFS&version=1.0.0&req
> > uest=GetFeature&typeName=geomesa:fmv&outputFormat=application/json
> > 
> >  
> > 
> > Our web-application (asynchronously) submits such URL requests using 
> > AJAX. The problem is that the request USUALLY (though not always) 
> > fails and “hangs” with the side-effect that the Accumulo Table-Server 
> > process will have terminated (possibly due to a memory-compaction 
> > error). At this point it is not possible to access Accumulo via 
> > GeoServer/GeoMesa-Plugin until the Zookeeper/Hadoop/Accumulo/GeoServer
> > stack is restarted.
> > 
> >  
> > 
> > However, if I manually send this request to GeoServer using the 
> > following wget(1) command, everything works fine:
> > 
> >  
> > 
> > wget -t 1 -O /tmp/gs-fmv.json
> > http://SERVER:8080/geoserver/geomesa/ows?service=WFS&version=1.0.0&req
> > uest=GetFeature&typeName=geomesa:fmv&outputFormat=application/json
> > 
> >  
> > 
> > I seem to be able to submit such requests at any frequency, even 
> > submitting multiple concurrent requests, and experience no such 
> > problems with the Accumulo Tablet Servers dying.
> > 
> >  
> > 
> > At this point, we’re stymied as to what might be happening and hoped 
> > that perhaps one of you had seen some similar behavior or might have a 
> > theory about what is happening, or even suggestions about how to debug 
> > this.
> > 
> >  
> > 
> > FYI, we have been running the following software versions in our 
> > “data-stack” since our work with GeoMesa/Accumulo technology began 
> > April/May 2014:
> > 
> >  
> > 
> > accumulo-1.5.1
> > 
> > geomesa 1.0.0
> > 
> > geoserver-2.5
> > 
> > /webapps/geoserver/WEB-INF/lib/geomesa-plugin-1.0.0-SNAPSHOT-geoserver
> > -plugin.jar
> > 
> > /webapps/geoserver/WEB-INF/lib/geomesa-core-1.0.0-SNAPSHOT.jar
> > 
> > hadoop-2.4.0
> > 
> > zookeeper-3.4.6
> > 
> >  
> > 
> > We would appreciate any help or insight you might be able to provide.
> > 
> >  
> > 
> > Bob Barnhart
> > 
> > Chief Systems Engineer | 858 826 5596 (Office) | 619 972 9489 (Mobile)
> > | Robert.M.Barnhart@xxxxxxxxxx
> > 
> >  
> > 
> >  
> > 
> >  
> > 
> > 
> > _______________________________________________
> > geomesa-users mailing list
> > geomesa-users@xxxxxxxxxxxxxxxx
> > To change your delivery options, retrieve your password, or 
> > unsubscribe from this list, visit 
> > http://www.locationtech.org/mailman/listinfo/geomesa-users
> 
> 
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit http://www.locationtech.org/mailman/listinfo/geomesa-users
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> http://www.locationtech.org/mailman/listinfo/geomesa-users