Re: [geomesa-users] Accumulo tablet-servers dying when accessing table v

["Barnhart, Bob M." <ROBERT.M.BARNHART@xxxxxxxxxx>]

Chris,

We're not [yet] talking "big data" here. We've seen failures trying to retrieve 55 FMV metadata records constituting ~68K.

I don't think we're running a web-server on this server--just GeoServer (etc.).

This morning one of our engineers used WizTools.org RESTClient 3.1 to hit our server every second and got 1 failure out of 50 requests:

<?xml version="1.0" ?>
<ServiceExceptionReport
   version="1.2.0"
   xmlns="http://www.opengis.net/ogc";
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
   xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wfs/1.0.0/OGC-exception.xsd";>
   <ServiceException>
      java.lang.OutOfMemoryError: unable to create new native thread
unable to create new native thread
</ServiceException></ServiceExceptionReport>

This seems to confirm our earlier suspicions that somehow, the JVM is [intermittently] running out of memory on our 16GB ("ram") VM.

Bob

-----Original Message-----
From: geomesa-users-bounces@xxxxxxxxxxxxxxxx [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Chris Eichelberger
Sent: Friday, December 19, 2014 8:55 AM
To: Geomesa User discussions
Subject: Re: [geomesa-users] Accumulo tablet-servers dying when accessing table via GeoServer using GeoMesa Plugin

Bob,

As this is a query for all locations and all time, how many records are being exchanged?  Is your web server also running on this VM?
GeoServer's WFS service and Accumulo+GeoMesa should be using iterators, but I wonder if some other down-stream process on the VM is trying to hold all of the response features in memory at one time, forcing something else to swap.  We *have* seen issues where the Accumulo server starts swapping, Zookeeper starts timing out, and then everything collapses.

Just a thought.

Sincerely,
  -- Chris


On Fri, 2014-12-19 at 16:43 +0000, Barnhart, Bob M. wrote:
> Thanks for responding Chris.
> 
> The query is identical to the ones we've used, except for the SERVER IP address.
> 
> I have examined the "tserver" logs and nothing really pops out except for a "failure during memory compaction". I'm not even positive that this was THE fatal error when the tserver terminates. 
> 
> This problem is really hard because it is not easily reproduced. Things work sometimes, but fail unpredictably. It may be exacerbated by the fact that we're running Zookeeper/Hadoop/Accumulo+GeoMesa/GeoServer and possibly Storm as well on an RHEL Server 6.5 VM with only 16GB of "ram".
> 
> -----Original Message-----
> From: geomesa-users-bounces@xxxxxxxxxxxxxxxx 
> [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Chris 
> Eichelberger
> Sent: Friday, December 19, 2014 8:18 AM
> To: geomesa-users@xxxxxxxxxxxxxxxx
> Subject: Re: [geomesa-users] Accumulo tablet-servers dying when 
> accessing table via GeoServer using GeoMesa Plugin
> 
> Bob,
> 
> I have not seen this type of behavior before, but I recommend reading through the logs on one of the tablet-servers that fails.  
> 
> Out of curiosity, were there geographic, time, or attribute constraints on your query that you left out for brevity in your note?
> 
> Best of luck!
> 
> Sincerely,
>   -- Chris
> 
> 
> On Thu, 2014-12-18 at 23:39 +0000, Barnhart, Bob M. wrote:
> > Our web-application uses GoogleEarth technology and has the ability 
> > to request from Accumulo (via GeoServer using the GeoMesa Accumulo
> > plugin) geospatial feature-data that has been ingested using GeoMesa.
> > Once a user has zoomed into a particular area of the map, the 
> > web-application uses AJAX to issue an HTTP request (such as the
> > following) in order to bring back meta-data (in this case for
> > “Full-Motion-Videos”) for any “features” that lie within the visible 
> > map area:
> > 
> >  
> > 
> > http://SERVER:8080/geoserver/geomesa/ows?service=WFS&version=1.0.0&r
> > eq 
> > uest=GetFeature&typeName=geomesa:fmv&outputFormat=application/json
> > 
> >  
> > 
> > Our web-application (asynchronously) submits such URL requests using 
> > AJAX. The problem is that the request USUALLY (though not always) 
> > fails and “hangs” with the side-effect that the Accumulo 
> > Table-Server process will have terminated (possibly due to a 
> > memory-compaction error). At this point it is not possible to access 
> > Accumulo via GeoServer/GeoMesa-Plugin until the 
> > Zookeeper/Hadoop/Accumulo/GeoServer
> > stack is restarted.
> > 
> >  
> > 
> > However, if I manually send this request to GeoServer using the 
> > following wget(1) command, everything works fine:
> > 
> >  
> > 
> > wget -t 1 -O /tmp/gs-fmv.json
> > http://SERVER:8080/geoserver/geomesa/ows?service=WFS&version=1.0.0&r
> > eq 
> > uest=GetFeature&typeName=geomesa:fmv&outputFormat=application/json
> > 
> >  
> > 
> > I seem to be able to submit such requests at any frequency, even 
> > submitting multiple concurrent requests, and experience no such 
> > problems with the Accumulo Tablet Servers dying.
> > 
> >  
> > 
> > At this point, we’re stymied as to what might be happening and hoped 
> > that perhaps one of you had seen some similar behavior or might have 
> > a theory about what is happening, or even suggestions about how to 
> > debug this.
> > 
> >  
> > 
> > FYI, we have been running the following software versions in our 
> > “data-stack” since our work with GeoMesa/Accumulo technology began 
> > April/May 2014:
> > 
> >  
> > 
> > accumulo-1.5.1
> > 
> > geomesa 1.0.0
> > 
> > geoserver-2.5
> > 
> > /webapps/geoserver/WEB-INF/lib/geomesa-plugin-1.0.0-SNAPSHOT-geoserv
> > er
> > -plugin.jar
> > 
> > /webapps/geoserver/WEB-INF/lib/geomesa-core-1.0.0-SNAPSHOT.jar
> > 
> > hadoop-2.4.0
> > 
> > zookeeper-3.4.6
> > 
> >  
> > 
> > We would appreciate any help or insight you might be able to provide.
> > 
> >  
> > 
> > Bob Barnhart
> > 
> > Chief Systems Engineer | 858 826 5596 (Office) | 619 972 9489 
> > (Mobile)
> > | Robert.M.Barnhart@xxxxxxxxxx
> > 
> >  
> > 
> >  
> > 
> >  
> > 
> > 
> > _______________________________________________
> > geomesa-users mailing list
> > geomesa-users@xxxxxxxxxxxxxxxx
> > To change your delivery options, retrieve your password, or 
> > unsubscribe from this list, visit 
> > http://www.locationtech.org/mailman/listinfo/geomesa-users
> 
> 
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or 
> unsubscribe from this list, visit 
> http://www.locationtech.org/mailman/listinfo/geomesa-users
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or 
> unsubscribe from this list, visit 
> http://www.locationtech.org/mailman/listinfo/geomesa-users


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit http://www.locationtech.org/mailman/listinfo/geomesa-users