[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [geomesa-users] Accumulo tablet-servers dying when accessing table via GeoServer using GeoMesa Plugin
|
Jim,
After noting that at least on one occasion I saw that Zookeeper ran out of memory, I went over the (memory) resource configurations for Accumulo, GeoServer and Zookeeper. Currently, we've been running with slightly larger parameters in Accumulo, but with more memory allocated to Zookeeper and since then have not experienced the problem. I'll keep you posted if we determine anything else.
I'll also keep the Tomcat recommendation in mind.
Thanks and Happy New Year!
Bob
-----Original Message-----
From: geomesa-users-bounces@xxxxxxxxxxxxxxxx [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Jim Hughes
Sent: Friday, January 02, 2015 10:19 AM
To: geomesa-users@xxxxxxxxxxxxxxxx
Subject: Re: [geomesa-users] Accumulo tablet-servers dying when accessing table via GeoServer using GeoMesa Plugin
Hi Bob,
Apologies for the delay in responding. I wanted to ask if you had figured out any more about the memory problems you had reported.
We typically have been running GeoServer with Tomcat. It may be worth trying out Tomcat or JBoss.
Thanks,
Jim
On 12/22/2014 05:57 PM, Barnhart, Bob M. wrote:
> Jim,
>
> I believe we have evidence that GeoServer AND the Accumulo TabletServer may be experiencing out of memory conditions. I'm experimenting with the memory-related configuration parameters, but since I can't duplicate the error without running our web-app, it's a bit difficult to diagnose.
>
> I think we're just running GeoServer via Jetty.
>
> Thanks,
> Bob
>
> -----Original Message-----
> From: geomesa-users-bounces@xxxxxxxxxxxxxxxx
> [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Jim
> Hughes
> Sent: Friday, December 19, 2014 2:24 PM
> To: geomesa-users@xxxxxxxxxxxxxxxx
> Subject: Re: [geomesa-users] Accumulo tablet-servers dying when
> accessing table via GeoServer using GeoMesa Plugin
>
> Hi Bob,
>
> Is that error happening on GeoServer or Accumulo? I imagine the GeoServer JVM is likely reporting its OOM.
>
> How much memory are giving GeoServer? Are you running it via Tomcat, Jetty, or JBoss?
>
> Thanks,
>
> Jim
>
> On 12/19/2014 01:45 PM, Barnhart, Bob M. wrote:
>> Chris,
>>
>> We're not [yet] talking "big data" here. We've seen failures trying to retrieve 55 FMV metadata records constituting ~68K.
>>
>> I don't think we're running a web-server on this server--just GeoServer (etc.).
>>
>> This morning one of our engineers used WizTools.org RESTClient 3.1 to hit our server every second and got 1 failure out of 50 requests:
>>
>> <?xml version="1.0" ?>
>> <ServiceExceptionReport
>> version="1.2.0"
>> xmlns="http://www.opengis.net/ogc";
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>> xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wfs/1.0.0/OGC-exception.xsd";>
>> <ServiceException>
>> java.lang.OutOfMemoryError: unable to create new native
>> thread unable to create new native thread
>> </ServiceException></ServiceExceptionReport>
>>
>> This seems to confirm our earlier suspicions that somehow, the JVM is [intermittently] running out of memory on our 16GB ("ram") VM.
>>
>> Bob
>>
>> -----Original Message-----
>> From: geomesa-users-bounces@xxxxxxxxxxxxxxxx
>> [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Chris
>> Eichelberger
>> Sent: Friday, December 19, 2014 8:55 AM
>> To: Geomesa User discussions
>> Subject: Re: [geomesa-users] Accumulo tablet-servers dying when
>> accessing table via GeoServer using GeoMesa Plugin
>>
>> Bob,
>>
>> As this is a query for all locations and all time, how many records are being exchanged? Is your web server also running on this VM?
>> GeoServer's WFS service and Accumulo+GeoMesa should be using iterators, but I wonder if some other down-stream process on the VM is trying to hold all of the response features in memory at one time, forcing something else to swap. We *have* seen issues where the Accumulo server starts swapping, Zookeeper starts timing out, and then everything collapses.
>>
>> Just a thought.
>>
>> Sincerely,
>> -- Chris
>>
>>
>> On Fri, 2014-12-19 at 16:43 +0000, Barnhart, Bob M. wrote:
>>> Thanks for responding Chris.
>>>
>>> The query is identical to the ones we've used, except for the SERVER IP address.
>>>
>>> I have examined the "tserver" logs and nothing really pops out except for a "failure during memory compaction". I'm not even positive that this was THE fatal error when the tserver terminates.
>>>
>>> This problem is really hard because it is not easily reproduced. Things work sometimes, but fail unpredictably. It may be exacerbated by the fact that we're running Zookeeper/Hadoop/Accumulo+GeoMesa/GeoServer and possibly Storm as well on an RHEL Server 6.5 VM with only 16GB of "ram".
>>>
>>> -----Original Message-----
>>> From: geomesa-users-bounces@xxxxxxxxxxxxxxxx
>>> [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Chris
>>> Eichelberger
>>> Sent: Friday, December 19, 2014 8:18 AM
>>> To: geomesa-users@xxxxxxxxxxxxxxxx
>>> Subject: Re: [geomesa-users] Accumulo tablet-servers dying when
>>> accessing table via GeoServer using GeoMesa Plugin
>>>
>>> Bob,
>>>
>>> I have not seen this type of behavior before, but I recommend reading through the logs on one of the tablet-servers that fails.
>>>
>>> Out of curiosity, were there geographic, time, or attribute constraints on your query that you left out for brevity in your note?
>>>
>>> Best of luck!
>>>
>>> Sincerely,
>>> -- Chris
>>>
>>>
>>> On Thu, 2014-12-18 at 23:39 +0000, Barnhart, Bob M. wrote:
>>>> Our web-application uses GoogleEarth technology and has the ability
>>>> to request from Accumulo (via GeoServer using the GeoMesa Accumulo
>>>> plugin) geospatial feature-data that has been ingested using GeoMesa.
>>>> Once a user has zoomed into a particular area of the map, the
>>>> web-application uses AJAX to issue an HTTP request (such as the
>>>> following) in order to bring back meta-data (in this case for
>>>> “Full-Motion-Videos”) for any “features” that lie within the
>>>> visible map area:
>>>>
>>>>
>>>>
>>>> http://SERVER:8080/geoserver/geomesa/ows?service=WFS&version=1.0.0&;
>>>> r
>>>> eq
>>>> uest=GetFeature&typeName=geomesa:fmv&outputFormat=application/json
>>>>
>>>>
>>>>
>>>> Our web-application (asynchronously) submits such URL requests
>>>> using AJAX. The problem is that the request USUALLY (though not
>>>> always) fails and “hangs” with the side-effect that the Accumulo
>>>> Table-Server process will have terminated (possibly due to a
>>>> memory-compaction error). At this point it is not possible to
>>>> access Accumulo via GeoServer/GeoMesa-Plugin until the
>>>> Zookeeper/Hadoop/Accumulo/GeoServer
>>>> stack is restarted.
>>>>
>>>>
>>>>
>>>> However, if I manually send this request to GeoServer using the
>>>> following wget(1) command, everything works fine:
>>>>
>>>>
>>>>
>>>> wget -t 1 -O /tmp/gs-fmv.json
>>>> http://SERVER:8080/geoserver/geomesa/ows?service=WFS&version=1.0.0&;
>>>> r
>>>> eq
>>>> uest=GetFeature&typeName=geomesa:fmv&outputFormat=application/json
>>>>
>>>>
>>>>
>>>> I seem to be able to submit such requests at any frequency, even
>>>> submitting multiple concurrent requests, and experience no such
>>>> problems with the Accumulo Tablet Servers dying.
>>>>
>>>>
>>>>
>>>> At this point, we’re stymied as to what might be happening and
>>>> hoped that perhaps one of you had seen some similar behavior or
>>>> might have a theory about what is happening, or even suggestions
>>>> about how to debug this.
>>>>
>>>>
>>>>
>>>> FYI, we have been running the following software versions in our
>>>> “data-stack” since our work with GeoMesa/Accumulo technology began
>>>> April/May 2014:
>>>>
>>>>
>>>>
>>>> accumulo-1.5.1
>>>>
>>>> geomesa 1.0.0
>>>>
>>>> geoserver-2.5
>>>>
>>>> /webapps/geoserver/WEB-INF/lib/geomesa-plugin-1.0.0-SNAPSHOT-geoser
>>>> v
>>>> er
>>>> -plugin.jar
>>>>
>>>> /webapps/geoserver/WEB-INF/lib/geomesa-core-1.0.0-SNAPSHOT.jar
>>>>
>>>> hadoop-2.4.0
>>>>
>>>> zookeeper-3.4.6
>>>>
>>>>
>>>>
>>>> We would appreciate any help or insight you might be able to provide.
>>>>
>>>>
>>>>
>>>> Bob Barnhart
>>>>
>>>> Chief Systems Engineer | 858 826 5596 (Office) | 619 972 9489
>>>> (Mobile)
>>>> | Robert.M.Barnhart@xxxxxxxxxx
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> geomesa-users mailing list
>>>> geomesa-users@xxxxxxxxxxxxxxxx
>>>> To change your delivery options, retrieve your password, or
>>>> unsubscribe from this list, visit
>>>> http://www.locationtech.org/mailman/listinfo/geomesa-users
>>> _______________________________________________
>>> geomesa-users mailing list
>>> geomesa-users@xxxxxxxxxxxxxxxx
>>> To change your delivery options, retrieve your password, or
>>> unsubscribe from this list, visit
>>> http://www.locationtech.org/mailman/listinfo/geomesa-users
>>> _______________________________________________
>>> geomesa-users mailing list
>>> geomesa-users@xxxxxxxxxxxxxxxx
>>> To change your delivery options, retrieve your password, or
>>> unsubscribe from this list, visit
>>> http://www.locationtech.org/mailman/listinfo/geomesa-users
>> _______________________________________________
>> geomesa-users mailing list
>> geomesa-users@xxxxxxxxxxxxxxxx
>> To change your delivery options, retrieve your password, or
>> unsubscribe from this list, visit
>> http://www.locationtech.org/mailman/listinfo/geomesa-users
>> _______________________________________________
>> geomesa-users mailing list
>> geomesa-users@xxxxxxxxxxxxxxxx
>> To change your delivery options, retrieve your password, or
>> unsubscribe from this list, visit
>> http://www.locationtech.org/mailman/listinfo/geomesa-users
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or
> unsubscribe from this list, visit
> http://www.locationtech.org/mailman/listinfo/geomesa-users
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or
> unsubscribe from this list, visit
> http://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit http://www.locationtech.org/mailman/listinfo/geomesa-users
