Re: [geomesa-users] Geomesa IN Query Performance

[Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx>]

    We use 'small' scans, for consistency in the rest of the API, and if
    I remember correctly to support BigTable. You could try changing it
    to a Get here:
    
    https://github.com/locationtech/geomesa/blob/geomesa_2.11-2.3.2/geomesa-hbase/geomesa-hbase-datastore/src/main/scala/org/locationtech/geomesa/hbase/data/HBaseIndexAdapter.scala#L331
    
    There might be some other places that would require changes as
    well...
    
    If you find that it makes a noticeable difference, please submit a
    PR or open a ticket to change to Gets.
    
    Thanks,
    
    Emilio

On 12/2/19 10:19 AM, Amit Srivastava wrote:

Is it possible in Geomesa to do HBase Get Query on ID index instead of Scan to get from Hbase? Single Get query is also fine for me.

On Mon, Dec 2, 2019 at 6:09 AM Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:

Hello,

You could double check the explain logging[1] to make sure that you have an ID index, however due to the nature of HBase, doing many single-row scans randomly distributed across your table is going to be slow. If there are any known characteristics to the features that you are updating, you can use a custom feature ID generator[2] to try to create feature IDs so that an update will hit a group of IDs close together in the index - that may improve performance some.

Thanks,

Emilio

[1]: https://www.geomesa.org/documentation/user/datastores/query_planning.html#explaining-query-plans
[2]: https://www.geomesa.org/documentation/user/datastores/runtime_config.html#geomesa-feature-id-generator

On 11/30/19 4:27 AM, Amit Srivastava wrote:

Hi All,

I am using Geomesa as a spatial temproral database. For one of the use-case. I need to do Geomesa Id based query (100 Ids in a batch). I am seeing high latency with IN query. Is there anyway to improve the query performance?

Below is the AuditLogger of query:

30 Nov 2019 09:04:05,481 [36m[DEBUG][m c7e42e76-074f-47e7-84d0-22b682912f6e (Coral Endpoint : 182) org.locationtech.geomesa.utils.audit.AuditLogger$: 
{
    "storeType": "hbase",
    "typeName": "OSMNodes",
    "date": 1575104645481,
    "user": "unknown",
    "filter": "IN ('FvUghvjl0iiMYMBsmQ_MoA#56144647','pY2sw3HeKtzeZJcu8u6ChQ#56144672','b7n6IdGTi1Kr_gCPMGCNGw#56144654','0O8cUGIwM5bREyJdr-e-3g#56854415','cWEPdm0iH82n9PHJjw9QgA#56144645','ls74D9mftAptPbT2-MVfdw#56854426','p9PI-J_sw78mI6bdre-DkQ#55772582','oypt0UG8kIc0Z5pZRRSXlg#56144681','0glVM6b1J9fcMu5QzxHBZg#56144648','EdF-Gs7RJNm-sd0NUhH_ew#56854451','73VBCwEWSJv0iXzhca27ZA#56854424','FKEIKipQ5jlTslj-cFqWnQ#57226766','tJj1QESy98YrFocAUsmAig#56854455','1fZLVqkokRrw1iHtKsqS7g#56854416','ynIlytnf75M80GEQmcq5xA#55772578','z3mJk8_OIetWvSfmQDe7Uw#57226757','wif0yWVwEDkfRJs6b1gkbA#56854420','hBNrWVxxWaUHbGpHX4-Jow#56854414','HMBhiho2059nBYoAlgR1YQ#56854454','nKEr_m7Vygdm0YAzHcIbEQ#57226786','nRK4k8BojfNJCHvkE0mpNQ#56854447','Eq7RmxvhEj5nnyUswNtXcQ#56144652','t23othKYQ5RBB_AwJKA4Mg#57501085','FYLA_y7cbwoQ9tOGREKjPQ#56854446','dWJA26NmvOW4R8dk_Q3oIg#55772579','IkV8h3GGrnCBBrxJ6bKXGA#56854449','XU8_WlQJsFSOehbECMuoSw#55772547','IiMN_m4zk222w4QB2vKIYg#56854453','dkJ3xvWVv4egVjRJVFDRPw#56854448','2X5UTSMeqy4PmicFGq6SOg#57226759','Wn6R1zwZlDntM030OYWAEw#56854417','eRMt9-jD6jfpfl3pWPU8-g#56854444','ENz8sxeSn_Axqno5TyvJlg#56144670','sOpAXO0OF6kpUYsCQgRk0w#57226792','FKXWV6W7J7u0LBheoYp9yA#56854452','p2fA2v4VRpr7rUKNwbkbtA#56144651','ivmsdZTFLoYoe8oMRDFRyg#56144677','a4WOED2Qdxdn_PSjszf7fQ#55772580','vYUTA-qnKSXHbg-DrBIFAw#56854422','u_KAFnJEgWbO15Vi_q68Kw#56144673','1Pg7ZOZwXdI7emfLC1AKIw#56854418','bg1yacaoD8Yd9sBZwSVkEA#56144657','s7XKcOn0oJheX7A6U1yPtw#55772545','JZgGfOHUbo0EHvXYBHjvww#56144643','pMbg6jyhLPLzA8rIYIk_sQ#55772551','YXgmVdu0vGYm8OmgGtDLvw#56144658','KiQlNIbMYqDbwM3qSTDXmA#55772544','HGb-AoEXrUU_0Kg4yu-qGw#56854419','fjMMZuleRhdfxSVcgQ7sZw#56144649','srPjx2xS_6NksOpGU8QWaA#56854427','4eLPlwOfe8qZ6CuWnKsWxg#56854413','qKfEGq-ZEwawh1RXxiU_XQ#57501084','0yzWN5nnP7TtuxVGjaxZkQ#56144656','uyT9DGm9AQdp0uemJhu_mw#55772555','MntPnTdv18HeQx2ehn9UAw#56144641','yKGpHSg_4lxMo1VwJ2EBuQ#56144642','DfKB6uXc-nlHUuCDQbm4oQ#56854429','kYy3W_Wa68xLYptUHUrAgw#55772543','FB56jakUAMX7J5JmnH_LMA#55772572','vc_AEf3AkHui4HV-R8cGlw#56144674','sQR0I2x5GNDQlGbzpwr00Q#56854456','bDGtKXBfCkVrq36Ji1SHmA#56854421','jo4yLNxLxW5y0lNb2IkOlQ#55772581','spEzC4ktdzzRnmjAEXLvRQ#57226768','fDmL3071ktD43MCMipT7WA#56144678','n52jno8zl0zsNbF3SuaH3A#57226788','6VGVgJE9UAmpRFKVjrDI_w#56854425','NhfrsSFm3mJQ4dKvGEj-FA#56144640','AFs_HHaLGS5yi0zgNEA1MQ#57226789','OPQxBKQrEiF-GA_-RmKxrQ#56854428','hM0M4H8JED-3IOR6ZyH32w#57226754','mrfglviJkne7p1oMJm8E7Q#56144679','jcwh12ux4ya_8KCMo6jIIQ#56854450','xTAqz6vj3zOodf76J6jvEQ#55772577','P5H_eqOTr1mZJgVLtzhIaw#55772549','PUfqg5qkObW1xZUtBOXK1Q#55772553','b4bgLu84Wd4-s744CuEKWw#56144644','I2giChiupW1WEVTx6n_-Jw#57226755','Ofr5hjw2-g_gc-UVzdNt3Q#56854445','bTuTODwuDGlHvD2M6WWuQg#56144646','fWut04swSSRQtJpqU3xnng#57501087','5vvznB3w2lNsB1z9Pq5o_Q#57226785','2_5O_a3FG5KvUGcr_HwRCA#56144650','FTRdnjQxj4tv-MB9mF_8dg#57501080','fByEYKqCAzHX9FvirTIB2g#57226791','NcXAPORmUL7RCwaCAVSraA#57226794','NqdWByyC16Yn1aEpRLGqnA#56144659','0kyUz8W8ORPyLzFjklE7HA#57226765','9BilERUylnD3Ek21WRItYQ#56144653','EaaAlAuPQSGytGrACFGnHg#55772583','AXUiHq1BTJuIt_Z34EIZpA#56144676','_IHdF_AV3GWcAVn1dk_xYA#55772584','bh3gPi7cmk9Qc7vOUnKCgA#57226763','97wjsuDzOZfJ-_yI9r0Xng#57226795','q9mhtt63Abi-dHWdqWGd1g#57501088','pI9Lv3Ms45TsEV1QWtPuKg#57226783','2Ts7hrzoQCimlCjjbAINig#57501083','u9y3jjiNKj1QqPf23Rz-xw#57501081','OS6UeBPsm2RqBxxr3O786w#56854423','Kck-bicWJJXW_9sZBfZRdQ#56144655')",
    "hints": "RETURN_SFT=*geometry:Point:srid=4326,ingestionTimestamp:Timestamp,nextTimestamp:Timestamp,serializerVersion:String,featurePayload:String",
    "planTime": 1,
    "scanTime": 78584,
    "hits": 99,
    "deleted": false
}

--

Regards,

Amit

--

Regards,

Amit Kumar Srivastava
+1 206 518 8339 / Mobile