RELATEED CONSULTING
相关咨询
选择下列产品马上在线沟通
服务时间:8:30-17:00
你可能遇到了下面的问题
关闭右侧工具栏

新闻中心

这里有您想知道的互联网营销解决方案
Elasticsearchperformancetestingfinding

Setup
1 Node cluster on my local laptop: 8core, Xms=8G, Xmx=8G

专注于为中小企业提供做网站、成都网站建设服务,电脑端+手机端+微信端的三站合一,更高效的管理,为中小企业临河免费做网站提供优质的服务。我们立足成都,凝聚了一批互联网行业人才,有力地推动了上1000家企业的稳健成长,帮助中小企业通过网站建设实现规模扩充和转变。

Indexing performance (Single index):
10 million payments, each one about 5KB, with batch size = 10000. Each batch takes roughly 2.5 s → 4 s, total time to index 10 million payment is around 50 min

Indexing performance (Multiple indices):
20 separate indices store totally 10 million payments. Indexing execution is slightly faster than single index case. Each batch takes roughly 1.7 s → 3.8 s, total time to index 10 million payment is around 38 min

Parameters required for bulk load operation
Elasticsearch config: http.max_content_length: 500mb

Client time out adjustment:

RestClient.builder(HttpHost("localhost", 9200))
        .setRequestConfigCallback {
    it.apply {
        this.setConnectTimeout(5000)
        this.setSocketTimeout(60000)
    }
}.setMaxRetryTimeoutMillis(60000))

Initially batch size is set to 100000, elastic search server becomes unstable with high GC frequency, occupying a large percent of CPU time. So larger batch size does not always imply higher performance

Query aggregation performance:

Test query: real aggregation query used by rule engine

{
  "aggregations": {
    "date_range": {
      "range": {
        "field": "createdAt",
        "ranges": [
          {
            "key": "LAST_7_DAYS",
            "from": 1544400968485,
            "to": 1545005768486
          }
        ],
        "keyed": false
      },
      "aggregations": {
        "filter_aggregator": {
          "filters": {
            "filters": {
              "602c7d66-e990-4dfb-b6e2-72b62ff159d5": {
                "terms": {
                  "beneficiaryId.keyword": [
                    "602c7d66-e990-4dfb-b6e2-72b62ff159d5"
                  ],
                  "boost": 1
                }
              },
              "67cab0c8-2510-443d-8f00-bce19c04815e": {
                "terms": {
                  "bankAccountUserId.keyword": [
                    "67cab0c8-2510-443d-8f00-bce19c04815e"
                  ],
                  "boost": 1
                }
              },
              "8da52e51-eabf-4f6c-b9f0-e222933c1cb7": {
                "terms": {
                  "payerId.keyword": [
                    "8da52e51-eabf-4f6c-b9f0-e222933c1cb7"
                  ],
                  "boost": 1
                }
              },
              "8da52e51-eabf-4f6c-b9f0-e222933c1cb7_602c7d66-e990-4dfb-b6e2-72b62ff159d5": {
                "bool": {
                  "filter": [
                    {
                      "terms": {
                        "payerId.keyword": [
                          "8da52e51-eabf-4f6c-b9f0-e222933c1cb7"
                        ],
                        "boost": 1
                      }
                    },
                    {
                      "terms": {
                        "beneficiaryId.keyword": [
                          "602c7d66-e990-4dfb-b6e2-72b62ff159d5"
                        ],
                        "boost": 1
                      }
                    }
                  ],
                  "adjust_pure_negative": true,
                  "boost": 1
                }
              },
              "9a1b4bad-ccf5-4c67-8718-02696cb351e4": {
                "terms": {
                  "clientId.keyword": [
                    "9a1b4bad-ccf5-4c67-8718-02696cb351e4"
                  ],
                  "boost": 1
                }
              },
              "9a1b4bad-ccf5-4c67-8718-02696cb351e4_602c7d66-e990-4dfb-b6e2-72b62ff159d5": {
                "bool": {
                  "filter": [
                    {
                      "terms": {
                        "clientId.keyword": [
                          "9a1b4bad-ccf5-4c67-8718-02696cb351e4"
                        ],
                        "boost": 1
                      }
                    },
                    {
                      "terms": {
                        "beneficiaryId.keyword": [
                          "602c7d66-e990-4dfb-b6e2-72b62ff159d5"
                        ],
                        "boost": 1
                      }
                    }
                  ],
                  "adjust_pure_negative": true,
                  "boost": 1
                }
              },
              "9a1b4bad-ccf5-4c67-8718-02696cb351e4_8da52e51-eabf-4f6c-b9f0-e222933c1cb7": {
                "bool": {
                  "filter": [
                    {
                      "terms": {
                        "clientId.keyword": [
                          "9a1b4bad-ccf5-4c67-8718-02696cb351e4"
                        ],
                        "boost": 1
                      }
                    },
                    {
                      "terms": {
                        "payerId.keyword": [
                          "8da52e51-eabf-4f6c-b9f0-e222933c1cb7"
                        ],
                        "boost": 1
                      }
                    }
                  ],
                  "adjust_pure_negative": true,
                  "boost": 1
                }
              },
              "9a1b4bad-ccf5-4c67-8718-02696cb351e4_8da52e51-eabf-4f6c-b9f0-e222933c1cb7_602c7d66-e990-4dfb-b6e2-72b62ff159d5": {
                "bool": {
                  "filter": [
                    {
                      "terms": {
                        "clientId.keyword": [
                          "9a1b4bad-ccf5-4c67-8718-02696cb351e4"
                        ],
                        "boost": 1
                      }
                    },
                    {
                      "terms": {
                        "payerId.keyword": [
                          "8da52e51-eabf-4f6c-b9f0-e222933c1cb7"
                        ],
                        "boost": 1
                      }
                    },
                    {
                      "terms": {
                        "beneficiaryId.keyword": [
                          "602c7d66-e990-4dfb-b6e2-72b62ff159d5"
                        ],
                        "boost": 1
                      }
                    }
                  ],
                  "adjust_pure_negative": true,
                  "boost": 1
                }
              }
            },
            "other_bucket": false,
            "other_bucket_key": "_other_"
          },
          "aggregations": {
            "beneficiary_amount": {
              "stats": {
                "field": "beneficiaryAmountUsd"
              }
            },
            "payer_amount": {
              "stats": {
                "field": "payerAmountUsd"
              }
            },
            "distinct_count_beneficiary": {
              "cardinality": {
                "field": "beneficiaryId.keyword"
              }
            },
            "distinct_count_payer": {
              "cardinality": {
                "field": "payerId.keyword"
              }
            },
            "distinct_count_client": {
              "cardinality": {
                "field": "clientId.keyword"
              }
            },
            "distinct_count_bank_acc": {
              "cardinality": {
                "field": "bankAccountUserId.keyword"
              }
            },
            "distinct_count_bene_country": {
              "cardinality": {
                "field": "beneficiaryCountry.keyword"
              }
            },
            "distinct_count_payer_country": {
              "cardinality": {
                "field": "payerCountry.keyword"
              }
            },
            "distinct_count_bene_currency": {
              "cardinality": {
                "field": "beneficiaryCurrency.keyword"
              }
            },
            "distinct_count_payer_currency": {
              "cardinality": {
                "field": "payerCurrency.keyword"
              }
            },
            "structured_payment_amount_personal": {
              "range": {
                "field": "payerAmountUsd",
                "ranges": [
                  {
                    "from": 9000,
                    "to": 9999.999
                  }
                ],
                "keyed": false
              }
            },
            "structured_payment_amount_company": {
              "range": {
                "field": "payerAmountUsd",
                "ranges": [
                  {
                    "from": 112500000,
                    "to": 124999999.999
                  }
                ],
                "keyed": false
              }
            }
          }
        }
      }
    }
  }
}

Test result : (Single Index)

ScenarioNumber of runExecution timesMinMaxAverage
Single thread Search result hit Result size unset 10 6s, 5.9s, 5.9s, 6s, 5.9s, 6.2s, 6.2s, 6.2s, 6.6s, 8.2s 5.9s 8.2s 6.93s
Single thread Search result hit Result size = 0 10 7200ms, 12ms, 19ms, 20ms, 23ms, 22ms, 28ms, 21ms, 19ms, 29ms 12ms 7200ms 19.3ms (First execution takes 7.2s, deviates too much from the rest, so exclude it)
Single thread Search result none hit Result size unset 10 510ms, 529ms, 549ms, 512ms, 489ms, 520ms, 506ms, 500ms, 493ms, 499ms 489ms 549ms 510.7ms
Single thread Search result none hit Result size = 0 10 389ms, 1ms, 2ms, 3ms, 3ms, 1ms, 1ms, 1ms, 2ms, 3ms 389ms 1ms 1.89ms (First execution takes 389ms, deviates too much from the rest, so exclude it)
20 threads Search result hit Result size unset 10 Exception, listener timeout after waiting for [60000] ms
20 threads Search result hit Result size = 0 10 18ms, 29ms, 61ms, 53ms, 93ms, 20ms, 39ms, 82ms, 17ms, 24ms 17ms 82ms 43.6ms
20 threads Search result none hit Result size unset 10 4.4s, 5.8s, 4.9s, 5.6s 4.6s, 5.4s, 5.3s 4.7s, 5.8s, 4.7s 5.8s 4.4s 5.1s
20 threads Search result none hit Result size = 0 10 1.4s, 25ms, 6ms, 22ms, 276ms, 250ms, 19ms, 30ms, 11ms, 18ms 1.4s 11ms 73ms (First execution takes 1.4s, deviates too much from the rest, so exclude it)

Test result: (Multiple Index)

ScenarioNumber of runExecution timesMinMaxAverage
Single thread Search result hit Result size unset 10 12.6s, 12.7s 12.6s 12.7s, 13.1s, 13.1s, 13.2s, 13,1s, 13.1s, 13.1s 12.6s 13.2s 12.9s
Single thread Search result hit Result size = 0 10 212ms, 147ms, 170ms, 272ms, 223ms, 222ms, 207ms, 255ms, 219ms, 219ms 147ms 272ms 214.6ms
Single thread Search result none hit Result size unset 10 1.2s, 1.1s, 1.1s, 1.1s, 1.1s, 1.2s, 1.1s, 1.1s, 1.2s, 1.1s 1.1s 1.2s 1.13s
Single thread Search result none hit Result size = 0 10 909ms, 23ms, 23ms, 35ms, 14ms, 26ms, 19ms, 12ms, 15ms, 22m 12ms 909ms 18.9ms (First execution takes 909ms, deviates too much from the rest, so exclude it)
20 threads Search result hit Result size unset 10 Exception, listener timeout after waiting for [60000] ms
20 threads Search result hit Result size = 0 10 18ms, 29ms, 61ms, 53ms, 93ms, 20ms, 39ms, 82ms, 17ms, 24ms 17ms 93ms 43.6ms
20 threads Search result none hit Result size unset 10 7.8s, 7.6s, 7.1s, 6.9s, 7.4s, 8s, 7.3, 7.2s, 7.3s, 7.4s 6.9s 8s 7.4s
20 threads Search result none hit Result size = 0 10 202ms, 91ms, 100ms, 146ms, 182ms, 131ms, 158ms, 48ms, 171ms, 152ms 48ms 202ms 138.1ms

Conclusion:
Aggregation performance hinges on the number of documents that matches the aggregation 

Result size parameter has significant impact on aggregation performance. not only because it skipped returning hit documents, but also because it enables caching for aggregation result, otherwise, you have to force result caching by explicitly setting request_cache=true

https://www.elastic.co/guide/en/elasticsearch/reference/6.6/shard-request-cache.html

Executing query concurrently can also have negative impact on performance

Increasing number of indices have positive impact on index speed but have large negative impact on aggregation if the aggregation is performed across indices


本文标题:Elasticsearchperformancetestingfinding
转载来于:http://lswzjz.com/article/jdeogp.html