Elasticsearch¶

Optimisations¶

Remove old indices’ replicas using Curator “action: replicas”

Best practices: 1) max heap size for java: 30-32GB 2) one shard per index per node 3) two replicas per index for failover

Get cluster stats:

curl localhost:9200/_cluster/stats?human&pretty
# /_cluster/stats/nodes/node1,node*,master:false

Get health:

curl localhost:9200/_cat/health?v
# Output
#epoch      timestamp cluster        status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
#1548094256 20:10:56  docker-cluster green           1         1    645 645    0    0        0             0                  -                100.0%

Get allocation:

curl localhost:9200/_cat/allocation?v

# Output:
#shards disk.indices disk.used disk.avail disk.total disk.percent host      ip        node
# 5         260b    47.3gb     43.4gb    100.7gb           46 127.0.0.1 127.0.0.1 CSUXak2

Get index settings:

curl "localhost:9200/<index>/_settings"

# multiple
/<index1>,<index2>/_settings
/_all/_settings
/log_2013_*/_settings

# filtering settings by name
/log_2013_-*/_settings/index.number_*

Create template with settings for all indices:

curl -XPUT "localhost:9200/_template/all" -H 'Content-Type: application/json' -d'
{
  "template": "*",
  "settings": {
    "number_of_replicas": 0,
    "number_of_shards": 1
  }
}
'

Basics¶

Templates:

# get
curl localhost:9200/_cat/templates?v
# check if the template exists
curl -I localhost:9200/_template/<template>
# delete
curl -XDELETE localhost:9200/_template/<template>

Clear all indexes:

curl -XDELETE localhost:9200/*

Delete index:

curl -XDELETE localhost:9200/<index>

Get all indexes and their sizes:

# more actual options here - https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-indices.html
curl localhost:9200/_cat/indices?v
    ?v          - show headers
    ?h=index    - show only indexes

Curator¶

# show elasticsearch indices
curator_cli show_indices

Installation¶

Ubuntu 16.04:

sudo sh -c "echo 'deb http://packages.elastic.co/curator/4/debian stable main' > /etc/apt/sources.list.d/elasticsearch-curator.list"
sudo apt update && sudo apt install elasticsearch-curator

Configuration¶

Configuration file /<HOME>/.curator/curator.yml:

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

Delete old Elasticsearch indices¶

Create action file. Example - del.yml:

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 30 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 30
      exclude:

Run action:

curator del.yml

# curator --dry-run del.yml