How To Install And Configure Elasticsearch Cluster In Linux

How To Install And Configure Elasticsearch Cluster In Linux

How To Install And Configure Elasticsearch Cluster In Linux

Hello Everyone

Welcome to CloudAffaire and this is Debjeet.

In this series, we will explore one of the most popular log management tools in DevOps better known as ELK (E=Elasticserach, L=Logstash, K=Kibana) stack.

What Is Elasticsearch?

Elasticsearch is a distributed, open source search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. Elasticsearch is built on Apache Lucene and was first released in 2010 by Elasticsearch N.V. (now known as Elastic). Known for its simple REST APIs, distributed nature, speed, and scalability, Elasticsearch is the central component of the Elastic Stack, a set of open source tools for data ingestion, enrichment, storage, analysis, and visualization. Commonly referred to as the ELK Stack (after Elasticsearch, Logstash, and Kibana), the Elastic Stack now includes a rich collection of lightweight shipping agents known as Beats for sending data to Elasticsearch.

Installing Elasticsearch Cluster:

Step 1: Configure yum repository for elasticsearch.

Step 2: Install elasticsearch cluster.

Configuring Elasticsearch Cluster:

Elasticsearch ships with good defaults and requires very little configuration. Most settings can be changed on a running cluster using the Cluster update settings API.

Config files location:

  • elasticsearch.yml for configuring Elasticsearch
  • jvm.options for configuring Elasticsearch JVM settings
  • log4j2.properties for configuring Elasticsearch logging

Note: These files are located in the config directory, whose default location depends on whether or not the installation is from an archive distribution (tar.gz or zip) or a package distribution (Debian or RPM packages).

For RPM based installation, Elasticsearch defaults to using /etc/elasticsearch for runtime configuration. Elasticsearch loads its configuration from the /etc/elasticsearch/elasticsearch.yml. The RPM also has a system configuration file located in /etc/sysconfig/elasticsearch

Elasticsearch Configuration Options:

Using /etc/elasticsearch/elasticsearch.yml:

  • path.data: Path to directory where to store the data (separate multiple locations by comma), default value: /var/lib/elasticsearch
  • path.log: Path to log files, default value: /var/log/elasticsearch
  • cluster.name: Name of your elasticsearch cluster, default value: my-application
  • node.name: Elasticsearch uses node.name as a human readable identifier for a particular instance of Elasticsearch so it is included in the response of many APIs. It defaults to the hostname that the machine has when Elasticsearch starts but can be configured explicitly in elasticsearch.yml using node.name.
  • network.host: By default, Elasticsearch binds to loopback addresses only — e.g. 127.0.0.1 and [::1]. This is sufficient to run a single development node on a server, in order to form a cluster with nodes on other servers, your node will need to bind to a non-loopback address using network.host: <HOST_IP_ADDRESS>
  • http.port: Elasticsearch uses port 9200 by default, you can change the default port using http.port:<CUSTOM_PORT>
  • discovery.seed_hosts: Pass an initial list of hosts to perform discovery when this node is started. Default value: 127.0.0.1, [::1].
  • cluster.initial_master_nodes: Bootstrap the cluster using an initial set of master-eligible nodes.
  • bootstrap.memory_lock: Lock the memory on startup
  • gateway.recover_after_nodes: Block initial recovery after a full cluster restart until N nodes are started

Using /etc/sysconfig/elasticsearch:

  • JAVA_HOME: Set a custom Java path to be used.
  • MAX_OPEN_FILES: Maximum number of open files, defaults to 65535.
  • MAX_LOCKED_MEMORY: Maximum locked memory size. Set to unlimited if you use the bootstrap.memory_lock option in elasticsearch.yml.
  • MAX_MAP_COUNT: Maximum number of memory map areas a process may have. If you use mmapfs as index store type, make sure this is set to a high value. For more information, check the linux kernel documentation about max_map_count. This is set via sysctl before starting Elasticsearch. Defaults to 262144.
  • ES_PATH_CONF: Configuration file directory (which needs to include elasticsearch.yml, jvm.options, and log4j2.properties files); defaults to /etc/elasticsearch.
  • ES_JAVA_OPTS: Any additional JVM system properties you may want to apply.
  • ES_HOME: Elasticsearch home directory, default value: /usr/share/elasticsearch
  • PID_DIR: Elasticsearch PID directory, default value: /var/run/elasticsearch
  • RESTART_ON_UPGRADE: Configure restart on package upgrade, defaults value: false.

Note: Distributions that use systemd require that system resource limits be configured via systemd rather than via the /etc/sysconfig/elasticsearch file.

Elasticsearch Directory Layout (RPM based):

  • home: Elasticsearch home directory or $ES_HOME, default location: /usr/share/elasticsearch
  • bin: Binary scripts including elasticsearch to start a node and elasticsearch-plugin to install plugins, default location: /usr/share/elasticsearch/bin
  • conf: Configuration files including elasticsearch.yml, default location: /etc/elasticsearch
  • conf: Environment variables including heap size, file descriptors, default location /etc/sysconfig/elasticsearch
  • data: The location of the data files of each index / shard allocated on the node. Can hold multiple locations, default location: /var/lib/elasticsearch
  • jdk: The bundled Java Development Kit used to run Elasticsearch. Can be overridden by setting the JAVA_HOME environment variable in /etc/sysconfig/elasticsearch. Default location: /usr/share/elasticsearch/jdk
  • logs: Log files location, default location: /var/log/elasticsearch
  • plugins: Plugin files location. Each plugin will be contained in a subdirectory, default location: /usr/share/elasticsearch/plugins

Step 3: View and change elasticsearch cluster configuration.

Step 4: Get elasticsearch cluster details.

Step 5: Get and Put data into your elasticsearch cluster.

To get more details on ELK, please refer below documentation.

https://www.elastic.co/guide/index.html

 

Leave a Reply

Close Menu