How To Install And Configure Logstash In Linux

Hello Everyone

Welcome to CloudAffaire and this is Debjeet.

In this series, we will explore one of the most popular log management tools in DevOps better known as ELK (E=Elasticserach, L=Logstash, K=Kibana) stack.

What Is Logstash?

Logstash is an open-source data collection engine with real-time pipelining capabilities. Logstash can dynamically unify data from disparate sources and normalize the data into destinations of your choice. Cleanse and democratize all your data for diverse advanced downstream analytics and visualization use cases.

While Logstash originally drove innovation in log collection, its capabilities extend well beyond that use case. Any type of event can be enriched and transformed with a broad array of input, filter, and output plugins, with many native codecs further simplifying the ingestion process. Logstash accelerates your insights by harnessing a greater volume and variety of data.

Logstash pipeline stages:

The Logstash event processing pipeline has three stages: inputs ==> filters ==> outputs. Inputs generate events, filters modify them and outputs ship them elsewhere. Inputs and outputs support codecs that enable you to encode or decode the data as it enters or exits the pipeline without having to use a separate filter. In layman term, you can compare a Logstash with an ETL tool in modern RDBMS systems.

Inputs:

Inputs are used to get data into Logstash. Logstash supports different input as your data source, it can be a plain file, syslogs, beats, cloudwatch, kinesis, s3, etc.

Filters:

Filters are intermediary processing devices in the Logstash pipeline. You can combine filters with conditionals to perform an action on an event if it meets certain criteria. Logstash supports different types of filters for data processing like gork, mutate, aggregate, csv, json, etc.

Outputs:

Outputs are the final phase of the Logstash pipeline. An event can pass through multiple outputs, but once all output processing is complete, the event has finished its execution. Logstash supports different types of outputs to store or send the final processed data like elasticsearch, cloudwatch, csv, file, mongodb, s3, sns, etc.

Logstash Directory Structure:

home: Home directory of the Logstash installation. Default location /usr/share/logstash
bin: Binary scripts including logstash to start Logstash and logstash-plugin to install plugins. Default location /usr/share/logstash/bin
settings: Configuration files, including logstash.yml, jvm.options, and startup.options. Default location /etc/logstash
conf: Logstash pipeline configuration files. Default location /etc/logstash/conf.d/*.conf
logs: Stores log files. Default location /var/log/logstash
plugins: Local, non Ruby-Gem plugin files. Each plugin is contained in a subdirectory. Default location /usr/share/logstash/plugins
data: Data files used by logstash and its plugins for any persistence needs. Default location /var/lib/logstash

Logstash Configuration Files:

Logstash has two types of configuration files: pipeline configuration files, which define the Logstash processing pipeline, and settings files, which specify options that control Logstash startup and execution.

Pipeline Configuration Files: You create pipeline configuration files when you define the stages of your Logstash processing pipeline. On deb and rpm, you place the pipeline configuration files in the /etc/logstash/conf.d directory. Logstash tries to load only files with .conf extension in the /etc/logstash/conf.d directory and ignores all other files.
Settings Files: The settings files are already defined in the Logstash installation. Logstash includes the following settings files:
- logstash.yml: Contains Logstash configuration flags.
- pipelines.yml: Contains the framework and instructions for running multiple pipelines in a single Logstash instance.
- jvm.options: Contains JVM configuration flags. Use this file to set initial and maximum values for total heap space.
- log4j2.properties: Contains default settings for log4j 2 library.
- startup.options (Linux): Contains options used by the system-install script in /usr/share/logstash/bin to build the appropriate startup script for your system.

How To Install And Configure Logstash In Linux:

Step 1: Configure yum repository for logstash.

#########################################################
## How To Install And Configure Logstash In Linux ##
#########################################################
## Prerequisites: One Linux system with internet access
## Linux OS: CentOs 7
## IP: 192.168.0.10

## ------------------------
## Configure yum repository
## ------------------------
## Download and install the public signing key
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

## Create the repository file
sudo vi /etc/yum.repos.d/logstash.repo
---------------------
[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
---------------------
:wq

#########################################################

## How To Install And Configure Logstash In Linux ##

#########################################################

## Prerequisites: One Linux system with internet access

## Linux OS: CentOs 7

## IP: 192.168.0.10

## ------------------------

## Configure yum repository

## ------------------------

## Download and install the public signing key

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

## Create the repository file

sudo vi /etc/yum.repos.d/logstash.repo

---------------------

[logstash-7.x]

name=Elastic repository for 7.x packages

baseurl=https://artifacts.elastic.co/packages/7.x/yum

gpgcheck=1

gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch

enabled=1

autorefresh=1

type=rpm-md

---------------------

:wq

Step 2: Install logstash.

## ---------------------
## Install Logstash
## ---------------------

## Install logstash
sudo yum install logstash

## Enable and start logstash
sudo systemctl daemon-reload
sudo systemctl enable logstash
sudo systemctl start logstash 
sudo systemctl status logstash

## View logstash configuration files
sudo ls /etc/logstash/

## ---------------------

## Install Logstash

## ---------------------

## Install logstash

sudo yum install logstash

## Enable and start logstash

sudo systemctl daemon-reload

sudo systemctl enable logstash

sudo systemctl start logstash

sudo systemctl status logstash

## View logstash configuration files

sudo ls /etc/logstash/

Step 3: Stash your first log from the command line using logstash.

## -------------------------------------
## Stash your first log from commandline
## -------------------------------------

## After starting Logstash, wait until you see "Pipeline main started" and then enter hello world at the command prompt
sudo /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'

hello world

## {
##      "@version" => "1",
##       "message" => "hello world",
##    "@timestamp" => 2020-02-16T06:49:51.384Z,
##          "host" => "system1"
## }

## To stop logstash press control + d

## -------------------------------------

## Stash your first log from commandline

## -------------------------------------

## After starting Logstash, wait until you see "Pipeline main started" and then enter hello world at the command prompt

sudo /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'

hello world

## {

## "@version" => "1",

## "message" => "hello world",

## "@timestamp" => 2020-02-16T06:49:51.384Z,

## "host" => "system1"

## }

## To stop logstash press control + d

To get more details on ELK, please refer below documentation.

https://www.elastic.co/guide/index.html