How To Install And Configure Logstash In Linux
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
In this series, we will explore one of the most popular log management tools in DevOps better known as ELK (E=Elasticserach, L=Logstash, K=Kibana) stack.
What Is Logstash?
Logstash is an open-source data collection engine with real-time pipelining capabilities. Logstash can dynamically unify data from disparate sources and normalize the data into destinations of your choice. Cleanse and democratize all your data for diverse advanced downstream analytics and visualization use cases.
While Logstash originally drove innovation in log collection, its capabilities extend well beyond that use case. Any type of event can be enriched and transformed with a broad array of input, filter, and output plugins, with many native codecs further simplifying the ingestion process. Logstash accelerates your insights by harnessing a greater volume and variety of data.
Logstash pipeline stages:
The Logstash event processing pipeline has three stages: inputs ==> filters ==> outputs. Inputs generate events, filters modify them and outputs ship them elsewhere. Inputs and outputs support codecs that enable you to encode or decode the data as it enters or exits the pipeline without having to use a separate filter. In layman term, you can compare a Logstash with an ETL tool in modern RDBMS systems.
Inputs:
Inputs are used to get data into Logstash. Logstash supports different input as your data source, it can be a plain file, syslogs, beats, cloudwatch, kinesis, s3, etc.
Filters:
Filters are intermediary processing devices in the Logstash pipeline. You can combine filters with conditionals to perform an action on an event if it meets certain criteria. Logstash supports different types of filters for data processing like gork, mutate, aggregate, csv, json, etc.
Outputs:
Outputs are the final phase of the Logstash pipeline. An event can pass through multiple outputs, but once all output processing is complete, the event has finished its execution. Logstash supports different types of outputs to store or send the final processed data like elasticsearch, cloudwatch, csv, file, mongodb, s3, sns, etc.
Logstash Directory Structure:
- home: Home directory of the Logstash installation. Default location /usr/share/logstash
- bin: Binary scripts including logstash to start Logstash and logstash-plugin to install plugins. Default location /usr/share/logstash/bin
- settings: Configuration files, including logstash.yml, jvm.options, and startup.options. Default location /etc/logstash
- conf: Logstash pipeline configuration files. Default location /etc/logstash/conf.d/*.conf
- logs: Stores log files. Default location /var/log/logstash
- plugins: Local, non Ruby-Gem plugin files. Each plugin is contained in a subdirectory. Default location /usr/share/logstash/plugins
- data: Data files used by logstash and its plugins for any persistence needs. Default location /var/lib/logstash
Logstash Configuration Files:
Logstash has two types of configuration files: pipeline configuration files, which define the Logstash processing pipeline, and settings files, which specify options that control Logstash startup and execution.
- Pipeline Configuration Files: You create pipeline configuration files when you define the stages of your Logstash processing pipeline. On deb and rpm, you place the pipeline configuration files in the /etc/logstash/conf.d directory. Logstash tries to load only files with .conf extension in the /etc/logstash/conf.d directory and ignores all other files.
- Settings Files: The settings files are already defined in the Logstash installation. Logstash includes the following settings files:
- logstash.yml: Contains Logstash configuration flags.
- pipelines.yml: Contains the framework and instructions for running multiple pipelines in a single Logstash instance.
- jvm.options: Contains JVM configuration flags. Use this file to set initial and maximum values for total heap space.
- log4j2.properties: Contains default settings for log4j 2 library.
- startup.options (Linux): Contains options used by the system-install script in /usr/share/logstash/bin to build the appropriate startup script for your system.
How To Install And Configure Logstash In Linux:
Step 1: Configure yum repository for logstash.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
######################################################### ## How To Install And Configure Logstash In Linux ## ######################################################### ## Prerequisites: One Linux system with internet access ## Linux OS: CentOs 7 ## IP: 192.168.0.10 ## ------------------------ ## Configure yum repository ## ------------------------ ## Download and install the public signing key sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch ## Create the repository file sudo vi /etc/yum.repos.d/logstash.repo --------------------- [logstash-7.x] name=Elastic repository for 7.x packages baseurl=https://artifacts.elastic.co/packages/7.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md --------------------- :wq |
Step 2: Install logstash.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
## --------------------- ## Install Logstash ## --------------------- ## Install logstash sudo yum install logstash ## Enable and start logstash sudo systemctl daemon-reload sudo systemctl enable logstash sudo systemctl start logstash sudo systemctl status logstash ## View logstash configuration files sudo ls /etc/logstash/ |
Step 3: Stash your first log from the command line using logstash.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
## ------------------------------------- ## Stash your first log from commandline ## ------------------------------------- ## After starting Logstash, wait until you see "Pipeline main started" and then enter hello world at the command prompt sudo /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }' hello world ## { ## "@version" => "1", ## "message" => "hello world", ## "@timestamp" => 2020-02-16T06:49:51.384Z, ## "host" => "system1" ## } ## To stop logstash press control + d |
To get more details on ELK, please refer below documentation.
https://www.elastic.co/guide/index.html