Filebeat and AWS Elasticsearch – Not Working


I have good experience in working with Elasticsearch, I have worked with version 2.4 and now trying to learn new Elasticsearch.
I am trying to implement Filebeat to send my apache and system logs to my Elasticsearch endpoint. To save my time I preferred to launch a t2.medium single node instance over AWS Elasticsearch Service under the public domain and I have attached the access policy to allow everyone to access the cluster.
The AWS Elasticsearch instance is up and running healthy.
I launched a Ubuntu(18.04) server, downloaded the filebeat tar and made the following configuration in filebeat.yml:

I enabled the required modules :

Then as per the filebeat documentation I changed the ownership of the filebeat file and started the filebeat with the following commands :

When I started the filebeat I faced the following permission and ownership issues :

To resolve this I changed the ownership for the files which were throwing errors.
When I restarted the filebeat service , I started facing the following issue :

Going through this link , I found that to work with AWS Elasticsearch I will need Beats OSS versions.
So I again downloaded the OSS version for beat from this link and followed the same procedure as above, but still no luck. Now I am facing the following errors :

Error 1:

Error 2:

From the second error I can understand that the geoip plugin is not available because of which I facing this error.

What else needs to be done to get this working?
Has anyone been to successfully connect Beats to AWS Elasticsearch?
What other steps I could to take to mitigate the above issue?

Envrionment Details:

  • AWS Elasticsearch Version : 6.7
  • File Beat : 7.2.0


First, you need to use OSS version of filebeat with AWS ES

Second, AWS ElasticSearch does not provide GeoIP module, so you will need to edit pipelines for any of the default modules you want to use, and make sure GeoIP is removed/commented out.

For example in /usr/share/filebeat/module/system/auth/ingest/pipeline.json (that’s the path when installed from deb package – your path will be different of course) comment out:

Repeat the same for apache module.

Leave a Reply