Logstash Too Many Files Opened

Too many open files on Elasticsearch

Just set elasticsearch - nofile to unlimited.
And make sure you running with the right user.

Elasticsearch too many open files

OK, so you have 5 nodes on the same host and you can have maximum 9000 open files. If you sum up the second column you're above that number, hence the error you're getting.

In order to see during startup how many max open files your ES is configured with, you can start your process with -Des.max-open-files=true and your log will show you how many max open files you can have.

Check here and here (depends on what Linux distro you have) on how to configure that settings for your Linux distro, but you might have to tweak /etc/security/limits.conf as well.

Logstash file limit?

If you archive the files from your server, logstash will stop watching them and release the file handle.

If you don't delete or move them, maybe there's a file pattern that only matches new files? Sometimes the current file is "foo.log" before being rotated to "foo.2014-11-17.log".

Logstash Forwarder (a lighter-weight shipper than the full Logstash) has a concept of "Dead Time", where it will stop watching a file if it's been inactive (default 24 hours).

Too many open files warning from elasticsearch

How to increase the maximum number of allowed open files depends slightly on your linux distribution. Here are some instructions for ubuntu and centos:

http://posidev.com/blog/2009/06/04/set-ulimit-parameters-on-ubuntu/
http://pro.benjaminste.in/post/318453669/increase-the-number-of-file-descriptors-on-centos-and

The elasticsearch documentation recommends setting the maximum file limit to 32k or 64k. Since you are at 16k and are already hitting a limit, I'd probably set it higher; something like 128k. See: http://www.elasticsearch.org/guide/reference/setup/installation/

After upping the number of open files and restarting elasticsearch you will want to verify that it worked by re-running the curl command you mentioned:

curl -XGET 'http://localhost:9200/_nodes?os=true&process=true&pretty=true'

As you add more indices (along with more documents), you will also see the number of files elasticsearch keeps track of increase. If you notice performance degradation with all of the indicies and documents, you can try adding a new node to your cluster: http://www.elasticsearch.org/guide/reference/setup/configuration/ - since you already have a sharded, replicated configuration, this should be a relatively painless process.

Errror for type and max_open_files on the logstash server's logs

The first error says, that logstash is trying to update the mapping for a specific index. This update will add a new mapping for the type "doc" but there is already a mapping for "messages" present. This would result in two mappings in same index, what is not supported anymore.
Please check the mapping for this index and the type of the documents you´re trying to index in your syslog-* indices. Maybe you have used the very same index already for some kind of documents with the type "message"?

The second error says, that the number of open files is reached. To increase it permanently, youl´ll need to follow this instructions (what you have partly applied already). Issue this changes not only on your elasticsearch server but also on the logstash host.

In order to apply this settings while the server is running you need to execute this command and restart the service:

sudo ulimit -n 65535

How to identify the bottleneck in slow Filebeat ingestion

At the time of writing, there is no specific logging in Filebeat or Logstash which makes it clear that there is 'backpressure' or where it is coming from. As was suggested on a few discussion forums, what you need to do is to run each component in isolation to benchmark the performance.

Set Filebeat to output to console and redirect the console to /dev/null. You can then use filebeat logs to measure how long Filebeat is taking to process a specific file.
Point Filebeat at Logstash, and redirect the Logstash output to /dev/null. Again, monitor the filebeat logs to see how long the processing takes. If it's slow at this point, then you may need to inspect the filters you are using. If you have simple or no filters, then maybe check if Logstash is CPU or memory constrained. Ideally there should be little/no slowdown processing through logstash vs just redirecting Filebeat to /dev/null
Point Logstash at Elasticsearch and see if the performance changes. If so, now is the time to tweak your logstash pipeline settings, changing bulk sizes, worker counts etc. You might also need to look at Elasticsearch performance, determining if you are CPU/Memory/Network or Disk I/O bound.

In my specific case, it was a mix of Logstash pipeline configuration and bumping of memory on my elasticsearch data nodes to solve my ingestion problem.