Elk Not Passing Metadata from Filebeat into Logstash

logstash metadata not passed to elasticsearch

You can create index like this

 index => "%{[beat][name]}-%{+YYYY.MM.dd}"

This would work definitely.

Logstash stopping when metadata is in output

You should not remove the @timestamp field since it's used for the %{+YYYY.MM.dd} part in the index name.

If you absolutely want to remove the @timestamp field, another way would be to add a new field for the index name before removing the @timestamp field.

Add this before remove_field:

add_field => { "index" => "beat-%{+YYYY.MM.dd}"}

And then use %{index} in your elasticsearch output.

Filebeat does not send logs to logstash

I was finally able to resolve my problem. First, the issue with container connection was resolved as mentioned in the UPDATE (Aug 15, 2018) section of my question.

The problem with Filebeat not sending logs over to Logstash was due to the fact that I had not explicitly specified my input/output configurations to be enabled (which is a frustrating fact to me since it is not clearly mentioned in the docs). So, changing my filebeat.yml file the following fixed did the trick.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - ${PWD}/filebeat-volume/data/*.txt

output.logstash:
  enabled: true
  hosts: ["elk:5044"]
  index: "your cusotm index"

setup.kibana:
  host: "elk:5601"

Filebeat with ELK stack running in Kubernetes does not capture pod name in logs

disclaimer: I'm a beats developer

What you want to do is not yet supported by filebeat, but definitely, it's something we want to put some effort on, so you can expect future releases supporting this kind of mapping.

In the meantime, I think your approach is correct. You can append the info you need to your logs so you have it in elasticsearch

how to remove filebeat metadata

You need to remove the additional add_host_metadata and add_cloud_metadata metadata you're adding explicitly and remove the remainder of the fields with the drop_field processor:

I've tested your configuration and changed the following:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
output.console:
  pretty: true
processors:
  - drop_fields:
      fields: ["agent", "log", "input", "host", "ecs" ]
  #- add_host_metadata: ~
  #- add_cloud_metadata: ~

The result:

{
  "@timestamp": "2020-11-27T15:55:17.098Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.10.0"
  },
  "message": "2020-11-27 00:29:58 status installed libc-bin:amd64 2.28-10"
}

According to the documentation, you can't remove some of the metadata, namely the @timestamp and type (which should include the @metadata field).

The drop_fields processor specifies which fields to drop if a certain
condition is fulfilled. The condition is optional. If it’s missing,
the specified fields are always dropped. The @timestamp and type
fields cannot be dropped, even if they show up in the drop_fields
list.

EDIT:

Since you appear to be running filebeat 5.2.1, I've tried the following configuration with even better success than filebeat 7.x:

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/*.log
output.console:
  pretty: true
processors:
  - drop_fields:
     fields: ["log_type", "input_type", "offset", "beat", "source"]

Result:

{
  "@timestamp": "2020-11-30T09:51:40.404Z",
  "message": "2020-11-27 00:29:58 status half-configured vim:amd64 2:8.1.0875-5",
  "type": "log"
}

EDIT2:

Conversely, because you've posted a filebeat 6.8.0 version output, I've also tested with this very same version:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
output.console:
  pretty: true
processors:
  - drop_fields:
      fields: ["beat", "source", "prospector", "offset", "host", "log", "input", "event", "fileset" ]
  #- add_host_metadata: ~
  #- add_cloud_metadata: ~

Output:

{
  "@timestamp": "2020-11-30T10:08:26.176Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.8.0"
  },
  "message": "2020-11-27 00:29:58 status unpacked vim:amd64 2:8.1.0875-5"
}