FileBeat 实战

序言

实际处理过程中的一些应用配置cuiyaonan2000@163.com

参考信息:

1. Change the output codec | Filebeat Reference [7.17] | Elastic

原始FileBeat的输入结构

{
	"@timestamp": "2022-08-16T06:48:44.846Z",
	"@metadata": {
		"beat": "filebeat",
		"type": "_doc",
		"version": "7.14.0"
	},
	"message": "2022-08-10 17:37:27.449 [] INFO  c.netflix.discovery.DiscoveryClient [929] - Completed shut down of DiscoveryClient",
	"input": {
		"type": "log"
	},
	"ecs": {
		"version": "1.10.0"
	},
	"host": {
		"ip": [
			"192.168.137.100",
			"fe80::91ba:8fe:7b85:759b",
			"172.17.0.1",
			"fe80::42:d8ff:fecd:466a",
			"fe80::cce1:cdff:fe00:20fe",
			"fe80::455:a9ff:fe61:152d"
		],
		"mac": [
			"00:0c:29:6a:28:0a",
			"02:42:d8:cd:46:6a",
			"ce:e1:cd:00:20:fe",
			"06:55:a9:61:15:2d"
		],
		"hostname": "localhost.localdomain",
		"name": "localhost.localdomain",
		"architecture": "x86_64",
		"os": {
			"platform": "centos",
			"version": "7 (Core)",
			"family": "redhat",
			"name": "CentOS Linux",
			"kernel": "3.10.0-1127.el7.x86_64",
			"codename": "Core",
			"type": "linux"
		},
		"id": "6aa46651a03248cdb16aedf495777b4a",
		"containerized": false
	},
	"agent": {
		"name": "localhost.localdomain",
		"type": "filebeat",
		"version": "7.14.0",
		"hostname": "localhost.localdomain",
		"ephemeral_id": "dd03f7db-d1c7-4e94-a611-06660c119476",
		"id": "8dc54ac8-aa86-4166-9aed-eb3ccaafaaf7"
	},
	"log": {
		"offset": 458376,
		"file": {
			"path": "/soft/cuiyaonan-logs/3.log"
		}
	}
}

控制output格式内容

output.console:
  codec.format:
    string: '%{[@timestamp]} %{[message]}'



#注释掉这些增强型的信息
processors:
#  - add_host_metadata:
#      when.not.contains.tags: forwarded
#  - add_cloud_metadata: ~
#  - add_docker_metadata: ~
#  - add_kubernetes_metadata: ~
 - drop_fields:
      fields: ["host", "log","agent","ecs","input"]

合并多行

修改input的如下选择:

• multiline.pattern: 指定正则表达式匹配
• multiline.negate: 匹配模式选择
• multiline.match: 向前合并or向后合并
• multiline.flush_pattern: 指定正则表达式，如果匹配则从内存中刷新当前行。
• multiline.max_lines:最大合并行数，如果超过，超过部分将丢弃，默认500。
• multiline.timeout: 合并行超时时间，默认5秒。这个会导致日志最后一行的延迟。

multiline.negatemultiline.match效果falseafter如果匹配，被追加前一行的后面falsebefore如果匹配，被追加下一行的前面trueafter如果不匹配，被追加前一行的后面truebefore如果不匹配，被追加下一行的前面