logstash6.5.4 解析 nginx 日志格式报错
log_format elk ‘{“time_local”:”$time_iso8601″,’
‘”remote_addr”:”$remote_addr”,’
‘”referer”:”$http_referer”,’
‘”request”:”$request”,’
‘”status”:$status,’
‘”bytes”:$body_bytes_sent,’
‘”agent”:”$http_user_agent”,’
‘”x_forwarded”:”$http_x_forwarded_for”,’
‘”up_addr”:”$upstream_addr”,’
‘”up_host”:”$upstream_http_host”,’
‘”reqeust_time”:”$request_time”‘
日志如下:
{“time_local”:”2020-05-16T12:43:48+08:00″,”remote_addr”:”192.168.5.148″,”referer”:”-“,”request”:”GET / HTTP/1.1″,”status”:304,”bytes”:0,”agent”:”Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36″,”x_forwarded”:”-“,”up_addr”:”-“,”up_host”:”-“,”reqeust_time”:”0.000″}
如果只是单纯解析时间,是没问题
input {
file {
path => “/var/log/nginx/access.elk.log”
}
}
filter {
grok {
match => [ “message”,”%{TIMESTAMP_ISO8601:locals}” ]
}
}
output {
stdout { codec => rubydebug }
}
结果:
“@version” => “1”,
“host” => “localhost.localdomain”,
“path” => “/var/log/nginx/access.elk.log”,
“message” => “{“time_local”:”2020-05-16T12:43:48+08:00″,”remote_addr”:”192.168.5.148″,”referer”:”-“,”request”:”GET / HTTP/1.1″,”status”:304,”bytes”:0,”agent”:”Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36″,”x_forwarded”:”-“,”up_addr”:”-“,”up_host”:”-“,”reqeust_time”:”0.000″}”,
“locals” => “2020-05-16T12:43:48+08:00”,
“@timestamp” => 2020-05-16T04:43:49.638Z
}
如果和解析 IP 一起使用,就报错:
input {
file {
path => “/var/log/nginx/access.elk.log”
}
}
filter {
grok {
match => [ “message”,”%{IP:client} %{TIMESTAMP_ISO8601:locals}” ]
}
}
output {
stdout { codec => rubydebug }
}
结果:
{
“@version” => “1”,
“host” => “localhost.localdomain”,
“path” => “/var/log/nginx/access.elk.log”,
“message” => “{“time_local”:”2020-05-16T12:50:00+08:00″,”remote_addr”:”192.168.5.148″,”referer”:”-“,”request”:”GET / HTTP/1.1″,”status”:304,”bytes”:0,”agent”:”Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36″,”x_forwarded”:”-“,”up_addr”:”-“,”up_host”:”-“,”reqeust_time”:”0.000″}”,
“tags” => [
[0] “_grokparsefailure”
],
“@timestamp” => 2020-05-16T04:50:01.476Z
}
有大神了解这是为什么吗,只要是和解析时间的表达式一起用,就报错。%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}”一起用就没问题