Fluent-bit, Opensearch and Nginx-Proxy-forwarder

May 15, 2023 joerg OpenSource, Self Hosting 0

As I managed to increase my home-server-farm with two new HP t630 ThinClients, I also have the space now to finally do more with Opensearch.

The components

  • Opensearch, in case you do not know, Opensearch is a full fork of Elasticsearch. AWS/Amazon has forked it during some license changes in Elastic in order to have a full open source version of it (Apache License 2.0)
    https://www.opensearch.org/
  • Fluent-Bit, is a log-processer written by the CNCF (Cloud Native Computing Foundation). You can compare it to logstash or filebeat. License is also Apache 2.0.
    https://fluentbit.io/
  • Nginx-Proxy-Manager, is an amazing docker container which lets you setup simple nginx based proxies including Lets Encrypt
    https://github.com/NginxProxyManager/nginx-proxy-manager

Fluent-Bit

In this blog post I want to focus on the fluent-bit part, honestly the Opensearch and Nginx setups are rather straight forward and I mainly struggled (for days) with getting the logs into Opensearch

[FILTER]
    Name modify
    Match *
    Add source nginx-proxy

[FILTER]
    Name parser
    Match nginx
    Key_Name log
    Parser nginx

[SERVICE]
    flush        1
    daemon       Off
    log_level    info
    parsers_file parsers.conf
    plugins_file plugins.conf

    http_server  Off
    http_listen  0.0.0.0
    http_port    2020
    Tag 	 nginx

    storage.metrics on
    storage.path /tmp/storage

[INPUT]
    name                  tail
    refresh_interval      5
    path                  /data/compose/4/data/logs/proxy-host-*.log
    read_from_head        true
    tag			  nginx


[OUTPUT]
    Name  opensearch
    Match *
    Host  IP-ADDRESS
    Port  9200
    Index logstash-
    tls        On
    tls.verify Off
    http_User USERNAME
    http_Passwd PASSWORD
    Suppress_Type_Name On

Step by Step

  • FILTER: The first filter applies a new tag to any (*) match. As this file is solely written for nginx log files it applies the nginx tag, so I can easily filter in Opensearch
  • FILTER: The second filter is to apply the regex to the input. The regex can be found in the parser configuration file in /etc
  • SERVICE: Service describes the general setup of fluent-bit
  • INPUT: This section describes the source of the files, as you can see here we basically tail all log files in the nginx logs folder
  • OUTPUT: Here you configure your Opensearch node. As you can see tls needs to be on.

A better regex for nginx-proxy

Fluent-bit already has a good parser for most webservers, but the nginx-proxy logs are not the same as nginx. While I also investigated if I can change the logging format somewhere inside the proxies docker container I ended up with a new regex.

The parser.conf file in /etc/fluent-bit/ has an entry for nginx, you can now decide to create a new parser module for the proxy or replace the nginx regex.

The following one has some room for optimization, but it works

\[(?<timestamp>\d{2}\/\w{2,4}\/\d{4}:\d{2}:\d{2}:\d{2} \+\d{4})\]\s*\-\s*(?<status_code>\d{3})\s*\d{3}\s*\-\s*(?<method>\w{3,5})\s*(?<protocol>\w{4,5})\s*(?<host>[\.\w]*)\s*\"(?<URI>[\/\w\-\.]*)\"\s*\[Client\s*(?<remote_ip>[\d\.]*)\]\s*\[Length\s*(?<length>\d*)\]\S*\s*\S*\s*\S*\s*\[Sent\-to\s*(?<send_to>[\d\.]*)\]\s*\"(?<User_Agent>[\w\s\/\.\(\;\:\)\d\,\+]*)\"