The elasticsearch-http() destination automatically sends multiple log messages in a single HTTP request, increasing the rate of messages that your Elasticsearch deployment can consume. For details on adjusting and fine-tuning the batch mode of the elasticsearch-http() destination, see the following section.

Batch size

The batch-bytes(), batch-lines(), and batch-timeout() options of the destination determine how many log messages syslog-ng OSE sends in a batch. The batch-lines() option determines the maximum number of messages syslog-ng OSE puts in a batch in. This can be limited based on size and time:

To increase the performance of the destination, increase the number of worker threads for the destination using the workers() option, or adjust the batch-bytes(), batch-lines(), batch-timeout() options.

Example: HTTP batch mode

In the following example, a batch consists of 100 messages, or a maximum of 512 kilobytes, and is sent every 20 seconds (20000 milliseconds).

destination d_elasticsearch-http {
    elasticsearch-http(url("http://your-elasticsearch-server:9200/_bulk")
        index("<elasticsearch-index-to-store-messages>")
        type("")
        url("http://your-elasticsearch-server:9200/_bulk")
        batch-lines(100)
        batch-bytes(512Kb)
        batch-timeout(10000)
    );
};

Load balancing between multiple Elasticsearch indexers

Starting with version 3.19, you can specify multiple URLs, for example, url(“site1” “site2”). In this case, syslog-ng OSE sends log messages to the specified URLs in a load-balance fashion. This means that syslog-ng OSE sends each message to only one URL. For example, you can use this to send the messages to a set of ingestion nodes or indexers of your SIEM solution if a single node cannot handle the load. Note that the order of the messages as they arrive on the servers can differ from the order syslog-ng OSE has received them, so use load-balancing only if your server can use the timestamp from the messages. If the server uses the timestamp when it receives the messages, the order of the messages will be incorrect.

CAUTION: If you set multiple URLs in the url() option, set the persist-name() option as well to avoid data loss.

Starting with version syslog-ng OSE version 3.22, you can use any of the following formats to specify multiple URLs:

url("server1", "server2", "server3"); # comma-separated strings
url("server1" "server2" "server3"); # space-separated strings
url("server1 server2 server3"); # space-separated within a single string

Example: HTTP load balancing

The following destination sends log messages to 3 different Elasticsearch indexer nodes. Each node is assigned a separate worker thread. A batch consists of 100 messages, or a maximum of 512 kilobytes, and is sent every 20 seconds (20000 milliseconds).

destination d_elasticsearch-http {
    elasticsearch-http(url("http://your-elasticsearch-server1:9200/_bulk" "http://your-elasticsearch-server2:9200/_bulk" "http://your-elasticsearch-server3:9200/_bulk")
        batch-lines(100)
        batch-bytes(512Kb)
        batch-timeout(20000)
        persist-name("d_elasticsearch-http-load-balance")
    );
};

If you are using load-balancing (that is, you have configured multiple servers in the url() option), increase the number of worker threads at least to the number of servers. For example, if you have set three URLs (url(“site1”, “site2”, “site3”)), set the workers() option to 3 or more.

Updated: