#include <logpipe.h>

Public Attributes
gboolean	ack_needed

gboolean	flow_control_requested

gboolean *	matched

const LogPathOptions *	lpo_parent_junction

Detailed Description

Processing pipeline

Within syslog-ng, the user configuration is converted into a tree-like structure. It's node in this tree is a LogPipe object responsible for queueing message towards the destination. Each node is free to drop/transform the message it receives.

The center.c module contains code that transforms the configuration into the log processing tree. Each log statement in user configuration becomes a linked list of pipes, then each source, referenced by the is piped into the newly created pipe.

Something like this:

source1 -> log statement1 |-> log statement2

E.g. each source is sending to each log path it was referenced from. Each item in the log path is a pipe, which receives messages and forwards it at its discretion. Filters are pipes too, which lose data. Destinations are piping their output to the next element on the pipeline. This basically means that the pipeline is a wired representation of the user configuration without having to loop through configuration data.

Reference counting

The pipes do not reference each other through their pipe_next member, simply because there'd be too much reference loops to care about. Instead pipe_next is a borrowed reference, which is assumed to be valid as long as the configuration is not freed.

Flow control

Flow control is the mechanism used to control the message rate between input/output sides of syslog-ng in order to avoid message loss. If the two sides were independent, the input side could well receive messages at a much higher rate than the destination is able to cope with.

This is implemented by allocating a per-source window (similar to a TCP window), which can be "filled" by the source without the danger of output queue overflow. Also, whenever a message is processed by the destination it invokes an ACK, which in turn increments the window size.

This basically boils down to the following:

the source is free to receive as much messages as fits into its window
whenever the destination has processed a message, this is signalled to freeing up a lot in its window
if the message is full, the source is suspended, no further messages are received.

This controls the message rate but doesn't completely ruin throughput, as the source has some space without being suspended, as suspension and resuming action takes considerable amount of time (mostly latency, but CPU is certainly also used).

There are currently two forms of flow control:

hard flow control
soft flow control

The first is the form of flow control present in earlier syslog-ng versions and was renamed as "hard" in order to differentiate from the other form. Hard means that the source is completely suspended until the destination indeed processed a message. If the network is down, the disk is full, the source will not accept messages.

Soft flow control was introduced when syslog-ng became threaded and the earlier priority based behaviour couldn't be mimiced any other way. Soft flow control cannot be configured, it is automatically used by file destinations if "hard" flow control is not enabled by the user. Soft flow control means that flow is only controlled as long as the destination is writable, if an error occurs (disk full, etc) messages get dropped on the floor. But as long as the destination is writable, the destination rate controls the source rate as well.

The behaviour in non-threaded syslog-ng was, that destinations were prioritized over sources, and whenever a destination was writable, sources were implicitly suspended. This is not easily implementable by threads and ivykis, thus this alternative mechanism was created.

Please note that soft-flow-control is a somewhat stronger guarantee than the earlier behaviour, therefore it is currently only used for destination files.

Plugin overrides

Various methods can be overridden by external objects within LogPipe and derived classes. The aim of this functionality to make it possible to attach new functions to a LogPipe at runtime.

For example, it'd make sense to implement the "suppress" functionality as such plugin, which is currently implemented in LogWriter, and in case a non-LogWriter destination would need it, then a separate implementation would be needed.

The way to override a method by an external object is as follows:

it should save the current value of the method address (for example "queue" for the queue method)
it should change the pointer pointing to the relevant method to its own code (e.g. change "queue" in LogPipe)
once the hook is invoked, it should take care about calling the original function

Member Data Documentation

◆ ack_needed

gboolean LogPathOptions::ack_needed

◆ flow_control_requested

gboolean LogPathOptions::flow_control_requested

◆ lpo_parent_junction

const LogPathOptions* LogPathOptions::lpo_parent_junction

◆ matched

gboolean* LogPathOptions::matched

The documentation for this struct was generated from the following file: