python(): writing server-style Python sources

This section describes server-style sources. For details on fetcher-style sources, see python-fetcher: writing fetcher-style Python sources.

The following points apply to using Python blocks in syslog-ng OSE in general:

Python parsers and template functions are available in syslog-ng OSE version 3.10 and later.

Python destinations and sources are available in syslog-ng OSE version 3.18 and later.
Supported Python versions: 2.7 and 3.4+ (if you are using pre-built binaries, check the dependencies of the package to find out which Python version it was compiled with).
The Python block must be a top-level block in the syslog-ng OSE configuration file.
If you store the Python code in a separate Python file and only include it in the syslog-ng OSE configuration file, make sure that the PYTHON_PATH environment variable includes the path to the Python file, and export the PYTHON_PATH environment variable. For example, if you start syslog-ng OSE manually from a terminal and you store your Python files in the /opt/syslog-ng/etc directory, use the following command: export PYTHONPATH=/opt/syslog-ng/etc.

In production, when syslog-ng OSE starts on boot, you must configure your startup script to include the Python path. The exact method depends on your operating system. For recent Red Hat Enterprise Linux, Fedora, and CentOS distributions that use systemd, the systemctl command sources the /etc/sysconfig/syslog-ng file before starting syslog-ng OSE. (On openSUSE and SLES, /etc/sysconfig/syslog file.) Append the following line to the end of this file: PYTHONPATH=”<path-to-your-python-file>“, for example, PYTHONPATH=”/opt/syslog-ng/etc”.
The Python object is initiated every time when syslog-ng OSE is started or reloaded.

CAUTION: If you reload syslog-ng OSE, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of syslog-ng OSE typically involves a reload.

The Python block can contain multiple Python functions.
Using Python code in syslog-ng OSE can significantly decrease the performance of syslog-ng OSE, especially if the Python code is slow. In general, the features of syslog-ng OSE are implemented in C, and are faster than implementations of the same or similar features in Python.
Validate and lint the Python code before using it. The syslog-ng OSE application does not do any of this.
Python error messages are available in the internal() source of syslog-ng OSE.
You can access the name-value pairs of syslog-ng OSE directly through a message object or a dictionary.
To help debugging and troubleshooting your Python code, you can send log messages to the internal() source of syslog-ng OSE. For details, see Logging from your Python code.

NOTE: Starting with 3.26, syslog-ng OSE assigns a persist name to Python sources and destinations. The persist name is generated from the class name.
If you want to use the same Python class multiple times in your syslog-ng OSE configuration, add a unique persist-name() to each source or destination, otherwise syslog-ng OSE will not start.

For example:
log {  
   source { python(class(PyNetworkSource) options("port" "8080") persist-name("<unique-string>); };
   source { python(class(PyNetworkSource) options("port" "8081")); };
};  
Alternatively, you can include the following line in the Python package: @staticmethod generate_persist_name. For example:
from syslogng import LogSource
   class PyNetworkSource(LogSource):
   @staticmethod
   def generate_persist_name(options):
       return options["port"]
   def run(self):
       pass
   def request_exit(self):
       pass

Declaration

Python sources consist of two parts. The first is a syslog-ng OSE source object that you define in your syslog-ng OSE configuration and use in the log path. This object references a Python class, which is the second part of the Python source. The Python class receives or fetches the log messages, and can do virtually anything that you can code in Python. You can either embed the Python class into your syslog-ng OSE configuration file, or store it in an external Python file.

source <name_of_the_python_source>{
    python(
        class("<name_of_the_python_class_executed_by_the_source>")
        options(
            "option1" "value1",
            "option2" "value2"
        )
    );
};
 
python {
    from syslogng import LogSource
    from syslogng import LogMessage

    class <name_of_the_python_class_executed_by_the_source>(LogSource):
        def init(self, options): # optional
            print("init")
            print(options)
            self.exit = False
            return True

        def deinit(self): # optional
            print("deinit")

        def run(self): # mandatory
            print("run")
            while not self.exit:
                # Must create a message
                msg = LogMessage("this is a log message")
                self.post_message(msg)

        def request_exit(self): # mandatory
            print("exit")
            self.exit = True
    };

Methods of the python() source

Server-style Python sources must be inherited from the syslogng.LogSource class, and must implement at least the run and request_exit methods. Multiple inheritance is allowed, but only for pure Python super classes.

You can implement your own event loop, or integrate the event loop of an external framework or library, for example, KafkaConsumer, Flask, Twisted engine, and so on.

To post messages, call LogSource::post_message() method in the run method.

init(self, options) method (optional)

The syslog-ng OSE application initializes Python objects every time when it is started or reloaded. The init method is executed as part of the initialization. You can perform any initialization steps that are necessary for your source to work.

When this method returns with False, syslog-ng OSE does not start. It can be used to check options and return False when they prevent the successful start of the source.

options: This optional argument contains the contents of the options() parameter of the syslog-ng OSE configuration object as a Python dictionary.

run(self) method (mandatory)

Use the run method to implement an event loop, or start a server framework or library. Create LogMessage instances in this method, and pass them to the log paths by calling LogSource::post_message().

Currently, run stops permanently if an unhandled exception happens.

For details on parsing and posting messages, see Python LogMessage API.

request_exit(self) method (mandatory)

The syslog-ng OSE application calls this method when syslog-ng OSE is shut down or restarted. The request_exit method must shut down the event loop or framework, so the run method can return gracefully. If you use blocking operations within the run() method, use request_exit() to interrupt those operations and set an exit flag, otherwise syslog-ng OSE is not able to stop. Note that syslog-ng OSE calls the request_exit method from a thread different from the source thread.

The deinit(self) method (optional)

This method is executed when syslog-ng OSE is stopped or reloaded. This method does not return a value.

For the list of available optional parameters, see python() and python-fetcher() source options.