• Skip to primary navigation
  • Skip to content
  • Skip to footer
syslog-ng documentation
    • syslog-ng Open Source Edition Administration Guides
      • What syslog-ng OSE is
      • What syslog-ng OSE is not
      • Why is syslog-ng OSE needed?
      • Who uses syslog-ng OSE?
      • Supported platforms
      • The philosophy of syslog-ng
      • Logging with syslog-ng
      • Modes of operation
      • Global objects
      • Timezones and daylight saving
      • Product licensing
      • High availability support
        • BSD-syslog or legacy-syslog messages
        • IETF-syslog messages
        • Enterprise-wide message model (EWMM)
      • Message representation in syslog-ng OSE
        • Specifying data types in value-pairs
      • Things to consider when forwarding messages between syslog-ng OSE hosts
      • Commercial version of syslog-ng
      • Compiling syslog-ng OSE from source
      • Compiling options of syslog-ng OSE
      • Uninstalling syslog-ng OSE
      • Configuring MSSQL Server
      • Configuring syslog-ng OSE on client hosts
      • Configuring syslog-ng OSE on server hosts
        • How relaying log messages works
      • Managing and checking syslog-ng OSE service on Linux
      • Location of the syslog-ng OSE configuration file
      • The configuration syntax in detail
      • Notes about the configuration syntax
      • Defining configuration objects inline
      • Using channels in configuration objects
      • Global and environmental variables
        • Loading modules
        • Listing configuration options
        • Visualize the configuration
        • Including configuration files
        • Reusing configuration blocks
        • Generating configuration blocks from a script
      • Python code in external files
      • Logging from your Python code
        • default-network-drivers() source options
        • internal() source options
        • file() source options
        • How file sources are followed
        • Notes on reading kernel messages
      • Arr logs
        • wildcard-file() source options
        • hypr-audit-trail() and hypr-app-audit-trail() source options
      • Jellyfin
        • kubernetes() source options
        • linux-audit() source options
        • Prerequisites to using the mqtt() source
        • Limitations to using the mqtt() source
        • Options of the mqtt() source
        • network() source options
          • The working mechanism behind the Proxy Protocol
          • Proxy Protocol configuration and output examples
        • nodejs() source options
        • darwin-oslog() source options
        • darwin-oslog-stream() source options
        • mbox() source options
        • osquery() source options
        • opentelemetry() source options
      • Pi-hole FTL
        • pipe() source options
        • pacct() options
        • program() source options
        • Server-style Python sources
        • Python LogMessage API
        • python() and python-fetcher() source options
        • Fetcher-style Python sources
      • qBittorrent
        • snmptrap() source options
        • sun-streams() source options
        • syslog() source options
        • syslog-ng-otlp() source options
        • system() source options
        • systemd-journal() source options
        • systemd-syslog() source options
        • Change an old source driver to the network() driver
        • UNIX credentials and other metadata
        • unix-stream() and unix-dgram() source options
        • stdin() source options
        • amqp() destination options
        • collectd() destination options
        • Discord destination options
        • Batch mode and load balancing with ElasticSearch
        • elasticsearch-http() destination options
        • file() destination options
        • bigquery() destination options
        • pubsub() destination options
        • graphite() destination options
        • graylog2() destination options
        • Prerequisites
        • How syslog-ng OSE interacts with HDFS
        • Storing messages with MapR-FS
        • Kerberos authentication with syslog-ng OSE hdfs() destination
        • HDFS destination options
        • HTTP destination options
        • Batch mode and load balancing with HTTP
        • HTTP-nonjava destination options
        • The Azure auth header plugin
        • The Python HTTP header plugin
        • Shifting from Java implementation to C implementation
        • Before you begin
        • Flow control in syslog-ng OSE and the Kafka client
        • Options of the kafka() destination’s C implementation
        • loggly() destination options
        • logmatic() destination options
        • loki() destination options
        • How syslog-ng OSE connects the MongoDB server
        • mongodb() destination options
        • Prerequisites to using the mqtt() destination
        • Limitations to using the mqtt() destination
        • Options of the mqtt() destination
        • Possible error messages
        • network() destination options
        • openobserve-log() destination options
        • Batch mode and load balancing with OpenSearch
        • OpenSearch() destination options
        • opentelemetry() destination options
        • osquery() destination options
        • pipe() destination options
        • program() destination options
        • pseudofile() destination options
        • python() destination options
        • Batch mode and load balancing with Redis
        • redis() destination options
        • riemann() destination options
        • Amazon s3 options
        • Slack destination options
        • smtp() destination options
        • Converting Cisco messages
        • snmp() destination options
        • splunk-hec-event: Send log messages to Splunk HEC
        • Using the sql() driver with an Oracle database
        • Using the sql() driver with a Microsoft SQL database
          • MySQL-specific interaction methods
          • MSSQL-specific interaction methods
        • sql() destination options
        • stdout() options
        • stomp() destination options
        • sumologic-http()
        • sumologic-syslog()
        • sumologic-http() destination options
        • sumologic-syslog() destination options
        • syslog() destination options
        • syslog-ng() destination options
        • syslog-ng-otlp() destination options
        • Change an old destination driver to the network() driver
        • telegram() destination options
        • unix-stream() and unix-dgram() destination options
      • usertty
      • Custom destinations
      • Client-side failover
          • Using embedded log statements
        • if-else-elif - Conditional expressions
        • Junctions and channels
        • Log path flags
        • Flow-control and multiple destinations
        • Configuring flow-control
        • Enabling reliable disk-based buffering
        • Enabling normal disk-based buffering
          • Information about disk-buffer files
          • Getting the status information of disk-buffer files
          • Getting the list of disk-buffer files
          • Printing the content of disk-buffer files
          • Orphan disk-buffer files
          • How to process messages from an orphan disk-buffer file
        • Enabling memory buffering
        • About disk queue files
        • Using filters
        • Combining filters with boolean operators
        • Comparing macro values in filters
        • Using wildcards in filters
        • Tagging messages
          • facility()
          • filter()
          • host()
          • in-list()
          • level() or priority()
          • match()
          • message()
          • netmask()
          • netmask6()
          • program()
          • rate-limit()
          • source()
          • tags()
      • Dropping messages
      • Global options
        • Configuring TLS on the syslog-ng OSE clients
        • Configuring TLS on the syslog-ng OSE server
        • Configuring syslog-ng OSE clients with mutual authentication
        • Configuring syslog-ng OSE servers with mutual authentication
      • Password-protected keys
      • TLS options
        • Formatting messages, filenames, directories, and tablenames
        • Templates and macros
        • Date-related macros
        • Hard versus soft macros
        • Macros of syslog-ng OSE
        • Example use case
        • Using template functions
        • Template functions of syslog-ng OSE
        • Modifying the on-the-wire message format
        • Replacing message parts
        • Setting message fields to specific values
        • Setting severity
        • Setting facility
        • Setting priority
        • Setting match variables
        • Unsetting message fields
        • Renaming message fields
        • Creating custom SDATA fields
        • Setting multiple message fields to specific values
        • map-value pairs
        • Conditional rewrites
        • Adding and deleting tags
        • Rewrite the timezone of a message
        • Anonymizing credit card numbers
          • The type() options of regular expressions
          • The flags() options of regular expressions
        • Optimizing regular expressions
        • Options of syslog-parser() parsers
        • Options of CSV parsers
        • Options of key=value parsers
        • Options of JSON parsers
        • Limitations of the XML parsers
        • Options of the XML parsers
        • Options of date-parser() parsers
      • Python parser
      • Parsing tags
        • Options of apache-accesslog-parser() parsers
        • Options of linux-audit-parser() parsers
      • Cisco parser
      • Parsing EWWM messages
      • iptables parser
      • Netskope parser
        • Message format parsed by panos-parser()
        • PAN-OS parser options
      • Sudo parser
        • PostgreSQL csvlog parser options
      • MariaDB parser
        • metrics-probe options
      • Websense parser
      • Windows XML Event Log (EVTX) parser
        • Fortigate parser options
      • Check Point Log Exporter parser
        • Options of Regular expression parsers
          • The structure of the pattern database
          • How pattern matching works
          • Artificial ignorance
          • Using parser results in filters and templates
          • Downloading sample pattern databases
          • Correlating log messages using pattern databases
          • Conditional actions
          • External actions
          • Actions and message correlation
          • Using pattern parsers
          • Pattern parsers of syslog-ng OSE
          • What’s new in the syslog-ng OSE pattern database format V5
              • patterns
                    • urls
                    • values
                      • example
                  • create-context
              • tags
      • OpenTelemetry parser
      • Correlating messages using the grouping-by() parser
      • Referencing earlier messages of the context
      • Options of grouping-by parsers
        • Using filters as selector
        • Shell-style globbing in the selector
        • Options of add-contextual-data()
        • Referring to parts of the message as a macro
        • Using the GeoIP2 parser
        • Transferring your logs to Elasticsearch using GeoIP2
        • Options of geoip2 parsers
      • Metrics and counters of syslog-ng OSE
      • Log statistics from the internal() source
      • Multithreading concepts of syslog-ng OSE
      • Configuring multithreading
      • Optimizing multithreaded performance
      • Possible causes of losing log messages
      • Creating syslog-ng OSE core files
      • Collecting debugging info
      • Running a failure script
      • Stopping syslog-ng
      • Reporting bugs and finding help
      • Recover data from orphaned diskbuffer files
      • Unusual storage directory
      • Unusual port number
      • Error messages
      • Using execmem prevented by SELinux
      • General recommendations
      • Handling large message load
        • Resolving hostnames locally
      • Collecting logs from chroot
      • Configuring log rotation
        • Load balancing with round robin
        • Configuration generator
      • The dqtool tool manual page
      • The loggen manual page
      • The pdbtool manual page
      • The secure-logging manual page
      • The slogencrypt manual page
      • The slogkey manual page
      • The slogverify manual page
      • The syslog-ng OSE control tool manual page
      • The syslog-debun manual page
      • The syslog-ng OSE manual page
      • The syslog-ng.conf manual page
      • CC by-nc-nd License
      • Documentation license
      • Glossary
      • Contributor License Agreement

    Multithreading and scaling in syslog-ng OSE

    On this page

    Starting with version 3.3, syslog-ng OSE can process sources and destinations in multithreaded mode to scale to multiple CPUs or cores for increased performance. Starting with version 3.6, this multithreaded mode is the default.

    Updated: October 21, 2024

    Previous
    Log statistics from the internal() source
    Next
    Multithreading concepts of syslog-ng OSE
    © 2024 Powered by Jekyll, Minimal Mistakes & Lunr.