Detailed background on syslog-gollector can be found on this blog post.

syslog-gollector is a Syslog Collector (sometimes called a Syslog Server), written in Go (golang), which has support for streaming received log messages to Apache Kafka, version 0.8. Log messages can be written to Kafka in parsed format, or written exactly as received.

The logs lines must be RFC5424 compliant, and in the following format:

<PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROC-ID MSGID MSG"

Consult the RFC to learn what each of these fields is. The TIMESTAMP field must be in RFC3339 format. Lines not matching this format are dropped by the syslog-gollector.

Checking out the "Running" section for hints on how to suitably configure Syslog clients.

The syslog-gollector supports multi-line log messages, so messages such as stack traces will be considered a single log message.

Parsing mode is enabled by default. In this mode, the Syslog header is parsed, and the fields become keys in a JSON structure. This JSON structure is then written to Kafka. If parsing mode is not enabled, the log line is written to Kafka as it was received.

For example, imagine the following log line is received by the syslog-gollector:

<134>1 2013-09-04T10:25:52.618085 ubuntu sshd 1999 - password accepted for user root

With parsing disabled, the line is written as is to Kafka. With parsing enabled, the following JSON object is instead written to Kafka:

{
    "priority":134,
    "version":1,
    "timestamp":"2013-09-04T10:25:52.618085",
    "host":"ubuntu",
    "app":"sshd",
    "pid":1999,
    "msgid": "-",
    "message": "password accepted for user root"
}

This parsed form may be useful to downstream consumers.

Tested on 64-bit Kubuntu 14.04.

mkdir ~/syslog-gollector # Or a directory of your choice.
cd ~/syslog-gollector
export GOPATH=$PWD
go get github.com/otoolep/syslog-gollector
go install github.com/otoolep/syslog-gollector

To run the tests execute:

go get gopkg.in/check.v1
go test github.com/otoolep/syslog-gollector/...

The binary will be located in the $GOPATH/bin directory. Execute

    syslog-gollector -h

for command-line options.

Make sure your Kafka cluster is up and running first. Point your syslog clients at the syslog-gollector, ensuring the log message format is what syslog-gollector expects. Both rsyslog and syslog-ng support templating, which make it easy to format messages correctly. For example, an rsyslog template looks like so:

$template SyslogGollector,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% - %msg%"

syslog-ng looks like so:

template SyslogGollector { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} - $MSG"); template_escape(no) };

The syslog-gollector exposes a number of HTTP endpoints, for configuration and management. The Admin server runs on localhost:8080 by default.

/statistics
/diagnostics

Adding the query parameter pretty to the URL will produce pretty-printed output. For example:

curl 'localhost:8080/statistics?pretty'

The most significant dependencies are:

• The Kafka 0.8 client sarama.
• The unit-test framework Package check.
• go-metrics for statistics support.

Thanks to the creators of these packages.

This code is still work-in-progress, and issues are being tracked. Other key tasks that span multiple issues include:

• Throughput needs to be measured.
• Run the program through Go's race-detector.

syslog-gollector reports a small amount anonymous data to Loggly, each time it is launched. This data is just the host operating system and system architecture and is only used to track the number of syslog-gollector deployments. Reporting can be disabled by passing -noreport=true to syslog-gollector at launch time.

Nothing to do with gollector/gollector.