To properly handle a folder (spool) of unified2 files in Unifiedbeat
- Filebeat's prospector/harvester approach is inappropriate for unifiedbeat
- while unifiedbeat does work as-is today, changes should recognize:
- the volume of data is a lot less than syslogs (given a well tuned sensor)
- there's only one folder containing unified2 files
- there's only one unified2 file that is active and should be tailed
- of course, the above is all wrong if a server is used to centralize unified2 files from multiple sensors
- then again, one could just run this version of unifiedbeat for each folder (i.e. each sensor) [KISS]
- while unifiedbeat does work as-is today, changes should recognize:
- add
os.Stat(r.reader.File.Name())
tospoolrecordreader.go
- to deal with renamed or deleted files
- change
Offset()
inspoolrecordreader.go
to return full path not justpath.Base
- add sample unified2 files in
sample_data
- add examples
clsreadu2.go
simple reader with countsclsspoolreader.go
to test SpoolRecordReader- use
CloseHook
to rename (archive) an indexed file- otherwise, an endless reading loop occurs (
openNext
flaw)
- otherwise, an endless reading loop occurs (
- use
- add new fields to the SpoolRecordReader struct:
FileSource
- the file currently being tailed (read)FileOffset
- the offset position into FileSource- these fields allow for a registry file -- to bookmark where we were if interrupted
- use these fields in
openNext
to set the offset in the call toNewRecordReader
A Go(lang) Library for decoding unified2 log files as generated by IDS applications such as Snort and Suricata.
go get github.com/jasonish/go-unified2
See https://godoc.org/github.com/jasonish/go-unified2
For more information on the unified2 file format see the Snort Manual.