Skip to content

shageman/loggregator

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

89 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Loggregator Build Status

Logging in the Clouds

Loggregator is the user application logging subsystem for Cloud Foundry.

Features

Loggregator allows users to:

  1. Tail their application logs.
  2. Dump a recent set of application logs (where recent is on the order of an hour).
  3. Continually drain their application logs to 3rd party log archive and analysis services.

Usage

$ cf logs --tail
Started GET "/" for 127.0.0.1 at 2013-04-05 13:14:58 -0700
Processing by Rails::WelcomeController#index as HTML
  Rendered /Users/tsaleh/.rbenv/versions/2.0.0-p0/lib/ruby/gems/2.0.0/gems/railties-4.0.0.beta1/lib/rails/templates/rails/welcome/index.html.erb (1.9ms)
Completed 200 OK in 12ms (Views: 11.3ms | ActiveRecord: 0.0ms)
Started GET "/assets/rails.png" for 127.0.0.1 at 2013-04-05 13:14:58 -0700
^C

$ cf logs --recent
Started GET "/" for 127.0.0.1 at 2013-04-05 13:14:58 -0700
Processing by Rails::WelcomeController#index as HTML
  Rendered /Users/tsaleh/.rbenv/versions/2.0.0-p0/lib/ruby/gems/2.0.0/gems/railties-4.0.0.beta1/lib/rails/templates/rails/welcome/index.html.erb (1.9ms)
Completed 200 OK in 12ms (Views: 11.3ms | ActiveRecord: 0.0ms)
Started GET "/assets/rails.png" for 127.0.0.1 at 2013-04-05 13:14:58 -0700

$

Constraints

  1. Loggregator collects STDIN & STDOUT from the customer's application. This may require configuration on the developer's side.
  2. A Loggregator outage must not affect the running application.
  3. Loggregator gathers and stores logs in a best-effort manner. While undesirable, losing the current buffer of application logs is acceptable.
  4. As much as possible, Loggregator should be disconnected from the rest of Cloud Foundry. Ideally, it's deployable outside of Cloud Foundry, entirely.
  5. The 3rd party drain API should mimic Heroku's in order to reduce integration effort for our partners. The Heroku drain API is simply remote syslog over TCP.

Architecture

Loggregator is composed of:

  • Sources: Logging agents that run on the Cloud Foundry components. They forward logs to:
  • Loggregator Catcher: Responsible for gathering logs from the sources, and storing in the temporary buffers.
  • Loggregator Server: Accepts connections from the cf CLI, allowing users to access their logs.
  • Loggregator Drainer: Implements the Heroku Drain API for 3rd party partners.

Source agents emit the logging data as protocol-buffers, and the data stays in that format throughout the system.

Loggregator Diagram

Development

Checkout

git clone https://github.com/cloudfoundry/loggregator
cd loggregator
git submodule update --init

Running tests

bin/test

Development binary builds

Note: This is linux amd64 only

About

Logging in the Clouds

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published