Skip to content

mozilla-services/reaper

AWS Reaper

About

The Reaper terminates forgotten AWS resources.

Reaper workflow:

  1. Find all enabled resources, filter them, then fire events based on config
    • Event types include sending emails, posting events to Datadog (Statsd), tagging resources on AWS, stopping or killing resources, and more
    • Reaper uses the Owner tag on resources to notify the owner of the resource with options to Ignore (for a time), Whitelist, Terminate, or Stop each resource they own
  2. Report statistics about the resources that were found
  3. Terminate or Stop abandoned resources after a set amount of time

Caution This app is experimental because:

  • doesn't have a lot of tests (Read: any), will add when SDK is updated

Building

  • install glide per https://github.com/Masterminds/glide
  • checkout repo
  • install dependencies: glide install
  • build binary: go build main.go
  • run binary: ./reaper -config config/default.toml
  • for command line options: ./reaper -help

Command Line Flags

  • config: required flag, the path to the Reaper config file. string (no default value)
  • dryrun: run Reaper in dryrun (no-op) mode. Events will not be triggered. boolean (default: true)
  • withoutCloudformationResources: skip checking for Cloudformation Resource dependencies (throttled by AWS, so it takes ages). boolean (default: false)

Creating a configuration file

Reaper configuration files should be in toml format. See config/default.toml for an example config.

  • Top level options
    • LogFile: the full filepath of the file that logs are written to. string
    • WhitelistTag: a string that will be used to tag resources that have been whitelisted. Defaults to REAPER_SPARE_ME. (string)
    • DefaultOwner: all unowned resources will be assigned to this owner. Can be an email address, or can be a username if DefaultEmailHost is specified. string
    • DefaultEmailHost: resources that do not have a complete email address as their owner will have this appended. Should be of the form "domain.tld". Works with DefaultOwner in the following way: DefaultOwner@DefaultEmailHost. string
    • EventTag: a tag that is added to all events that support tagging. Should be of the form key1:value1,key2:value2. string
  • HTTP options (under [HTTP])
    • TokenSecret: the secret key used to secure web requests. string
    • ApiURL: used to generate URLs for Reaper's HTTP API. Should be of the form protocol://host:port. string
    • Listen: where the HTTP server will listen for requests. Should be of the form host:port. string
    • Token: TODO
    • Action: TODO
  • Logging (under [Logging])
    • Extras: enables or disables extra logging, such as dry run notifications for EventReporters not triggering. boolean
  • States (under [States])
    • Interval: the interval between Reaper's scans for resources. The time format must be a duration parsable by Go's time.ParseDuration. See: http://godoc.org/time#ParseDuration. Example: 1h. string
    • FirstStateDuration: the length of the first state assigned to resources that match filters. The time format must be a duration parsable by Go's time.ParseDuration. See: http://godoc.org/time#ParseDuration. Example: 1h. string
    • SecondStateDuration: the length of the second state assigned to resources that match filters. The time format must be a duration parsable by Go's time.ParseDuration. See: http://godoc.org/time#ParseDuration. Example: 1h. string
    • ThirdStateDuration: the length of the third state assigned to resources that match filters. After the Third state elapses, resources move to a permanent final state. The time format must be a duration parsable by Go's time.ParseDuration. See: http://godoc.org/time#ParseDuration. Example: 1h. string
  • Events (under [Events])
    • Datadog ([Events.Datadog])
      • Enabled: enables or disables the Datadog EventReporter. Note: Datadog statistics and Event depend on this. boolean
      • Triggers: states for which Datadog will trigger Reapable Events. Can be any/all/none of first, second, third, final, or ignore. []string
    • Tagger ([Events.Tagger])
      • Enabled: enables or disables the Tagger EventReporter. boolean
      • Triggers: states for which Tagger will trigger Reapable Events. Can be any/all/none of first, second, third, final, or ignore. []string
    • Reaper ([Events.Reaper])
      • Enabled: enables or disables the Reaper EventReporter. boolean
      • Triggers: states for which Reaper will trigger Reapable Events. Can be any/all/none of first, second, third, final, or ignore. []string
      • Mode: when the Reaper EventReporter is triggered on a Reapable Event, it will Stop or Terminate Reapables per this flag. Note: modes must be capitalized. string
    • Email ([Events.Email])
      • Enabled: enables or disables the Email EventReporter. boolean
      • Triggers: states for which Email will trigger Reapable Events. Can be any/all/none of first, second, third, final, or ignore. []string
      • Host: the mailserver Reaper will use
      • AuthType: the type of authentication used by the mailserver. Should be one of none, md5 or plain. string
      • Port: the port used by the mailserver. int
      • Username: the username to use for the mailserver. string
      • Password: the password to use for the nmailserver. string
      • From: the address that Reaper will send mail from, must be parsable by Go's mail.ParseAddress. See: http://godoc.org/net/mail#ParseAddress. string
  • All Supported AWS Resource types have these properties
    • Enabled: enables or disables reporting of this resource type. Note: resources will still be queried for as they inform Reaper about the dependencies of other resources. boolean
    • FilterGroups (under [ResourceType.FilterGroups]): FilterGroups are sets of filters that can be applied to resources. In order for a resource to match a FilterGroup, it must match all filters in the FilterGroup. If an resource matches any FilterGroup, it has satisfied Reaper's filters. []FilterGroup
      • Example FilterGroup:

        [ResourceType.FilterGroups.Example]
                [ResourceType.FilterGroups.Example.Filter1]
                    function = "IsDependency"
                    arguments = ["false"]
                [ResourceType.FilterGroups.Example.Filter2]
                    function = "Running"
                    arguments = ["true"]
        
      • In this example, we see a FilterGroup named "Example" that has two Filters, Filter1 and Filter2.

      • A FilterGroup is a []Filter, and a Filter has two components, a function and arguments. The function is the name of the filtering function for the associated resource type (string), and arguments is a slice of arguments to that function ([]string).

  • Currently supported AWS Resource types:
    • SecurityGroups (under [SecurityGroups])
    • Cloudformations (under [Cloudformations])
    • AutoScalingGroups (under [AutoScalingGroups])
    • Instances (under [Instances])
    • Volumes (under [Volumes])