This tool provides a simple way to bootstrap AWS environments. It supports simple commands to create, destroy or package system configuration (user defined scripts). S3 is used to store configuration data which is referenced by the initial configuration script using the UserData feature of AWS.
The primary motivation for a tool like this is the ability to easily hack up test environments or maintain repeatable and easy to boostrap virtual environments.
NOTE: This is a very, very early WIP. I wouldn't use it for anything. :)
I needed a simple AWS bootstrap tool that meets the following criteria:
- All CM materials housed in a git/hg/svn repo, pushed to and deployed from S3.
- Facilitate immutable infrastructure without relying on AMIs or baked images.
- Simple configuration file that controls destruction/creation of AWS resources.
- Cross platform (windows/linux).
- Less configuration is more, specify only what you need. Assume sane defaults for as much as possible. Further configuration should be handled downstream in scripts, anyway.
A JSON config file provides EC2 and other service information.
Commands available:
- saws -a create
- saws -a destroy
- saws -a pack
- saws -a push
- saws -a stop
- saws -a start
Create creates any services (e.g., EC2 nodes) defined in JSON.
Destroy terminates all ec2 services.
Pack creates a zip of configuration data to prep for upload to S3, which the intial-config scripts reference.
Push publishes the package.zip to the configured S3 repo.
Stop/Start start and stop all EC2 nodes.
The saws.json file defines the VPC and associated AWS services and is the only configuration input for saws
. Saws expects to find the saws.json (or specifically overloaded via -c) file in the current directory along with the associated scripts and data used to initialize the nodes.
The configuration file is designed to contain the minimum amount of information necessary to determinately recreate an environment. Important JSON configuration params:
- initialconfig: file to use for initial UserData. This file will be templated with key AWS information: SAWS_S3_ACCESSKEY, SAWS_S3_SECRETKEY, SAWS_S3BUCKET, SAWS_HOSTNAME, SAWS_VPC
- s3bucket: the S3 bucket to store central configuration data. This will be automatically created if it does not exist
- vpc: the CIDR range of the VPC. This should encompass any private/public subnet CIDR range.
- privatenet: the CIDR range of the private subnet in the VPC
- publicnet: the CIDR range of the public subnet in the VPC
- allsecuritygroups: defines specific security groups. The only options are 'name' and 'tcpport' which affect ingress security group filters.
- keypair: specify which key to use, if it doesn't exist it will be created
- destroypolicy: when running destroy, if set to 'nuke', will destroy VPCs and RDS instances. Otherwise, it will only destroy EC2 nodes. This is helpful to persist data and for creation/destruction iteration time.
- ec2: an array of ec2 definitions
- instancetype: AWS instance type
- name: name of the EC2 instance, which will be tagged with a Name tag
- ami: AMI to use when creating the instance
- keyname: the SSH key/value pair used to access the instance, option if 'keypair' is specified globally
- securitygroups: a list of strings to map into security groups. 'default' is always available but others must match the 'allsecuritygroups' setting
- hasexternalip: if set to true, the instance will be given an external ip (optional)
- isnat: if set to true, the instance will have traffic routed to it in order to NAT for other instances (optional)
- initialconfig: an overload to use a specific initialconfig template rather than the global one (optional)
- rds: an array of rds definitions
- dbinstanceidentifier: the name of the RDS instance
- dbname: the default DB name, if applicable
- engine: string representing the engine, see AWS docs. e.g., "MySQL"
- dbinstanceclass: instance class of the RDS instance, e.g., "db.t1.micro"
- masterusername: admin DB user
- masteruserpassword: admin user password
- elb: an array of elb definition
- name: name of elb
- instanceport: port the instance is listening on, assumed to be the LB port
- protocol: "HTTP" or "HTTPS"
- securitygroups: array of security groups for the LB that must match the name of the 'allsecuritygroups' array
- instances: array of instance names to be included in the LB
Requires a Go compiler.
$ git clone https://github.com/jamesunger/saws $ cd saws $ export GOPATH=$PWD $ go get github.com/aws/aws-sdk-go/service/ec2 $ go build saws.go
The demo saws.json creates three Debian GNU/Linux instances and a Windows 2012 instance. It also create a basic load balancer and a MySQL RDS DB. The base scripts provided will bootstrap salt on each node and expose nginx to test the LB on the two Debian hosts.
First, set the following AWS vars in your shell for the AWS lib to work:
- AWS_SECRET_ACCESS_KEY
- AWS_REGION
- AWS_DEFAULT_REGION
- AWS_ACCESS_KEY_ID
- SAWS_S3_SECRET_KEY
- SAWS_S3_ACCESS_KEY
SAWS_S3_SECRET_KEY and SAWS_S3_ACCESS_KEY are specific keys for saws and specify the AWS keys instances will use when clients download from S3, so minimal access should be used.
$ ./saws -a create Created keypair: dummykey Key saved to file: dummykey.key Created new VPC: vpc-89b953ed Created MySQL RDS instance: someid Created instance salt: i-24bd088f Created instance debiantest1: i-d1bc097a Created instance debiantest2: i-4cbc09e7 Created instance wintest1: i-3122bae3 Created elb: testlb-1869659592.us-east-1.elb.amazonaws.com Waiting for remaining 5 creation steps to complete... 1: External IP for salt assigned: 52.21.104.223 2: Configured for NAT: salt 3: External IP for debiantest2 assigned: 52.7.38.215 4: External IP for debiantest1 assigned: 52.21.105.3 5: Endpoint for RDS instance someid: someid.cqxtcggij89r.us-east-1.rds.amazonaws.com
Now we can login to the salt host and verify Salt is working and accept the minion keys.
$ ssh -i dummykey.key admin@52.21.104.223 The authenticity of host '52.21.104.223 (52.21.104.223)' can't be established. ECDSA key fingerprint is 4f:e8:b5:17:29:2d:ae:66:10:2f:b3:da:94:ee:33:6f. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '52.21.104.223' (ECDSA) to the list of known hosts. Linux ip-192-168-98-201 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u1 x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. admin@salt:~$ sudo salt-key -A -y The following keys are going to be accepted: Unaccepted Keys: debiantest1 debiantest2 wintest1 Key for minion debiantest1 accepted. Key for minion debiantest2 accepted. Key for minion wintest1 accepted. admin@salt:~$ sudo salt '*' grains.item os wintest1: ---------- os: Windows debiantest1: ---------- os: Debian debiantest2: ---------- os: Debian admin@salt:~$
And lets verify the LB is up and running.
$ curl testlb-1869659592.us-east-1.elb.amazonaws.com <html> <head> <title>Welcome to nginx!</title>> </head> <body bgcolor="white" text="black"> <center><h1>Welcome to nginx!</h1></center> </body> </html>