Listen a SQS queue, execute a command when received. A message body is passed as STDIN to the command.

(in Japanese) SQS を監視しメッセージを取得したら指定のコマンドを実行します。 メッセージの内容はコマンドの標準入力として渡します。

$ go get github.com/koron/sqs-notify

or update

$ go get -u github.com/koron/sqs-notify

Require git command for dependencies.

$ go get github.com/goamz/goamz

From online help.

Usage: sqs-notify [OPTIONS] {queue name} {command and args...}

OPTIONS:
  -region {region} :    name of region (default: us-east-1)
  -worker {num} :       num of workers (default: 4)
  -nowait :             didn't wait end of command to delete message
  -ignorefailure:       don't care command results, treat it as success always
  -retrymax {num} :     num of retry count (default: 4)
  -msgcache {num} :     num of last received messages in cache (default: 0)
  -redis {path} :       use redis as message cache (default: disabled)
  -logfile {path} :     log file path ("-" for stdout)
  -pidfile {path} :     pid file path (available with -logfile)

  -mode {mode} :        pre-defined set of options for specific usecases

Source: https://github.com/koron/sqs-notify

Basic usage:

sqs-notify [-region {region}] {queue name} {command and args}

1. Prepare AWS auth information.

   1. Use ~/.aws/credentials (recomended).

      sqs-notify supports ~/.aws/credentials file, and use info from sqs-notify or default section in this order. Example:

      [sqs-notify]
      aws_access_key_id=foo
      aws_secret_access_key=bar

   2. Use two environment variables.

      • AWS_ACCESS_KEY_ID
      • AWS_SECRET_ACCESS_KEY

2. Run sqs-notify

   $ sqs-notify your-queue cat
   

   This example just copy messages to STDOUT. If you want to access the queue via ap-northeast-1 region, use below command.

   $ sqs-notify -region ap-northeast-1 your-queue cat
   

• us-east-1 (default)
• us-west-1
• us-west-2
• eu-west-1
• ap-southeast-1
• ap-southeast-2
• ap-northeast-1
• sp-east-1

When -logfile {FILE PATH} is given, all messages which received are logged into the file. If FILE PATH is -, it output all logs to STDOUT not file.

Using -pidfile {FILE PATH} with -logfile, sqs-notify writes own PID to the file. You can send SIGHUP to that PID to rotate log.

Log example:

2014/05/28 00:13:29 EXECUTED queue:kaoriya body:"abc\n" cmd:cat status:0
2014/05/28 00:14:27 EXECUTED queue:kaoriya body:"def\n" cmd:cat status:0
2014/05/28 00:14:54 EXECUTED queue:kaoriya body:"foo\nbar\n" cmd:cat status:0

Each items in one log are separated by tab char (\t).

When message doesn't have LF at EOF (end of file/message), the last line can't be handled by read shell command or so. This is limitation of read command, not sqs-notify. Therefore this kind of scripts don't work correctly for messages without LF at EOF:

#!/bin/sh
while read line
do
  echo "received: $line"
done

To work around this problem, use xargs like this.

#!/bin/sh
xargs -0 echo | (
while read line
do
  echo "received: $line"
done
)

You can use /dev/stdin pseudo file, if your system support it, like this:

#!/bin/sh

data=`cat /dev/stdin`

# do something for data.

sqs-echo is useful for debugging received SQS message with sqs-notify. It just shows date, time, byte num and contents of received messages. Example output is below:

2015/05/07 12:43:08 (7) "foo\nbar"
2015/05/07 12:43:12 (3) "qux"

You can install sqs-echo with below command.

$ go install github.com/koron/sqs-notify/cmd/sqs-echo

To suppress executions of the command for duplicated (seen recently) messages, please use -msgcache {num} option. {num} means cache size for messages.

Below command shows how to enable suppression with cache size 10.

$ sqs-notify -msgcache 10 -logfile - my-queue sqs-echo

You'll get log like this when sqs-notify detect duplicated message.

2015/05/07 12:13:08 EXECUTED queue:my-queue body:"foo\n" cmd:cat status:0
2015/05/07 12:43:11 SKIPPED  queue:my-queue body:"foo\n"

sqs-notify stores hashs (MD5) of message to detect duplication.

sqs-notify では -redis オプションに設定ファイル(JSON)を指定すると、メッセージ ハッシュの保存先をRedisに変更できます。

実行例:

$ sqs-notify -redis redis-example.json -logfile - my-queue sqs-echo

JSON内で使用できる設定項目は以下のとおりです。

redis-example.json には最小限の設定が記載されています。

at-most-once mode provides at-most-once command execution.

$ sqs-notify -mode at-most-once -msgcache 10 my-queue sqs-echo

This mode implies -ignorefailure option, excludes -nowait option, and requires one of -msgcache or -redis option.