A relatively simple Redis-backed reliable task queue and state machine.
Its made up of four simpleq's: todo, doing, failed, and done. Tasks will never be dropped on the floor even if a processing server crashes because all operations are atomic. Tasks can be represented as any data type.
Note: relyq assumes all tasks have IDs. Thus you must pass in a pointer to an object that has an Id() method (or its element has an Id() method.)
There are a few redis clients for Go but this package uses redigo
go get github.com/Rafflecopter/golang-relyq/relyq
import (
"github.com/garyburd/redigo/redis"
"github.com/Rafflecopter/golang-relyq/relyq"
redisstorage "github.com/Rafflecopter/golang-relyq/storage/redis"
)
func CreateRelyQ(pool *redis.Pool) *relyq.Queue {
cfg := &relyq.Config{
Prefix: "my-relyq", // Required
Delimiter: ":", // Defaults to :
IdField: "id", // ID field for tasks
UseDoneQueue: false, // Whether to keep list of "done" tasks (default false)
KeepDoneTasks: false, // Whether to keep the backend storage of "done" tasks (default false)
}
storage := redisstorage.New(redisstorage.JSONMarshaller, pool, cfg.Prefix, cfg.Delimiter)
return relyq.New(pool, storage, cfg)
}
func QuickCreateRelyQ(pool *redis.Pool) *relyq.Queue {
return relyq.NewRedisJson(pool, &relyq.Config{Prefix: "my-relyq"})
}Create your task types:
type Task struct {
relyq.StructuredTask
IdField string
SomethingElse *OtherStruct
}Basic use:
q := CreateRelyQ(redisPool)
// Push a task on to Todo queue
err := q.Push(&Task{SomethingElse: &OtherStruct{}})
// Move a task from Todo to Doing
task := new(Task)
ok, err := q.Process(task) // ok=false means nothing in task
// Block and process a task
err := q.BProcess(task)
// Finish a task
err := q.Finish(task)
// Or fail it
err := q.Fail(task)
// Remove a task from the Failed queue
err := q.Remove(q.Failed, task)
// Eventually
err := q.Close()Or use a listener:
q := CreateRelyQ(redisPool)
var example *Task
l := q.Listen(example)
go func() {
for err := range l.Errors {
// Do something with errors
}
}()
go func() {
for task := range l.Tasks {
// Do something with tasks
err := q.Finish(task.(*Task))
}
}()
// Eventually
err := q.Close()go test
Normal operation stores the full task description or object in the queue itself. This can be inefficient for LREM operations. Sometimes one might even want to store task descriptions in a separate datastore than redis to save memory. Custom backends have been created for this purpose.
The Redis backend stores serialized task objects in Redis. Each options object also accepts the storage_prefix field to set the prefix for where task objects are stored. Documentation
Create a storage object:
storage := redisstorage.New(marshallers.JsonMarshaller, pool, cfg.Prefix, cfg.Delimiter)Or, for skipping the storage step, use this handy shortcut
q := relyq.NewRedisJson(pool, cfg)- deferred tasks
- recurring tasks
See LICENSE file.