[DEPRECATED] - Using Channels in the core broker-interface was a poor decision in hindsight. ¯_(ツ)_/¯ check out github.com/fortytw2/hoplite

A Library for implementing background-job-processing systems. Think of it as the implementation of the business-logic of sidekiq, without any convenience methods/helpful scheduling logic. Just raw workers, pools, and queues.

A Processor is the function handed to a worker, of the type

type Processor func(*Task) ([]*Task, error)

Returning more tasks from a processor is optional, but useful in many cases. I would recommend implementing your Processor as a small wrapper that performs type-casting of Task.Info, an interface, to the type you actually want to work with.

You should be using thermocline through the Pool API, as using individual workers may be a bit more error-prone/harder to work with. Basic Pool use shown below.

// A Broker is a simple message-queue implementation, an interface
// with only three functions to write to use your own.
var b thermocline.Broker
b = mem.NewBroker()

// create a new task every 10ms
ticker := time.NewTicker(time.Millisecond * 10)
go func() {
    w, err := b.Write("test", thermocline.NoVersion)
    if err != nil {
        t.Fatalf("cannot get write chan %s", err)
    }
    for {
        select {
        case t := <-ticker.C:
            tk, _ := thermocline.NewTask(t)
            w <- tk
        }
    }
}()

var worked int64
// create a worker pool on the unversioned queue "test", with
p, err := thermocline.NewPool("test", thermocline.NoVersion, b, func(task *thermocline.Task) ([]*thermocline.Task, error) {
    atomic.AddInt64(&worked, 1)
    return nil, nil
}, 30)
if err != nil {
    t.Errorf("cannot create pool %s", err)
}

time.Sleep(500 * time.Millisecond)
ticker.Stop()
err = p.Stop()
if err != nil {
    // oh no a nasty error
    return err
}

fmt.Println(atomic.LoadInt64(&worked))

The number of workers in a given pool can be easily tuned at runtime via pool#Add(workers int), which can both add and remove workers from a pool.

To stop a pool, simply call pool.Stop(), which will close out all workers and wait for their exit.

MIT