The idea behind this work is to provide a simple way to build highly scalable messaging microservices. These message microservices use different transports such as Kafka or Redis for communication. The ideas is to deploy those microservices on Docker, Kubernetes or any other container system.

This work was inspired by the awesome Spring Cloud Stream project. However, streaming modules written in Java are pretty heavyweight, therefore we chose to implement the base framework in Go because it is lightweight and blazingly fast.

Our key design goals are to be minimalist and lightweight but still compatible with the Spring Cloud Stream Modules in terms of the most important command line arguments so they can be used interchangeably. This does not mean that go-cloud-stream supports all the options that Spring Cloud Stream supports. However, the key aspects of a modules are supported (e.g., queue and topic bindings, Redis and Kafka as transports, ...)

Combined with Spring Cloud Dataflow or your own "composition" and deployment framework, this work can easily be used to build highly scalable messaging microservices.

This project is intended to be used as "library" to build messaging microservices on top of either Kafka or Redis as transport layer. However, Go does not supported shared libraries that can be dynamically linked, we ship the code as is (without main) and compiles is as part of so-called "Go Cloud Stream Modules". See some concrete examples here: https://github.com/frosenberg/go-cloud-stream-modules.

If you want to test and extend the code, you have start a Redis, Kafka and Zookeeper as they are needed in the tests.

We support both, Redis and Redis Sentinel mode, so start both of them with:

$ redis-server
$ redis-sentinel etc/sentinel.conf

$ cd $KAFKA_HOME/bin/zookeeper-server-start.sh &
$ cd $KAFKA_HOME/bin/kafka-server-start.sh

$ go test -v ./...

See here for examples: https://github.com/frosenberg/go-cloud-stream-modules