a general purpose flexible leader election utility

elector is a general purpose leader election utility, providing a pluggable backend, as well as a flexible handlers for election events.

The cli for elector accepts a leader-begin-command and leader-end-command; the former executed during transitions to leader state, and the latter during transitions from leader state. (leader-begin-command and leader-end-command are shell commands)

The elector library defines two basic interfaces; ElectionBackend, a backend that sources leader election events from a backend system, and Handler, a function executed after state transitions to perform arbitrary commands.

An ElectorBackend defines ElectionLoop, which is not expected to return, barring unrecoverable errors. It should continue to source election events and send elector.State messages to the updates channel (StateLeader, StateNotLeader, StateError). The reconciliation loop handles the transitions between states and executes the necessary handlers.

A Handler is executed by the reconciliation loop when state transitions occur to and from leader state. It is a function that accepts no arguments and returns and error in case it cannot complete it's task. Handlers should be resilient, i.e. if a handler receives an error that is recoverable it should handle it's own retry logic. The elector will attempt retries in a failure, but because it has not knowledge of the underlying command, has less flexibility in its operation. In case of an error, the reconciliation loop will transition to StateError, ensuring that the elector relinquishes master state and waits a given timeout before resuming candidacy.

To define an ElectorBackend take a look at the console implementation - it is very straightforward.

To define custom handlers take a look at the pre-defined handlers, also very straightforward.

For custom usage of an elector, take a look at it's usage in main. Create an elector.Elector, define it's backend and handlers, and call Run().

The etcd-lock backend is supported by etcd-lock. It acquires a lock on a node (keyspace) in etcd and continually updates the ttl on that node to maintain leader state. It supports full tls security.

The console backend is used for testing proper execution of state transitions and handlers. It sources election events from the console, i.e. your terminal. It allows input of text events (LEADER, NOTLEADER, ERROR).

Because elector manages local state with the state of a remote leader election, it requires logic to reconcile local state machine with messages from the remote system. The reconciliation loop behaves as follows:

    message rcvd:
		case StateLeader:
			if state == StateNotLeader
				err = exec BeginLeaderHandler
				if not err
					state = StateLeader
				if err
					<- StateError

		case StateNotLeader:
			if state == StateLeader
				err = exec EndLeaderHandler
				if not err
					state = StateNotLeader
				if err
					<- StateError

		case Error:
			if state == StateLeader
				err = exec EndLeaderHandler
				if not err
					state = StateError
					exec ErrorHandler (timeout)
					state = StateNotLeader
				if err
					<- StateError

WARNING: after a StateError message, if the EndLeaderHandler fails repeatedly, there is a condition where the local system may continue to operate as leader (as is the case with any failure of the EndLeaderHandler). The loop will continue to retry every 5s for 12 attempts, after which it will exit the system and log failure messages.