I've been thinking a bit about roles and responsibilities for the primary and secondary nodes of a log. Here I'm sketching a model that is mostly compatible with the current replication protocol, and which makes the nodes a bit more independent (e.g., could be run by different organizations).
A log instance consists of a primary node and (ideally) several secondary nodes. A log is identified by its key, i.e., the key that is used to sign the log's advertised tree heads (the same tree heads that are cosigned by witnesses, and for which the log operator's states intended reliability etc). Each node is identified by a separate node key.
* Local trees
Each node (including the primary) keeps its own local tree. That tree is possibly larger (but not smaller, except when a new node is starting up) than the log's advertised tree. Each node is identified by its node key. The node key is used to sign the tree heads of its local tree. These signatures must not be confused with the log's signed tree heads; if it's not enough that separate keys are used, they could use a separate signature namespace.
The semantics of the signatures on local trees is that the node promises that it's local tree is append-only, and that all data covered by the signed tree head is committed to local storage. I.e., the tree should survive events like a local power outage. However, reliability is best effort. If the node suffers a disk failure, or is decommissioned for any other reason, the contents of the tree may be lost (except for parts of it replicated elsewhere, as described below).
* Primary node
The primary node's responsibility is to accept new leaves from users, commit into its local tree, and sign resulting local tree using its node key. Periodically, it queries the signed tree heads of the secondary nodes' trees, checks consistency, and publishes new versions of the *log*'s signed tree head once data is replicated to all secondaries. (If we have a larger number of secondaries, we could consider allowing the primary to proceed even in the case that a single secondary is behind or unreachable).
* Secondary nodes
Secondaries only accept new leaves from the primary. A secondary that is new or for some reason is behind, will first get the log's signed tree head, and retrieve all leaves it is missing. It must check inclusion and consistency before committing the leaves to its local tree and underlying storage. Next, it will periodically get the primary node's local tree head (verifying the signature using the node key of the node that is the current primary), and similarly incorporate after inclusion and consistency checks pass. Periodically, or when asked by the primary, it will sign the head of its local tree using its own node key.
So at all time we have this relation between tree sizes:
log's tree <= each secondary node tree <= primary node tree
Extensions: It may be useful to enable secondaries to also act as mirrors, republishing the latest tree head it has received from the primary node, together with available cosignatures. It may be possible to distribute new leaves in more of a peer-to-peer fashion, instead of each secondary retrieving them directly from the primary.
* Migration on primary failure
What needs to happen when a primary fails or is to be replaced? We need the following steps:
0. If possible, the primary node's access to the log signing key should be removed.
1. Each secondary must be configured that the primary is down. This must likely be a manual procedure, with a human determining that the primary should no longer be used. On each secondary, this means that the node key of the old primary is removed from the configuration.
2. Once all the secondaries agree that there is no longer any primary node, one of the secondaries can become new primary. If the local trees of the secondaries are of different sizes, the one with the largest tree should be selected as the new (interrim) primary, but not yet with access to the log's signing key. (If, for some reason, a different node is chosen, the nodes that are ahead of the chosen node must be reset: Discard the extra leaves, destroy previous node key and create a new one).
3. The secondaries that were not chosen as primary are now reconfigured to use the chosen node (identified by node key, as usual) as primary, and retrieve all leaves and commit them to their local trees.
4. After some time, nodes should all be in sync. If desired, the chosen node can now be demoted back to secondary (after which all the other secondaries will again be reconfigured that there is no primary), and a new primary node can be selected.
5. Finally, the new primary should be given access to the log's signing key and start normal operation (accept leaves from users, advertise new tree heads, request cosignatures, etc).
If we are willing to have secondaries coordinate with eachother, part of this process could potentially be automated. If all nodes are connected to each other (with the exception of the failing primary, which is explicitly and manually removed from the set of nodes in step (1) above), we could maybe have a protocol that lets nodes first agree that there is no primary, and then elect a new primary based on tree size, and which nodes are configured as candidates for getting access to the log's signing key.
Regards, /Niels