Socket Communication

As you’ve seen elsewhere in this documentation, the Cicada Communicator interface defines a set of Communication Patterns that can be used by algorithms, with SocketCommunicator providing an implementation that uses standard operating system sockets and TCP/IP networking for the transport layer. While advanced users with specialized hardware may wish to write their own communicators from scratch, there is much more to SocketCommunicator than you’ve seen so far; this article will introduce you to the full generality and flexibility of socket-based communication in Cicada.

SocketCommunicator Creation

You may-or-may-not have noticed that none of the examples in this documentation create an instance of the SocketCommunicator class directly - in some cases the communicator is created implicitly as part of another call (SocketCommunicator.run), and in others a factory function returns a new communicator, ready for use (SocketCommunicator.connect, SocketCommunicator.split). This is because an instance of SocketCommunicator doesn’t just wrap a single socket, it wraps a collection of sockets, allowing every player to communicate directly with every other player. Creating those sockets and setting-up the connections is a complex process involving careful coordination among players, which is why Cicada divides SocketCommunicator creation into separate high-level and low-level APIs. The low-level APIs create and initialize sockets that are only handed off for use by a new instance of SocketCommunicator once they’re ready.

High Level API

The high level API provides the following methods:

SocketCommunicator.run

SocketCommunicator.run is a static method that creates a set of player processes on the local machine, each executing a function that you provide. Before your function is executed, the underlying sockets are created with the low-level API and SocketCommunicator instances are created and passed as the first argument to your function. This method is widely used for development, debugging, and throughout this documentation.

SocketCommunicator.connect

SocketCommunicator.connect is a static factory function that returns a new instance of SocketCommunicator. It must be called by every player that will be members of the new communicator. The sockets are created using the low level API and parameters you provide, or environment variables set by the cicada run and cicada start commands. This is a good choice for running your programs in production on separate hosts. See Running Cicada Programs for examples.

Note that you can call SocketCommunicator.connect more than once within a program to create multiple communicators, which is more flexible (but less convenient) than using SocketCommunicator.split.

SocketCommunicator.split

SocketCommunicator.split partitions an existing communicator’s players into groups, and returns a new instance of SocketCommunicator for the new group containing the calling player, if any. It’s a good choice when you need to split a group of players into smaller subgroups to work on separate tasks, and it must be called by every member of the existing communicator. See Multiple Communicators for examples.

SocketCommunicator.shrink

SocketCommunicator.shrink is intended for use when failures have occurred - it should be called by as many members of the existing communicator as possible, and returns a new instance of SocketCommunicator that contains the subset of players that are still alive and responding. Note that SocketCommunicator.shrink cannot guarantee that all remaining players will be included in the new communicator, so you would likely never want to use it outside of failure recovery situations.

Low Level API

The low level API is located in the cicada.communicator.socket.connect module, and includes the following:

Timer

The Timer class is used to manage timeouts during initialization. Callers typically create one instance of Timer and pass it the other low level API functions, to specify an overall timeout for the entire process.

listen

The listen() function provides a standardized way to create a listening (server) socket, ready and waiting to accept connections. Since SocketCommunicator depends on connections between every pair of players, every player must create a listening socket in order to setup the network. Typically, the listening socket is passed to direct() or rendezvous() and then discarded; however, advanced users may wish to use the listening socket even after establishing the rest of the network. For example, it can be used to setup an MPC Service where the players act as servers that listen for requests from clients, performing MPC calculations on their behalf.

direct

The direct() function is used to create the complete network of connected sockets required to create an instance of SocketCommunicator, when you know the address of every player in advance. It must be called by every player that will become a member of the new communicator. This is useful in managed or datacenter environments where addresses and port numbers can be specified up-front, which greatly simplifies and streamlines the setup process.

rendezvous

The rendezvous() function is used to create the complete network of connected sockets required to create an instance of SocketCommunicator, when only the address of the root (rank 0) player is known in advance. It must be called by every player that will become a member of the new communicator. This is more convenient when setting-up a group of players in an ad-hoc environment, but requires more time and provides more points for failure.

There are several other helper functions and classes in cicada.communicator.socket.connect that will not be covered here.

Addressing

Network addresses throughout the low-level and high-level APIs must be specified using string URLs. The use of URLs makes the choice of address family unambiguous, and allows for possible future expansion. Currently, two types of address family are supported. For TCP/IP networking, use the following:

  • tcp://{ip address}:{port} # e.g. tcp://192.168.0.6:25252

  • tcp://{ip address} # e.g. tcp://192.168.0.7

When using the first form, a port number is specified explicitly. With the second, a port number will be chosen at random by the operating system. Note that in some contexts the API will not allow you to use the second form - for example, direct() requires explicit ports for every address, while rendezvous() only requires an explicit port for player 0’s address.

Alternatively, you can use Unix domain sockets for connections between players running on the same machine. To do so, use URLs like the following:

  • file://{path} # e.g. file:///tmp/run5/player3

Transport Layer Security (TLS/SSL)

Cicada’s socket communicator infrastructure supports Transport Layer Security (formerly known as Secure Sockets Layer) for encrypted communication and player authentication.

To enable TLS when creating a communicator, each player must supply the following:

  • An identity - a private key and a certificate that the player will use to identify themselves to others.

  • A set of trusted identities (certificates) with which the player will communicate. These might be player certificates, a certificate authority used to sign player certificates, or any combination thereof.

The cicada credentials command can be used to generate player identity and certificate files. The only required argument is the player rank, so three players could generate their own credentials as follows:

$ cicada credentials --rank 0

$ cicada credentials --rank 1

$ cicada credentials --rank 2

This would create an identity file and a certificate file in the current directory for each player. The identity file player-{rank}.pem contains the player’s private key and certificate in PEM format, and must be safeguarded by the player to prevent any other party from assuming their identity. The certificate file player-{rank}.cert contains just the player’s certificate in PEM format, and should be distributed to the other players so they can recognize the player at runtime.

Tip

cicada credentials generates 2048-bit empty-passphrase RSA private keys and self-signed certificates for players as a convenience. Advanced users will likely want to use their own existing tools and workflows to create proper credentials with passphrases that are signed by real certificate authorities.

Once credentials have been created for every player, they can be supplied during startup to create encrypted communicators. To use TLS with the high-level API, callers pass the filesystem paths to player identities and certificates. For example, to use TLS with SocketCommunicator.run, you might do something like the following:

def main(communicator):
    pass

world_size=3
identities = [f"player-{rank}.pem" for rank in range(world_size)]
trusted = [f"player-{rank}.cert" for rank in range(world_size)]
SocketCommunicator.run(world_size=world_size, fn=main, identities=identities, trusted=trusted)

Warning

This example assumes that all of the credential files - including the players’ private keys - are in the current directory. This might be acceptable for testing with SocketCommunicator.run but should never happen in production, since it would allow players to assume each others’ identities by loading other players’ identity files.