{ "cells": [ { "attachments": {}, "cell_type": "raw", "id": "858e1818-e86b-4a0f-b0a2-f34f29986a54", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ ".. _transcripts:\n", "\n", "Transcripts\n", "===========\n", "\n", "In this section we extend Cicada's basic :ref:`logging` functionality to support advanced use cases including debugging, network traffic analysis, consistency verification, and zero-knowledge proofs. We refer to these more detailed logs as *transcripts*, and generate them using the :mod:`cicada.transcript` module. Transcripts supplement Cicada's normal logging with an optional recording mechanism that generates new categories of log messages, and a set of :mod:`logging` components to filter and format those messages in straightforward ways. \n", "\n", ".. tip::\n", "\n", " You'll need a good working knowledge of the Python :mod:`logging` module to fully understand the following examples!\n", "\n", "For the rest of this article, we'll be working with the following Cicada program, which adds two numbers using additive secret sharing:" ] }, { "cell_type": "code", "execution_count": 1, "id": "f01f1b88-43f9-4e7e-ba4c-a1abcf081d79", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "import numpy\n", "\n", "from cicada.additive import AdditiveProtocolSuite\n", "from cicada.communicator import SocketCommunicator\n", "from cicada.encoding import FixedPoint\n", "\n", "def main(communicator):\n", " encoding = FixedPoint(precision=2)\n", " protocol = AdditiveProtocolSuite(communicator, order=127, encoding=encoding)\n", " a_share = protocol.share(src=0, secret=numpy.array(2), shape=())\n", " b_share = protocol.share(src=1, secret=numpy.array(3), shape=())\n", " c_share = protocol.add(a_share, b_share)\n", " c = protocol.reveal(c_share)" ] }, { "cell_type": "markdown", "id": "0fa12ec6-c26f-408d-8c5b-3ba7797cd500", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "Note that we're using a very low-precision encoding (`precision=2`) and a very small field (`order=127`) to make the values in the logged output easier to read." ] }, { "cell_type": "raw", "id": "e97445e0-af85-4eae-992e-a19c4e0a490b", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "Network Traffic\n", "---------------\n", "\n", "To see how Cicada transcripts work, let's generate a transcript of the network messages generated by our program:" ] }, { "cell_type": "code", "execution_count": 2, "id": "bc827c1a-e555-4357-b784-f1f306bcae28", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Player 0: --> 1 PRZS 2113258192961202993\n", "Player 2: --> 0 PRZS 4989078122295468691\n", "Player 1: --> 2 PRZS 1641890102022822472\n", "Player 0: <-- 2 PRZS 4989078122295468691\n", "Player 2: <-- 1 PRZS 1641890102022822472\n", "Player 1: <-- 0 PRZS 2113258192961202993\n", "Player 2: --> 0 GATHER 93\n", "Player 0: --> 0 GATHER 15\n", "Player 1: --> 0 GATHER 39\n", "Player 2: --> 1 GATHER 93\n", "Player 0: <-- 2 GATHER 93\n", "Player 1: --> 1 GATHER 39\n", "Player 2: --> 2 GATHER 93\n", "Player 1: <-- 2 GATHER 93\n", "Player 0: <-- 0 GATHER 15\n", "Player 2: <-- 2 GATHER 93\n", "Player 0: <-- 1 GATHER 39\n", "Player 1: <-- 1 GATHER 39\n", "Player 0: --> 1 GATHER 15\n", "Player 0: --> 2 GATHER 15\n", "Player 1: <-- 0 GATHER 15\n", "Player 2: <-- 0 GATHER 15\n", "Player 1: --> 2 GATHER 39\n", "Player 2: <-- 1 GATHER 39\n" ] } ], "source": [ "import logging\n", "\n", "from cicada import transcript\n", "\n", "with transcript.record():\n", " handler = transcript.net_handler()\n", " transcript.set_handler(logging.getLogger(), handler)\n", " SocketCommunicator.run(fn=main, world_size=3);" ] }, { "cell_type": "raw", "id": "0488191e-79f8-453c-9c99-e0473d49380f", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "We begin by calling :any:`transcript.record` before running our program, to enable generation of the additional log messages supported by transcription.\n", "\n", "Then, we use :any:`transcript.net_handler` to create a :class:`~logging.StreamHandler`, configured to display networking messages to the console using a default human-readable representation where each line of output represents one message that is sent or received. For example, in the above output:\n", "\n", " Player 1: --> 2 GATHER X\n", "\n", "means that player 1 has sent a GATHER message to player 2, with a payload of \"X\" (if you run these examples yourself, the payloads will be different due to random seeding).\n", "\n", "Note that interpreting message traffic can be extremely difficult without the context of the program generating them. For this reason, transcript output can include arbitrary \"context\" messages that you insert from your application yourself:" ] }, { "cell_type": "code", "execution_count": 3, "id": "ed0574ad-49ff-4934-8f45-a037b2d223dd", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Player 0: Let's setup additive sharing!\n", "Player 2: Let's setup additive sharing!\n", "Player 1: Let's setup additive sharing!\n", "Player 0: --> 1 PRZS 4612915327354008253\n", "Player 1: --> 2 PRZS 7414186504898461884\n", "Player 2: --> 0 PRZS 1882202436348525527\n", "Player 1: <-- 0 PRZS 4612915327354008253\n", "Player 2: <-- 1 PRZS 7414186504898461884\n", "Player 0: <-- 2 PRZS 1882202436348525527\n", "Player 1: Let's share some secrets!\n", "Player 2: Let's share some secrets!\n", "Player 0: Let's share some secrets!\n", "Player 1: Let's add some secrets!\n", "Player 2: Let's add some secrets!\n", "Player 0: Let's add some secrets!\n", "Player 1: Let's reveal the results!\n", "Player 2: Let's reveal the results!\n", "Player 0: Let's reveal the results!\n", "Player 1: --> 0 GATHER 6\n", "Player 2: --> 0 GATHER 75\n", "Player 0: --> 0 GATHER 66\n", "Player 1: --> 1 GATHER 6\n", "Player 2: --> 1 GATHER 75\n", "Player 0: <-- 1 GATHER 6\n", "Player 2: --> 2 GATHER 75\n", "Player 1: <-- 1 GATHER 6\n", "Player 0: <-- 2 GATHER 75\n", "Player 2: <-- 2 GATHER 75\n", "Player 1: <-- 2 GATHER 75\n", "Player 0: <-- 0 GATHER 66\n", "Player 0: --> 1 GATHER 66\n", "Player 0: --> 2 GATHER 66\n", "Player 1: <-- 0 GATHER 66\n", "Player 2: <-- 0 GATHER 66\n", "Player 1: --> 2 GATHER 6\n", "Player 2: <-- 1 GATHER 6\n" ] } ], "source": [ "def main(communicator):\n", " transcript.log(\"Let's setup additive sharing!\")\n", " encoding = FixedPoint(precision=2)\n", " protocol = AdditiveProtocolSuite(communicator, order=127, encoding=encoding)\n", " transcript.log(\"Let's share some secrets!\")\n", " a_share = protocol.share(src=0, secret=numpy.array(2), shape=())\n", " b_share = protocol.share(src=1, secret=numpy.array(3), shape=())\n", " transcript.log(\"Let's add some secrets!\")\n", " c_share = protocol.add(a_share, b_share)\n", " transcript.log(\"Let's reveal the results!\")\n", " c = protocol.reveal(c_share)\n", "\n", "with transcript.record():\n", " handler = transcript.net_handler()\n", " transcript.set_handler(logging.getLogger(), handler)\n", " SocketCommunicator.run(fn=main, world_size=3);" ] }, { "cell_type": "raw", "id": "3846f4ff-c4f5-44dd-a9cc-c528d4397243", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "Now, we see the context messages originating directly from our script, interleaved with the transcribed network traffic. Context messages logged by the application should be chosen to provide information about what the program is doing at a high level, which will make it much easier to understand the rest of the transcript output.\n", "\n", ".. admonition:: By the way ...\n", "\n", " If you examine the message transcript closely, you may be surprised to see that our program doesn't send any messages while generating secret shares. Does that surprise you? It's because Cicada uses a cool trick - pseudorandom zero sharing - to generate additive secret shares without communication. Similarly, you can see that communication isn't needed to add secrets, when using additive secret sharing." ] }, { "cell_type": "raw", "id": "740f55e8-081d-4d94-82ce-2a4d46840c37", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "If you want to customize the transcript, you can override the network message format string when configuring the handler, using a rich set of fields including `net.arrow` (message direction, relative to the local rank), `net.comm.name` (communicator name), `net.dir` (message direction, relative to the local rank), `net.dst` (message destination), `net.other` (the other player sending or receiving with the local rank), `net.payload` (message payload), `net.comm.rank` (local player), `net.src` (message source), `net.tag` (message type), and `net.verb` (\"sent\" or \"received\", depending on whether the local player is sending or receiving).\n", "\n", "Note in the above example that the rank of the player logging the event is always on the left, whether they are the sender or the receiver. With the arrows indicating which direction the message is travelling, this is intuitive for a person to understand, but it can complicate programmatically parsing the transcript. As an alternative, you could use some of the alternative fields to unconditionally put the sender and recipient in fixed positions, regardless of which player is logging the event. As an example, let's try formatting our network messages as CSV for easier ingestion in some hypothetical analysis workflow (note that we also override the context message formatting so the context messages are valid CSV, too):" ] }, { "cell_type": "code", "execution_count": 4, "id": "61d85723-16d7-485f-8898-91fe4a8b2846", "metadata": { "editable": true, "raw_mimetype": "", "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "# Player 0: Let's setup additive sharing!\n", "# Player 1: Let's setup additive sharing!\n", "# Player 2: Let's setup additive sharing!\n", "Player 0,sent,0,1,PRZS,6501041567845331950\n", "Player 2,sent,2,0,PRZS,6064374732292475082\n", "Player 1,sent,1,2,PRZS,3210179197484905169\n", "Player 1,received,0,1,PRZS,6501041567845331950\n", "Player 0,received,2,0,PRZS,6064374732292475082\n", "Player 2,received,1,2,PRZS,3210179197484905169\n", "# Player 1: Let's share some secrets!\n", "# Player 0: Let's share some secrets!\n", "# Player 2: Let's share some secrets!\n", "# Player 1: Let's add some secrets!\n", "# Player 2: Let's add some secrets!\n", "# Player 0: Let's add some secrets!\n", "# Player 1: Let's reveal the results!\n", "# Player 2: Let's reveal the results!\n", "# Player 0: Let's reveal the results!\n", "Player 1,sent,1,0,GATHER,95\n", "Player 2,sent,2,0,GATHER,12\n", "Player 0,sent,0,0,GATHER,40\n", "Player 1,sent,1,1,GATHER,95\n", "Player 2,sent,2,1,GATHER,12\n", "Player 0,received,1,0,GATHER,95\n", "Player 1,received,1,1,GATHER,95\n", "Player 0,received,2,0,GATHER,12\n", "Player 2,sent,2,2,GATHER,12\n", "Player 1,received,2,1,GATHER,12\n", "Player 2,received,2,2,GATHER,12\n", "Player 0,received,0,0,GATHER,40\n", "Player 0,sent,0,1,GATHER,40\n", "Player 0,sent,0,2,GATHER,40\n", "Player 1,received,0,1,GATHER,40\n", "Player 2,received,0,2,GATHER,40\n", "Player 1,sent,1,2,GATHER,95\n", "Player 2,received,1,2,GATHER,95\n" ] } ], "source": [ "with transcript.record():\n", " fmt = \"# {processName}: {msg}\"\n", " netfmt = \"{processName},{net.verb},{net.src},{net.dst},{net.tag},{net.payload}\"\n", " handler = transcript.net_handler(fmt=fmt, netfmt=netfmt)\n", " transcript.set_handler(logging.getLogger(), handler)\n", " SocketCommunicator.run(fn=main, world_size=3);" ] }, { "attachments": {}, "cell_type": "markdown", "id": "4e8bc2bb-8acb-4e6a-944a-798006ce813b", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "If you look carefully, you can see that this produces two nearly identical events for each message (once when the message is sent, and once when the message is received). If you wish to eliminate the duplication, e.g. by only logging messages when they're sent, you can specify that too:" ] }, { "cell_type": "code", "execution_count": 5, "id": "b061ce36-05e3-4073-8a1a-fb8a93df845d", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "# Player 0: Let's setup additive sharing!\n", "# Player 1: Let's setup additive sharing!\n", "# Player 2: Let's setup additive sharing!\n", "Player 0,sent,0,1,PRZS,4923441390019161378\n", "Player 2,sent,2,0,PRZS,3792132382029304949\n", "Player 1,sent,1,2,PRZS,8009281797746407522\n", "# Player 0: Let's share some secrets!\n", "# Player 1: Let's share some secrets!\n", "# Player 2: Let's share some secrets!\n", "# Player 0: Let's add some secrets!\n", "# Player 2: Let's add some secrets!\n", "# Player 1: Let's add some secrets!\n", "# Player 0: Let's reveal the results!\n", "# Player 2: Let's reveal the results!\n", "# Player 1: Let's reveal the results!\n", "Player 0,sent,0,0,GATHER,99\n", "Player 2,sent,2,0,GATHER,64\n", "Player 1,sent,1,0,GATHER,111\n", "Player 2,sent,2,1,GATHER,64\n", "Player 1,sent,1,1,GATHER,111\n", "Player 0,sent,0,1,GATHER,99\n", "Player 2,sent,2,2,GATHER,64\n", "Player 0,sent,0,2,GATHER,99\n", "Player 1,sent,1,2,GATHER,111\n" ] } ], "source": [ "with transcript.record():\n", " fmt = \"# {processName}: {msg}\"\n", " netfmt = \"{processName},{net.verb},{net.src},{net.dst},{net.tag},{net.payload}\"\n", " handler = transcript.net_handler(received=False, fmt=fmt, netfmt=netfmt)\n", " transcript.set_handler(logging.getLogger(), handler)\n", " SocketCommunicator.run(fn=main, world_size=3);" ] }, { "cell_type": "raw", "id": "f7a1ee7e-8296-48ab-9b1c-0b5aeaafcd8e", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "Now the transcript only contains sent messages. Of course, you could use `sent=False` instead, if you prefer to only see messages when they're received." ] }, { "cell_type": "raw", "id": "8d73bf51-f378-4363-8e53-9fee10efc017", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "Consistency Verification\n", "------------------------\n", "\n", "In addition to logging application context and network message events, Cicada's transcription framework can generate a set of Python statements that summarize the computation being performed in a way that supports consistency verification, along with forms of zero-knowledge proof such as MPC-in-the-head [#]_.\n", "\n", "To get started, let's switch to a much simpler computation, one that only requires a single player and no communication:" ] }, { "cell_type": "code", "execution_count": 6, "id": "acce87e7-4aaf-425d-af1c-77d6ad1be3d1", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).ones(shape=3), numpy.array([1, 1, 1], dtype='object'))\n", "\n", "bg = numpy.random.PCG64()\n", "bg.state = {'bit_generator': 'PCG64', 'state': {'state': 310581584614395710612860853258973098112, 'inc': 87454387631014545490430812487103061493}, 'has_uint32': 0, 'uinteger': 0}\n", "cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).uniform(size=3, generator=numpy.random.Generator(bg)), numpy.array([24, 47, 119], dtype='object'))\n", "\n", "lhs = numpy.array([1, 1, 1], dtype='object')\n", "cicada.arithmetic.Field(order=127).inplace_add(lhs=lhs, rhs=numpy.array([24, 47, 119], dtype='object'))\n", "cicada.transcript.assert_equal(lhs, numpy.array([25, 48, 120], dtype='object'))\n", "\n" ] } ], "source": [ "from cicada.arithmetic import Field\n", "\n", "with transcript.record():\n", " handler = transcript.code_handler()\n", " transcript.set_handler(logging.getLogger(), handler)\n", " \n", " f = Field(order=127)\n", " a = f.ones(3)\n", " b = f.uniform(size=3, generator=numpy.random.default_rng())\n", " f.inplace_add(a, b)" ] }, { "cell_type": "raw", "id": "fa473501-a918-4aa9-aa38-dce8c958e557", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "Note that we've switched our logging setup from :any:`net_handler` to :any:`code_handler`, to make the code output visible.\n", "\n", "It's important to understand that the consistency verification output isn't simply a copy of the code being executed; rather, it's a set of derived statements that can be executed to verify that the computation that was performed is logically consistent. In particular, there *is not* a one-to-one relationship between lines of original source code and lines of verification code. For example, the first line of the computation::\n", "\n", " f = Field(order=127)\n", "\n", "in the original source doesn't generate any verification output at all, because the verification code is designed to be (relatively) stateless. Instead, the next line::\n", "\n", " a = f.ones(3)\n", "\n", "generates a single verification statement::\n", "\n", " cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).ones(shape=3), numpy.array([1, 1, 1], dtype=object))\n", "\n", "... which uses :func:`cicada.transcript.assert_equal` to test that the left- and right-hand expressions are equal. Note that the expressions have been expanded and \"flattened\" so that variable assignments are replaced with temporary objects, and all arguments are explicitly specified. If you used the Python interpreter to :func:`exec` this entire expression, it would return without error, since the expression::\n", "\n", " cicada.arithmetic.Field(order=127).ones(shape=3)\n", "\n", "does, in-fact, produce an array of three ones::\n", "\n", " numpy.array([1, 1, 1], dtype=object)\n", "\n", "Let's look at the next line in the program::\n", "\n", " b = f.uniform(size=3, generator=numpy.random.default_rng())\n", "\n", "This code is slightly tricky, since we're creating a random number generator using numpy's default random initialization, which will produce different results every time it is run. In this case, we get three lines of verification code (if you run this code yourself, the output will be different due to the random initialization)::\n", "\n", " bg = numpy.random.PCG64()\n", " bg.state = {'bit_generator': 'PCG64', 'state': {'state': ..., 'inc': ...}, 'has_uint32': 0, 'uinteger': 0}\n", " cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).uniform(size=3, generator=numpy.random.Generator(bg)), numpy.array([...], dtype=object))\n", "\n", "The first two lines of code are used to carefully initialize a random number generator with state identical to that when the original code was run, so that it will produce the same output. Again, if you ran these three lines using :func:`exec`, they would return without error, because the the expression::\n", "\n", " cicada.arithmetic.Field(order=127).uniform(size=3, generator=numpy.random.Generator(bg))\n", "\n", "initialized with the correct state, should produce a numpy array with the given values as output.\n", "\n", "Finally, the original statement::\n", "\n", " f.inplace_add(a, b)\n", "\n", "also produces three lines of consistency ouput::\n", "\n", " lhs = numpy.array([1, 1, 1], dtype=object)\n", " cicada.arithmetic.Field(order=127).inplace_add(lhs=lhs, rhs=numpy.array([59, 60, 55], dtype=object))\n", " cicada.transcript.assert_equal(lhs, numpy.array([...], dtype=object))\n", "\n", "... where the additional lines are necessary to work around the fact that :func:`~cicada.arithmetic.Field.inplace_add` doesn't have a return value, since it modifies its input in-place." ] }, { "cell_type": "raw", "id": "8f9765f6-2f8f-4d70-841f-c44f84a2af17", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "Let's confirm that we can execute the consistency verification code without error. We'll buffer the transcript to a string instead of writing it to the console, so we can execute it. To do this, you can supply your own choice of :mod:`handler` to :func:`~cicada.transcript.code_handler` or :func:`~cicada.transcript.net_handler` to replace the default:" ] }, { "cell_type": "code", "execution_count": 7, "id": "1b99ec72-3b43-4c43-8b90-f3105327fa87", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "import io\n", "\n", "with transcript.record():\n", " buffer = io.StringIO()\n", " handler = transcript.code_handler(logging.StreamHandler(buffer))\n", " transcript.set_handler(logging.getLogger(), handler)\n", " \n", " f = Field(order=127)\n", " a = f.ones(3)\n", " b = f.uniform(size=3, generator=numpy.random.default_rng())\n", " f.inplace_add(a, b)" ] }, { "cell_type": "raw", "id": "59cf60e1-762d-4110-8b33-053b75ddd13d", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "Let's inspect the buffered transcript, to ensure that it contains what we expect:\n", "\n", ".. danger::\n", "\n", " Executing code from other parties (even Cicada) is a serious security risk! Always carefully review a transcript before executing it!" ] }, { "cell_type": "code", "execution_count": 8, "id": "26c535ac-5522-494e-a47e-d8407697c55a", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).ones(shape=3), numpy.array([1, 1, 1], dtype='object'))\n", "\n", "bg = numpy.random.PCG64()\n", "bg.state = {'bit_generator': 'PCG64', 'state': {'state': 294111206801531245155334061750484527776, 'inc': 44482354329879700285959243340315682281}, 'has_uint32': 0, 'uinteger': 0}\n", "cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).uniform(size=3, generator=numpy.random.Generator(bg)), numpy.array([42, 25, 0], dtype='object'))\n", "\n", "lhs = numpy.array([1, 1, 1], dtype='object')\n", "cicada.arithmetic.Field(order=127).inplace_add(lhs=lhs, rhs=numpy.array([42, 25, 0], dtype='object'))\n", "cicada.transcript.assert_equal(lhs, numpy.array([43, 26, 1], dtype='object'))\n", "\n" ] } ], "source": [ "buffer.seek(0)\n", "for line in buffer:\n", " print(line.strip())" ] }, { "cell_type": "markdown", "id": "e264c773-39b1-4a1a-9d04-94e1205e831d", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "Manual inspection shows that the transcript contains the expected output. Now, we can execute the buffered transcript, one-line-at-a-time, and confirm that none of the assertions raise exceptions:" ] }, { "cell_type": "code", "execution_count": 9, "id": "9e8ef542-dcd5-4aa4-8d25-f57a7d7ff947", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Consistency verification succeeded.\n" ] } ], "source": [ "import cicada.transcript\n", "\n", "try:\n", " buffer.seek(0)\n", " for line in buffer:\n", " exec(line)\n", "except Exception as e:\n", " print(f\"Consistency verification failed: {e}\")\n", "else:\n", " print(\"Consistency verification succeeded.\")" ] }, { "cell_type": "markdown", "id": "ad5917f4-f586-402e-a63c-fb023bc5575a", "metadata": {}, "source": [ "Let's go back to our original example. Because it requires three players, we'll setup Cicada to write a separate transcript for each player: " ] }, { "cell_type": "code", "execution_count": 10, "id": "46137d16-acfe-4955-a5e9-344d2703c645", "metadata": {}, "outputs": [], "source": [ "def main(comm):\n", " handler = transcript.code_handler(logging.FileHandler(f\"player-{comm.rank}.py\", \"w\"))\n", " transcript.set_handler(logging.getLogger(), handler)\n", "\n", " transcript.log(\"Let's setup additive sharing!\")\n", " encoding = FixedPoint(precision=2)\n", " protocol = AdditiveProtocolSuite(comm, order=127, encoding=encoding)\n", " transcript.log(\"Let's share some secrets!\")\n", " a_share = protocol.share(src=0, secret=numpy.array(2), shape=())\n", " b_share = protocol.share(src=1, secret=numpy.array(3), shape=())\n", " transcript.log(\"Let's add some secrets!\")\n", " c_share = protocol.add(a_share, b_share)\n", " transcript.log(\"Let's reveal the results!\")\n", " c = protocol.reveal(c_share)\n", "\n", "with transcript.record():\n", " SocketCommunicator.run(fn=main, world_size=3);" ] }, { "cell_type": "markdown", "id": "a1d6e65a-148a-4a1d-9d8c-7c7658cc2a07", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "Note that we've moved our transcript configuration code inside `main`, which allows us to generate a separate filename for each transcript based on player rank. Also note that we've left our context messages in-place. Let's see what the results look like for one of the players:" ] }, { "cell_type": "code", "execution_count": 11, "id": "6563d816-8098-425c-aab7-7e4f3ac80e8a", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "# Let's setup additive sharing!\n", "# Let's share some secrets!\n", "# cicada.additive.AdditiveProtocolSuite().share(src=0, secret=numpy.array(2, dtype='int64'), shape=(), encoding=None)\n", "\n", "bg = numpy.random.PCG64()\n", "bg.state = {'bit_generator': 'PCG64', 'state': {'state': 275934374581578202693231533907928662192, 'inc': 74303973817490204264452453842712767295}, 'has_uint32': 0, 'uinteger': 0}\n", "cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).uniform(size=(), generator=numpy.random.Generator(bg)), numpy.array(62, dtype='object'))\n", "\n", "bg = numpy.random.PCG64()\n", "bg.state = {'bit_generator': 'PCG64', 'state': {'state': 45413300204548699694289800305975195387, 'inc': 122458647806018768901565485422780565793}, 'has_uint32': 0, 'uinteger': 0}\n", "cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).uniform(size=(), generator=numpy.random.Generator(bg)), numpy.array(117, dtype='object'))\n", "\n", "lhs = numpy.array(62, dtype='object')\n", "cicada.arithmetic.Field(order=127).inplace_subtract(lhs=lhs, rhs=numpy.array(117, dtype='object'))\n", "cicada.transcript.assert_equal(lhs, numpy.array(72, dtype='object'))\n", "\n", "cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).__call__(object=numpy.array(8, dtype='object')), numpy.array(8, dtype='object'))\n", "\n", "cicada.transcript.assert_equal(cicada.encoding.FixedPoint(precision=2).encode(array=numpy.array(2, dtype='int64'), field=cicada.arithmetic.Field(order=127)), numpy.array(8, dtype='object'))\n", "\n", "lhs = numpy.array(72, dtype='object')\n", "cicada.arithmetic.Field(order=127).inplace_add(lhs=lhs, rhs=numpy.array(8, dtype='object'))\n", "cicada.transcript.assert_equal(lhs, numpy.array(80, dtype='object'))\n", "\n", "# cicada.additive.AdditiveProtocolSuite().share(src=1, secret=numpy.array(3, dtype='int64'), shape=(), encoding=None)\n", "\n", "bg = numpy.random.PCG64()\n", "bg.state = {'bit_generator': 'PCG64', 'state': {'state': 286236090826420737711002355196760144559, 'inc': 74303973817490204264452453842712767295}, 'has_uint32': 1, 'uinteger': 1242373202}\n", "cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).uniform(size=(), generator=numpy.random.Generator(bg)), numpy.array(82, dtype='object'))\n", "\n", "bg = numpy.random.PCG64()\n", "bg.state = {'bit_generator': 'PCG64', 'state': {'state': 66772109718043022979323393862998920392, 'inc': 122458647806018768901565485422780565793}, 'has_uint32': 1, 'uinteger': 123532106}\n", "cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).uniform(size=(), generator=numpy.random.Generator(bg)), numpy.array(74, dtype='object'))\n", "\n", "lhs = numpy.array(82, dtype='object')\n", "cicada.arithmetic.Field(order=127).inplace_subtract(lhs=lhs, rhs=numpy.array(74, dtype='object'))\n", "cicada.transcript.assert_equal(lhs, numpy.array(8, dtype='object'))\n", "\n", "# Let's add some secrets!\n", "# cicada.additive.AdditiveProtocolSuite().add(lhs=cicada.additive.AdditiveArrayShare(storage=80), rhs=cicada.additive.AdditiveArrayShare(storage=8), encoding=None)\n", "\n", "# cicada.additive.AdditiveProtocolSuite().field_add(lhs=cicada.additive.AdditiveArrayShare(storage=80), rhs=cicada.additive.AdditiveArrayShare(storage=8))\n", "\n", "cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).add(lhs=numpy.array(80, dtype='object'), rhs=numpy.array(8, dtype='object')), numpy.array(88, dtype='object'))\n", "\n", "# Let's reveal the results!\n", "# cicada.additive.AdditiveProtocolSuite().reveal(share=cicada.additive.AdditiveArrayShare(storage=88), dst=None, encoding=None)\n", "\n", "lhs = numpy.array(88, dtype='object')\n", "cicada.arithmetic.Field(order=127).inplace_add(lhs=lhs, rhs=numpy.array(0, dtype='object'))\n", "cicada.transcript.assert_equal(lhs, numpy.array(88, dtype='object'))\n", "\n", "lhs = numpy.array(88, dtype='object')\n", "cicada.arithmetic.Field(order=127).inplace_add(lhs=lhs, rhs=numpy.array(59, dtype='object'))\n", "cicada.transcript.assert_equal(lhs, numpy.array(20, dtype='object'))\n", "\n", "cicada.transcript.assert_equal(cicada.encoding.FixedPoint(precision=2).decode(array=numpy.array(20, dtype='object'), field=cicada.arithmetic.Field(order=127)), numpy.array(5.0, dtype='float64'))\n", "\n" ] } ], "source": [ "with open(\"player-0.py\") as stream:\n", " for line in stream:\n", " print(line.strip())" ] }, { "cell_type": "raw", "id": "96db2be2-02e0-434f-955a-1f86a2973ffd", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "As you can see, the context messages are incorporated into the transcript as comments. Because we're transcribing a full-fledged Cicada program, there are additional nuances to be aware of. First, you will notice that there are comments besides those that we explicitly generate from our code, including::\n", "\n", " # cicada.additive.AdditiveProtocolSuite().share(src=0, secret=numpy.array(2, dtype='int64'), shape=(), encoding=None)\n", "\n", "and::\n", "\n", " # cicada.additive.AdditiveProtocolSuite().reveal(share=cicada.additive.AdditiveArrayShare(storage=91), dst=None, encoding=None)\n", "\n", "You might wonder why these high-level statements are converted to comments instead of executable code. This is because consistency verification checks for these operations would require a full-fledged multi-player environment with networking, communicators, and-so-on. Instead, the transcript contains all of the low-level field arithmetic operations that underlie these operations, while the commented code provides context for why those particular field operations are being performed.\n", "\n", ".. note::\n", "\n", " We expect the consistency verification code output from Cicada to grow and evolve over time as we continue to develop this new capability. In particular, the current set of output is defined using whitelists that will likely change with experience. There are no guarantees that the generated code will not change from one version of the library to the next, only that it should continue to run without raising exceptions.\n", "\n", "Now, let's run all three transcripts to verify their consistency:" ] }, { "cell_type": "code", "execution_count": 12, "id": "a27c2f60-9010-45cc-adff-3542c5b838fb", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Player 0 consistency verification succeeded.\n", "Player 1 consistency verification succeeded.\n", "Player 2 consistency verification succeeded.\n" ] } ], "source": [ "import traceback\n", "\n", "for rank in range(3):\n", " try:\n", " with open(f\"player-{rank}.py\") as stream:\n", " for line in stream:\n", " exec(line)\n", " except Exception as e:\n", " print(f\"Player {rank} consistency verification failed: {e}\")\n", " print(traceback.format_exc())\n", " else:\n", " print(f\"Player {rank} consistency verification succeeded.\")" ] }, { "cell_type": "raw", "id": "452e3c56-e9ba-4ade-8e39-2b5ccddf9c1f", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "So, that worked perfectly. Let's consider some other options you can use when working with code outputs. First, you can choose to have network messages interleaved with the code, using `sent=True` and `received=True`, which behave similarly to what we saw earlier with :func:`~cicada.transcript.net_handler`. Let's try turning them on. We'll also go back to sending the transcript directly to the screen, but this time we'll use a :class:`~logging.NullHandler` to disable the output for every player except player 0:" ] }, { "cell_type": "code", "execution_count": 13, "id": "4f3bc0cf-5ad7-4091-a3f2-4ea52394632d", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "# Let's setup additive sharing!\n", "# 0 <-- 2 PRZS 2484391363636162676\n", "# 0 --> 1 PRZS 3705543425192515578\n", "# 0 <-- 2 GATHER 11\n", "# Let's share some secrets!\n", "# cicada.additive.AdditiveProtocolSuite().share(src=0, secret=numpy.array(2, dtype='int64'), shape=(), encoding=None)\n", "\n", "# 0 <-- 1 GATHER 95\n", "bg = numpy.random.PCG64()\n", "bg.state = {'bit_generator': 'PCG64', 'state': {'state': 70824561701714392247314131268118952641, 'inc': 248539783406888609260376891470122436953}, 'has_uint32': 0, 'uinteger': 0}\n", "cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).uniform(size=(), generator=numpy.random.Generator(bg)), numpy.array(31, dtype='object'))\n", "\n", "bg = numpy.random.PCG64()\n", "bg.state = {'bit_generator': 'PCG64', 'state': {'state': 327741436583401028878067909547269207792, 'inc': 66951744883520492504918500539108410445}, 'has_uint32': 0, 'uinteger': 0}\n", "cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).uniform(size=(), generator=numpy.random.Generator(bg)), numpy.array(43, dtype='object'))\n", "\n", "lhs = numpy.array(31, dtype='object')\n", "cicada.arithmetic.Field(order=127).inplace_subtract(lhs=lhs, rhs=numpy.array(43, dtype='object'))\n", "cicada.transcript.assert_equal(lhs, numpy.array(115, dtype='object'))\n", "\n", "cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).__call__(object=numpy.array(8, dtype='object')), numpy.array(8, dtype='object'))\n", "\n", "cicada.transcript.assert_equal(cicada.encoding.FixedPoint(precision=2).encode(array=numpy.array(2, dtype='int64'), field=cicada.arithmetic.Field(order=127)), numpy.array(8, dtype='object'))\n", "\n", "lhs = numpy.array(115, dtype='object')\n", "cicada.arithmetic.Field(order=127).inplace_add(lhs=lhs, rhs=numpy.array(8, dtype='object'))\n", "cicada.transcript.assert_equal(lhs, numpy.array(123, dtype='object'))\n", "\n", "# cicada.additive.AdditiveProtocolSuite().share(src=1, secret=numpy.array(3, dtype='int64'), shape=(), encoding=None)\n", "\n", "bg = numpy.random.PCG64()\n", "bg.state = {'bit_generator': 'PCG64', 'state': {'state': 33710239833072880162431526170503648606, 'inc': 248539783406888609260376891470122436953}, 'has_uint32': 1, 'uinteger': 1396278564}\n", "cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).uniform(size=(), generator=numpy.random.Generator(bg)), numpy.array(36, dtype='object'))\n", "\n", "bg = numpy.random.PCG64()\n", "bg.state = {'bit_generator': 'PCG64', 'state': {'state': 93867515633347282979830946528566031101, 'inc': 66951744883520492504918500539108410445}, 'has_uint32': 1, 'uinteger': 1422560118}\n", "cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).uniform(size=(), generator=numpy.random.Generator(bg)), numpy.array(118, dtype='object'))\n", "\n", "lhs = numpy.array(36, dtype='object')\n", "cicada.arithmetic.Field(order=127).inplace_subtract(lhs=lhs, rhs=numpy.array(118, dtype='object'))\n", "cicada.transcript.assert_equal(lhs, numpy.array(45, dtype='object'))\n", "\n", "# Let's add some secrets!\n", "# cicada.additive.AdditiveProtocolSuite().add(lhs=cicada.additive.AdditiveArrayShare(storage=123), rhs=cicada.additive.AdditiveArrayShare(storage=45), encoding=None)\n", "\n", "# cicada.additive.AdditiveProtocolSuite().field_add(lhs=cicada.additive.AdditiveArrayShare(storage=123), rhs=cicada.additive.AdditiveArrayShare(storage=45))\n", "\n", "cicada.transcript.assert_equal(cicada.arithmetic.Field(order=127).add(lhs=numpy.array(123, dtype='object'), rhs=numpy.array(45, dtype='object')), numpy.array(41, dtype='object'))\n", "\n", "# Let's reveal the results!\n", "# cicada.additive.AdditiveProtocolSuite().reveal(share=cicada.additive.AdditiveArrayShare(storage=41), dst=None, encoding=None)\n", "\n", "# 0 --> 0 GATHER 41\n", "# 0 <-- 0 GATHER 41\n", "lhs = numpy.array(41, dtype='object')\n", "cicada.arithmetic.Field(order=127).inplace_add(lhs=lhs, rhs=numpy.array(95, dtype='object'))\n", "cicada.transcript.assert_equal(lhs, numpy.array(9, dtype='object'))\n", "\n", "lhs = numpy.array(9, dtype='object')\n", "cicada.arithmetic.Field(order=127).inplace_add(lhs=lhs, rhs=numpy.array(11, dtype='object'))\n", "cicada.transcript.assert_equal(lhs, numpy.array(20, dtype='object'))\n", "\n", "# 0 --> 1 GATHER 41\n", "# 0 --> 2 GATHER 41\n", "cicada.transcript.assert_equal(cicada.encoding.FixedPoint(precision=2).decode(array=numpy.array(20, dtype='object'), field=cicada.arithmetic.Field(order=127)), numpy.array(5.0, dtype='float64'))\n", "\n" ] } ], "source": [ "def main(comm):\n", " handler = logging.StreamHandler() if comm.rank == 0 else logging.NullHandler()\n", " handler = transcript.code_handler(handler, sent=True, received=True)\n", " transcript.set_handler(logging.getLogger(), handler)\n", "\n", " transcript.log(\"Let's setup additive sharing!\")\n", " encoding = FixedPoint(precision=2)\n", " protocol = AdditiveProtocolSuite(comm, order=127, encoding=encoding)\n", " transcript.log(\"Let's share some secrets!\")\n", " a_share = protocol.share(src=0, secret=numpy.array(2), shape=())\n", " b_share = protocol.share(src=1, secret=numpy.array(3), shape=())\n", " transcript.log(\"Let's add some secrets!\")\n", " c_share = protocol.add(a_share, b_share)\n", " transcript.log(\"Let's reveal the results!\")\n", " c = protocol.reveal(c_share)\n", "\n", "with transcript.record():\n", " SocketCommunicator.run(fn=main, world_size=3);" ] }, { "cell_type": "raw", "id": "34d5ea54-588c-4ff8-91b1-09899076453a", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "Now we can see the network messages, formatted as Python comments just like context messsages, so they won't interfere with executing the consistency verification code.\n", "\n", ".. important::\n", "\n", " You should understand that messages from other players arrive when they arrive, without regard to what the local player is doing at the moment. It is not at all unusual to see messages associated with a given operation arriving from another player long before the current player has reached that point in the code. This happens when the other player is \"running ahead\" of the current player due to quirks of process sceduling and network timing, and is completely normal. Welcome to distributed computation!\n", "\n", "You can configure the context and network message formatting using the same `fmt` and `netfmt` parameters covered earlier; if you do, it'll be up to you to include the \"#\" in your format strings, to maintain the executability of the consistency verification output." ] }, { "cell_type": "raw", "id": "0ebeadfa-d4eb-4d05-a659-e9cdeea8c56f", "metadata": { "editable": true, "raw_mimetype": "text/restructuredtext", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ ".. seealso:: :ref:`logging`\n", "\n", ".. rubric:: Footnotes\n", "\n", ".. [#] Ishai, Yuval, et al. \"Zero-knowledge from secure multiparty computation.\" *Proceedings of the thirty-ninth annual ACM symposium on Theory of computing. 2007.*" ] } ], "metadata": { "celltoolbar": "Raw Cell Format", "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.2" } }, "nbformat": 4, "nbformat_minor": 5 }