[viff-devel] VIFF benchmarks

[Thomas Jakobsen]

Hi VIFF'ers,

I've written a VIFF application that should make it easy to create,
run and visualize benchmarks for VIFF protocols.

As an illustrating example, all you have to do in order to benchmark
the Toft05 comparison:

1) Define the benchmark. This means implementing a subclass of
VIFFBenchmark that specifies the runtime class to be used as well as
what should be measured. Here we simply measure the time it takes to
compare two random shared integers using the Toft05 comparison
protocol.

class BenchmarkToft05(VIFFBenchmark):
    def get_runtime_class(self):
        return Toft05Runtime
    def protocol(self, rt, attr):
        Zp = GF(int(attr['modulus']))
        a = rt.prss_share_random(Zp)
        b = rt.prss_share_random(Zp)
        c = gather_shares([a,b])
        c.addCallback(self.start_timer)
        d = (a <= b)
        d.addCallback(self.stop_timer)
        return d

2) Run the benchmark. This is done by setting up a benchmark suite
that has a connection to a database where it can store the results.
Multiple benchmarks with various parameters can then be added to the
suite before it is run.

d = Database([host="localhost", db="benchmark", user="benchamark",
passwd = "secret")
s = Suite(revision="ec70dc5ac213",
              database=d,
              user="smpc",
              work_dir = "some/dir/for/temporary/files",
              hosts=config['hosts'])

(The config['hosts'] reads a config file where host adresses and ports
are specified.)

s.add_benchmark(BenchmarkToft05({
    'runs': 100,
    'n': 3,
    'threshold': 1,
    'modulus': 2342342348883
    'use_ssl': True}))

s.run()

In this case, the Toft05 comparison is run 100 times with 3 players
and security threshold 1 using ssl_connection between the players.
Multiple benchmarks can be added to the suite before running it. The
resulting timings are stored in the database. Any parameter that is
specified when adding a benchmark will be available to the running
benchmark, e.g. in the protocol()-method.

3) Visualize the results. Data can be fetched directly from the
database or through some helper methods (not quite written yet...) and
visualized using e.g. matplotlib. Each single result is stored, so
statistical analysis such as computing confidence intervals, is
possible.


That's it. If you want to benchmark something that does not involve a
protocol, e.g. just the time it takes to do a local multiplication (or
even something unrelated to VIFF), you can subclass Benchmark instead
of VIFFBenchmark. It works almost the same way execpt that the
benchmark is only run on one machine and there is no runtime
available.

I've used the application to set up a few benchmarks and run it on our
new benchmark servers. The autogenerated result can be seen on
http://www.daimi.au.dk/~tpj/web. It shows two benchmarks for the tip
revision. There's also a "history" graph showing execution time of the
Toft05 comparison back in time. Each dot represents a mercurial
commit. Luckily it gets faster and faster :-) I admit there's not much
explanation to the graphs yet, but I hope it's enough to illustrate
the use of the application.

The script that generated the graphs can easily be set up to run
regularly as a cronjob. I'm about to do that such that we always have
an up-to-date view on the performance of VIFF. You are welcome to use
the application for your self, but if you have a benchmark that you
will like to be included in the "nightly benchmark" - and preferably,
some code or concrete idea how to do the benchmark, you are welcome to
contanct me and I will try to include it.

If you are interested in the source and have access to the DAIMI file
system, it is located at /users/tpj/viff-repos/viff-benchmark. I will
later put it on viff.dk. If you have problems getting it to work, I'll
like to hear about that, too.

Disclaimer: The code is still very much "alpha".

Regards,
Thomas