[viff-devel] inlineCallbacks (was: VIFF and large scale programs -- is VIFF really asynchronous?)

[Martin Geisler]

T.Toft at cwi.nl writes:

Hi everybody,

It occurred to me that there is a tool we haven't talked about here.
Let me try to describe it so that you can have it in mind... I think
the change would be simple, the code would change from

> x, y = runtime.shamir_share([1, 2], Zp, input)
> z = x*y
> for i in xrange(100):
>   z=z*z

to this

  x, y = runtime.shamir_share([1, 2], Zp, input)
  z = x*y
  for i in xrange(100):
    z = (yield z * z)

The yield keyword turns a function into a generator -- the function
execution is paused at the place of the yield and the caller gets an
iterator back. This can be used to produce a stream of values:

  def N():
    i = 0
    while True:
      i += 1
      yield i

This is used as

  integers = N()
  for i in integers:
    if i > 42:
      break

This is the basic use of generators and is already quite cool :-) But
it gets more cool when we use them to implement coroutines:

  http://en.wikipedia.org/wiki/Coroutine

In comparison to a subroutine (normal function calls), the execution
of a coroutine can be interleaved with other coroutines -- this is
what we want for VIFF.

Python got support for coroutines in version 2.5:

  http://docs.python.org/whatsnew/2.5.html#pep-342-new-generator-features

and Twisted has a decorator for using them with Deferreds:

  http://twistedmatrix.com/documents/current/api/twisted.internet.defer.html#inlineCallbacks

This decorator allows a function to yield a Deferred value and have
the reactor restart the function when the Deferred has fired. The
function is restarted with *the value* of the Deferred. From the point
of view of the function, this means that we are *waiting on the
Deferred* -- please note that this is normally impossible!

In the above code it would mean that the reactor gets a chance to send
and receive values after each multiplication. But it also means that
the multiplications will be scheduled sequentially.

That is why I haven't introduced inlineCallbacks before: they simplify
some code at the expense of concurrency. For a radical example, please
see this blog post:

  http://blog.mekk.waw.pl/archives/14-Twisted-inlineCallbacks-and-deferredGenerator.html


So... more food for thoughts :-)


-- 
Martin Geisler