[viff-devel] VIFF and large scale programs -- is VIFF really asynchronous?

[Martin Geisler]

Mikkel Krøigård <mk at cs.au.dk> writes:

> However, the focus is on handling the memory issue.
>
> While you can save memory by splitting up the computation, I would
> much prefer it if you could write code like the stuff Tomas wrote
> without thinking about memory-saving tricks. Ideally, it should be
> completely straightforward and without tricks.

I agree. But it is tricky... consider a "good" program like this:

  x = a * b
  y = c * d

and a "bad" program like this:

  x = a * b
  y = x * c

In the first we *want* both multiplications to be scheduled, but in the
second case we would prefer to wait with the calculation of y until x is
here. One problem is that to "wait with the calculation of y" sort of
means that we want y to be a Deferred :-)

> We just discussed if the Twisted reactor could somehow get control
> before we are done scheduling. In the case below, we would like to
> start getting some of the older stuff done before piling on new
> computations, and if the reactor could just send and receive once in a
> while, then maybe that could happen.
>
> However, Twisted is apparently not the kind of scheduling system we
> would need for that. I guess my question for this list is if my
> understanding here is correct.

Definitely, Twisted doesn't do any scheduling.

> Martin told me that there may be a way to reduce the memory usage by
> minimizing the size of deferreds, and that this is already mentioned
> in the bug tracker.

Yes, I have been talking to the Twisted guys about this. Please see this
bug report and especially this comment:

  http://twistedmatrix.com/trac/ticket/3245#comment:13

I have also been adding support for tracking the memory usage. Simply
add --track-memory to any program using VIFF and it will sample the
memory every 0.1 seconds and print out a line when the values change.

By doing that I found that the multiplication benchmark uses an
additional *6 KiB* for *each* additional multiplication I ask it to do.
Here I'm talking about the peak memory usage of the process, so this
includes the Deferreds, their values, their pointers to each other, etc.

> However, add an extra zero to the number of iterations and we're back
> to square one.

True. But high-level software takes up memory. We have a very elegant
in-memory program structure, but it is wasteful with memory unless the
programmer takes special care not to start too much at the same time.

-- 
Martin Geisler
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL: <http://lists.viff.dk/pipermail/viff-devel-viff.dk/attachments/20090126/769b1023/attachment.pgp>