alexn.org

Monix Task vs Scalaz vs Future, The Benchmark

Monifu square

The Monix Task was heavily inspired by scalaz.concurrent.Task and by scala.concurrent.Future. That's not a secret and I'll be forever grateful to their authors. I've ran a benchmark and I'm glad to report that the Monix Task beats in performance both.

Such results are actually unexpected, because the Monix Task has to do tricks in order to be "cancelable", a trait that allows it to close opened resources when race conditions happen, which really means extra footwork. But no, right now, it beats both in performance and by quite the margin.

Details:

  • Benchmark used is TaskGatherBenchmark in the repository
  • Monix version: 2.0-RC13
  • Scalaz version: 7.2.4
  • Scala version: 2.11.8
  • Java version: 1.8.0_60
  • OS: OS X El Captain, version 10.11.6

Sequence

The purpose of this test is the performance of flatMap, or in other words the performance of the run-loop, on both normal/synchronous tasks and tasks that are forked in separate (logical) threads. So in other words:

Task.sequence(tasks)

Which is translated more or less into this:

tasks.foldLeft(init)((acc,et) => acc.flatMap(b => et.map(e => b += e)))

So for synchronous tasks (that evaluate immediately), and note here that Scala's Future is not applicable since Future is not trampolined:

Source Type Operation Score Error Units
Monix Sync Sequence 6716.906 157.947 ops/s
Scalaz Sync Sequence 3518.888 167.148 ops/s

And for tasks that fork threads on execution:

Source Type Operation Score Error Units
Monix Forked Sequence 2044.624 24.852 ops/s
Scalaz Forked Sequence 1090.355 15.851 ops/s
S.Future Forked Sequence 1753.614 20.871 ops/s

As you can see, the Monix Task has twice the throughput of Scalaz and fares quite better compared with Scala's standard Future.

Gather

The gather operation would be:

Task.gather(tasks)

This works like sequence, except that the evaluation has non-ordered effects. What this means is that, if the tasks are forking threads, then they get executed in parallel.

For synchronous/immediate tasks the numbers are:

Source Type Operation Score Error Units
Monix Sync Gather 3800.559 341.509 ops/s
Scalaz Sync Gather 2152.441 13.569 ops/s
S.Future Forked Sequence 1753.614 20.871 ops/s

And for forked tasks:

Source Type Operation Score Error Units
Monix Forked Gather 1396.797 17.098 ops/s
Scalaz Forked Gather 1014.452 13.569 ops/s
S.Future Forked Sequence 1753.614 20.871 ops/s

Including the results of Future.sequence as well, because Future has strict evaluation and it can be used to execute futures in parallel. The performance of gather can be worse than Future.sequence, because of the execution model. But if it executes tasks that have immediate execution, or a mixed batch, then it is much better.

Gather Unordered

The gatherUnordered operation would be:

Task.gatherUnordered(tasks)

This behaves like gather, except that it does not care for the order in which the results are served. Can have much better performance if you don't care about order.

For synchronous/immediate tasks:

Source Type Operation Score Error Units
Monix Sync Unordered 5654.462 150.792 ops/s
Scalaz Sync Unordered 3340.645 244.145 ops/s
S.Future Forked Sequence 1753.614 20.871 ops/s

For forked tasks:

Source Type Operation Score Error Units
Monix Forked Unordered 1658.055 12.114 ops/s
Scalaz Forked Unordered 1657.454 35.218 ops/s
S.Future Forked Sequence 1753.614 20.871 ops/s

Again, performance is really good for synchronous tasks, whereas for forked tasks it evens out with the performance of Future.sequence.

Raw output

[info] # Run complete. Total time: 00:04:28
[info]
[info] Benchmark                              Mode  Cnt     Score     Error  Units
[info] TaskGatherBenchmark.gatherMonixA      thrpt   10  1396.797 ±  17.098  ops/s
[info] TaskGatherBenchmark.gatherMonixS      thrpt   10  3800.559 ± 341.509  ops/s
[info] TaskGatherBenchmark.gatherScalazA     thrpt   10  1014.452 ±  13.569  ops/s
[info] TaskGatherBenchmark.gatherScalazS     thrpt   10  2152.441 ±  24.811  ops/s
[info] TaskGatherBenchmark.sequenceFutureA   thrpt   10  1753.614 ±  20.871  ops/s
[info] TaskGatherBenchmark.sequenceMonixA    thrpt   10  2044.624 ±  24.852  ops/s
[info] TaskGatherBenchmark.sequenceMonixS    thrpt   10  6716.906 ± 157.947  ops/s
[info] TaskGatherBenchmark.sequenceScalazA   thrpt   10  1090.355 ±  15.851  ops/s
[info] TaskGatherBenchmark.sequenceScalazS   thrpt   10  3518.888 ± 167.148  ops/s
[info] TaskGatherBenchmark.unorderedMonixA   thrpt   10  1658.055 ±  12.114  ops/s
[info] TaskGatherBenchmark.unorderedMonixS   thrpt   10  5654.462 ± 150.792  ops/s
[info] TaskGatherBenchmark.unorderedScalazA  thrpt   10  1657.454 ±  35.218  ops/s
[info] TaskGatherBenchmark.unorderedScalazS  thrpt   10  3340.645 ± 244.145  ops/s

Cheers!