Dmitry Ivanov bio photo

Dmitry Ivanov

Sr Engineer at TomTom, Amsterdam. Nerd and Scala geek. AWS and DevOps devotee.

Email Twitter Facebook Google+ LinkedIn Instagram Github Stackoverflow

ScalaDays 2015 in Amsterdam

Amsterdam loves Scala!

Last week Amsterdam was hosting ScalaDays 2015 conference. I’m really happy that a big Scala event has finally reached The Netherlands.

Amsterdam Scala

There is a huge Scala community with relatively long history in here. First Scala user groups existed here since 2009-2010. Nowadays the biggest one, Amsterdam.Scala meetup, has more than 500 members. With events happening every 2-3 weeks (hackathons, talks, workshops) and awesome speaker line-ups123. It’s far from the biggest Scala community in Europe, but definitely one of the most active ones (hard to compete with London Scala UG though :)

So I’d like to share my thoughts on some talks I found particularly interesting.

Martin Odersky and future of Scala

It all started with Martins keynote. As usual, he spent some time explaining how language was evolving and what exactly he understands by ‘Scalable language’ term.

I really admire some irony he usually adds to his talks:

"Scala - where it came from, where it's going" - Martin Odersky.

But I guess the most interesting part was about Scala future.

Making Scala TASTY

Meet TASTY (Typed Abstract Syntax Trees) - an intermediate format for compiled Scala classes that aims to bring more backward compatibility for Scala libraries.

It’s not suprising at all that it’s super important for Typesafe to solve binary compatibility issue, in order to satisfy Enterprise players and defeat this long-time counter-Scala argument. Overall, it looks to me as a better way to address that rather than just slowing down innovation as was suggested in ‘Scala-should-be-boring’ Rod Jonhsons talk that was given at ScalaDays 2013.

Better type system with Dotty

Dotty - is a EPFL playground language platform for new generation of Scala with new and refined features. With its generalised union/intersection types, implicit function types and etc it surely attracts a lot of attention amongsts typelevel people.

It seems natural that two initiatives (improving binary compatibility and evolving the language) are coming together. And the former one should ease introducing the latter. Nevetheless it will probably take at least a couple of years till we get a chance to start playing with new Scala major version.

Expressing effects in types

Another interesting concept that Martin mentioned is a ‘better treatment of effects’ which is essentially about being able to express code side effects (throwing exceptions, IO, null-dereferencing) via type system.

"Better treatment of effects" - Martin Odersky.

So on the second slide throws keyword from the first method signature is actually just a type throws[R, Exception] that requires to have an implicit CanThrow[Exception], so called capability, in scope. Looks cool, isn’t it? :)

However, only time will show if it’s feasible to make it ‘non-boilerplatty’ and useful.


Jonas Bonér: “Life Beyond the Illusion of Present”

The next keynote by Typesafe CTO Jonas Bonér was about essential concepts of distributed systems design. I always enjoy his talks on this topic. They are great at introducing people to new ways of thinking and reasoning about state and events in distributed systems.

"Life Beyond the Illusion of Present" - Jonas Bonér.

“Tracking time is actually tracking causality.”

That’s what important: Causality and events Ordering.

  • There is no such thing as reliable time or guaranteed message delivery.
  • Think in facts.
  • Never do in-place updates, just collect and apply new events/facts.
  • CRDTs to the rescue!

Event sourcing is the key. Updating the locally observed state is simply merging new facts with old ones:

val newLocalPresent = 
            observedPasts.foldLeft(oldLocalPresent) { _ merge _ }


Akka Actors vs Types

Rolands talk (“Project Gålbma: Actors vs. Types”) was actually my biggest highlight of this conference. A new iteration of a massive work to make Akka Actors Typed was introduced.

I’m a true fan of Akka framework. It is a great piece of software and it really helps building scalable applications faster and more effectively. But lack of type safety when working with actors was always a bit … strange :).

Development with Akka nowadays largely depends on knowing common conventions and patterns. Could be cool if a type system, rather than just docs (which are awesome for Akka) and blogs, guides you in the right direction.

Typed Channels in Akka 2.1, - Roland Kuhn.

New typed Actor API proposal made me think: “Yes, that’s a proper way to do it in Scala, language with a rich and strong type system.”

Akka Typed API proposal, - Roland Kuhn.
  • Actor is characterised from now by a Behavior[T] represented as a partial function Message | Event => Behavior
  • No more overridable methods like preStart, postStop, etc: you can define reaction to these Signals in behavior function.
  • No more direct reference to actor context.
  • No more built-in sender(): you have to encode a sender reference to your message API.

Latest Akka Typed docs and examples can be found here:

Lambda Architecture & Spark Streaming

Another notable talk was by Helena Edelson - “Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka and Scala”.

She demonstrated how you can build stream processing pipelines leveraging high composability of this stack.

Althought a bit too much Cassandra advertising (I like C*, but still), it was cool to see some snippets how you can glue these things together and what are potential use-cases. And of course Spark connector for Cassandra is pretty awesome. Now other NoSQL vendors start understanding4 necessity of Spark-support as well. :)

Few years ago Storm was almost a standard component of ‘Lambda architecture’. Event the term itself was coined by the creator of Storm. Now Spark Streaming together with Akka Streams, Kafka and others might get this place.


Monad Transformers for Mere Mortals

How often do you have to deal with nested containers? For example, Future[Option[T]] or Future[Either[L,R]]. Composing them together in for-comprehensions is not than easy. Consider this code sample from Erik Bakkers presentation “Options in Futures, how to unsuck them”:

val fa: Future[Option[Int]] = ???
val fb: Future[Option[Int]] = ???

// Problem, `a` and `b` are Option[Int], and not Int
val result = 
  for {
  	a <- fa
  	b <- fb
  } yield a - b

To solve this, people usually end up creating a set of utility functions, implicit transformers, like FutureOption[T], that allow you to flatten containers. Althought it works, you still have to steal^Wwrite these.

Consider another solution using Scalaz OptionT, a so called Monad Transformer:

val fa: Future[Option[Int]] = ???
val fb: Future[Option[Int]] = ???

// Here, a and b are Int, extracted from both the Future and Option!
val finalOptionT = 
	for {
		a <- OptionT(fa)
		b <- OptionT(fb)
	} yield a - b

// And to get back to the normal structure
val finalFutureOption: Future[Option[Int]] =

Often people in Scala community think that Scalaz or Shapeless are too hardcore and avoid them.

Sometimes it’s a valid point :), but I prefer to borrow just essential stuff. Like awesome ValidationNEL of Scalaz, or HList of Shapeless, or these Monad Transformers OptionT, \/ that Erik demonstrated.

Slides from Eriks workshop:

I can also recommend another great presentation on Scalaz ‘gateway drugs’ by Brendan McAdams:

Functions-passing style

Followed by their work on Spores Heather Miller and Philipp Haller presented this year a new distributed programming model based on Spores: “Function-Passing Style, A New Model for Asynchronous and Distributed Programming”.

  • In this model, unlike with traditional data passing, you actually send a serialized clojures that are executed on the remote node against target data.
  • The data holder, or Silo, resides on the node that has target data. Location transparency in system is achieved by introducing a SiloRef which encapsulates lazy application of the clojure to target data in remote Silo.
  • Serializability of clojures is guaranteed by Spores.
  • There is a limitation of Spores: nodes running application code should have the same classpath (otherwise deserialization will fail).

It’s still unclear to me how to handle concurrent updates to the same data. Probably it’s out of the models scope and can be tackled by using CRDT-like data structures or just immutable data.

"Function-Passing Style, A New Model for Asynchronous and Distributed Programming" - Philipp Haller and Heather Miller.

Overall I think it’s great and makes a lot of sense for data-crunching systems like Spark where data locality is the key and sending it over the wire is not an option (because of the volumes).


Rapture Result and Modes

Jon Pretty presentation with a quite complicated name (“Delimited dependently-typed monadic checked exceptions in Scala”) attracted my attention because of two things:

  • Interesting concept of abstracting the actual return type of library API (using so called Modes).
  • A new Result container similar to Either, ValidationNEL and others.
"Delimited dependently-typed monadic checked exceptions in Scala" - Jon Pretty. Modes allow users to choose the return type of the library API:

import modes.returnEither._
Json.parse("[1, 2, 3]") // will have return type Either[ParseException, Json]


import modes.returnFuture._
Json.parse("[1, 2, 3]") // will immediately return a Future[Json]

It’s looks like a cool magic, but I see a couple of different scenarios where this might be useful:

  • Turning library API to blocking for tests and to async in your main code.
  • Switching between, for example, Either-based or Try-based API without having to write your own wrappers.

The tri-state Return type looks something between Either and ValidationNEL:

val result: Result[Json, ParseException] = ...

// result might be: 
// `Answer` - successful result
// `Errata` - contains _collected_ failures
// `Unexpected` - contains a signle (non-recoverable?) exception

I like the trick of simulating union types using ‘with’:

// Read 'with' as 'or'
val result: Result[Json, DatabaseException with IOException with SerializationException] = ...

This is useful here as Errata is actually a typed Multimap.


Kamon and Akka metrics

There is a huge demand for Akka monitoring nowadays. Typesafe console was looking great, but tracing performance was … awful.

Authors of Kamon are trying to meet that demand with their toolkit for collecting actor metrics and traces.

Ivan Topolnjak showed in his talk, “Kamon: Metrics and traces for your reactive application”, how they were able to minimise performance impact of metrics collection to the system. It was interesting to know that they are using HdrHistogram for this.

HdrHistogram (High Dynamic Range Histogram) is “A Histogram that supports recording and analyzing sampled data value counts across a configurable integer value range with configurable value precision within the range”5. And it is really-really fast. :)

"Kamon: Metrics and traces for your reactive application" - Ivan Topolnjak.

Tracing actor message flows is another important problem which is not fully solved yet. There is a still experimental Akka Tracing extension. And Kamon also has some of these capabilities: It would be cool to eventually have a visualization for flows like the one Twitter Zipkin provides.

Also would be nice to be able to trace composite Futures.

Other notable talks

I will update this post with the links to videos as soon as they become available on Parleys.