Alexandru Nedelcu - Blog

Copyright is Dead

Alexandru Nedelcu — Fri, 06 Mar 2026 07:10:02 +0000

Thomas, my tomcat, trying to fit into a box of tissues, visibly distressed that he can't fit. This is a metaphor, not sure if it's working 😛

If copyrighted works can be whitewashed / reimplemented via AI, then copyright is dead. And now we have a case in public view, the reimplementation of chardet, relicensed from LGPL to MIT, causing upheaval.

This is a yet another opinion piece on AI, sorry, I’m taking a 40-days break from social media, which prompts me to write more on my blog. At least I get to be less superficial about it.

The chardet reimplementation was created by pointing the AI at the project’s test suite that works as the specification. However, if the project’s source code was in the LLM’s training data, then one can argue that this wasn’t a “clean room” reimplementation. You don’t necessarily need to copy/paste code from the original project in order to create a derivative work — people that get exposed to original source code can’t really make the claim that they could execute a clean room reimplementation, which is why the guidelines for contributing to reverse engineering efforts (like the Wine or Mono projects) contained warnings against viewing Microsoft’s proprietary code.

And many people are now worried that if this is legal, then copyleft is dead. And they should be worried, but it’s not just copyleft fans that should be worried, but rather all creative industries that depend on copyright. Because if this passes, then copyright is as good as dead.

Well, I for one don’t think this is all that bad. Sure, it can potentially leave us without a job, but copyright has always been … fake. Copyright violations are not really theft, because copyright is not like actual private property — because if you’re infringing on someone’s copyright, you aren’t depriving the owner of their possesion. Copyright can only work because it’s a government-granted monopoly. And globally, it can only work because the US and Europe could use their economic strength to impose it via treaties.

One can make the case that it deprives the owner of future revenue and recognition, which is the case made by the likes of RIAA and MPAA, which is also why we ended up with DRM, i.e., software defective by design and by law. Are we really siding with the RIAA, MPAA and Disney now? That should make us, Open Source fans, question our beliefs, despite the many benefits that copyright also bestowed on us.

And “copyleft” happened as a counter-reaction of hackers that used the law to turn copyright on its head. The story goes that Richard Stallman, the guy behind the GPL license and the FSF, wanted to improve his printer’s driver and he couldn’t do that, because printer drivers started being proprietary, with no available source-code. If the guy had the possibility of reimplementing a printer driver for cheap, we probably wouldn’t have a copyleft movement.

Copyleft hasn’t been suitable for protecting many of authors’ wishes and interests for some time. An early example of this was tivoization, with GPL being patched in GPLv3. But then there was Linus Torvalds that came out against it, in an interview, saying:

“To me, the GPL really boils down to “I give out code, I want you to do the same.” The thing that makes me not want to use the GPLv3 in its current form is that it really tries to move more toward the “software freedom” goals. For example, the GPLv2 in no way limits your use of the software. If you’re a mad scientist, you can use GPLv2’d software for your evil plans to take over the world (“Sharks with lasers on their heads!!”), and the GPLv2 just says that you have to give source code back. And that’s OK by me. I like sharks with lasers. I just want the mad scientists of the world to pay me back in kind. I made source code available to them, they have to make their changes to it available to me. After that, they can fry me with their shark-mounted lasers all they want.” — Linus Torvalds, 2006

Another example was the proliferation of “source available” licenses that are not Open Source or Free software. The first instance I remember is Microsoft’s Shared Source Initiative, back when Microsoft was treating Linux as “cancer”. And such initiatives are now back in full force, because the era of interest-free money may be over, companies building Open Source struggling to survive and flipping, e.g., MongoDB, Elasticsearch, Akka. Their reasoning for doing so may be legitimate — copyleft doesn’t work given the “SaaS loophole”, and if you try selling your Open Source product as SaaS, Amazon may be able to do it better and cheaper.

In fairness, they didn’t get the memo — Open Source, in the context of a business, can only work as a complementary to a proprietary product, as a way to bring costs down, or as a strategy to get rid of your competition.

But wait, in a world in which the cost of cloning stuff is going down, even if proprietary, or in a world in which laymen can just solve their own, simpler problems, without buying into a whole SaaS deal, avoiding to get stuck with everything but the kitchen sink … SaaS is probably dead, too.

And yet another example are all the people concerned that their software ends up being used in a way they disagree with (e.g., by the military, or for surveillance). Famously, a license like the Software shall be used for good, not evil is not compatible with Open Source / Free Software, like all restrictions on usage. Although, to be frank, you may try banning the military from using and abusing your software, but that’s a bit naive, because the state, by design, has a monopoly on violence, copyright is only possible due to this social construct, and you can’t expect the military to show restraint when it comes to copyright infringements, especially during war or “special operations”.

I mentioned that it’s not all doom and gloom because …

If you’re an anarchist loving the ideas behind Open Source, you should like some of these developments, because yes, you may now be able to fix your printer driver. And this ability may not be restricted just to software developers, but to laymen as well. “Democratizing” this or that should have happened for software development, and instead, the world moved to protected, defective by design, consumption-only devices, in order for industry to protect copyright, the world converging to a handful of monopolies built on copyright and patents.

Well, I may worry about my future, but my anarchist heart is glad to see the days of copyright numbered.

The article Copyright is Dead first appeared on alexn.org.

Boring Tech May Become Irrelevant Tech

Alexandru Nedelcu — Thu, 05 Mar 2026 16:12:06 +0000

Thomas, self-employed tomcat, worried about the evolution of the software industry.

You may have read the “Boring” Tech Stack in 2015, a view with which I’ve wholeheartedly agreed. And if you didn’t agree with this view, the corporation you’re working for surely agreed. It’s why companies like Google have had a list of “approved” languages, for example. But is this view relevant in 2026? I think not.

This is a continuation of Programming Languages in the Age of AI Agents.

Companies pick “boring” tech because boring is reliable, it does the job, and most importantly, it minimizes risk. Companies don’t want to be locked into tech stacks that are obsolete, because:

Hiring developers becomes more challenging, and historically, it’s been good to pick tooling that optimizes for “horizontal scalability” of development — i.e., optimize for head count — which makes sense in a growing industry, given there are more beginners than seniors, beginner-friendly tech wins, and popular tech is beginner-friendly, due to it being old, popular and targeting simple apps.
Projects get stuck on obsolete tech, companies being forced to support that tech stack on their own. Project rewrites are super risky, the industry being filled with stories of failed rewrites. As such, we now have it ingrained that a project rewrite is something we should never do. So, companies want tech stacks that get supported for decades.
A special mention here goes for maintenance — older tech tends to be more stable, whereas newer tech tends to break compatibility more often, and the upgrade treadmill sucks.
The YAGNI mentality is most often good, our industry’s history being filled with stories of projects preparing for “big data” with data that could easily fit in RAM.

But in 2026 we are facing the reality of AI/LLMs. I’ll use the example of less popular programming languages, such as Rust, Scala, OCaml, F#, Haskell, and others. Here’s why:

The cost of refactoring for switching tech stacks, programming languages included, is going to zero.
The cost of training new people on your tech stack also converges to zero — if your project has the right skills promoted and the guardrails, does it matter if novices aren’t “functional programming” wizards from day 1?
The cost of dependency upgrades goes to zero, becoming an automated process, even when it involves API breakage.
Frontier models are good, but they still do dumb shit all the time, so having better tech, like a potent compiler to watch your back, matters.
The feedback loop matters, and static, expressive type systems provide the most bang for the buck (optimal token usage).
Top programming languages have the advantage of a large corpus available for training, however, that’s also a liability, as the common denominator of all code seen in the wild is not very good (which is why by simply telling the LLM to use FP, it will generate better code for the simple fact that FP code seen in the wild tends to be higher quality).

People are already talking about programming languages designed for LLMs, not humans. I think that’s nonsense; we’ll use languages designed for LLMs and humans, because even if you are an AI maximalist, you still need to communicate with the AI system efficiently, and to have the ability to review what it did. And what will such a language look like?

Well, IMO (YMMV), such a language will be high-level, but it will also be very precise, very static, very expressive, in order to optimize the feedback loop, to optimize tokens, to minimize mistakes, to force LLMs and humans alike to apply mathematical reasoning. It’s hard to picture us settling on natural languages (English); that boat sailed in the ~16th century when symbolic notation was adopted for math. And it’s hard for me to see very popular languages like Java, Python or C++ surviving either, due to all their inherent unsafety.

Anyway, more to the point — given the costs of software maintenance, refactoring or adoption are significantly going down, converging to zero, then old lessons no longer make sense, and companies are already on outdated best practices regarding the adoption of tech stacks.

The next time you’re in a meeting with stakeholders, they probably have LLMs/AI in their KPIs 😉, so now you have the words to fight back against Java, Spring, Hibernate and Oracle DB 😈

The article Boring Tech May Become Irrelevant Tech first appeared on alexn.org.

Tapir Server with Cats-Effect and Pekko HTTP (snippet)

Alexandru Nedelcu — Thu, 05 Mar 2026 12:27:35 +0000

At work, we have been using Akka/Pekko HTTP for some servers, but one of the foundations is Cats-Effect, and I wanted to use Tapir for refactoring one of our microservices.

As you may have noticed, I sometimes use this blog as a replacement for GitHub Gist.

Here’s a simple snippet for using Tapir, with business logic driven by Cats-Effect, using Akka/Pekko HTTP as a backend:

#!/usr/bin/env -S scala shebang

//> using scala "3.8.2"
//> using dep com.softwaremill.sttp.tapir::tapir-core:1.13.10
//> using dep com.softwaremill.sttp.tapir::tapir-pekko-http-server:1.13.10
//> using dep org.typelevel::cats-effect:3.6.3
//> using dep org.apache.pekko::pekko-http:1.3.0
//> using dep org.apache.pekko::pekko-actor-typed:1.4.0

import cats.effect.*
import cats.effect.std.Dispatcher
import org.apache.pekko.actor.typed.ActorSystem
import org.apache.pekko.actor.typed.scaladsl.Behaviors
import org.apache.pekko.http.scaladsl.Http
import org.apache.pekko.http.scaladsl.server.Route
import scala.concurrent.ExecutionContext
import sttp.tapir.*
import sttp.tapir.server.pekkohttp.PekkoHttpServerInterpreter

object Main extends IOApp.Simple {

  // Endpoint definition (pure tapir, no effect type yet)
  val helloWorldEndpoint =
    endpoint.get
      .in("hello" / "world")
      .in(query[String]("name"))
      .out(stringBody)
      .errorOut(stringBody)

  // Business logic described via Cats Effect IO functions
  def greetLogic(name: String): IO[Either[String, String]] =
    IO.println(s"Saying hello to: $name")
      .as(Right(s"Hello, $name!"))

  // Bridge: IO logic → Future, as required by the Pekko HTTP interpreter
  def helloWorldRoute(using Dispatcher[IO], ExecutionContext): Route =
    PekkoHttpServerInterpreter().toRoute(
      helloWorldEndpoint.serverLogic(name => 
        summon[Dispatcher[IO]].unsafeToFuture(greetLogic(name))
      )
    )

  override def run: IO[Unit] = {
    val res =
      for {
        given Dispatcher[IO] <- Dispatcher.parallel[IO]
        given ExecutionContext <- Resource.eval(IO.executionContext)
        given ActorSystem[Nothing] <- Resource(IO {
          val system = ActorSystem(Behaviors.empty, "tapir-pekko-sample")
          val cancel = IO.fromFuture(IO {
            val f = system.whenTerminated
            system.terminate()
            f
          })
          (system, cancel.void)
        })
        _ <- Resource(IO {
          // Starts server
          val bound = Http().newServerAt("localhost", 8383)
            .bind(helloWorldRoute)
          val cancel =
            IO.fromFuture(IO {
              bound.flatMap(_.unbind())
            });
          (bound, cancel.void)
        })
      } yield ()

    res.use { _ =>
      for {
        _ <- IO.println(
          "Server running at http://localhost:8383 — press ENTER to stop"
        )
        _ <- IO.readLine.void
      } yield ()
    }
  }
}

The article Tapir Server with Cats-Effect and Pekko HTTP (snippet) first appeared on alexn.org.

Hybrid IO-driven Promise (Scala snippet)

Alexandru Nedelcu — Tue, 17 Feb 2026 12:03:04 +0000

Alternative to Scala’s Promise and Cats-Effect’s Deferred, exposing a hybrid API, useful for interoperability between imperative libraries (e.g., Akka/Pekko) and Cats-Effect.

Compared with Deferred, this IOPromise allows for completing the promise via unsafeTryComplete, which can be used in an imperative (non-IO) context, but it still exposes an IO-driven API.

Pros:

The advantage of using this, instead of just wrapping scala.concurrent.{ Promise, Future } references in cats.effect.IO is that the IO value returned by awaitOrGet doesn’t leak memory (being able to unregister its callback on cancellation).
The advantage of using this, over cats.effect.Deferred, is that it allows for completing the promise in an imperative context, without needing to instantiate a Dispatcher or having to import cats.effect.unsafe.global, thus being safer, more efficient and with a lighter API.

//> using dep "org.typelevel::cats-effect:3.6.3"
//> using scala "3.3.7"

import cats.effect.IO
import java.util.concurrent.atomic.AtomicReference
import scala.annotation.tailrec
import scala.collection.immutable.Queue
import scala.concurrent.ExecutionContext
import scala.concurrent.Future

/** This is a replacement for `scala.concurrent.Promise` and
  * `cats.effect.Deferred`, exposing a hybrid API, useful in 
  * interop scenarios.
  *
  * All methods exposed have both pure/safe and impure/unsafe variants:
  *   - [[tryComplete]] and [[unsafeTryComplete]]
  *   - [[awaitOrGet]] and [[unsafeAwaitOrGet]]
  */
final class IOPromise[A] private (ref: AtomicReference[IOPromise.State[A]]) {
  import IOPromise.*

  /** Attempts to complete the promise with the given result.
    *
    * It's UNSAFE, in the sense that it's side-effecting. This can be a feature,
    * as it can be used in a non-`IO` context.
    *
    * @return
    *   `true` if the promise was successfully completed with the given result,
    *   `false` if the promise was already completed by another
    *   thread/actor/fiber.
    */
  @tailrec
  def unsafeTryComplete(result: Either[Throwable, A]): Boolean = {
    val state = ref.get()
    state.tryComplete(result) match {
      case None => false
      case Some((task, newState)) =>
        if (ref.compareAndSet(state, newState)) {
          task.run()
          true
        } else {
          // Retry...
          unsafeTryComplete(result)
        }
    }
  }

  /** Variant of [[unsafeTryComplete]]. */
  def unsafeTrySuccess(value: A): Boolean =
    unsafeTryComplete(Right(value))

  /** Variant of [[unsafeTryComplete]]. */
  def unsafeTryFailure(ex: Throwable): Boolean =
    unsafeTryComplete(Left(ex))

  /** Safe (IO-driven) version of [[unsafeTryComplete]]. */
  def tryComplete(result: Either[Throwable, A]): IO[Boolean] =
    IO(unsafeTryComplete(result))

  /** Safe (IO-driven) version of [[unsafeTrySuccess]]. */
  def trySuccess(value: A): IO[Boolean] =
    tryComplete(Right(value))

  /** Safe (IO-driven) version of [[unsafeTryFailure]]. */
  def tryFailure(ex: Throwable): IO[Boolean] =
    tryComplete(Left(ex))

  /** Returns an `IO` that completes when the promise is completed.
    *
    * @see
    *   [[unsafeAwaitOrGet]] for the side-effecting version of this method.
    */
  def awaitOrGet: IO[A] = {
    def cancel(cb: Callback[A]): IO[Unit] =
      IO.defer {
        val current = ref.get()
        current.unregister(cb) match {
          case Some(update) if !ref.compareAndSet(current, update) =>
            cancel(cb) // retry
          case _ =>
            IO.unit
        }
      }

    @tailrec
    def tryCompleteOrEnqueue(cb: Callback[A]): Option[IO[Unit]] = {
      val state = ref.get()
      state.getResult(cb) match {
        case Right(newState) =>
          if (ref.compareAndSet(state, newState)) {
            Some(cancel(cb))
          } else {
            tryCompleteOrEnqueue(cb)
          }
        case Left(task) =>
          task.run()
          None
      }
    }

    IO.executionContext.flatMap { ec =>
      IO.async { cb =>
        IO(tryCompleteOrEnqueue((ec, cb)))
      }
    }
  }

  /** Awaits the completion of the promise, producing a result.
    *
    * It's UNSAFE because it's side-effecting, and `Future` itself has its own
    * warts.
    *
    * @see
    *   [[awaitOrGet]] for the safe (IO-driven) version of this method.
    * @return
    *   a `Future` that completes when the promise is completed.
    */
  def unsafeAwaitOrGet(using ec: ExecutionContext): Future[A] = {
    @tailrec
    def tryCompleteOrEnqueue(cb: Callback[A]): Unit = {
      val state = ref.get()
      state.getResult(cb) match {
        case Right(newState) =>
          if (!ref.compareAndSet(state, newState)) {
            tryCompleteOrEnqueue(cb)
          }
        case Left(task) =>
          task.run()
      }
    }

    val promise = scala.concurrent.Promise[A]()
    tryCompleteOrEnqueue((ec, r => promise.complete(r.toTry)))
    promise.future
  }
}

object IOPromise {
  /** Creates a new `IOPromise`.
    *
    * @see
    *   [[unsafe]] for the side-effecting version of this method.
    */
  def apply[A](): IO[IOPromise[A]] =
    IO(unsafe())

  /** Creates a new `IOPromise` in a side-effectful way.
    *
    * UNSAFE, because it's side-effecting, meaning that it allocates mutable
    * state.
    *
    * @see
    *   [[apply]] for the safe (IO-driven) version of this method.
    */
  def unsafe[A](): IOPromise[A] =
    new IOPromise(new AtomicReference(State.Pending(Queue.empty)))

  private type Callback[A] = (ExecutionContext, Either[Throwable, A] => Unit)

  sealed private trait State[A] extends Product with Serializable {
    /** @return
      *   either `Some((task, newState))`, in case the state was `Pending`, and
      *   we have to call all the registered callbacks. Or `None`, in case the
      *   state was already `Completed`, and the caller has nothing left to do.
      */
    def tryComplete(result: Either[Throwable, A]): Option[(Runnable, State[A])] =
      this match {
        case State.Pending(callbacks) =>
          val task: Runnable = () =>
            for ((ec, notify) <- callbacks)
              ec.execute(() => notify(result))
          Some((task, State.Completed(result)))
        case State.Completed(_) =>
          None
      }

    /** Attempts to register the given callback, if the state is still
      * `Pending`, returning the new state with the callback registered.
      *
      * If the state is already completed, then the callback isn't registered,
      * the caller returning a task that should be executed. Note that this
      * method is supposed to be pure, which is why we are not calling the
      * callback directly.
      */
    def getResult(cb: Callback[A]): Either[Runnable, State[A]] =
      this match {
        case State.Pending(callbacks) =>
          Right(State.Pending(callbacks.enqueue(cb)))
        case State.Completed(result) =>
          Left { () =>
            val (ec, notify) = cb
            ec.execute(() => notify(result))
          }
      }

    /** Removes a callback from the queue, used for cancellation. */
    def unregister(cb: Callback[A]): Option[State[A]] =
      this match {
        case State.Pending(callbacks) =>
          val filtered = callbacks.filterNot(_ == cb)
          Some(State.Pending(filtered))
        case State.Completed(_) =>
          None
      }
  }

  private object State {
    /** Represents a promise that is still pending, and has a queue of callbacks
      * waiting on completion.
      */
    final case class Pending[A](callbacks: Queue[Callback[A]]) extends State[A]

    /** Represents a promise that is already completed, with the result
      * memoized.
      */
    final case class Completed[A](result: Either[Throwable, A]) extends State[A]
  }
}

As a usage example, here’s a KillSwitch implementation on which we can await the shutdown via cats.effect.IO, very useful for Akka/Pekko+Cats-Effect interop.

//> using dep "org.apache.pekko::pekko-stream-typed:1.4.0"

import cats.effect.kernel.Resource
import cats.effect.IO
import org.apache.pekko.stream.FlowShape
import org.apache.pekko.stream.Graph
import org.apache.pekko.stream.KillSwitch
import org.apache.pekko.stream.KillSwitches
import org.slf4j.LoggerFactory

/** Replacement for Akka/Pekko `SharedKillSwitch`.
  *
  * It's primary innovation is the [[awaitShutdown]] method.
  *
  * NOTE: the use of `SharedKillSwitch` in this project is now banned ;-)
  */
trait BetterKillSwitch extends KillSwitch {
  def flow[T]: Graph[FlowShape[T, T], BetterKillSwitch]
  def awaitShutdown: IO[KillSwitchOutcome]
}

object BetterKillSwitch {
  def unsafe(name: String): BetterKillSwitch = {
    val underlying = KillSwitches.shared(name)
    val promise = IOPromise.unsafe[KillSwitchOutcome]()
    new BetterKillSwitchEfficient(underlying, promise)
  }

  def apply(name: String): IO[BetterKillSwitch] =
    IO(unsafe(name))

  def resource(name: String): Resource[IO, BetterKillSwitch] =
    Resource(apply(name).map { ks =>
      (ks, IO(ks.shutdown()))
    })
}

final private class BetterKillSwitchEfficient(
  underlying: org.apache.pekko.stream.SharedKillSwitch,
  promise: IOPromise[KillSwitchOutcome]
) extends BetterKillSwitch {
  override def awaitShutdown: IO[KillSwitchOutcome] =
    promise.awaitOrGet

  override def shutdown(): Unit =
    try
      underlying.shutdown()
    finally {
      val _ = promise.unsafeTrySuccess(KillSwitchOutcome.Completed)
    }

  override def abort(ex: Throwable): Unit =
    try
      underlying.abort(ex)
    finally
      if (!promise.unsafeTrySuccess(KillSwitchOutcome.Errored(ex)))
        LoggerFactory.getLogger(getClass).error(
          "Kill switch was already shutdown when aborting with error",
          ex
        )

  override def flow[T]: Graph[FlowShape[T, T], BetterKillSwitch] =
    underlying.flow[T].mapMaterializedValue { it =>
      if (it == underlying) BetterKillSwitchEfficient.this
      else new BetterKillSwitchEfficient(it, promise)
    }
}

/** Kill-switches can be shutdown gracefully (i.e., by completing the stream),
  * or via an error.
  *
  * The effect on the downstream is not the same, as `onError` short-circuits
  * the stream, while `shutdown` allows the in-process messages to complete.
  *
  * This type is used to signal to shut down listener whenever the kill-switch
  * is shutdown gracefully or aborted via an error.
  */
enum KillSwitchOutcome {
  case Completed
  case Errored(e: Throwable)

  def toEither: Either[Throwable, Unit] =
    this match {
      case Completed => Right(())
      case Errored(e) => Left(e)
    }
}

Note that when using a SharedKillSwitch (the standard implementation provided by Akka/Pekko), we could already do something like the following, by leveraging Akka/Pekko streams, but it’s definitely heavier, API-wise as well (notice the Materializer requirement):

import cats.effect.syntax.*
import org.apache.pekko.stream.Materializer
import org.apache.pekko.stream.SharedKillSwitch
import org.apache.pekko.stream.scaladsl.Source

def awaitShutdown(
  ref: SharedKillSwitch
)(using Materializer): IO[Either[Throwable, Unit]] = {
  val start = IO {
    val done = Source.never[Unit]
      .via(ref.flow)
      .run()
    (done, IO.unit)
  }
  IO.fromFutureCancelable(start)
    .void.attempt
}

The article Hybrid IO-driven Promise (Scala snippet) first appeared on alexn.org.

New Year, Old Me

Alexandru Nedelcu — Thu, 08 Jan 2026 10:10:32 +0000

Me and my wife on our winter retreat, somewhere in Brașov, Romania.

In 2025, I was happy. Like, really happy. I wasn't miserable before, far from it, but this past year was strange, as I've been having repeated episodes when I felt my heart was bursting with happiness. It's been going on for months now, and it's not subsiding, the feeling and frequency are only intensifying.

The relationship I have with my wife is great. We started dating in 2001, got married in 2007, so that’s almost 19 years of marriage, and 25 years of loving each other, and I love her even more. The relationship I have with my son is great. He’s 15, now a freshman in high school, and I feel a lot of accomplishment from teaching him math, computer science, or my values; plus I’m really proud of him, having bittersweet moments as I witness the man he’s growing into. Work is good. I’m a “Chapter Lead”, which is a fancy way of saying that I’m a cross-team manager while still having individual contributions. I still have a lot of passion for programming. And my winter break is now over, I’ve spent it with family and friends, and I’ve also read 5 fiction books in 2 weeks (yay!).

This is noteworthy for me because I’m usually grumpy and a pessimist. I was never diagnosed with depression, but I went through at least two burnouts; and the pandemic, with the subsequent war at our borders, hit me hard. I’ve always joked that having a family to take care of wouldn’t allow me to reach the bottom. And I do have a tendency for doom-and-gloom, which is why I’m lucky to have married an optimist.

I think that what changed is that I’ve started being more grateful. Getting older, you start realizing some obvious truths, such as the fact that life is actually good as long as you have your health, and your basic needs met, like food and shelter. And having fun, in the modern sense (e.g., travel, parties), is actually optional. Recharging batteries is important, but I’ve come to realize that what actually brings happiness is living with purpose and gratitude, which is hard work.

I’m now afraid it won’t last, nothing lasts, but right now each day for me is a gift, and I’m glad that I came to this realization before falling seriously ill, or some other tragedy that’s waiting around the corner. Being raised as an Orthodox Christian (non-practicing, and I was mostly agnostic in the past), I’ve never learned the true power of prayer — instilling gratitude by thanking God daily for the gifts I’ve received. This is similar to meditation sessions, focusing on breathing, to remind you that you’re still here and alive, but I find such meditation to be more shallow, as feeling alive doesn’t lead me to gratitude, unless I’m faced with serious life problems.

Disconnecting from web slop (like world news) certainly helped. I’ve come to realize that all online social media platforms are toxic, including Mastodon or Bluesky. To tell you the truth, these aren’t any less toxic than 𝕏/Twitter, their only redeeming quality being the ability to avoid the algorithmic feed, and thus curation works slightly better (but only slightly). The Web is dying due to AI slop, bots and negative people, but we can still find communities that are islands, tuned to our values and interests. And being too-online is bad for our mental health.

I maintain a social media presence strictly for learning programming, honing my craft, and for finding out more about interesting hobbies, or books recommendations. You know, the good stuff. I’ve had occasional doom-scrolling episodes, or political posts that I then deleted within 10 minutes. Someone is always wrong or an asshole on the Internet, some tragedy is always happening in other parts of the world, but growing older, I care less and less. I’m still not able to care only about what I can influence, but this Stoic philosophy of life is certainly my ideal.

Back in 2024, I had New Year’s resolutions that had to do with habits. What a waste. New Year’s should be a time for reflection, not for TODO lists. My 2026 New Year’s resolution is to keep doing what I’ve been doing — to be grateful, to strive to be a good person, to live with purpose, to maintain curiosity, to perfect my craft, to strive to be the best husband, father, colleague, and friend that I can be. I don’t have recipes for these, and I’ll fail a lot, but this path is what brings me joy.

The article New Year, Old Me first appeared on alexn.org.

How To Become a DevOps Engineer

Alexandru Nedelcu — Wed, 26 Nov 2025 00:00:00 +0000

The TLDR: Eat your own dogfood by practicing DevOps on your own infrastructure, change mentality to one oriented towards automation, and grow your hard engineering skills, such as learning to do programming, because it’s in the job description.

While working in this industry, at some point it becomes clear that good DevOps engineers may be more rare than good developers. Senior people end up interviewing for engineering roles, and as an interviewer, I started being happy to see people knowing what $? is for.

I may even understand why that happens. Depending on the organization, the job of a DevOps is less about engineering per se, and more about dealing with manual investigations and paperwork. Few devs have the stomach to become DevOps, so the normal pipeline for creating DevOps engineers goes through Support departments. DevOps was coined to highlight that Operations teams need to invest in automation and fast problem detection mechanisms, which requires software development. But in many projects it becomes clear that the “dev” in DevOps may be missing entirely.

Be as it may, if you want to grow as a DevOps engineer, you need to grow your inner “dev”. Knowing how to use your Windows laptop and browser isn’t enough of a hard skill to get by. So here’s what you need to do:

1) Use the Linux command line every day

First, Linux dominates on the server-side. If you’re administering Windows servers, the days of your job are numbered, even if you’re a believer in Microsoft’s tech.

I can’t stress this enough. It’s embarrassing to notice people not being able to do basic tasks, such as:

Navigate the file system, find files, view file contents, edit files.
Handle tar.gz archives.
Finding out if a package is installed and what version.
Finding if a certain process is running.
Getting the logs of a process.
Filter errors from those logs. Read those errors.
Kill that process.
Investigate CPU, RAM or disk usage.
Check that a server is up and running.

The easiest way to do that is to install Linux on your laptop. If you can’t do that on your work laptop, maybe because your organization has rigid ideas about security and support, go for Apple’s macOS, or try WSL, or at the very least, install Linux on your personal laptop.

Then force yourself to use Unix/Linux command line tools, every day. Given the advent of LLM tools, it’s easier than ever to get answers.

If you’re Romanian, check out “Utilizarea sistemelor de operare”. This is a book used as support for a course at University Politehnica of Bucharest, however, it’s also used by 15-year-olds participating in the AcadNet competition. I think that professional DevOps engineers should have more knowledge than high-schoolers participating in local competitions 🧑‍🎓

2) Have your own Virtual Private Server (VPS) and self-host stuff

There are cheap VPS providers around, like Hetzner or Netcup. Get your own VPS. Then work your way to self-hosting your own services like:

Static websites, served by Nginx (easiest)
VPN (WireGuard)
WordPress
Isso commenting widget
Matomo
Linkding
Vaultwarden
FreshRSS
Mastodon
NextCloud

There are many services you might want to host, depending on what you fancy (not email, don’t do that). Providing your family with online services might be good motivation. In the process you may learn how to:

Keep backups, after you lose everything, which will happen at least once, multiple kinds of backups, actually.
Harden your Linux server, after you leave a Redis port open on the Internet, and your server gets infected by malware mining for Bitcoin (true story).
Do automated upgrades and cleanups, because you don’t have the time or the attention span to keep it up to date, or to intervene when that server crashes due to low disk space.
Simplify hosting by using Docker containers, because you want easy installations, and a predictable environment that can be replicated.
Reinstall that Linux server from scratch, every once in a while, by switching VPS providers or taking advantage of lowering prices.
Write portable scripts for automated provisioning of new Linux servers, because remembering stuff just won’t cut it.
Keep those scripts in a Git repository, alongside other configuration or documentation you need; you need the ability to rebuild that server from scratch in a couple of hours, tops.
Find ways for continuous monitoring, to make your server as resilient as possible, because your personal brand or your family depends on it.
Find ways to block AI bots that can consume your bandwidth, or DDoS attacks; or optimize the server to withstand traffic sent due to your links trending on social media.

NOTE: Pain is good! Embrace it, witness it, but don’t normalize it. It tells you when you need to find better solutions. Have the guts to say “this sucks, I can do better!” Pain-driven development, as I like calling it 😄

WARN: You can also do home-automation, which can be a great motivation, but unless a Linux server is involved, a server that you administer, and that’s exposed to danger, it doesn’t count. Home automation stuff can be very user friendly and you won’t learn much.

3) Learn Programming

The job of a DevOps includes “dev”. It’s in the job description. You mustn’t be just a user.

The mentality of a DevOps engineer should be “automate and monitor all the things”, with everything else being just red tape and drudgery getting in the way, also ripe for automation. So of course you need programming.

As programming languages, at the very least, you need Bash and Python. Bash sucks, everyone hates it, but it’s everywhere and works for very simple scripts. You’ll need more than Bash, though.

But don’t stop at just superficial knowledge about syntax. Learn basic algorithms and data structures, because you need to get into the right mindset. If you need motivation, consider that you need the knowledge of a 15-year-old in their freshman year at a high-school with a computer-science curriculum (from Romania, at least, I don’t know what high-schools teach in other countries). And don’t fool yourself about the existence of LLMs (AKA “AI”), because at the very least you need to know what to ask for, or to review the generated code.

The good news is that it’s easier than ever to learn, the Internet being filled with (often free) books or video tutorials, and you can also use ChatGPT for learning (when used in the right way, it can be useful). You can also join like-minded communities and seek help from others. Teach others, too, as becoming a teacher is the best way to go deep.

WARN: you may expect your employer or your government to invest in your training, and that surely would be nice and wise, but it’s not really how the free market works. If you stagnate, or if you find yourself unable to get new jobs, it’s not your past or present employers’ responsibility, it’s not your government’s responsibility, it’s mostly yours. Sorry, I don’t make the rules.

4) Be a Cat owner

You’re not a true DevOps if you don’t have a cat occasionally sitting on your keyboard, wreaking havoc.

Cat sitting on Macbook keyboard (source)

Now go forth, self-host stuff, do programming, have fun, pet your cat, and learn some hard engineering skills 💪

The article How To Become a DevOps Engineer first appeared on alexn.org.

Programming Languages in the Age of AI Agents

Alexandru Nedelcu — Sun, 16 Nov 2025 06:14:15 +0000

In the age of “AI” Agents generating the code, is your programming language choice still relevant? Or will we just converge on using the top 5 programming languages, because that’s what the “AI” has been trained on? You can also ask the same question for any other piece of tech from your stack.

At work, I’ve built several Python scripts doing various boring chores. GitHub’s Copilot, powered by GPT or Claude Sonnet, does a reasonably good job at writing Python scripts. Furthermore, I’ve seen Python scripts developed by rookies that are worse than what these agents can generate. A programming language’s popularity is a feedback loop. These days, because Python is popular, having available a huge training corpus, “AI” agents can be successful at generating Python scripts that work, making Python even more popular.

For programming language aficionados, for people loving their craft, while there is reason to worry that “AI” Agents can kill the long tail, I also think there are reasons to celebrate the dawn of “AI”. Here’s why:

1) Having a compiler with an expressive static type system helps the “AI” Agent converge on a solution with a much shorter feedback loop.

“If it compiles, it runs” was never fully true, but in programming languages like Scala, Haskell, or Rust, our confidence in the code is indeed much higher once the code compiles.

To give an example, Scala 3 has been on the market for a while, but there’s not much public code published that deals with Scala 3’s new macro system, and yet, “AI” Agents are able to generate code that works. This is very visible when using VS Code with GitHub Copilot, because it’s integrated with the LSP server, so you see its mistakes in real-time (Metals being quick to highlight compilation errors); and so the agent goes through several iterations trying to fix whatever compiler error it sees; with the result most often being code that compiles.

This ability to iterate and converge towards a working solution based on external feedback (from a compiler, from unit tests) is what makes “AI” Agents usable. And, having an expressive/powerful static type system provides much faster feedback than other validation types, such as unit tests. When people complain about Scala’s or Rust’s compilers being slow, they forget that they eliminate the need for unit tests guarding against entire classes of errors. And we need all the help we can get to guard against “AI” hallucinations 😉

2) We need the ability to review what the “AI” Agent does. Can you reason about the generated code?

Programmers may describe “what they want” instead of “how” in natural language, however, they still need to ensure that they’re getting what they’ve asked for. Just executing the program, and looking at the output, is a very superficial way to test it, as we should all know by now. So at the very least you want tests, and tests are boring, we want them automated, so you want to see the tests that the agent has generated. Did the agent generate tests for the obvious edge cases? You can’t really say that without looking at the code.

And there’s another looming issue that’s happening with agent-generated code that people aren’t aware of, yet: the comprehension debt. You see, the software project isn’t just about telling the computer how it needs to behave, in order to produce some desired outcome. It’s also about the knowledge you’ve built along the way. And if you have no people left on the team that understand the inner workings of a project, you’re screwed. And “AI” Agents won’t help, because they have a limited context window that tends to vanish. You won’t keep around the dialogs you have with the agent, and even if you do, that kind of documentation is poor and can be misinterpreted; context poisoning is also a thing.

Peter Naur writes in Programming as Theory Building that:

“Although it is essential to upgrade software to prevent aging, changing software can cause a different form of aging. The designer of a piece of software usually had a simple concept in mind when writing the program. If the program is large, understanding that concept allows one to find those sections of the program that must be altered when an update or correction is needed. Understanding that concept also implies understanding the interfaces used within the system and between the system and its environment.”

“Changes made by people who do not understand the original design concept almost always cause the structure of the program to degrade. Under those circumstances, changes will be inconsistent with the original concept; in fact, they will invalidate the original concept. Sometimes the damage is small, but often it is quite severe. After those changes, one must know both the original design rules, and the newly introduced exceptions to the rules, to understand the product. After many such changes, the original designers no longer understand the product. Those who made the changes, never did. In other words, nobody understands the modified product. Software that has been repeatedly modified (maintained) in this way becomes very expensive to update. Changes take longer and are more likely to introduce new “bugs”. Change induced aging is often exacerbated by the fact that the maintainers feel that they do not have time to update the documentation. The documentation becomes increasingly inaccurate thereby making future changes even more difficult.”

…

“At least with certain kinds of large programs, the continued adaptation, modification, and correction of errors in them, is essentially dependent on a certain kind of knowledge possessed by a group of programmers who are closely and continuously connected with them.”

“Programming should be regarded as an activity by which the programmers form or achieve a certain kind of insight, a theory, of the matters at hand. This suggestion is in contrast to what appears to be a more common notion, that programming should be regarded as a production of a program and certain other texts.”

At the same time, it’s still true that the source of truth is the source code itself, always has been, always will be. So how do you preserve knowledge, despite the natural churn and evolution that happens in software projects?

Well, one way of doing it is with great source code that makes design and intent clear. And great source code is like math, ageless, describing what you want, not how, with an architecture that allows for evolution, but at the same time makes clear the design’s invariants, i.e., its so-called laws. Programming in a higher-level language matters for “AI” Agents as well. It matters if the source code fully describes the specifications (e.g., the original intent or design constraints) or not. “AI” Agents may be good at assembly language, but they can’t be that good, given that serializing specs in assembly is a lossy process.

And for reviewing the source code, deductive reasoning will never go out of fashion, because it’s how the human brains work. Therefore, functional programming with its “equational reasoning” is still valuable in the age of “AI” Agents, and arguably, even more so, because you’re dealing with an inconsistent idiot that can’t learn.

The article Programming Languages in the Age of AI Agents first appeared on alexn.org.

Try-catch-finally in Java is Cursed

Alexandru Nedelcu — Mon, 10 Nov 2025 09:54:08 +0000

Java has a usable interruption protocol, and that’s good. However, one problem with it is that it relies on InterruptedException, and it can be caught and ignored. Being a “checked exceptions”, many developers simply ignore it

BlockingQueue queue;
// ...
while (isActive) {
  try {
    var task = queue.take();
    process(task);
  } catch (InterruptedException e) {
    // ignore
  }
}

There are reasons for why you may want to react to a thread interrupt. Maybe you want to close some resources. What’s bad about it is that you can continue as if nothing happened.

But it’s not just the interruption protocol that’s problematic here, but the semantic of try-catch-finally as well (sample attribution):


void infiniteLoop() {
  while (true) {
    try { return; } 
    finally { continue; }
  }
}

Yep, you can ignore return, too. This works as well:

try {
  return 42;
} finally {
  // What actually gets returned
  return 0;
}

Also, exception suppression — in this case, the client won’t get FileNotFoundException, not even as a “suppressed” exception.

try {
  // can throw FileNotFoundException
  return Files.readString(Path.of("missing.txt")); 
} finally {
  throw new RuntimeException("Cleanup failed");
}

If you think this can’t happen, consider the classic resource-usage pattern:

InputStream in = ...;
try {
  process(in);
} finally {
  // can throw IOException, masking any exception 
  // that happened in `process`
  in.close(); 
}

Thankfully, this is improved by try-with-resources, in which case the exception thrown by close() gets added as a “suppressed” exception to the one thrown by the try block.

The article Try-catch-finally in Java is Cursed first appeared on alexn.org.

Scala vs F#

Alexandru Nedelcu — Sat, 01 Nov 2025 06:58:37 +0000

Scala logo on the left, F# logo on the right.

Which language leans more towards functional programming? In this binary choice, people have perceived F# to be that language, due to its ML roots, but I have a different perspective...

Before bringing your pitchforks: I’m an expert in Scala, but I only have superficial experience with F#, so take this opinion for what it is. I also think both languages are great, and this is just an opinion on which one I prefer.

F# is a wonderful language, and the primary choice between Scala and F# would actually be driven by your preferred choice of platform and ecosystem (JVM vs .NET). That being said,

Scala is actually a more “functional” language than F# because it wins in the expressiveness and the FP culture departments.

First, in Scala, you can work with higher-kinded types and type classes. To understand why that’s important, consider that Scala does not need .NET’s flavor of reification for generics because what it has is far more potent. These are features that Don Syme has been explicitly against because, according to him, it complicates the type system. This is partially true, I’m actually inclined to agree; however, in my experience, the actual problems in programming come from elsewhere, code reuse and static type-safety being good. And I’d also argue that this opinion against type-classes and higher-kinds may also be due to bias, with .NET itself making higher-kinds more complicated due to its runtime-based reification (you can do type-erasure on .NET, but people and libraries expect reification, which would mean interop would suffer). The irony of .NET’s original marketing is that the JVM has been an easier-to-target platform for languages that are not Java or C#.

To give an example of a type-class in action:

// Scala 3
def sumList[A](list: List[A])(using Numeric[A]): A =
    val n = summon[Numeric[A]]
    list.foldLeft(n.zero)(n.plus)

I hope I’m not getting this wrong, it’s been a while since I worked with F#, but its standard library does something like this, which isn’t something you actually see in common code:

// F#
let inline sumList (list: ^T list) : ^T 
    when ^T : (static member (+) : ^T * ^T -> ^T) 
    and ^T : (static member Zero : ^T) =
    List.fold (+) LanguagePrimitives.GenericZero list

To note:

Scala’s version is a simple function making use of implicit parameters, Numeric being a type-class;
F#’s version is an inline function (due to adding restrictions not natively supported by .NET), and the restrictions refer to “static members” — note that Scala no longer has the notion of “static methods” in the language (although the JVM does).

You can express this in the latest C#, actually, as they’ve added “abstract static methods” (link), which are frowned upon in F#, the general advice for F# devs being to steer clear of them, unless they need it for interop. So, C# almost has type classes.

// C#
interface INumeric where T : INumeric
{
    static abstract T Zero { get; }
    static abstract T Plus(T a, T b);
}

static T SumList(List list) where T : INumeric
{
    return list.Aggregate(T.Zero, (acc, x) => T.Plus(acc, x));
}

But, let’s go one more level — we can work on any list-like type, that we can express as a type parameter, and note that working with a generic type restricted to implementing Foldable is different in nature to using Iterable / Enumerator (OOP subtyping):

// Scala 3
import cats.Foldable
import cats.syntax.all.given

def sumAll[F[_], A](list: F[A])(using Foldable[F], Numeric[A]): A =
    val n = summon[Numeric[A]]
    list.foldLeft(n.zero)(n.plus)

// Now it works for any sequence
sumAll(Vector(1,2,3))
sumAll(List(1,2,3))
sumAll(Array(1,2,3))

The true power of higher-kinds manifests when we have to keep that list type in the returned value, something that OOP subtyping (or .NET’s generics reification) can’t achieve:

// Scala 3
import cats.{Foldable, MonoidK}

def flatten[F[_], A](list: F[F[A]])(using Foldable[F], MonoidK[F]): F[A] =
    val m = summon[MonoidK[F]]
    list.foldLeft(m.empty[A])(m.combineK)

// Note how the type gets preserved
val l1: List[Int] = 
    flatten(List(List(1,2,3), List(4,5,6)))
val l2: Array[Int] =
    flatten(Array(Array(1,2,3), Array(4,5,6)))

And there’s a lot more to talk about here, such as the ability to auto-derive type-class instances, which in Scala is great. For instance, in Scala, you can develop something like Kotlin Serialization as a library, a library that can work for basic needs and that’s very type-safe, in a day. You don’t even need macros for it, unless you want customizability (e.g., by annotations).

This difference is not only academic, manifesting itself for example in F#’s AsyncSeq, which is a library combining Async with Seq. This is nice and all, but in Scala, all the features in this AsyncSeq are expressed more generically in the Typelevel Cats library, increasing reuse for other types. To make a comparison, it’s like in Go, when people needing generics were simply duplicating the code for all the types they care about. This arguably works for some common cases, but also sucks.

F# does have Hindley-Milner type inference, however, it only works as long as OOP subtyping isn’t involved, and due to its .NET interop and dependence, it has a lot of OOP subtyping. Hindley-Milner in general, being a type system with global type inference, makes error messages hard to read and understand, although, granted, this is more relevant in languages with more advanced type systems. Even in languages with HM (e.g., Haskell), the general advice is for public functions to have explicit types; otherwise the contract exposed is fragile. And also, the more advanced the type system, the more difficult HM becomes. For example, the more advanced features of Haskell you use, the more HM breaks down. See also Idris, in which type inference is undecidable in general.

I always felt that this is a general problem with F#, inspired by two separate worlds, and not trying hard enough to combine them. For example, its generic types can have restrictions that can’t be expressed in .NET (like in the sumList above). Therefore, to use them, you can only use inline functions, which are functions that are not seen by the .NET runtime, so they can’t be passed around as values. In fairness, Scala 3 makes heavy use of inline functions as well, for compile-time magic and macros, but in Scala you can use inline functions to reify types and then expose any compile-time information to normal functions, via implicit parameters. So it doesn’t have the mixture I’ve felt while using F#.

F# has a sort of mixed personality to it, not trying too much to mix the functional aspects with OOP, like Scala or OCaml are trying. This can be considered a feature, except, one interacts with .NET and OOP a lot in F#, more than in your typical Scala FP project (IMO, YMMV). To give one example, when I last tried, in F# serialization was still done using runtime reflection, whereas in Scala we have libraries like Circe which work entirely at compile-time, providing static type-safety, and exposing a very FP API. If you compare .NET’s solutions for serialization with Java’s Jackson, for sure, .NET wins due to generics reification, but if you compare .NET with Scala’s solutions, or even with Kotlin’s Serialization, it leaves something to be desired (in the type-safety department, at least).

Back to the FP aspects, Scala has a very healthy ecosystem of FP libraries. FP has been so successful in Scala that it ended up with two mature and competing ecosystems for it, i.e. Typelevel and ZIO. Scala has one of the best books on FP around: FP in Scala. And it has also inspired other ecosystems, see Effect.

Note that there are aspects of F# that I like. Off the top of my head, I can think of:

Type providers — you could build something similar in Scala in a library, actually, however it’s challenging, and I haven’t seen something polished yet;
Computation expressions — Scala has “for comprehensions”, but F#’s computation expressions are more evolved; this is for working with “monadic” or “applicative” types, but note that Scala also has plans to go the way of Kotlin and provide direct style support (thus providing an alternative to monads because it was too boring before 😜);
LINQ support — Scala has had libraries that are equivalent, such as Quill, and while it’s nice that in Scala we can bring the full power of LINQ as a library, I also feel that proper support needs to be baked into the language, or it’s forever plagued with issues;
Active patterns, although Scala’s pattern matching is quite nice, too.

F# also has features that Scala can do better, but that are nice to have regardless, and worth mentioning, such as “units of measure” or “code quotations”.

F# resembles Python, in the sense that its designers have introduced features to solve concrete use-cases. But, just like in Python, it has features that don’t feel orthogonal or very generic; in Python, for instance, I can think of several features that were added to avoid adding multi-line lambdas. So much for the “one way of doing things”.

But overall, F# is a language I would enjoy very much, if I wanted to target .NET. And .NET can be … an acquired taste.

I’ve always preferred the Java ecosystem because it has always been closer in spirit and culture to Linux and Open-Source. This has pros and cons. .NET feels more like a cathedral and Java feels more like a bazaar. In terms of ecosystem, the difference is night and day. For instance, Oracle has a lot of power, as they still develop and own Java, but they don’t provide the IDE or the libraries or the tooling most people use. They don’t even have the most popular OpenJDK distribution. And if they stop being good stewards, OpenJDK can get forked, as they aren’t the only contributors to it. I like having competition, choices, communities (plural), instead of having Microsoft impose its solution du jour.

The JVM also has had a great evolution, in some ways leapfrogging .NET (e.g., runtime optimizations, the “pauseless” GC implementations Project Loom, GraalVM), and in some ways closing the gaps (upcoming Valhalla, which finally brings value types and maybe generics specialization). There’s still plenty of healthy competition going on. .NET going for ‘abstract static methods’, being close to type-classes, is forcing Java to push for type-classes support. Fun times.

People do have good reasons to love the .NET ecosystem as well, since it has evolved a lot, too. It’s now open-source, multi-platform, performant, and it’s still the preferred high-level platform for games — which is why I may teach my son some C# or F# 😁

The article Scala vs F# first appeared on alexn.org.

Scala 3 / HOCON Parsing

Alexandru Nedelcu — Thu, 30 Oct 2025 17:33:31 +0000

How to parse HOCON in Scala 3? How to use Circe for HOCON? How to work with multiple JSON codecs per data structure?

This article is using Scala 3 with -no-indent. And this file is executable with Scala CLI’s Markdown support 🤖

scala --power --enable-markdown \
  https://raw.githubusercontent.com/alexandru/alexn.org/refs/heads/main/_posts/2025-10-30-scala-3-hocon-parsing.md

I’ve just started migrating our project at work to Scala 3. It’s a work-in-progress, but it’s been going well, and thus far the biggest roadblock has been the PureConfig library, an awesome library, but it didn’t seem to have complete support for Scala 3. In particular, it doesn’t seem to have support for ConfigWriter derivation, yet. So I’ve started to think of alternatives.

UPDATE: PureConfig apparently has ConfigWriter derivation in the codebase, as part of a different module, found out about it from this Reddit comment: see source code, added in this PR. I don’t know how well it works, but it’s encouraging. Unfortunately, I checked the status by just looking at outdated docs 🤦‍♂️

One solution I’ve found is to use the circe library, with circe-config. And here-in lie challenges:

Rewriting the configuration files is not an option:
- PureConfig used kebab-case for its keys;
- It also allowed us to use type = type-name for discriminating union types.
- Any solution we pick needs to be configured for these conventions.
Deriving the type-class instances should be painless — people shouldn’t need to remember an import.
We also do JSON encodings with camelCase via Circe, sometimes for types that are also used in configuration files, therefore those types need different HOCON config vs JSON encodings.

So in summary:

Circe needs to be configured to parse and generate HOCON;
The type-class instances need to be different, but still globally visible — to avoid conflicts and preserve “type-class coherence”.

Starting with some dependencies:

//> using scala "3.3.7"
//> using dep "io.circe::circe-core:0.14.15"
//> using dep "io.circe::circe-config:0.10.2"
//> using options -no-indent

In order to respect our HOCON’s conventions, we needed a Circe Configuration:

import io.circe.derivation.Configuration

given hoconConventions: Configuration = {
  val kebabCase: String => String =
    _.replaceAll("([a-z])([A-Z])", "$1-$2").toLowerCase

  Configuration.default
    .withTransformMemberNames(kebabCase)
    .withDiscriminator("type")
    .withTransformConstructorNames(kebabCase)
}

Note that with Circe, the auto-derivation we want goes something like this:

case class HttpConfig(
  hostname: String,
  port: Int,
  contextPath: Option[String]
)

object HttpConfig {
  given Codec[HttpConfig] = 
    Codec.AsObject.derivedConfigured
}

As mentioned before, I’d rather have a different type-class, and I don’t want to have to remember to import hoconConventions, but we can just create different type-classes that wrap Circe’s:

import io.circe.{Encoder, Decoder, Codec}

trait ConfigEncoder[A] {
  def encoder: Encoder.AsObject[A]
}

trait ConfigDecoder[A] {
  def decoder: Decoder[A]
}

trait ConfigCodec[A]
  extends ConfigEncoder[A]
  with ConfigDecoder[A] {

  def codec: Codec.AsObject[A]
}

The difficult part is auto-deriving these, but thankfully, Scala 3 makes it easy:

import scala.deriving.Mirror

object ConfigEncoder {
  // Calling `derivedConfigured` ourselves, then wrapping it
  //
  inline def derived[A](using inline A: Mirror.Of[A]): ConfigEncoder[A] = {
    // IMPORTANT — Scala needs to know how to derive
    // a Circe `Encoder` from a `ConfigEncoder`.
    import Givens.given
    
    val underlying = io.circe.Encoder.AsObject.derivedConfigured[A]
    Derived(underlying)
  }

  final private case class Derived[A](encoder: Encoder.AsObject[A])
    extends ConfigEncoder[A]
}

Importantly, Scala must find the defined encoders for a class’s fields, therefore it must have a way to convert from ConfigEncoder into io.circe.Encoder and from ConfigDecoder to io.circe.Decoder. But these need to be imported only in the lexical scope of those inline def derived functions:

private object Givens {
  given circeEncoder[A](using encoder: ConfigEncoder[A]): Encoder.AsObject[A] =
    encoder.encoder

  given circeDecoder[A](using decoder: ConfigDecoder[A]): Decoder[A] =
    decoder.decoder
}

Similarly, we describe the derivation logic for ConfigDecoder and ConfigCodec:

object ConfigDecoder {
  inline def derived[A](using inline A: Mirror.Of[A]): ConfigDecoder[A] = {
    // IMPORTANT — Scala needs to know how to derive
    // a Circe `Encoder` from a `ConfigEncoder`.
    import Givens.given
    val underlying = Decoder.derivedConfigured[A]
    Derived(underlying)
  }

  final private case class Derived[A](decoder: Decoder[A])
    extends ConfigDecoder[A]
}

object ConfigCodec {
  inline def derived[A](using inline A: Mirror.Of[A]): ConfigCodec[A] = {
    // IMPORTANT — Scala needs to know how to derive
    // a Circe `Encoder` from a `ConfigEncoder`.
    import Givens.given
    val underlying = Codec.AsObject.derivedConfigured[A]
    Derived(underlying)
  }

  final private case class Derived[A](codec: Codec.AsObject[A]) 
    extends ConfigCodec[A] {
    override def decoder = codec
    override def encoder = codec
  }
}

OK, so I think we can do better than this with some utilities. This is a copy/paste-able snippet:

import com.typesafe.config.Config
import com.typesafe.config.ConfigFactory
import com.typesafe.config.ConfigRenderOptions
import com.typesafe.config.ConfigValue
import com.typesafe.config.ConfigValueFactory
import io.circe.derivation.Configuration
import io.circe.syntax.given
import io.circe.Codec
import io.circe.Decoder
import io.circe.Encoder
import io.circe.Json
import scala.deriving.Mirror
import scala.jdk.CollectionConverters.*

trait ConfigEncoder[A] {
  def encoder: io.circe.Encoder.AsObject[A]

  extension (a: A) {
    def toRawConfig: Config = {
      given Encoder.AsObject[A] =
        encoder
      ConfigFactory.empty().withFallback(
        jsonToConfigValue(a.asJson)
      )
    }

    def renderConfigString: String =
      toRawConfig.root().render(renderOptions)
  }
}

object ConfigEncoder {
  inline def derived[A](using inline A: Mirror.Of[A]): ConfigEncoder[A] = {
    import Givens.given
    val underlying = io.circe.Encoder.AsObject.derivedConfigured[A]
    Derived(underlying)
  }

  final private case class Derived[A](encoder: Encoder.AsObject[A])
    extends ConfigEncoder[A]
}

trait ConfigDecoder[A] {
  def decoder: io.circe.Decoder[A]

  final def decodeConfig(
    config: com.typesafe.config.Config
  ): Either[io.circe.Error, A] = {
    import io.circe.config.parser
    parser.decode[A](config)(using decoder)
  }

  final def parseConfigString(
    configString: String
  ): Either[io.circe.Error, A] = {
    val config = ConfigFactory.parseString(configString)
    decodeConfig(config)
  }
}

object ConfigDecoder {
  def apply[A](using ConfigDecoder[A]): ConfigDecoder[A] =
    summon[ConfigDecoder[A]]

  inline def derived[A](using inline A: Mirror.Of[A]): ConfigDecoder[A] = {
    import Givens.given
    val underlying = Decoder.derivedConfigured[A]
    Derived(underlying)
  }

  final private case class Derived[A](decoder: Decoder[A])
    extends ConfigDecoder[A]
}

trait ConfigCodec[A]
  extends ConfigEncoder[A]
  with ConfigDecoder[A] {

  def codec: Codec.AsObject[A]
}

object ConfigCodec {
  inline def derived[A](using inline A: Mirror.Of[A]): ConfigCodec[A] = {
    import Givens.given
    val underlying = Codec.AsObject.derivedConfigured[A]
    Derived(underlying)
  }

  final private case class Derived[A](codec: Codec.AsObject[A]) extends ConfigCodec[A] {
    override def decoder =
      codec
    override def encoder =
      codec
  }
}

private object Givens {
  given circeEncoder[A](using encoder: ConfigEncoder[A]): Encoder.AsObject[A] =
    encoder.encoder

  given circeDecoder[A](using decoder: ConfigDecoder[A]): Decoder[A] =
    decoder.decoder

  given hoconConventions: Configuration =
    Configuration.default
      .withTransformMemberNames(kebabCase)
      .withDiscriminator("type")
      .withTransformConstructorNames(kebabCase)

  private val kebabCase: String => String =
    _.replaceAll("([a-z])([A-Z])", "$1-$2").toLowerCase
}

private def jsonToConfigValue(json: Json): ConfigValue =
  json.fold(
    ConfigValueFactory.fromAnyRef(null),
    boolean => ConfigValueFactory.fromAnyRef(boolean),
    number =>
      number.toLong match {
        case Some(long) => ConfigValueFactory.fromAnyRef(long)
        case None => ConfigValueFactory.fromAnyRef(number.toDouble)
      },
    str => ConfigValueFactory.fromAnyRef(str),
    arr => ConfigValueFactory.fromIterable(arr.map(jsonToConfigValue).asJava),
    obj => ConfigValueFactory.fromMap(obj.toMap.view.mapValues(jsonToConfigValue).toMap.asJava)
  )

private val renderOptions =
  ConfigRenderOptions
    .defaults()
    .setOriginComments(false)
    .setComments(true)
    .setFormatted(true)
    .setJson(false)

To test it, we first define a more complex data structure:

// Providing both HOCON and JSON codecs, separately 
// (to prove it works)

final case class AppConfig(
  appName: String,
  http: HttpConfig,
  inputConfig: InputConfig
) derives ConfigCodec, Codec.AsObject

final case class HttpConfig(
  hostname: String,
  port: Int,
  contextPath: Option[String]
) derives ConfigCodec, Codec.AsObject

sealed trait InputConfig derives ConfigCodec, Codec.AsObject

object InputConfig {
  final case class Kafka(
    bootstrapServers: String,
    topic: String,
    groupId: String
  ) extends InputConfig derives ConfigCodec, Codec.AsObject

  final case class IbmMq(
    queueManager: String,
    channel: String,
    connectionName: String,
    queueName: String
  ) extends InputConfig derives ConfigCodec, Codec.AsObject
}

And then we can test it by decoding and the re-encoding, for both HOCON and JSON, to prove a point:

import cats.syntax.all.*

val appConfigHocon = """
app-name = "my-app"

http {
  hostname = "localhost"
  port = 8081
  context-path = "/mq"
}

input-config {
  type = ibm-mq
  queue-manager = "QM1"
  channel = "CHANNEL1"
  connection-name = "localhost(1414)"
  queue-name = "VESPER.QUEUE"
}
"""

val appConfigFromHocon = ConfigDecoder[AppConfig]
  .parseConfigString(appConfigHocon)
  .valueOr(throw _)
// encode it again and print it
println("FROM HOCON:\n---------")
println(appConfigFromHocon.renderConfigString)

// Proving that we have different codecs for JSON 
// (with different conventions):
val appConfigJson = """
{
  "appName" : "vesper-app-mq",
  "http" : {
    "hostname" : "localhost",
    "port" : 8081,
    "contextPath" : "/mq"
  },
  "inputConfig" : {
    "IbmMq" : {
      "queueManager" : "QM1",
      "channel" : "CHANNEL1",
      "connectionName" : "localhost(1414)",
      "queueName" : "VESPER.QUEUE"
    }
  }
}
"""

import io.circe.parser.decode
import io.circe.syntax.*

val appConfigFromJson = 
  decode[AppConfig](appConfigJson).valueOr(throw _)

println("FROM JSON:\n---------")
println(appConfigFromJson.asJson.spaces2)

NOTE: you just learned to provide multiple JSON encodings for THE SAME data-structures. Pretty cool, huh? 😉

The article Scala 3 / HOCON Parsing first appeared on alexn.org.

'AI' Sucks the Joy Out of Programming

Alexandru Nedelcu — Mon, 27 Oct 2025 00:00:00 +0000

Talking with my dogs brings me greater pleasure, and I think it helps me get more work done.

I now think that AI/LLMs suck the joy out of programming. I’ve used spicy auto-complete, as well as agents running in my IDE, in my CLI, or on GitHub’s server-side. I’ve been experimenting enough with LLM/AI-driven programming to have an opinion on it. And it kind of sucks.

I started programming when I was around 15 years old, and I’ve had about 28 years of pleasure from doing it. What I like the most is when things finally end up working, in the way I want them to work, and that happens after gradually building an understanding of the problem space, and of the algorithms used. It’s not just the destination, but the journey for getting there. There is nothing more gratifying than mastering some algorithm or technique, or finally understanding the problem that I need to solve.

What I don’t like about programming is when things fail, and they fail for stupid reasons, such as a small detail I missed, and it nearly always happens either because I’m not paying attention, or because my understanding is incomplete, and I just copied some solution from somewhere, and it was a bad solution to begin with. For example, concurrency or performance problems are the worst, especially because they tend to be non-deterministic, and the documentation isn’t easily accessible, so you put together bits and pieces from the Internet. And I hate such parts because I end up with a trial-and-error workflow where I desperately try things out, in what can only be described as “shooting bullets in the dark”.

When I tell an LLM-driven Agent to do something, it may take care of the easy parts. Sometimes, those parts are boring, such as interacting with some 3rd party and proprietary API, but they are easy nonetheless. And for the hard parts? It fails, and it fails hard. The first impulse is to tell the LLM that it made mistakes, giving it high-level details about what to do next, in a very slow and error-prone feedback loop, and then I watch the LLM fail again. And because it got the easy parts right, I feel compelled to give it another chance, again telling it to fix its crap, and then I watch it fail again, and it fails harder this time.

Being very frustrated, already, I checkout the code from the repository, giving the code a shot myself, because at least I know how to investigate and come up with a reasonable solution. But then I discover that the code is … hard to maintain crap. And the problem is that it doesn’t look like crap during the initial PR(s), but crap builds up (with crappy comments too), because I’m too afraid of telling it to make the code more maintainable, due to the high probability of mistakes.

Even when things end up working, I’m left with the feeling that I could’ve done a better job, and this feeling permeates everything but insignificant scripts.

With an LLM-driven workflow, I’m getting all the bad parts about programming, like the stress generated by not being in control when things aren’t working, desperately trying to shoot bullets in the dark and fix unmaintainable code, and without any of the gratification, which comes from the journey itself. We’re replacing that journey and all the learning, with a dialogue with an inconsistent idiot.

The article 'AI' Sucks the Joy Out of Programming first appeared on alexn.org.

Scala 3 / No Indent

Alexandru Nedelcu — Sun, 26 Oct 2025 08:35:53 +0000

Scala 3 came with significant indentation (🏛️). And I still dislike it, despite trying hard. So I’m switching back to braces.

Oh, no 😱 not this kind of rant again! Sorry, but it must be said about once per year 💪 — the rants will continue until morale improves!

Here are some reasons:

It can lead to compilation errors that have weird explanations (e.g., types don’t match). My number one concern with significant indentation is ambiguity. CoffeeScript suffered from ambiguity (thank goodness it’s dead), YAML suffers from ambiguity, and Python doesn’t allow multi-line expressions (without a \\) or multi-line lambdas to avoid ambiguity.
Copy/paste doesn’t work well, because when inserting a newline, editors have the tendency to keep the indentation of the previous line, so in most cases, when pasting, you’re at the wrong indentation level, and depending on the editor’s behavior, the first line of the pasted block might shift relative to the rest of the block. Scalafmt won’t help as it chokes on invalid syntax. My text editing skills are relatively good, and I still find myself unable to efficiently copy/paste blocks of code with significant indentation. Note that with braces, pasting at the wrong indentation level simply doesn’t matter and Scalafmt can do its job.
Editors now have to be smarter about jumping to the beginning and end of a block or expression. E.g., these classic Vim shortcuts no longer work for navigation:
- [{ and [( for jumping to the opening brace or paren;
- ]} and ]) for jumping to the closing brace or paren;
- % for jumping to matching brace or paren.
You could avoid some braces before, e.g., by being more expression-oriented. Braces are a visual aid for blocks using statements, and for example, you could easily spot functions described by single expressions (with a high likelihood that they are pure), versus functions using statements.
Braces are a visual aid for lexical scope — they make it easier to explain what scopes or blocks of code are to beginners. I’ve heard the argument that students have an easier time with significant indentation, but that’s just NOT true in my experience, quite the opposite. You can feel the tension due to the existence of (optional) end markers, as if non-English students needed yet another English keyword.
Whitespace changes make diffs (in PR reviews) noisy. With braces, we do have formatting changes, but we can ignore it when insignificant.
Like it or not, braces are a de facto standard due to the popularity of the C/C++ family. Even relatively newer languages, like Rust, use braces. Folks, sorry, but Pascal is dead, ML-languages are still niche, despite first appearing in the 1970s, and everyone hates CoffeeScript.
Scala 2.x codebases will be with us for a very long time, which is why we’ll have to live with both old-style and new-style syntax, forever. As with Perl’s TMTOWTDI, having the ability to use the Klingon language is cool, but the risk/reward ratio isn’t great.

No-Indent 💪

Take a stand and reject significant indentation. Add this to your build.sbt file:

ThisBuild / scalacOptions ++= Seq(
  "-rewrite",
  "-no-indent",
)

And if you’re using Scalafmt, an OK configuration for me in .scalafmt.conf is this:

runner.dialect = scala3

# Even with old syntax, I dislike redundant braces, although there
# might be a case for adding them to aid clarity for big expressions
rewrite.rules = [RedundantBraces]
rewrite.redundantBraces.maxBreaks = 10

rewrite.scala3.convertToNewSyntax = true
rewrite.scala3.removeOptionalBraces = false

UPDATE — As an alternative, Jakub Kozłowski suggests disabling “significant indentation” in Scalafmt, unsure if this is better or worse, but note that the dialect override may prevent Scalafmt from recognizing code using significant indentation as being valid Scala code:

# Alternative to the above config
runner.dialect = scala3
runner.dialectOverride.allowSignificantIndentation = false
# allows `if x then y`
runner.dialectOverride.allowQuietSyntax = true

Note: Scalafmt doesn’t appear to have the ability to rewrite the code from significant indentation to old-syntax, but it can rewrite from old-syntax to significant indentation (which is why we have to rely on Scalac’s -no-indent -rewrite). This suggests Scalafmt has more information when braces are available, and so do we 😉

The article Scala 3 / No Indent first appeared on alexn.org.

Scala 3 / Match Types

Alexandru Nedelcu — Sat, 25 Oct 2025 00:00:00 +0000

Scala has a neat feature called match types. Let’s play…

This article is using Scala 3 syntax and features using -no-indent.

Here’s an example:

type Head[T] = T match {
  case Array[a]        => Option[a]
  case List[a]         => Option[a]
  case String          => Option[Char]
  case NonEmptyList[a] => a
  case Map[k, v]       => Option[(k, v)]
  case k *: v          => k
  case _               => Nothing
}

def head[T](t: T): Head[T] = t match {
  case ref: Array[a] =>
    ref.headOption
  case ref: List[a] =>
    ref.headOption
  case ref: String =>
    ref.headOption
  case ref: NonEmptyList[a] =>
    ref.head
  case ref: Map[k, v] =>
    ref.headOption
  case ref: (k *: v) =>
    ref.head.asInstanceOf[Head[k *: v]]
  case _ =>
    throw IllegalArgumentException()
}

// Using it works as expected, with the compiler
// correctly inferring the types in each case:
val v1: Option[Int] =
  head(Array(1, 2, 3))
val v2: Option[String] =
  head(List("a", "b", "c"))
val v3: Option[Char] =
  head("hello")
val v4: Int =
  head(NonEmptyList.of(10, 20, 30))
val v5: Option[(String, Int)] =
  head(Map("a" -> 1, "b" -> 2))
val v6: String =
  head("hello" *: 42 *: true *: EmptyTuple)
val v7: Nothing =
  head(())
val v8: Nothing =
  head(123)

The above seems to achieve dependent typing.

Scala isn’t alone in having this ability, here’s a TypeScript equivalent using its conditional types (archive), although note that Scala’s match types are more powerful (and better looking):

// TypeScript code

type Head =
  // Warn: tuples are arrays at runtime
  T extends [infer K, ...infer _] ? K :
  T extends Array ? A | undefined :
  T extends string ? string | undefined :
  T extends Map ? [K, V] | undefined :
  never;

function head(t: T): Head {
  if (Array.isArray(t)) {
    // covers arrays and tuples at runtime
    return (t[0] ?? undefined) as Head;
  }
  if (typeof t === "string") {
    return (t[0] ?? undefined) as Head;
  }
  if (t instanceof Map) {
    const it = t.entries().next();
    return (it.done ? undefined : it.value) as Head;
  }
  throw new Error("Unsupported type");
}

// Examples:
const v1: number | undefined =
    head([1, 2, 3]);
const v2: string | undefined =
    head(["a", "b", "c"]);
const v3: string | undefined =
    head("hello");
const v4: [string, number] | undefined =
    head(new Map([["a", 1], ["b", 2]]));
const v5: string = // tuple
    head(["hello", 42, true] as [string, number, boolean]);

Recursivity

Scala’s match types can also have recursive definitions, so for example:

type AtomOf[T] = T match {
  case Iterable[a] => AtomOf[a]
  case t => t
}

// These are valid
val x: AtomOf[List[List[String]]] = 
  "atom"
val y: AtomOf[Int] = 
  42
val z: AtomOf[List[Set[Map[String, Int]]]] = 
  ("key", 1)

This makes inferring types related to tuples easier, e.g., straight from that documentation page, we can see that tuple concatenation can be expressed:

type Concat[Xs <: Tuple, Ys <: Tuple] = 
  Xs match {
    case EmptyTuple => Ys
    case h *: t     => h *: Concat[t, Ys]
  }

If you want to waste time on things that bring you joy, you can now do type-level arithmetic (already expressed in scala.compiletime.ops.int.+), taking inspiration from Peano’s axioms:

Which can be expressed almost literally:

import scala.compiletime.ops.int.S as Succ

type Sum[A <: Int, B <: Int] <: Int =
  A match {
    case 0        => B
    case Succ[a0] => Sum[a0, Succ[B]]
  }

val x: Sum[3, 4] = 7 // works
val y: Sum[3, 4] = 8 // fails with a compile-time error

As a note, one would think that Succ could be defined like this:

type Succ[A <: Int] <: Int =
  A match {
    case 0 => 1
    case 1 => 2
    case 2 => 3
    //...
    case 2147483646 => 2147483647
  }

But I couldn’t make it work, maybe some kind soul from the Internet could explain why. Thankfully, it’s already defined by the Scala standard library at scala.compiletime.ops.int.S, and this one works. And note that the Scala library defines types in terms of S, with gems such as this one:

// Standard library
package scala

object Tuple {
  /** Literal constant Int size of a tuple */
  type Size[X <: Tuple] <: Int = 
    X match {
      case EmptyTuple => 0
      case x *: xs => S[Size[xs]]
    }
}

//...
val x: Tuple.Size[(Int, String, Double)] = 3

You can just smell the Turing completeness 😁

Caveat

Match types without exhaustive matches don’t really work. And the behavior changed somewhat in later Scala versions. For example, the following would trigger a compile time error in Scala 3.3.7 (LTS), but not in the latest 3.7.3:

type Head[T] = T match {
  case Array[a] => Option[a]
  case List[a]  => Option[a]
}

def head[T](t: T): Head[T] = 
  t match {
    case ref: Array[a] =>
      ref.headOption
    case ref: List[a] =>
      ref.headOption
  }

// ... these calls would trigger compile-time errors in 3.3.7,
// but not in later versions...
head(Iterator(1))
head("a string")

The issue here is that Head[String], in the above definition, is not allowed to exist in Scala 3.3.7 (LTS), but is allowed to exist in later versions. For me, this was surprising. But note the LTS is not free of surprises, as this sample compiles just fine on Scala 3.3.7 (LTS), when my intuition suggests it should not:

class Foo[T](val value: T)

type Unpacked[T] = 
  T match {
    case Foo[a] => a
  }

def unpack[T](t: T): Unpacked[T] = 
  t match {
    case ref: Foo[a] => ref.value
  }

// No compile-time errors whatsoever, in either versions,
// just runtime  exceptions:
unpack("a string")
unpack(List(1,2,3))

The kicker here is that you can make the above fail at compile-time, on Scala 3.3.7 LTS, if you add the final keyword to that Foo.

final class Foo[T](val value: T)

type Unpacked[T] = T match {
  case Foo[a] => A
}

// Compile-time error in Scala 3.3.7
// Compilation passes in Scala 3.7.3
unpack("a string")
unpack(List(1,2,3))

My shoddy rationalization goes like this: If Foo is an open class (a class that can be extended or an interface/trait), it means that String could eventually inherit from it in later Java/Scala versions. That’s highly unlikely, but the compiler probably has no way of knowing that. But it’s a poor rationalization, because TBH, I don’t see a problem with upgrades to String making Head[String] exist. It would be worse if it happened the other way around, IMO. This may be one of those cases in which the compiler team couldn’t make it work well, so decided to just shrug. So we’ll have to live with it.

The solution to the above is to just allow it to exist, by having a case _ => Nothing branch:

type Head[T] = T match {
  case Array[a] => Option[a]
  case List[a]  => Option[a]
  case _        => Nothing
}

// Type allowed to exist, but it reduces to Nothing
def x: Head[String] = throw IllegalArgumentException("Boo")
val y: Nothing = x

And keep in mind that this match type is a return type that’s equivalent to a “union type” function parameter, which is how you can protect functions:

type Head[T] = T match {
  case Array[a] => Option[a]
  case List[a]  => Option[a]
  case _        => Nothing
}

// Protecting the function call by using an 
// "untagged union type" as the "upper bound" 
// of our type parameter.
def head[T <: Array[?] | List[?]](t: T): Head[T] = 
  t match {
    case ref: Array[a] =>
      ref.headOption
    case ref: List[a] =>
      ref.headOption
    case _ =>
      // Note we don't really need this branch, but 
      // the compiler can't see it, and it also gives 
      // us a really scary compiler error, if we try 
      // removing it.
      throw IllegalArgumentException("Boo")
  }

The article Scala 3 / Match Types first appeared on alexn.org.

Update on Math Formulas via Copilot

Alexandru Nedelcu — Tue, 21 Oct 2025 00:00:00 +0000

Or how I'm using GitHub's Copilot in Agent mode for yak shaving, evolving my Jekyll-powered website for rendering mathematical formulas by SVG images, instead of heavy JavaScript.

This blog uses Mathjax for rendering math formulas, see my posts on Math. But Mathjax is a JavaScript library, and it was running client-side, in the browser. And I tend to avoid JavaScript for this particular website.

Given that I intend to write more math formulas, it bothered me that the dependence on JavaScript is hurting this blog’s RSS feed, as RSS feed items can’t have JS in them. Also, the page rendering is slow and dirty, hurting the user experience due to that brief moment between the static HTML being loaded and MathJax doing its job, during which one can see the raw TeX syntax, and the page layout shifts to accommodate the JS-powered rendering.

I’ve now switched the implementation to one that uses Mathjax on the server-side, when the website is built, translating those formulas to SVG images. Here’s how:

PR, started by Copilot
Commit fixing RSS and dark-mode, because life can’t be simple.

Here’s a sample. You can inspect it or “view source” for confirmation:

As a side-note, this is one of those times when I appreciate having full control over the implementation of my blog, instead of using WordPress or other, more evolved off-the-shelf solutions (I’m thinking here of my previous article on outsourced voices).

I used GitHub’s Copilot, running as an Agent on GitHub, to start the PR and take care of the most difficult parts. It worked, but it needed supervision. Therefore, this is also a tale of how I used Copilot to modify my blog’s implementation.

For instructing Copilot, I started with this prompt:

This website currently uses Mathjax for rendering math formulas. However, I’d like those math formulas to be more portable, i.e., to not require JavaScript for rendering.

Modify the website such that math formulas get rendered to SVG images, using the original TeX formula as “alt” text for the visually impaired.

Suggestions:

Use mathjax-node for the rendering to SVG files (placed in /assets/math at build time).

Use the formula’s hashcode as the SVG file name.

Reuse already generated files for repeated formulas.

Introduce a Jekyll plugin that searches and replaces math/TeX code to image tags pointing to these SVG files, also generating the files when needed.

Upon inspecting the result, which wasn’t bad, I then followed up with a change request:

Unfortunately, I now saw that mathjax-node is an old package.

Drop it from the project and use the latest stable version of Mathjax and whatever it is needed to make it work.

After reviewing the generated code, I noticed Copilot didn’t do the right thing. It broke the command meant for development, jekyll serve. It certainly helped that I know how to do the job well:

There is a problem with this approach. These SVG files shouldn’t be generated directly in the project’s sources, for one, because I don’t want them to be added to the repository, but also, because it breaks jekyll serve.

These files should be generated using Jekyll’s mechanism, by appending to site.static_files a Jekyll::StaticFile instance. There is already an example in managed_js.rb.

The result of this should be that /assets/math should be available only in the _site directory after a jekyll build and the images should be served via jekyll serve as well (so without an explicit _site directory being built).

Also, remove the current assets/math directory from this PR.

Plenty of micro-management here, telling it precisely what to do. And then the result was buggy:

I discovered that it doesn’t escape the SVG’s title properly, and followed up with a fix myself.
The width and height attributes were not calculated correctly by my setup, due to Mathjax preferring ex as the unit type, referring to the width and height of an x character of a font (weird). I did plenty of CSS tweaks that I’m not sure Copilot could’ve done.
The RSS feed wouldn’t work in dark mode, because the SVGs are black text on transparent background, and I couldn’t find a way to add CSS styling to the RSS feeds. Here, Copilot was completely disappointing. My final solution was to generate “transparent background” versions for the website and “light background” versions for the RSS feed.

I used the “agent mode” in Visual Studio Code as well. The results when using GPT-4.1 are very poor, which is a pity, as this is the model with an unlimited number of requests on GitHub Copilot’s Pro plan. Claude Sonnet fared much better (at coding tasks in agent mode, but it’s poor at others, such as high-school math).

At the time of writing, I now have another attempt at using Copilot Agent for optimizations: Optimize Jekyll build time for Mathjax and image processing; because the build times have gone up and that’s bad, too.

It certainly has potential.

The article Update on Math Formulas via Copilot first appeared on alexn.org.

Math Pill #2: Square Roots

Alexandru Nedelcu — Wed, 15 Oct 2025 00:00:00 +0000

Simplifying radicals is strangely satisfying.

I’m helping my son with mathematics, and I have to re-learn it, but this is fun. Hence, I’m hoping to start a series of articles.

Problem statement

Given:

Show that:

Solution

Starting with:

Often such sums can be rewritten like this:

We observe the formula:

Therefore, we end up with the following system of equations:

The solutions may or may not jump at you, but there’s a little Vieta’s formula that can help — we can find a and b as being the solutions to this quadratic equation:

Solving that:

Threfore:

Going back to our original value, we rewrite it:

We still have a square of a sum, but this time we can spot the formula easily:

Next, for simplifying A, we’ll focus on rewriting this:

Same trick applies, so we’re dealing with this system:

Which will have solutions generated by this quadratic equation:

Solving it:

Therefore our sum can be rewritten as (we remember that we have a minus sign though 🙂):

We can now rewrite our A:

And finally:

The article Math Pill #2: Square Roots first appeared on alexn.org.

Math Pill #1: Sums

Alexandru Nedelcu — Wed, 15 Oct 2025 00:00:00 +0000

My son started high-school, so I’m helping him with mathematics. It’s been a long time, and I have to re-learn. But it’s fun, and I’m hoping to start a series of articles.

Here’s an interesting sum:

In order to write it with the summation notation, we first need to observe that the difference between elements in this number sequence is 4, therefore 4k is involved in the formula of the general term:

Such sums are actually telescopic. And when you have fractions like these with a constant numerator, the general term can be split in 2 fractions, with the help of an equation, needing to find an a satisfying this:

The sum becomes:

Expanding it:

The terms get cancelled, the result being:

More samples

We can apply this solution to other similar sums as well, for example:

In which case the fraction decomposition can be found with this equation:

And it works with more denominator factors as well:

With the equation for finding the fraction decomposition being:

Non-constant numerators

For this one we no longer have a constant numerator and the above solution no longer works:

To do this fraction decomposition, we now need to find 2 constants, a and b, the equation now being:

To solve it, after eliminating the denominator, the trick is to group by powers of k:

k being variable, it means that, in order for the above equation to have solutions, we need the constants to nullify k, so we have this system of equations:

And now we can finally write the sum as:

The article Math Pill #1: Sums first appeared on alexn.org.

Outsourced Voices, Outsourced Minds

Alexandru Nedelcu — Mon, 13 Oct 2025 05:24:14 +0000

My cat, Thomas, sitting in a box, telling me that maybe it's better if I went to touch some grass.

We were fooled.

Jekyll, the most popular static site generator, was released in 2008. That’s around the time I registered the alexn.org domain, although I wouldn’t start blogging on it until later. I had another blog before this, it was powered by WordPress, hosted by a local service provider, but it died due to neglect, the articles I had on it lost forever. And before turning it off, I discovered that it had been infested with malware.

That’s how I could see the value in a static website generator like Jekyll: a git repository seemed immortal, I could no longer lose my old thoughts, and the cheaper it is to host, the higher the probability that it would stay alive and kicking, just waiting for me to come back and share new articles. And nothing could be more secure than a static website.

This was happening around 2010, it was right when people were starting to use social media, alongside forums, like Hacker News, Reddit, to distribute and comment on links. So tech blogs stopped having their own commenting widget, and those that continued to have comments, outsourced them to services such as Disqus. Alongside comments, amongst dynamic features of blogs, other casualties were trackbacks and pingbacks (archive), which were the equivalent of seeing who mentioned or quoted you on social media. And yet another was the mailing list, outsourced to Feedburner, then Mailgun. In optimizing my online presence, I achieved what I wanted, but in retrospect, I now realize that optimizing for hosting costs is foolish.

Up until around 2010, it seemed like RSS/Atom feeds were thriving, but then in 2013 Google killed Reader, being perceived as the death of RSS/Atom feed readers. They were trying to create their own Facebook competitor, Google+, and Reader was standing in their way. I can’t tell you how much joy the death of Google+ brought, I’ve never been happier about an online service dying, but I digress. Well, feed readers are still alive and thriving, just with a much smaller audience. The web is dying, and Google Reader proved to be the canary in the coal mine.

It’s 2025, and you’re still posting and commenting on links, but you’re most likely doing it on social media. Facebook, 𝕏/Twitter, LinkedIn are deprioritizing links, which means that posts containing links will have a lower reach. Let me repeat that — social media silos are fighting against web links, the foundation of the open web. They do so to “encourage” content and discussions on their own platforms, and now these platforms are filled with politics, rage, and AI slop.

You’re no longer in control of what you consume on the web. Algorithms are now in control, and those algorithms want you to be a mindless zombie that consumes ads, ads that serve malware, scams, or politics. Not to be misunderstood, I recently bought theater tickets to a show via an ad, curated ads can clearly be valuable, but curation is incompatible with the ambitions of these social networks.

I’d rather not leave the original sin out of this discussion, of course: Google Search becoming increasingly filled with SEO spam, and again, index curation was incompatible with Google’s ambitions due to the conflict of interest generated by their DoubleClick acquisition. After that, everything else was inevitable, such as Google increasingly stealing website content to show it directly in results, due to how poisoned the index became.

It wasn’t always like this. I remember a young Google filled with interesting websites, a young Twitter filled with enthusiasts talking about programming, gardening, other hobbies, a young Facebook filled with photos and life updates of acquaintances. And that was the norm, not the exception. But following 100 people on social media, waiting for their status updates, wouldn’t generate much engagement, so the platforms started force-feeding people the status updates of strangers and bots, discussions converging inevitably to politics, sprinkled with rage, and low-quality humor, as that’s what keeps people engaged.

There are still people online with interests other than politics, but I bet that many of you, even if you aren’t contributing political sludge, at the very least you’re engaging in doomscrolling, as that’s what the algorithm optimizes for. To make matters worse, even when there’s no algorithm, people are now so trained on this behavior that it persists, as seen on Mastodon or Bluesky. You could do anything you want, and you could be anyone you’d like when going online, and yet, many willingly choose to be armchair political commentators. In fairness, Godwin’s law was always a thing. And it’s easier to keep your feed curated when there’s no algorithm, but only if you’re willing to unfriend/unfollow acquaintances who have gone off the deep end.

And now social media platforms are being filled with AI slop, with the platform owners providing the tools for it. Meta/Facebook, 𝕏/Twitter, LinkedIn are literally encouraging AI-generated posts, because it’s a cheap way of keeping you active on their platforms, in a zombie state, ready to consume whatever ads they throw at you. For these companies, it’s a good thing if we are getting dumber due to ChatGPT.

We’ve outsourced our voices on the Internet, because it was convenient. We’ve outsourced our minds as well.

But we are people with ideas, feelings, aspirations, lives. We are not machines, we are not automatons, we have souls. And we deserve to own our identity, our voice, and our thoughts. We deserve access to an open web, a web that’s good for connecting with like-minded people, a web good for learning and exploring our passions.

We were fooled, but we can fight back. Don’t let algorithms control your actions, don’t outsource your voice and your mind. Choose wisely, as in the future we may find ourselves completely unable to make choices.

The article Outsourced Voices, Outsourced Minds first appeared on alexn.org.

Effects and Concurrency in Functional Programming

Alexandru Nedelcu — Wed, 17 Sep 2025 14:43:37 +0000

Video link (open on YouTube.com)

I've delivered (sometime this year) this presentation at a local meetup (ING Hubs Romania). The target was mainstream developers, with the intent to give them a taste of FP and working with I/O in Scala.

Video link (open on YouTube.com)

I’ve posted this on my social media accounts in September, but I’m including it on my blog for posterity. Ignore the noise if you’ve already seen it.

The article Effects and Concurrency in Functional Programming first appeared on alexn.org.

Scala's Gamble with Direct Style

Alexandru Nedelcu — Fri, 29 Aug 2025 05:33:16 +0000

The staircase at EPFL that inspired Scala's logo.

This is a more superficial article, i.e., an opinion piece. I’m making it my mission, again, to publish more thoughts on my own blog.

Scala has had a wide ecosystem for functional programming™️ directly inspired by Haskell and OCaml, due to its expressive type system and support for both type class encodings and OOP. As such it has inspired not one, but multiple runtimes for monadic IO, such as Cats-Effect, ZIO, Kyo. And these libraries served as an inspiration for others, see Effect-TS.

Yet, Scala 3, the language, does not move in the direction of more monadic IO, but rather in the direction of “direct style”, preferring continuations to monads, but without providing support for continuations out of the box. There are some attempts to fill that gap:

The dotty-cps-async approach is IMO the best bet, even for retrofitting monadic IO to direct style. But in my limited experience, it has edge cases, and in the projects making use of it, like Cats-Effect (possibly ZIO or Kyo as well), this support isn’t used much. For example, one reason this happens is because the interruption model is different from that of the JVM, e.g., if you don’t turn a cancellation into an InterruptedException, it won’t blend well with the language’s constructs, like try-catch-finally. But also, support not being part of the actual language, in this case, may mean the implementation is cursed to battle edge cases and bugs forever.
gears builds on top of virtual threads for the JVM, and it supports Scala Native as well. It can support WASM via WASM JSPI, possibly in the next release. So it builds on the runtime’s support for either blocking threads or (one-shot) continuations. That’s incredibly limiting. For example, it has no JavaScript support and blocking threads is still very taxing on JVM-like platforms that don’t have virtual threads, such as Android. And obviously, execution cannot be fine-tuned, and you don’t have the abilities of a user-space runtime such as that of Cats-Effect or ZIO.
ox is a JVM library for blocking threads and doesn’t make any attempts of supporting anything else but the JVM. If you work on the JVM, I guess that’s fine, although the library will be somewhat less useful after Structured Concurrency lands in Java 25. If we want to go back to blocking threads (with mostly the same caveats applying), I guess that’s acceptable with the support for virtual threads since Java 21, but what about Scala Native, JS or Wasm? Ox is a cool library and may be right for a lot of projects, but for the ecosystem, if Scala doesn’t escape the JVM, or at least JVM-isms, it’s far less interesting than the next versions of Java (which will also have type-classes BTW).

Keep the above in mind and compare with Kotlin:

Kotlin Coroutines have multi-platform support, and this means — JVM, Android, iOS (Native), JS and WasmGC.
- Watch Structured concurrency by Roman Elizarov to get a sense of the design considerations that went into it. This isn’t your grandpa’s async/await.
- Those coroutines, along with its support for context parameters, can be leveraged not just for I/O, but also for handling of resources in general, matching Scala’s libraries such as Cats-Effect and ZIO (see Arrow).
Multiplatform support is top-notch. It has a growing ecosystem of libraries, with Compose Multiplatform having stable iOS support and getting contributions from Google and others.
It’s a language that evolves as well. For instance, context parameters are almost here, rich errors as well, and it may even get better immutability support before Scala.

Scala obviously relies more on community and less on commercial support. I actually loved that about Scala, despite it being unable to remain viable for targeting mobile devices, but…

Scala 2.x thrived due to making it saner to work with asynchronous I/O and concurrency. And yet the world isn’t standing still and alternatives have improved. Scala could’ve taken the path of F#’s computation expressions, thus improving the ergonomics of working with monadic IO. Scala could also include support for continuations out-of-the-box. Scala 3 does neither of those things, which means monadic IO is not getting the support it needs in order to go mainstream and the “direct style” approaches are currently in limbo.

I understand why Scala’s designers may prefer a “direct style” path. I’m not sure if I believe in the long-term success of monadic IO. Undoubtedly, it’s currently a marketplace failure, and when you solve concurrency issues by other means, it’s debatable if monadic IO is still worth it. Even those that swear by Effect-TS have to realize that its apparent success only has to do with how much async/await/Promise in JavaScript sucks, and if that pain is ever fixed, the project’s growth will likely stagnate.

But Scala’s evolution is currently alienating the part of the community that builds cutting-edge, user-space I/O runtimes that are the envy of the industry, while not providing the support required for making “direct style” work for the folks that would rather prefer that over monads. And I think that’s bad, especially as alternatives exist, one of those alternatives being Java 25.

I still ❤️ Scala, it’s a productive language, and I believe it will be even more awesome with capture checking. But making programming safer is just one aspect of what makes or breaks a language. There are other aspects, such as the platforms and problem domains a language is able to target. Most problems we solve on a daily basis are I/O-related problems and without a consistent story that targets the mainstream, I fear for its future.

The article Scala's Gamble with Direct Style first appeared on alexn.org.

Backup Script for My Cloud Storage

Alexandru Nedelcu — Fri, 18 Apr 2025 15:53:12 +0000

I currently have around ~370 GB stored in OneDrive, including the family’s photo archive that I’d rather not lose. It needs backups, and currently I don’t have a home NAS. I used to just sync my files, via rclone to Backblaze B2, which can keep older versions of files.

Recently, I moved my backups to a Hetzner’s Storage Share, which is a managed Nextcloud service — I prefer this because it’s a reasonably priced EU service, and it’s based on FOSS that I could self-host. But the problem is: how to keep older versions of files? This is important as backups are meant to prevent accidents, such as accidental deletions, or data degradation.

The trick is in using the --backup-dir option of rclone sync (see documentation). This tells the rclone command to copy any files that have changed or were deleted in the specified directory.

Here’s a script, built for my own needs, use with care (i.e., don’t execute it if you don’t understand what it does).

#!/usr/bin/env python3

import argparse
import datetime
import os
import re
import subprocess
import sys

RSOURCE = "onedrive"
RDEST = "nextcloud"
SYNC_PARAMS = "--delete-excluded -c --track-renames --onedrive-hash-type sha1"

EXCLUDE_PATTERNS = [
    ".git",
    ".DS_Store",
    ".localized",
    "*.swp",
    ".#*",
]

def execute(command, verbose):
    if verbose:
        sys.stdout.write("--------------------------------------------------------------------------\n")
        sys.stdout.write(command + "\n")
        sys.stdout.write("--------------------------------------------------------------------------\n")
    r = os.system(command)
    if r != 0:
        sys.stderr.write(f"Command '{command}' failed with exit code {r}\n")
        sys.exit(r)

def execute_capture_output(command):
    r = subprocess.run(
        command,
        capture_output=True,
        text=True
    )
    if r.returncode != 0:
        sys.stderr.write(r.stderr)
        sys.stderr.write("\n")
        sys.exit(r.returncode)
    return r.stdout

def list_dirs(label):
    r = execute_capture_output(["rclone", "lsd", f"{label}:"])
    names = []
    for l in r.splitlines():
        parts = re.split(r'\s+', l, maxsplit=5)
        if parts[-1].startswith("."): continue
        names.append(parts[-1])
    return names

def main():
    parser = argparse.ArgumentParser(description='Backs up files to cloud storage.')
    parser.add_argument('-d', '--dry-run', action='store_true')
    parser.add_argument('-q', '--quiet', action='store_true')
    args = parser.parse_args()

    date = datetime.datetime.now().strftime("%Y-%m-%d.%H-%M-%S")
    extra_params = [item
        for list in [
            [f"--exclude \"{e}\"" for e in EXCLUDE_PATTERNS],
            ["--verbose"] if not args.quiet else ["-q"],
            ["--dry-run"] if args.dry_run else [],
        ]
        for item in list
    ]

    for dirname in list_dirs(RSOURCE):
        backup_dir = f"{RDEST}:Backups/OneDrive/{date}/{dirname}"
        execute(
            f"rclone sync \"{RSOURCE}:{dirname}\" \"{RDEST}:{dirname}\" {SYNC_PARAMS} \"--backup-dir={backup_dir}\" {" ".join(extra_params)}",
            not args.quiet
        )

if __name__ == "__main__":
    main()

The article Backup Script for My Cloud Storage first appeared on alexn.org.

Tagged vs Untagged Unions (in Scala)

Alexandru Nedelcu — Wed, 02 Apr 2025 08:30:19 +0000

What’s the difference between Option[A] and A | Null? How about between Either[A, B] and A | B?

Scala has had the Option data-type for dealing with null values:

val person: Option[Person] = ???

// You can pattern match on it:
person match {
  case Some(e) => updateState(e)
  case None => doNothing
}

// You can combine it with other values, sequentially:
for {
  p <- person
  a <- p.address
  t <- transaction
} yield updateState(a, t)

// ... or in parallel (with the ability to get all error messages):
import cats.syntax.all.*

(person, transaction).mapN { (p, t) =>
  updateState(p, t)
}

The big benefit of Option is that it’s a “monadic type”, which also has to do with the fact that it’s a “boxed” type, meaning that Option[Option[T]] is not the same as Option[T]. This is very important, as that means that Option has predictable behaviour (it’s lawful), and can be used in instances in which Option[Option[T]] makes sense:

case class Person(
  name: String,
  address: Option[Address]
)

// For an HTTP PATCH, if a field is provided, it should be 
// updated, otherwise, if `None`, then the old value is kept.
case class PersonHttpPatchRequest(
  name: Option[String],
  address: Option[Option[Address]]
)

Scala 3 also has A | Null as a type, with the explicit-nulls compiler option being required to make it useful. This is similar to what Kotlin provides, the so-called A?, although slightly less mature.

The downside of A | Null is that it’s not a monadic type. It’s not lawful in the same way as Option is. And that’s because it’s not a boxed type. A | Null | Null is the same as A | Null. Therefore, this no longer works:

case class Person(
  name: String,
  address: Address | Null
)

// Oops! This isn't right:
case class PersonHttpPatchRequest(
  name: String | Null,
  address: String | Null | Null
)

Another downside is that you can’t really use A | B for expressing Either[Error, Result]. This is because you don’t know which is the error and which is the result.

trait Parser[A] {
  // Doesn't work as `parse` could produce a `Right(ParseError)`, 
  // after all, I can't see why errors shouldn't be serializable 
  // and deserializable.
  def parse(input: String): ParseError | A 
}

This has 2 advantages:

Scala is an OOP language, and we care about the variance of type parameters, so A | B is better than AA >: A in the type signature of methods.
A | Null being a supertype it means that using it can preserve backwards-compatibility.

For the first point, consider this:

// we've got a covariant type parameter
sealed trait List[+A] { 

  def prepend1[B >: A](value: B): List[B] = ???

  def prepend2[B](value: B): List[A | B] = ???
}

Which of these two methods is better? IMO, it’s the second one because it’s more expressive, whereas the first one is losing information. In fairness, the Scala compiler can and probably does infer that B >: A can be A | B, so we’re essentially comparing Scala 2 with Scala 3 here. And I’d rather talk about a List[Int | String], for example, than a List[Any].

For the second point, given these methods:

// For testing contra-variance
def foo(value: A): B

If we change its signature to using A | Null, then we aren’t breaking compatibility, because all the call sites still work:

// Contra-variance, still compatible
def foo(value: A | Null): B

// ....
// Still works in all the older code
val a: A = ???
foo(a)

But if we’d use Option[A] instead, we’d break source and binary compatibility, which is especially problematic on the JVM due to all the dynamic linking of transitive dependencies:

def foo(value: Option[A]): B

// ...
// This now breaks
val a: A = ???
foo(a) // Error

Of course, the old fashioned way of dealing with this is to use method overloading, assuming you can do it (unless type erasure gets in the way):

def foo(value: A): B = foo(Some(value))
def foo(value: Option[A]): B

Also, it wouldn’t work for return types, so for example:

// If we are forced to keep this signature
def foo(value: A): Option[B]

// We can't add an overload like this
def foo(value: A): B

So there you have it. Currently, there’s no free lunch and I actually like both.

The article Tagged vs Untagged Unions (in Scala) first appeared on alexn.org.

My Favorite Firefox Extensions

Alexandru Nedelcu — Sat, 15 Mar 2025 18:29:27 +0000

In my previous post, I mentioned that I now use Firefox as my main browser and you should too. Firefox’s strength is its extensions, with many available on Android devices as well. Here’s a list of my favorites.

On both the desktop and in Firefox for Android 🤖:

Dark Reader: an extension that modifies the styles of websites to make them dark-themed. It can also auto-switch depending on the system’s theme (light vs. dark), or detect the theme of the website in order to not interfere. It’s the best at what it does.
LeechBlock NG: an extension that helps me with building healthy habits by blocking distracting websites during certain hours of the day, or how long I’ve spent on those websites.
SponsorBlock - Skip Sponsorships on YouTube: as the name suggests, it skips the sponsorships. I already pay for YouTube Premium, I’d rather not watch any ads, even when embeded in the video.
Stylus: Helps with modifying the styles of websites. I rarely need it, but it’s so nice to have when I do.
uBlock Origin: The best ad-blocker, period. Chrome stopped supporting it, now it’s only available on Firefox. For example, it’s the only one that attempts ad-blocking Facebook.
Web Archives: View archived and cached versions of web pages on various search engines, such as the Wayback Machine and Archive․is.
Yang!: An extension that brings client-side search bangs (which first appeared in DuckDuckGo). You can keep using your favorite search engine (e.g., Google), it avoids an extra server roundtrip (which may be problematic for privacy as well), it’s faster, and it’s supported on Android.

For the desktop only:

1Password: The best password manager, kind of expensive, though.
Activate Reader View: Sometimes Firefox’s Reader View can’t be activated due to it not detecting an article on the page. This extension forces the “reader view” to be activated.
AudioContext Suspender: This fixes an issue with Firefox due to how websites wrongly use AudioContexts, preventing battery drain.
Auto Tab Discard: Inactivates tabs that haven’t been used in a while, freeing up memory and CPU. Chromiums started adding this as a feature, but this extension has customization options, like what tabs should never be discarded.
Clickbait Remover for Youtube: Replaces thumbnails and modifies titles on YouTube to remove clickbait. It works.
Close Tabs Shortcuts + Toggle Pin Tab: I need Alt-P for (un)pinning the current tab, and Alt-W for “close other tabs”. Firefox, unfortunately, doesn’t allow for adding or customizing shortcuts for such tab actions, out of the box, but there’s nothing that an extension can’t fix.
Floccus: This can sync your bookmarks between browsers, so for example, you can maintain the same set of bookmarks between Firefox and Chromium. It can use a GitHub repository as the backend.
LanguageTool: I’m not a native English speaker, so this helps me with grammar and spelling. The server is FOSS and can be installed offline, but I pay for Premium. The company is from the EU, so that’s a plus.
Linkding extension + SingleFile: I have my own Linkding server, that works like Pinboard, a repository of searchable links, some of which are shared. The Linkding extension also has integration with “SingleFile”, which is another extension that can save a copy of the page you’re viewing as a single HTML file, such that it’s archived on the Linkding server.
Old Reddit Redirect: Redirects Reddit links to old.reddit.com because the new UI sucks.
RSSPreview: Finds and previews RSS feeds on websites. Useful if you use a feed reader.
Sideberry: Vertical tabs tree. I don’t use “tree style tabs” all the time, but (compared with tab groups) trees are automatic, and when I need this sidebar, it’s one Ctrl-E away.
StreetPass for Mastodon: Discover people you’d like to follow on Mastodon, by the websites you visit. It’s based on the WebFinger protocol. So if you’re vising my website, you should get a suggestion for @alexelcu@social.alexn.org.
Substitoot — improved Mastodon federation: This extension loads missing replies and stats for posts on Mastodon. My instance is a small one (it’s only me on it) and without this extension I wouldn’t see the replies of the posts from my feed, which is somewhat of a flaw in the distributed nature of Mastodon.
Tabliss: New Tab page with beautiful backgrounds and a clean design.
Vimium: Vim-like keybindings for Firefox. With it, you can discard the mouse or touchpad, and use just the keyboard to navigate the web. It’s great and very addictive.

What are your favorite Firefox extensions? Any goodies I should try?

The article My Favorite Firefox Extensions first appeared on alexn.org.

Use Firefox in 2025

Alexandru Nedelcu — Mon, 03 Mar 2025 21:51:40 +0000

I grew up with the Internet, since before people had Internet connections at home or in their pocket. The browser, being the window to the open web, holds a special place in my heart. In this article I’m suggesting the use of Firefox in 2025, for both technical and political reasons, as it’s still the “user agent” that it set out to be.

On the desktop

Firefox remains the only one supporting uBlock Origin, the version that’s not defective by the design of the browser. Many may not notice a difference by using its Lite version, the one based on Manifest v3, but I can tell you that most ad-blocking tech is easy to circumvent, whereas the full version of uBlock Origin has been the nightmare of online advertisers.

Firefox can have extensions that expose a sidebar. This is how Sideberry or Tree Style Tab can work. Tab grouping from Chrome is nice, but you have to make a conscious effort to use groups, whereas trees are automatic, and you can simply hide that sidebar if you’re not interested in it. It actually works well with Firefox’s vertical tabs, with Ctrl-E toggling my “Sideberry” panel whenever I need it, otherwise seeing a nice vertical column of icons.

Vertical tabs have been officially released in Firefox 136, and the implementation seems to be quite good. Tab groups (experimental) can be enabled if you toggle browser.tabs.groups.enabled in about:config.

Ctrl-Tab in Firefox has the best behavior, and it’s nice that if you keep Ctrl pressed it shows you a graphical switcher previewing around 7 tabs on my laptop’s screen. I’ve only seen something equivalent in Vivaldi.

Firefox has great privacy, with features such as Total Cookie Protection which isolate third-party cookies without breaking websites. It also blocks many trackers by default, although note that I don’t enable “Strict” mode, as it breaks websites (and honestly, uBlock Origin is enough). It’s still nice that it blocks the worst privacy offenders out of the box.

Multi-account Containers are really nice when you have to manage multiple accounts for the same online service (e.g., AWS). Although, note that it doesn’t do much for privacy (ever since Total Cookie Protection) and the feature provides less isolation than when using “profiles”.

The suggestions in the address bar, aka “Awesome Bar”, are the best. Don’t get me wrong, Chromium’s Omnibar is pretty good, as well, but despite its improvements, IMO it’s still prioritizing search over bookmarks and history.

Firefox’s history synchronization works better than in all Chromium browsers I’ve tested. Chrome’s synchronized history is missing items when you’re enabling encryption, like for example when you’re jumping from page to page by clicking links. What gets synchronized are the pages and the searches that you enter directly in the address bar. And the time window is more limited. Firefox’s history of visited URLs, and its bookmarks, are helpful when using its address bar, as it can reliably give you suggestions, thus avoiding Google searches.

Firefox supports DNS over HTTPS with the best customizability — i.e., it can fall back to the system’s DNS. This is important for me in case I’m connected to the corporate VPN in charge of serving corporate resources, but I still want my regular DNS queries to go through my own DoH service, encrypted as well.

Firefox has offline translations via offline AI. I remember disliking Google Translate because it is slow, and I also worried about my privacy, as you’re sending your web page to a server. Speaking of, if you think AI isn’t useful in a browser and that Mozilla shouldn’t invest in AI, well, here’s one sample where you’re definitely wrong.

Other things I like:

Firefox has had the best Linux and BSD support. For example, many switched to it due to its support for Wayland.
The PDF viewer has support for editing PDF documents. On macOS this doesn’t sound like much (due to Preview), but on Windows or Linux it’s a blessing.
UI of Picture-in-Picture is nice and PiP can be automatic when you switch tabs (in preview / labs).
The Reader View is good and reliable (also see this extension to force activate it in the rare cases where article detection fails).
Has great customizability, even though it feels limited at times — there’s nothing that an extension, a custom stylesheet or some setting in about:config can’t fix.

For Android (Mobile)

Firefox on Android supports extensions, and here are the ones I’m using:

Some of these features may be built into some of the available browsers, but not all of them, plus the functionality provided is better, for example:

Dark Reader is better than the “apply dark theme to websites” checkbox from Chrome (activated via chrome://flags), it usually gets out of the way, but also allowing customizations for when it fails; I’m also one of those users that auto-switches to dark mode only at night.
Cookie AutoDelete is built into the Brave browser, but with fewer customizations and functionality; being a power user, I prefer Firefox’s extension.

Other features of Firefox for Android that I like:

“Open in another app” can be disabled, or Firefox can ask what you want. It also provides a neat shortcut in the menu if you change your mind.
New Tab page is helpful.
Synchronization of tabs with the desktop works well, and the UI is nice (I like that I can find where it is).
Reader Mode works well. Compared with the desktop, it can’t do text to speech on its own, but there are other solutions for that, including built-in Android functionality or Pocket.

Firefox for Android has issues, such as poor performance on some websites (e.g., Mastodon, I even opened an issue) or poor PWA support. But I use my mobile browser primarily for reading articles, and for that, having a good ad-blocker, dark mode (for when needed), or a working reader mode are much more important.

Firefox, the User Agent

The user agent is a piece of software responsible for interacting with the web, such as a browser, but it’s critical that this agent represents the user and its interests. A user agent shouldn’t represent the interests of the publishers or that of the advertisers while harming the user. Publishers and advertisers are important as well, but the user should come first.

Think of Chrome. When you log into Chrome, it automatically logs you into your Google account on the web. It also activates a setting named “improve search suggestions” by which Chrome shares your navigation history with Google. It doesn’t matter if you have encryption enabled, that feature will be on for every new device, as it’s not synchronizing from your account’s settings either. Chrome is adversarial towards users. On every released version, any privacy conscious user would have to carefully read its release notes and look at its settings page, thinking about “how will these new features screw me in the future?” Do you like uBlock Origin? Google has deprecated it and lied about the reasoning for doing so, knowing full well that the replacement is much less capable. We’re talking about the desktop because Chrome on Android does not support extensions, even though it could without much effort, being the same codebase.

Chrome is an excellent browser, but when the user’s interests conflict with that of Google, it’s the interests of Google that win.

A browser being a good user agent isn’t solely about trust, but about ability as well. A browser isn’t a good user agent if it can’t be extended in ways that conflicts with the interests of its maker. You need extensions to retake your browser. I believe Firefox is still a worthy User Agent, despite Mozilla’s blunders. This isn’t just a feeling, but a fact based in Firefox’s current technical abilities.

But for Firefox’s sake, I hope Mozilla understands why many people still use Firefox because once trust is lost, it may never return.

Politics

I tried staying apolitical for a long time, but this is no longer possible, as this is now about my self-interest. And given the worldwide politics, especially in light of the US administration becoming hostile towards the European Union, I’m becoming increasingly more uncomfortable using or recommending products or services from the US’s Big Tech companies. Likewise, I’m uncomfortable with products built by MAGA figures. When other people place their country first, voting for an administration that turns against allies, I have to remember that I’m a European first, sorry.

You may disagree, and that’s fine by me. Software is usually amoral, and your threat model may be different from mine. But being EU-friendly is important to me for obvious reasons and unfortunately, in the world we now inhabit, I find myself increasingly wary about the software that I’m adopting. And don’t get me wrong, I try avoiding Russian and Chinese software and services as well 🤷‍♂️

Other browsers that would bring me some peace of mind would be Tor, Mullvad, or Vivaldi. There may be others, like Zen. As a general rule, they either have to be community maintained or backed by an EU company, with Firefox being the only exception that I’m going to allow, for now.

Note that I’m not naive about where the major contributions are coming from. There are only 3 major browser engines, all 3 developed by US companies, all 3 primarily funded by Google’s Ads. But I can live with Open-Source and I think the forks can do a reasonable job at disabling hostile functionality, such as all the phoning to the mothership, with mixed results.

Controversies

Mozilla isn’t without controversy, and lately, they’ve doubled down on them diversifying their revenue via more advertising, updating their privacy to make it unambiguous that they intend to share your personal data with advertisers. Yikes!

Many people view this as a betrayal of their values, and there’s some truth to that. Take a look at this old ad, when they seemed to have more of a drive for doing the right thing:

Firefox a different kind of browser (open on YouTube.com)

But when talking about Mozilla’s direction, one has to keep in mind that to develop and maintain a browser, you need a lot of money. In our country, we have a saying like “you’re my brother, but cheese costs money”. Chrome’s estimates, for example, exceed 1 billion $ per year in development costs. People don’t want to pay for browsers, you can’t rely on the donations of people preferring to ad-block YouTube instead of paying for Premium. And asking for governments, like the EU, to fund it is not a great idea for obvious reasons, such as Mozilla not being an EU entity or the fact that the state serves the interest of their taxpayers, and not the interest of the whole world.

Mozilla has been surviving on Google’s Ads ever since the Search deal. Google, at this point, is funding all 3 major browser engines, and this includes Safari, given that Google is paying Apple around 20 billions per year in the search deal. This has consequences. For example, even though both Mozilla and Apple have been singing the privacy tune, they never did anything to upset their cash cow, which is Google Search. So let’s be honest — Firefox has been funded by ad-tech since 2006, and Mozilla diversifying their revenue can’t be a worse betrayal of their values than what happened since then.

What’s important for me aren’t the words, but what they actually accomplish. Thus far, Firefox has gone to further lengths than any other browser to preserve privacy in ways that actually matter. For example, the aforementioned offline translation feature that doesn’t leak your web page to an online server. I also remember when they released PDF.js, so back when the PDF viewer in Chrome was a proprietary blob (FoxIt, does it still ship with it?), Mozilla was like “hold my beer” and built an Open-Source PDF viewer in freaking JavaScript and HTML. Their history is filled with great feats, both old and recent, that just can’t be easily erased.

In fact, the introduction of their privacy-preserving attribution feature is far more worrying than any of the new words in their Privacy Policy. On the other hand, I can give them the benefit of the doubt, at least for now. At the very least, I know that they’ll try making PPA privacy-preserving. Mostly because I don’t have much choice left.

The article Use Firefox in 2025 first appeared on alexn.org.

Self-hosted Bookmarks Manager

Alexandru Nedelcu — Fri, 14 Feb 2025 05:03:54 +0000

I like having a database of links I encounter on the web, like a searchable database, complete with archive links as well. I found a solution.

Storing bookmarks in your browser is OK, especially if you use a solution for synchronizing that archive, such as floccus (GitHub), but that’s not enough, as I also want it to be searchable (with tags and descriptions) and shareable. I used Pinboard, Pocket, private notes, wiki notes and I even tried a links section on this blog.

I finally found a Pinboard-like replacement that I can easily self-host and that’s just perfect for my needs: linkding (GitHub link).

My links are now hosted here: links.alexn.org (RSS feed).

Setup

The official installation instructions are pretty good, but in case it helps, what follows is my setup.

Entry in my docker-compose.yaml:

linkding:
  container_name: linkding
  image: sissbruecker/linkding:latest
  restart: unless-stopped
  healthcheck:
    test: ["CMD-SHELL", "curl --silent --fail http://localhost:9090/health || exit 1"]
  ports:
    - "3009:9090"
  volumes:
    - 'linkding:/etc/linkding/data'
  networks:
    - external_network

My Nginx reverse-proxy setup:

location / {
    root   /var/www/links.alexn.org;
    include ./snippets/cloudflare-ips.conf;
    
    proxy_pass http://0.0.0.0:3009;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;

    # Prevent Cloudflare from caching while allowing client caching
    add_header Cache-Control "private, no-transform";
    add_header CDN-Cache-Control "no-store, no-cache";  # Cloudflare-specific
    add_header Cloudflare-CDN-Cache-Control "no-store"; # Cloudflare-specific
}

location /static/ {
    root   /var/www/links.alexn.org;
    include ./snippets/cloudflare-ips.conf;
    
    proxy_pass http://0.0.0.0:3009;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    
    # Nginx caching
    proxy_cache CACHE;
    proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
    proxy_cache_valid 200 302 30d;
    proxy_cache_valid 404 1m;
    proxy_cache_key $host$request_uri;
    
    # Cache settings for browsers and CDN
    expires 30d;
    add_header Cache-Control "public, no-transform";
    add_header Vary Accept-Encoding;
    add_header X-Cache-Status $upstream_cache_status; # Optional: helps with debugging
}

I’m also doing targeted server backups, so here’s the script that I’m periodically executing via cron:

#!/bin/bash

set -e

FILENAME="linkding-$(date +"%Y-%m").zip"
FILEPATH="/var/lib/my-backups/$FILENAME"
CONTAINER_BACKUP="/etc/linkding/data/backup.zip"

if [ "$(docker inspect -f '{{.State.Running}}' linkding)" ]; then
  echo "[$(date +"%Y-%m-%d %H:%M:%S%z")] Generating $FILEPATH"
  docker exec -it linkding python manage.py full_backup "$CONTAINER_BACKUP"
  docker cp linkding:"$CONTAINER_BACKUP" "$FILEPATH"
  docker exec linkding rm -f "$CONTAINER_BACKUP"

  if [ -f "$FILEPATH" ]; then
    rclone copy "$FILEPATH" "backup:AlexnOrg/Linkding/"
  fi
fi

rm -f /var/lib/my-backups/linkding-*.zip

With the cron entry being like:

50 */6 * * * root cronic /path/to/bin/vm-backup-linkding

Ensure to auto-upgrade it as well.

The article Self-hosted Bookmarks Manager first appeared on alexn.org.

Auto-updating Docker containers

Alexandru Nedelcu — Fri, 14 Feb 2025 05:00:00 +0000

I'm self-hosting stuff using Docker on my own server. One problem that comes up for a personal server is to keep it up to date, as old software has security issues. And we don't want to self-host servers that can become vulnerable due to neglect.

There are a bunch of solutions floating around, but for a personal server, the most pragmatic solution I found is a simple script:

#!/usr/bin/env bash
set -e

cd "$(dirname "$0")" || exit 1
BINDIR="$(pwd)"
SC='\033[0;36m' # Cyan (0;36)
NC='\033[0m' # No Color

printf "${SC}------------------------------------------------${NC}\n"
printf "${SC}Updating & Cleaning Docker — $(date +"%Y-%m-%d %H:%M:%S %Z")${NC}\n"
printf "${SC}------------------------------------------------${NC}\n"

printf "\n${SC}> docker image prune -af${NC}\n\n"
docker image prune -af  2>&1

printf "\n${SC}> docker images | grep -v REPOSITORY | awk '{print \$1\":\"\$2}' | xargs -Iname docker pull name${NC}\n\n"
docker images | grep -v REPOSITORY | awk '{print $1":"$2}' | xargs -Iname docker pull --quiet name 2>&1

printf "\n${SC}> docker compose up -d --remove-orphans${NC}\n\n"
docker compose up -d --remove-orphans  2>&1

printf "\n${SC}> docker image prune -af${NC}\n\n"
docker image prune -af  2>&1

printf "\n${SC}> docker system prune -f${NC}\n\n"
docker system prune -f  2>&1

You can then add this to /etc/cron.d/ to run once per day:

0 3 * * * root cronic /path/to/bin/vm-docker-update-all

Note that I’m using cronic for getting alerted over email when my cron scripts are failing. You can ignore that part.

The astute reader will wonder — doesn’t this aggressive policy have the potential to break your setup? Upgrades for docker images, especially latest images, aren’t necessarily security upgrades. So the answer is: of course, new versions can break backwards compatibility with your current setup (e.g., database schemas), but I’d rather have a broken server than a vulnerable one. And I think the only problem I encountered thus far was when my Mastodon instance was automatically upgraded from 4.2.x to 4.3.x, but then I quickly got an email from my Monit instance.

With upgrade policies in place, health monitoring and backups, self-hosting stuff is cheap and joyful.

The article Auto-updating Docker containers first appeared on alexn.org.

Join the Open Web

Alexandru Nedelcu — Wed, 20 Nov 2024 07:43:54 +0000

"Discussing The Divine Comedy With Dante", a painting by Dai Dudu, Li Tiezi and Zhang An

Now that you've seen the dangers of social media, the question is, what are you going to do about it?

Social media networks are built to be walled gardens, and come with all the problems of walled gardens. For instance, social media networks can be bought and remade in the image of their owners, they can deprioritize your posts if you’re not paying, they can indoctrinate you via algorithms meant for maximizing engagement, they can ban you for wrongthink.

Maybe it’s time for you to adopt the open web, in these (possibly not so) easy steps:

Adopt an RSS/Atom feed reader.
Own your online identity, by starting a blog on your domain.
Break up with your smartphone.

Social media posts are shallow, whereas more thought and details goes into website articles and blog posts. Your favorite people on the web have better thoughts when they blog. And RSS/Atom feeds are still the best way to consume the web.

You don’t want your livelihood to depend on the whims of algorithms or polarized mobs or tech owners that want to do politics. Starting a blog on your domain is still the best way to have an online voice.

Blogs are still relevant and awesome. Initially, you may not have much of an audience, but you need to remember that the primary audience for your blog is you. The more you write, the more you clarify your thoughts, and also, the more you remember. My blog is also a personal wiki. That you get an audience out of it, that’s a bonus. Although, note that people who have had blogs for a while actually get more traffic on their blogs; that’s because blogs get linked in documentation, or are indexed by search engines, or are subscribed (by RSS or email) by loyal followers, so the traffic you get is constant, whereas the traffic for social media posts is fleeting.

I like alternatives like Mastodon and Bluesky, but make no mistake about it — the perils are still there because it’s the same format that drives people to shallowness and polarization, and their distribution and federation story is less than ideal. Yes, join Mastodon or Bluesky, but use social networks primarily for the distribution of web links and for the discovery of blogs that you can follow. Adopt POSSE!

Social media is driven by its always-on availability due to the internet-connected device in our pocket. It’s junk food for socialization or for staying up to date. If we remove social media from our phone, its appeal drops. Therefore, we have to adopt digital minimalism and make our smartphone a dumber one by uninstalling all addictive slot machines from it.

Write and read blogs, you’ll be happier for it 😉

The article Join the Open Web first appeared on alexn.org.

Cancellable HTTP requests via Scala's Tapir

Alexandru Nedelcu — Sun, 29 Sep 2024 08:21:34 +0000

This sample shows an HTTP server and an HTTP client, that can cleanly cancel requests, both on the client-side and on the server-side. This is needed for safe disposal of resources.

On the client-side, it’s important to close or reuse the connection early, to avoid connection leaks. And on the server-side, the cancellation signal can be received when the client closes its connection, so the server may choose to cancel the processing of the request, since there’s no longer a client waiting for the response.

This sample is using Tapir for describing HTTP endpoints. For the server backend, Tapir is configured to use Netty. And for the client making HTTP requests, it’s using Sttp, powered by the standard async-http-client. And I’m also using Cats-Effect, as the effect system because it rocks, with its interruption model being best in class.

#!/usr/bin/env -S scala shebang

//> using scala "3.3.4"
//> using dep "com.softwaremill.sttp.tapir::tapir-netty-server-cats:1.11.5"
//> using dep "com.softwaremill.sttp.tapir::tapir-sttp-client:1.11.5"
//> using dep "com.softwaremill.sttp.client3::async-http-client-backend-cats:3.9.8"
//> using dep "org.typelevel::cats-effect:3.5.4"
//> using dep "org.slf4j:slf4j-nop:2.0.16"

import cats.effect.*
import cats.effect.std.Dispatcher
import sttp.tapir.*
import sttp.tapir.server.netty.cats.NettyCatsServer
import sttp.client3.*
import sttp.client3.asynchttpclient.cats.AsyncHttpClientCatsBackend
import sttp.tapir.client.sttp.SttpClientInterpreter
import scala.concurrent.duration.*
import scala.concurrent.TimeoutException

object Endpoints:
  type Type = Endpoint[Unit, Unit, Unit, String, Any]

  val slow: Type = endpoint.get
    .in("slow")
    .out(stringBody)

  val fast: Type = endpoint.get
    .in("fast")
    .out(stringBody)

def startServer: Resource[IO, Unit] =
  val fastEndpoint = Endpoints.fast.serverLogic: _ =>
    IO.pure(Right("fast response"))

  val slowEndpoint = Endpoints.slow.serverLogic: _ =>
    IO.monotonic.flatMap: startedAt =>
      val task =
        for
          _ <- IO.println("[Server] Received ping request!")
          _ <- IO.sleep(10.seconds)
        yield Right("slow response")

      task.onCancel:
        for
          now <- IO.monotonic
          elapsedSecs = (now - startedAt).toNanos / 1_000_000_000.0
          _ <- IO.println(
            f"[Server] Request cancelled after $elapsedSecs%.2f seconds!"
          )
        yield ()

  for
    d <- Dispatcher.parallel[IO]
    _ <- Resource
      .make:
        NettyCatsServer[IO](d)
          .port(8080)
          .addEndpoints(List(fastEndpoint, slowEndpoint))
          .start()
      .apply: s =>
        for
          _ <- IO.println("[Server] Shutting down...")
          _ <- s.stop()
          _ <- IO.println("[Server] Bye, bye 👋")
        yield ()
  yield ()

def makeRequest(e: Endpoints.Type, backend: SttpBackend[IO, Any]): IO[Unit] =
  val send = SttpClientInterpreter()
    .toRequest(e, Some(uri"http://localhost:8080"))
    .apply(())
    .response(asStringAlways)
    .send(backend)
    .flatMap: response =>
      IO.println(s"[Client] Response: ${response.body}")
    .timeout(1.second)
    .recoverWith:
      case _: TimeoutException =>
        IO(System.err.println("[Client] ERROR: Request timed out!"))
      case e =>
        IO.apply:
          System.err.println(s"[Client] ERROR Request failed!")
          e.printStackTrace()
  for
    _ <- IO.println(s"[Client] Sending request to ${e.show}")
    _ <- send
  yield ()

object Main extends IOApp.Simple:
  override def run: IO[Unit] =
    val resources =
      for
        _ <- startServer
        clientBackend <- AsyncHttpClientCatsBackend.resource[IO]()
      yield clientBackend

    resources.use: clientBackend =>
      for
        _ <- makeRequest(Endpoints.fast, clientBackend)
        _ <- makeRequest(Endpoints.slow, clientBackend)
      yield ()

To run this sample, you can use Scala CLI, which is the default scala launcher since Scala 3.5.0. For macOS:

# Installs the latest Scala via Homebrew
brew install scala

# Makes the above script executable
chmod +x ./tapir-client-server.scala

# Runs the above script
./tapir-client-server.scala

The article Cancellable HTTP requests via Scala's Tapir first appeared on alexn.org.

Trusting Open-Source

Alexandru Nedelcu — Fri, 06 Sep 2024 00:00:00 +0000

My cat, Thomas, trying to understand "source available" licenses

“Shrink-wrapped”, “off-the-shelf” apps, or what the young have come to know as premium apps from the app store, are destined to languish, to become expensive subscriptions, or to be sold to shady companies that will sell your data or make your computer part of a botnet. Proprietary apps are destined to either disappear, most of them taking your data with them, or to rip you off.

I use plenty of proprietary software, my days of FOSS idealism are long gone. But I still prefer Open-Source apps when suitable choices are available. Even if I have to pay for a subscription (e.g., Newsblur), even if it is ads funded (e.g., all browsers), and sometimes even when the product is inferior, as long as it meets my needs (e.g., Gimp). This need is amplified when we build products because I want sane foundations to build on. I would never willingly pick a programming language with a proprietary compiler, a proprietary database system, or depend on proprietary libraries.

This is because Open-Source can accept contributions, or it can be forked. If you need control or responsibility, you can assume it. If you don’t have the expertise for it, you can pay others to provide service. You may not want control, or you may not have the resources, but that possibility is there. Open-Source software is not immortal, but as long as there’s interest, people can find ways to keep it going.

The point of Open-Source isn’t and has never been “source available”. That’s just a prerequisite and a nice to have. The purpose has always been giving users the freedom to use for whatever purpose, or to fork the software, which in turn translates to lower development costs for the software makers. Software makers have to give up control, the bargain they have to make to receive contributions. This obviously wouldn’t be a good business model, unless you treat the software as a gift to the world, and as a commoditized complement of other products or services. Beware of companies selling support for Open-Source software, as this creates the perverse incentive to keep the software difficult to use and insecure.

There are multiple issues with “source available” licensing.

Firstly, Open-Source licensing gives you peace of mind because they are old and tested. You know exactly what you’re getting. The problem with freemium licensing is that money doesn’t change hands, there’s no contract, no negotiation, so the license is all you have, and that license is open to interpretation. Legal departments in corporations may ban copyleft licenses, but copyleft licenses are well understood, whereas licensing that’s designed to ban AWS is not.

As a side-note, I find it worrying that idealism in youth, nowadays, is less about freedom, and more about hatred against the rich and powerful. On computer forums, whenever you see someone defending Free Software / Open-Source, that person is likely a graybeard that remembers the times before Open-Source software became this common infrastructure we all rely on.

You’d think that the source being available is still a good thing because you can read it, helping with understanding and debugging. But that just opens you up to copyright or patent lawsuits, should you end up being inspired by that source code in your own work. This was a thing back when Microsoft had their Shared Source initiative, a company that has continuously threatened Linux and Android phone makers with patent lawsuits, even post-Ballmer, while their “Microsoft changed” marketing campaign was ongoing.

You may be able to contribute to the project, but then you’re just foolishly doing unpaid labor, as you won’t be able to assume control, should you desire it. The raison d’être of Open-Source is gone. And if the software maker goes under, if you built your business on top, forking the software won’t be possible, or if it is under some restricted freedoms, few will want to touch it.

Note that BSL, a license designed for delayed Open-Source publication, is at least more honest and better than source-available licensing designed to ban a company’s competition, like SSPL. Due to the breach of trust, software like MongoDB, Redis or Elasticsearch are dead to me. Interestingly, Elasticsearch recently returned to Open-Source, re-licensed as AGPL, although, I don’t understand why anyone would want to return to it, instead of the more free and more secure Opensearch.

Proprietary software has a place in this world. We all have to put food on the table. Open-Source is infrastructure, it’s “the commons”, but it doesn’t have good business models, and we don’t need control for everything. Some tools can be rented instead of owned. But trust in business is everything. And once gone, it’s gone, although it seems like the young may have to relearn history’s lessons the hard way 🥲

The article Trusting Open-Source first appeared on alexn.org.

1Password vs. Bitwarden

Alexandru Nedelcu — Tue, 20 Aug 2024 06:41:41 +0000

Optimising costs and my workflow is a personal obsession of mine. I've been a 1Password user for years, and I keep trying Bitwarden. If you want to pick between the two, here are the differences between them at the time of writing.

Take this comparisson for what it is: a snapshot in time. In a year from now, both would have evolved, and some of these items may no longer be valid.

I’m not enumerating here what they both do well, such as trustworthiness, the ability to reliably autofill login credentials, or the cross-platform and cross-browser coverage. These are their pros and cons relative to each other:

1Password

Significantly better UI and UX — and this isn’t a subjective opinion.
Sort by date created / updated — useful to find the items that have changed. This seems like a small thing, but it’s difficult to extract this info from Bitwarden (see workaround).
Full history of items, available on the web UI — Bitwarden only keeps a history of the main password.
Attachments management (that doesn’t suck).
Flawless Passkey support — I did not bump into any issues with it.
Auto-filling works for TOTP codes, and multi-step dialogs are now automated, no longer requiring intervention (i.e., UIs that first ask for email, then for password, then for TOTP).
Useful CLI, very straightforward to use for managing secrets in your local environment.
Flawless unlocking with Touch ID, including the CLI.
SSH key management.
Share items with anyone.
Global shortcut (⌘\) that’s very efficient, on macOS at least; also on macOS, 1Password can autofill in other apps, not just the browser, easily invoked via this global shortcut.
Keyboard shortcuts that work really well — for both the desktop app and the browser extension.
“Show in large type”.
More document types — e.g. bank accounts, software licenses, Wi-Fi networks.
Offline support — possible to edit or save new items while offline.
Nicer integration with Fastmail’s masked emails.
Managing the vaults of your organization or family is very intuitive.
Exports (for backups or migrations) contain everything, including shared vaults or attachments.

As downsides for 1Password:

The autofill behavior for the provided “websites” are limiting — for example, I would have liked URL-prefix matching.
Android has some issues — e.g., the aforementioned customizable autofill behavior (i.e., host vs base domain) does not work (I hope they fix it), and sometimes, the UI failed to open, so I had to restart the app.
High cost — I pay 68 EUR per year for the family plan (5.65 EUR per month), and I only share the subscription with my son — for professionals this is cheap, but for students, unemployed folks, or the average Joe in general, it’s definitely expensive.
Proprietary software, closed-source — this isn’t a problem for me, except…
Subscription-based — they used to have a standalone version that was connecting to Dropbox, and it felt like a betrayal when they dropped it.
Good, reliable software, but I have trouble trusting companies that took a lot of venture capital and are pressured to grow.

Bitwarden

Open Source — being FOSS at the very least protects it somewhat from the company dying or introducing unreasonable prices.
- It has a server-side clone, Vaultwarden, that can be easily self-hosted for $0 — but be warned, the third-party audits do not apply to it.
- The company may not keep their Open-Source policy (see below).
Usable free plan.
Very cheap Premium for solo professionals ($10 / year).
Username generator.
Usable CLI, although a little awkward, and lacking Touch ID support, AFAIK. You need jq and general wizardry with Unix command-line tools for it to be usable.
Send text or files.
Translated in more languages — e.g., Romanian.
Usable Passkey support (but see below).
The settings for URL matching are more flexible and work on Android as well (vs 1Password, which is experiencing issues).

As downsides for Bitwarden:

Deplorable UI — there’s no other way to describe it; I tried getting my father used to it, but it was mission impossible.
- You have to use both the desktop app and the web vault because both suck in different ways, and neither is sufficient, so you have to throw the CLI in the mix for common maintenance tasks.
No usable offline support — you can disconnect from the network, but it becomes read-only, and the UI is confusingly letting you edit items, and you can lose those edits.
Managing organizations is less than ideal. E.g., if you move items in an organization, you can only change their ownership afterward by cloning them, and you can only do that on the web UI.
The browser extension fails to auto-complete TOTP codes, you have to rely on those codes being copied in the clipboard.
Almost no usable keyboard shortcuts — i.e., Cmd+Shift+Y in the browser, for activating the extension, is unusable because it doesn’t let you do anything via the keyboard next. Their documentation should warn people that the handful of available shortcuts are not usable in keyboard-only flows. The desktop app is really bad at it, e.g., when editing an item, you can’t even press Cmd/Ctrl+S for save, or Esc for cancel, like in every other app. This isn’t just a nice to have, for people with visual impairment being an accessibility disaster.
Managing attachments is very unintuitive. And you can’t view attached PDFs without downloading them.
The browser extension is difficult to configure, e.g., to use Touch ID, although this has more to do with it being more conservative when asking for permissions.
Passkey support isn’t as flawless, I bumped into some issues with it when generating passkeys.
CLI utility is very slow; it’s as if it interrogates the server on every command, yet it still needs bw sync for the synchronization of the latest changes.
CLI utility does not have integration with the desktop app’s biometrics login, so you end up with a BW_SESSION in your bash/zsh profile, making it insecure, or you have to introduce your master password every time, which is annoying.
Exports (for backups or migrations) don’t contain attachments, and you need separate exports for organizations — makes it unintuitive and hard to backup your data. And most of their documentation articles on exports don’t mention that organizations need separate exports.
On Android, the UI can become very slow, if the server is slow — at the time of writing, when using the vault.bitwarden.eu server, the app was unusable for me due to how slow it was, with Android sometimes complaining that the app should be killed, but it worked significantly better with my self-hosted Vaultwarden instance. So the app is very dependent on the server’s latency, and doesn’t have good cache management.
On Android, the UX for auto-completion is not ideal, with the accessibility draw-over being frequently in the way.
Bitwarden took VC capital as well, and while it’s Open-Source nature may protect the project’s future…
Their new Secrets Manager is, apparently, not fully open source and its UI couldn’t be used by Vaultwarden. This at least points to the very real possibility that the company may pull a bait and switch in the future, like so many other companies that build their popularity via FOSS licensing.

I’ve tried self-hosting Vaultwarden, the clone, via my Docker-enabled server, and it was effortless to install, and the running process very efficient. I’m impressed, that’s how all FOSS projects meant for self-hosting should be. Furthermore, I want more stuff built with Rust + SQLite. The Fediverse should take notes 😉

For me, Bitwarden’s Open-Source nature almost makes up for the above deficiencies. Almost, but not quite. I can see myself using Bitwarden, if I were solo, but I’m trying to get my family to use password managers, and right now the UX makes that difficult. I’ll try it again next year.

The article 1Password vs. Bitwarden first appeared on alexn.org.

Cryptographically Strong Random on the JVM

Alexandru Nedelcu — Tue, 13 Aug 2024 08:17:53 +0000

When generating random numbers for certain use-cases, such as when generating keys / IDs, it’s recommended for the random function to be “cryptographically strong”. Otherwise, attackers could predict random values, enabling serious security vulnerabilities.

Instances created via Java’s Random, or ThreadLocalRandom, or Scala’s Random are not cryptographically strong. How such random functions usually work is that they use a “pseudo-random number generator” algorithm that starts with a seed initiated from the current timestamp. Therefore, if you can guess the timestamp and know the algorithm, you can predict the entire sequence of pseudo-random numbers.

It’s probably good to avoid them by default, unless being insecure (and fast) is the requirement. Prefer the use of SecureRandom.

import java.security.SecureRandom

final var secureRandom = new SecureRandom()
// ...
secureRandom.nextInt()

SecureRandom supports several algorithms, see list.

new java.security.SecureRandom().getAlgorithm()
// Output: NativePRNG

The default on my machine is NativePRNG. This should mean that nextBytes uses /dev/urandom (shouldn’t block), but generateSeed() uses /dev/random, so it can block. You can specify an algorithm explicitly:

final var secureRandom = SecureRandom.getInstance("NativePRNGNonBlocking");

Depending on the algorithm used, these calls can do I/O that can block the current thread in case the random stream has less entropy than requested (see Wikipedia entry).

What about multi-threading? SecureRandom instances should be thread-safe; however, the implementation could do expensive synchronization, so I find it better to distribute the contention:

// Java
import java.security.SecureRandom;

public class ThreadLocalSecureRandom {
  private static final int COUNT = 100;
  private static final SecureRandom[] INSTANCES = new SecureRandom[COUNT];

  static {
    for (int i = 0; i < COUNT; i++) {
      INSTANCES[i] = new SecureRandom();
    }
  }

  public static SecureRandom current() {
    var threadId = Thread.currentThread().getId();
    var index = Math.floorMod(threadId, COUNT);
    return INSTANCES[index];
  }
}

Note that UUID.randomUUID is cryptographically strong, using SecureRandom under the hood. This also means that it shares the issues of SecureRandom — the small possibility that the thread underneath gets blocked when generating random UUIDs. This only happens under certain conditions, however, when we generate UUIDs, we generate a lot of them, in horizontally scaled nodes being started as Docker instances, so the likelihood of hitting this in production increases.

In our Scala project, we have this helper that redirects the generation of new UUIDs to threads that can be blocked (by semantically marking the operation as “blocking”):

// Scala code
import cats.effect.IO
import java.util.UUID

object UUIDUtils {
  /**
    * `UUID.randomUUID` is a risky operation, as it can block the current thread.
    * This function redirects such calls to the thread-pool meant for blocking IO.
    */
  def generateRandomUUID: IO[UUID] =
    IO.blocking(UUID.randomUUID())
}

This can be easily done with Kotlin’s Coroutines as well:

// Kotlin code
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import java.util.UUID

suspend fun generateRandomUUID(): UUID = 
    withContext(Dispatchers.IO) {
        UUID.randomUUID()
    }

The article Cryptographically Strong Random on the JVM first appeared on alexn.org.

On Advertising and Tracking

Alexandru Nedelcu — Mon, 12 Aug 2024 08:46:00 +0000

Ads on 𝕏 (formerly Twitter) can be blocked. You can use uBO, uBO Lite, or Brave. And on Android, the web interface is a decent alternative to their app, and can be used as a PWA (I recommend Brave for Android, not Firefox). This, of course, is a cat and mouse game. For instance, Facebook has made it hard for ad-blockers to keep up, they probably have a team of engineers working on anti-ad-blocking tech; uBO still works, but I wonder for how long. For the most part, ad-blocking in the browser works well.

My wish is to write more and to publish more on my blog, and less on social media. This may mean more noise for you, my dear reader. I try maintaining a useful set of tags, so for example, you may want only my articles on Programming or Scala. As a PRO-tip, many feed readers can filter feeds by tags; personally, I use Newsblur, which has excelent filtering.

Last night, I was watching “House of the Dragon” on MAX (former HBO MAX), and they’ve introduced ads in their “Basic” plan. I did not mind an ad for cleaning products, although a little obnoxious, but I did mind their ad for a sports betting app. “Recorder”, an online publication from Romania, has recently covered a vast international online fraud operating in Eastern Europe, gaining access to people’s bank accounts via social engineering, and stealing their money. The way they reach people is via Google and Facebook ads, linking to deep fakes of public figures. And I’ve seen those ads with my own eyes, in my own Facebook feed, I’ve reported them repeatedly, and Facebook rejected my reports.

Ads are now, unfortunately, a vehicle for malware, scams, or services that are deceptive and barely legal, which is why many secretly wish for the entire business model to burn to the ground. Also, ads are very annoying. So the answer to this question should be settled already:

Should we block ads?

Unfortunately, there are second-order effects to blocking ads…

The free web was built on ads. Yes, there was a web before ads, but speaking as someone who has been on the Internet since the late 90s, it was a shitty web. You may not like it, but some online services are costly to host. A service like YouTube isn’t possible without a sustainable business model, and it’s really difficult to build YouTube alternatives because Google can only make it work at their huge scale. And peer-to-peer alternatives are a romantic notion that will just not happen. Therefore, a completely ads-free YouTube would be a service inaccessible to the poor. Even if your government started collecting taxes for keeping certain Internet services free, that could mean poorer countries may get blocked from those services. I remember being blocked from a BBC show because I’m not a UK citizen. And wanting the Internet accessibility to depend on whomever is in power is foolish.

The fact of the matter is that, in many ways, advocating for an ads-free Internet is currently advocating against the notion that information wants to be free.

Ad-blocking, just like piracy before it, hurts alternatives and helps incumbents. People used to pirate Microsoft Windows, or Adobe Photoshop, back in the day, even if alternatives such as Linux or Gimp were more than suitable for the average Joe’s needs. Companies can’t compete with free alternatives, unless they establish a monopoly. And that’s precisely what happened, current day monopolies having been greatly helped by piracy. It’s hard nowadays to be an indie game or software maker, which is why I can’t be so harsh against DRM-enabled distribution platforms. We helped the rise of DRM and of centralized distribution channels by pirating software.

Just as with piracy, blocking ads is keeping users from trying ads-free alternatives. People are less likely to try Mastodon or Bluesky, if their 𝕏 feed isn’t filled with annoying ads, and the issue is that they are still contributing to that platform’s success by engaging or distributing content. “Voting with your wallet” no longer works, and then we start wishing for some government to step in and break monopolies that we helped create.

Ad-blockers are contributing to the death of the open Web. You should have noticed that the web is less popular on mobile phones. Services push people to use native apps. For example, the Reddit app is being forced on users, despite the web interface being more than adequate.

And the reason is simple — you can’t block ads in apps. If your Pi-hole-based solution works right now for some use-cases, it won’t work forever. Pi-hole can’t work for 1st party ads, and apps can use their own DNS-Over-HTTPS endpoints.

The web works so well because the user agent is ours, acting on our behalf. We are in control, and the browser is the perfect sandbox. But with all that power comes great responsibility, and the trend that has been happening is companies driving users away from the web browser and into privacy-invasive apps. This is why browsers have tried addressing privacy-concerns in ways that don’t invalidate ads-driven business models, or that breaks the web, such as Chrome’s Privacy Sandbox initiative, or Firefox with its privacy-preserving attribution API. Such initiatives have been met with hostility by the extremely online laptop class, without them suggesting any alternative, and with the discourse highlighting that for them the issue isn’t with privacy at all.

Good and bad reasons to block ads

Blocking ads is legitimate if you want to protect yourself or others from scams, malware, or privacy-invasive tracking. Personally, I think I can protect myself, although absolutely everyone can fall for scams, it’s just an issue of resources and the right timing. And I certainly would like to protect my son or my father from scams or malware. Yes, I block all the ads that I can block.

Blocking ads is immoral if you just want access to free shit.

YouTube has a quite reasonably priced Premium option. If you’re not paying for YouTube Premium, and you’re blocking their ads, you’re just an entitled freeloader that wants to get other people’s work for free. I’d say the same thing for 𝕏, except that I think their Premium+ tier isn’t reasonably priced — OTOH, if you’re not paying at least for “Basic”, then you’re a freeloader, and with 𝕏 this is more problematic, as you could contribute content to the Fediverse; and instead you’re just giving 𝕏 your attention, while rambling about how Elon Musk is driving it into the ground.

Many people just want to receive the hard labor of others for free. If you’re one of those people ranting about “late-stage capitalism”, I’m sorry to tell you comrade, but even if the revolution comes, you’ll still have to work for a living. If you’re one of those people that hates the business model, wanting to drive it into the ground, then why not go for the subscription-based alternatives? Why not encourage sustainable business models that aren’t ads-based?

Indie developers or corporations are just people that have to work for a living, and by depriving them of revenue, you’re either contributing to their unemployment, or to a world of locked-down devices. Stealing from a corporation is still stealing from people that have to make ends-meet. As much as I hate DRM, or the walled gardens created by Apple, Google, Amazon, Spotify et al., it shouldn’t be difficult to see why creators have preferred the walled gardens.

TLDR — blocking ads is recommended, with the industry certainly needing a wake-up call given current practices, but don’t be a freeloader because you’re contributing to the death of the open web, and to the disappearance of small creators, while growing existing monopolies.

The article On Advertising and Tracking first appeared on alexn.org.

Scala's Future

Alexandru Nedelcu — Wed, 10 Jan 2024 17:37:53 +0000

The Scala logo, in the shape of a hexagon, meant to fit the logos of certain well known libraries, as a hint 😜

As software developers, we invest a lot in our tools. Time, energy, and feelings. We recommend tools to others; we build on top; we belong to a community; we contribute; hence we're eventually becoming stakeholders. And there's no other tool more clamored or risky in our belt than the programming language. Well, gather around, kids, let me tell you two stories from my past with the software industry…

The Dot-Com Bubble

I joined the workforce in the aftermath of the dot-com boom and bust. It was quite a bad time to be a rookie in the software industry, especially in Romania, a country still recovering from the dissolution of the Soviet Union. My first salary was $150 per month. And I remember a depressing article on java.net, advising Java developers that were left without a job to go back to school. The general feeling was that Java was dying, but as times proved, news of its death are always greatly exaggerated. And this was during the time I was learning Java, hoping to replace PHP, which I deeply disliked, but paid the bills. PHP is still one of the most popular programming languages on the web. We’ll never get rid of it.

Déjà vu: We just experienced a bubble bursting. The year 2023 was no longer an employee’s market. Have you noticed all the high-profile layoffs? In 2023, more than 260,000 people lost their job in tech, from over 1100 companies, surpassing the layoffs of 2022. The numbers are shocking because the industry seemed to be invincible during the pandemic. And these were just the numbers visible to Western media; e.g., countries from Eastern Europe don’t have good stats, and markets like China’s are opaque. Certainly, things aren’t as bad as in 2001, with cause for optimism due to the AI hype, but the big layoffs and the downsizing are hard facts you can’t ignore.

When you compare the activity on job websites that most people ignore, or the activity on Reddit, or on X/Twitter, or elsewhere, the activity for most programming languages seems to be declining. On StackOverflow too, and this one is interesting because activity for mainstream languages is also going down due to most interesting questions being answered. It may not be just the tech bubble bursting, but also the pandemic-induced fatigue, with many people being tired of spending time online.

In the Scala community, we sometimes talk of Rust or Kotlin as being shiny lights, but the proxies used for assessing a language’s popularity show that both Rust and Kotlin have stalled growth. It’s funny how perception works. Don’t get me wrong, I think Rust has had and will continue to have good growth and has a great future, and that’s because it’s positioned to compete with C++, which otherwise has no real competition. Rust is not a competitor of garbage-collected languages, because it’s more difficult to use. For instance, it makes FP hard, actually, never mind the interactive development people are used to in dynamic languages. And also, your average code has a better chance of having good throughput in a managed language like Java. Which is counterintuitive for people romanticizing about native code and “zero-cost abstractions”, and you can see how the hype cycles get fueled by people’s hopes and dreams instead of hard facts, which are missing (computer science is still not a science).

This is my personal experience which you should take with a grain of salt. There’s an observational study that disagrees with the point on productivity, from Google, which you should also take with a grain of salt. The field is so devoid of data, that results of surveys from beginners are interpretted as rigurous research, but at least Google tries to have some data, and we should encourage companies to do more of this.

All languages are having setbacks in this climate. Another example is TypeScript, which had some noteworthy libraries that decided to drop it from their codebase, like Svelte, Drizzle, or Turbo. Time may prove that there’s more going on, but if we don’t put any observations we make in context, we may miss the confounders, and it’s a shame, given Scala’s awesome trajectory thus far.

Python’s Plateau of Productivity

I’ve also been a Python and a Ruby developer, and I remember well the onslaught of Ruby on Python’s seemingly fragile market share. Python (and Java) fans were actually afraid of Ruby burgeoning on the scene, and they had good reasons, because the languages seemed to be used for the same kind of problems, and are similar, despite the somewhat different philosophies. Ruby was fresh, Ruby seemed better for DSLs, and it has had “Ruby on Rails” as the killer app, which is a great showcase of what Ruby can do. At that time, Python fans were patting themselves on the back, reminding one another of the hype cycle, hoping for the promised “plateau of productivity”. I’m pretty sure that’s where Li Haoyi got the idea for his Scala article on the same topic, since he’s also a Python developer and fan. And yes, the “plateau of productivity” folks were right.

Python also went through a really slow and painful migration to Python 3 (AKA Python 3000). Version 3 came with a bunch of breaking changes, and because the language is dynamic and dependent on C extensions, it took forever for libraries to migrate. My sore spot was Django. As a side note, Django was also tightly coupled with the native MySQL client, which also made it hard to introduce async I/O via monkey patching, but that’s another story. Python’s reliance on native libraries is both a blessing and a curse. Consider that the last official Python 2.7.x release was in 2020, 12 year after the Python 3 release. And in 2023 it finally reached EOL on Ubuntu, while many organizations still use Python 2, unable to migrate.

In the meantime, Python became the world’s most popular programming language. It had to do with market fitness, of course, being the language of choice for data science, due to projects such as Numpy, Scipy, Matplotlib, TensorFlow, Jupyter, others. And note that Python was not targeted at data science, it just happened organically. And data science is not the only domain Python is good at. For many years, Python has been one of the languages recommended for developing Linux applications, being distributed by default in most Linux distros, with GUI bindings being well-maintained. Python is also very decent for doing web development, with the aforementioned Django also being a killer app, despite its limitations.

Python had a BDFL, and was definitely designed with taste; however its direction went where the community wanted it, by open consensus, not committee. People just used Python because it solved their needs, and it grew from there. Do you know what Python.org’s selling point is? That’s a rhetorical question because you didn’t know before I asked, and nobody cares. Nobody picks Python because of the official marketing brochure.

Also, when Python 3 was released, it was ignored. But in time, more and more new features were added to Python 3, making migration more enticing, such as optional type hints, or couroutines/tasks for async I/O, thus removing the need for previous hacks. When you introduce breaking changes, you need carrots to entice people to migrate despite the pain. Sorry, I don’t make the rules 😉

Déjà vu: Doesn’t the above story sound similar with Scala 3 and the fears some of you have regarding it? Of course, Python isn’t Scala, and the past doesn’t necessarily repeat itself.

However, here are some lessons we can learn from Python’s history:

Community-driven direction, simply by people doing their thing, is good; you don’t need a committee to decide what people build with the language, and complaining that multiple options exist, or complaining about certain libraries or techniques being more popular than others is silly.
Having multiple choices is not bad — Python may pride itself on being a “batteries included” language with an “only one way of doing things” philosophy, but those are just aspirations, as in practice it couldn’t be further from the truth, Python having had multiple well-maintained libraries for doing the same thing, and non-orthogonal language features that were added to cope with the fact that the language can’t have multi-line anonymous functions.
We want good build tools, but build tools don’t make or break a language; here we should look at Python’s easy_install, pip, pyenv, virtualenv, pyenv-virtualenv, scons, etc. or how they haven’t solved yet the problem of depending on native libraries, therefore they need Docker for reproducible builds and deployments; the irony here being that Python needs a standard library with “batteries included” because depending on libraries turns out to be challenging for rookies. Having tools is great, but having fun and ways to get things done is more important.
Languages that break compatibility need new features to entice the community into migrating. When the migration isn’t painless, the greatest competition comes from older versions of the language (i.e., Scala 2.13.x is already quite good, just as Python 2.17 was already quite good). And we need to be open to new developments, while giving constructive feedback; otherwise we sound like a bunch of old farts complaining about the changing times and the kids these days. The FUD around new features may end up discouraging some really cool developments that may even help the status quo.
Migrations, given breaking changes, can take a really long time, but established languages with active projects can afford to wait. What are companies going to do? Throw everything in the bin? That would be silly, and for this reason, incumbents rarely wither away, with very few notable exceptions. Instead, well maintained programming languages age like fine wine, only growing in popularity.

To keep in mind, at least until AGI takes over 🤖

How’s Scala doing?

YMMV, but here are my impressions:

Scala 3 is objectively a better language, with much stronger compatibility guarantees.
Tooling can always improve, but it has some great tooling already that other languages can only dream of (Scala CLI, Metals, IntelliJ IDEA, Scalafmt, Scala.js, etc.).
It has several ecosystems of libraries that are mostly community-driven (Typelevel, com-lihaoyi, ZIO, etc.), distributing the risks; I mean, holy cow, the community may seem small, but it’s still super productive, and this isn’t a top-down cathedral (e.g., dotNET), it’s a bazaar.
Corporate support can be better, but we should also be thankful for Lightbend’s contributions, for VirtusLab stepping up, and other languages with their communities would be lucky to have Scala Center. If this is what an academic language looks like, I think I want more academic languages.
Scala is a mainstream language and was used in many projects that aren’t going away.
Scala is the most popular FP language, and it has some of the best resources for practicing and learning FP. It’s also one of the most Python-like languages I’ve used. There’s no doubt in my mind that it will continue to be in demand.

It’s not all roses; tools could be improved, libraries could be more stable, jobs may be harder to find these days, but Scala developers are among the best paid on average, and the projects I see are quite exciting. I tried to jump ship just before the pandemic, to startups doing JavaScript, TypeScript, and Python. It was awful, and maybe I was just unlucky, but Scala jobs still have this je ne sais quoi that keeps me hooked. Perhaps it’s because we aren’t doing dumb web UIs to a database all the time, which also explains the attraction of some towards Rust 🤷‍♂️

How do I feel about Scala? I feel quite good. You see, I get bored easily, and I have a track record of jumping from tech stack to tech stack, just because. On the job, I’ve used PHP, Java, C#, C++, Perl, Python, Ruby, JavaScript, TypeScript, and I’ve also learned and played in my free time with many others, including the darlings du jour. With Scala, I did not have time to be bored, either because of involvement in FOSS, or because of the awesome projects I was hired to work on. It’s a language that has grown with me, and the ecosystem always provides something new to learn, or some new library to get excited about, while matching my aspirations. And I’m not alone. If you take a look at the history of old-timers in the Scala community, you’ll notice a clear progression in their coding style, as it’s a language and ecosystem that grows with you. I always find it amazing how the same guy that worked on Scalatra, later worked on Http4s, and that’s the kind of evolution you seldom see elsewhere. I’m confident Scala 3 will give us new opportunities to learn, grow and build.

The “scalable” language, indeed 😍

The article Scala's Future first appeared on alexn.org.

Update NextDNS with a Scala CLI script

Alexandru Nedelcu — Wed, 10 Jan 2024 11:14:14 +0000

Today I was reminded how awesome Scala is for scripting, via Scala CLI. And it goes beyond having “batteries included”.

Problem: I have a NextDNS account, as my DNS provider, for privacy and for blocking ads. My work computer has VPN software on it, that overrides the Wi-Fi’s DNS servers in order to access internal company resources. But I still want to configure NextDNS in my browser, and the problem is that my Chromium browser doesn’t fall back to the system DNS when I configure DNS-over-HTTPS.

Solution: a Scala script that updates the NextDNS configuration with overrides for the internal resources that I care about, as NextDNS has an HTTP API that I can use.

The following script uses Scala 3, with 4-spaces for significant indentation. It’s executable, and running it requires just having Scala CLI installed.

#!/usr/bin/env -S scala-cli shebang

//> using scala "3.3.1"
//> using toolkit latest
//> using lib "com.monovore::decline:2.4.1"

import cats.syntax.all.given
import com.monovore.decline.*
import sttp.client4.quick.*
import upickle.default.*
import java.net.InetAddress

val domainsToCleanUp = List(
    "corporate.net"
)

val toUpdate = List(
    "confluence.corporate.net",
    "kibana.corporate.net",
    "elk.corporate.net",
    "elk.uat.corporate.net",
    //...
)

def run(apiKey: String, profileId: String, dryRun: Boolean) =
    case class GetResponse(data: List[Map[String, String]])
        derives ReadWriter

    lazy val getProfile =
        val resp = quickRequest
            .get(uri"https://api.nextdns.io/profiles/$profileId/rewrites")
            .header("X-Api-Key", apiKey)
            .send()
        upickle.default.read[GetResponse](resp.body.toString)

    def post(name: String, content: String) =
        println(s"Adding $name (ip: $content)")
        if (!dryRun) then quickRequest
            .post(uri"https://api.nextdns.io/profiles/$profileId/rewrites")
            .header("X-Api-Key", apiKey)
            .header("Content-Type", "application/json")
            .body(upickle.default.write(
                Map(
                    "name" -> name,
                    "content" -> content
                )
            ))
            .send()

    def patch(id: String, name: String, content: String) =
        println(s"Updating $name (ip: $content)")
        if dryRun then quickRequest
            .patch(uri"https://api.nextdns.io/profiles/$profileId/rewrites/$id")
            .header("X-Api-Key", apiKey)
            .header("Content-Type", "application/json")
            .body(upickle.default.write(
                Map(
                    "content" -> content
                )
            ))
            .send()

    def delete(id: String, name: String) =
        println(s"Deleting $name")
        if !dryRun then quickRequest
            .delete(uri"https://api.nextdns.io/profiles/$profileId/rewrites/$id")
            .header("X-Api-Key", apiKey)
            .send()

    def skip(name: String, content: String) =
        println(s"Skipping $name (ip: $content)")

    for name <- toUpdate do
        val address = InetAddress.getByName(name)
        val ip = address.getHostAddress.nn

        getProfile.data.find: entry =>
            entry.get("name").contains(name)
        match
        case None =>
            post(name, ip)
        case Some(entry) if !entry.get("content").contains(ip) =>
            patch(entry("id"), name, ip)
        case Some(_) =>
            skip(name, ip)

    for domain <- domainsToCleanUp do
        val toDelete = getProfile.data.filter: entry =>
            entry.getOrElse("name", "").endsWith(domain) &&
            !toUpdate.contains(entry("name"))
        for entry <- toDelete do
            delete(entry("id"), entry("name"))

object Main extends CommandApp(
    name = "nextdns-vpn-update",
    header = "Update NextDNS's rewrites based on the current DNS (corporate VPN)",
    main =
        val apiKey = Opts
            .option[String]("api-key", help = "NextDNS API key")
            .orElse(Opts.env[String]("NEXTDNS_API_KEY", help = "NextDNS API key"))
        val profileId = Opts
            .option[String]("profile-id", help = "NextDNS profile ID")
            .orElse(Opts.env[String]("NEXTDNS_PROFILE_ID", help = "NextDNS profile ID"))
        val dryRun = Opts
            .flag("dry-run", help = "Dry run").orFalse

        (apiKey, profileId, dryRun).mapN(run)
)

Of note, the requirements for such a script:

Doing HTTP requests;
Composing and parsing JSON documents;
Parsing command-line arguments.

My script depends on Scala Toolkit, which is a set of libraries that can take care of doing HTTP requests (via sttp) and parsing JSON (via upickle). The script also depends on decline, my favorite library for parsing command line arguments. I don’t need to install these separately, as Scala CLI takes care of managing these dependencies (unlike some of the popular scripting languages).

The script can be executed directly, if you have Scala CLI installed, as it includes a shebang. Just paste the above in a file on your PATH, like ~/bin/nextdns-vpn-update.scala. Then make it executable:

chmod +x ~/bin/nextdns-vpn-update.scala

And Scala’s Metals, which I use with VS Code, is just wonderful for such scripts, with auto-completion, GH Copilot and everything. It took me at most 15 minutes to write and test this.

Python and Ruby are definitely dethroned for all my future scripting needs.

The article Update NextDNS with a Scala CLI script first appeared on alexn.org.

New Year Resolutions

Alexandru Nedelcu — Mon, 01 Jan 2024 09:21:44 +0000

A new year is upon us, and it's customary to make resolutions. The problem with new year resolutions is that we tend to forget them in about a week. So, how to make resolutions that stick?

For example, I’d like to lose weight and be stronger. You may not have the same goals, but you probably have similar ones, and the same thought process applies.

In summary:

Focus on habits, not goals, consistency is key;
Pick easy to achieve daily habits;
Use “habit stacking”;
Have clarity about what you’re doing by measurement;
Control your environment;
Look for the root cause of bad habits.

Focus on habits

While goals are useful for thinking about a destination, for one, they are too abstract and not actionable. “Losing weight” is too abstract, because you end up focusing on a measurement given by the scale, however, that number is just a proxy for things you want. I gained more than 20 Kg (44 pounds) since the pandemic, and I personally want to lose weight because my belly fat is making it uncomfortable for me to tie my shoes, I also snore loudly since gaining weight, and I fear sleep apnea, my allergic asthma has gotten worse, I’m uncomfortable when doing activities that I liked, such as riding my bike, and finding clothes that fit has gotten harder. And all of these side effects are having a negative impact on interactions with my family and friends.

Note that I went through multiple episodes of losing weight and gaining it back, because my genetics work against me, as I have an increased appetite. I’m also not a professional. Therefore, take this advice with a grain of salt.

One of the reasons for why diets fail is because the measurement of success is the weight itself, as seen on the scale, and not the habits that lead to losing weight and then maintaining it via a healthy lifestyle. If you set a goal like, “by this date I want to be 70 Kg (154 pounds)”, you’re setting yourself up for failure. Crash diets are terrible because they don’t lead to healthy habits, quite the contrary, the implementation resembles an eating disorder, and you can’t wait to get it over with and return to your old habits. And then you regain all the weight, the infamous yo-yo effect.

Focusing on habits is harder, because you need some knowledge, however, people underestimate the impact of small periodic investments that pay dividends. You don’t need to change much about your daily routine, and having smaller daily goals makes it easier to achieve them consistently. Consistency is key.

Make habits easy

I want to be more active. Setting a daily goal of 10,000 steps isn’t always achievable, and is also relying on technology, even if it’s the ideal, but setting a daily goal of going for at least one walk, or doing more house chores, is doable. When working from home, you could even walk around the room while on a call, which is how many managed to stay active during the pandemic. You can also look for opportunities to walk more. If you drive a car for your daily commute, then you may be able to park further from your destination.

I want to be stronger, to grow my muscle mass. A cheap way to get started is calisthenics, i.e., using your own body weight for strength training. Push-ups are a great exercise. Committing to 20 push-ups per day may be too much, either due to lack of time or strength. The habit should be something so easily achievable that you’re left with no excuse for not doing it daily. You can commit to 2 push-ups per day. And if you lack the strength for push-ups, you can do wall push-ups. You can do more than 2 push-ups per day, but anything extra is a bonus. Your daily goal should be to do at least 2 push-ups per day, with no progression, until the end of time.

Stack habits

You can have a habit of doing a couple of push-ups whenever you pour a glass of water or a cup of coffee. Drinking more water is a good habit, that may need some effort on its own, but it’s a great excuse to get up from your desk regularly, which prevents back pain. Or you could do some push-ups after you brush your teeth, or after some other daily habits that you have.

I’d also like to learn to swim better and have that as a weekly practice. I already have a habit of driving my son to his swimming lessons, which he practices 2 times per week, for an hour per session. During that time, I can also hit the nearby gym’s swimming pool. The expended effort doesn’t matter, I’ll be happy if I can get in 10 minutes of swimming, but do so consistently.

The above are examples of habit stacking. If you already have a habit, you can use that as the prompt for a new habit.

Measure

As any professional worth their salt will tell you, regular exercise doesn’t really help with losing weight, although it helps in managing it. You still need to eat fewer calories than you expend, and regular exercise can increase your appetite. Our body has evolved to adapt to higher energy expenditure, either via appetite, or via slowing down our metabolism at rest (e.g., less fidgeting, less body heat), which is also why crash diets are terrible, because the experience is terrible and motivation is fleeting. Strength training may help, in time, by increasing your BMR, but that too isn’t enough. A good diet is about getting away with a modest daily caloric deficit, fooling the body into thinking that it got enough, making the whole experience bearable long term.

There are multiple ways of achieving modest caloric deficits. One way is to restrict the food groups you eat. In the past, I tried the “ketogenic diet”, which works because you increase the protein, while decreasing palatability. It was a terrible experience for me. I no longer have the strength to do anything like it, even if I’d like to focus more on protein, and vegetables.

We need clarity about what we are doing on a daily basis, in order to break bad habits.

For dieting, what I’m doing right now is to track my daily caloric intake. This may not be a long-term habit, or an easy one, however, I need to calibrate my intuition of what to prefer eating, while ensuring that I eat enough. People struggling with obesity tend to underestimate the calories they eat, and that eating impulse needs to be more rational, as “eating when hungry, stopping when full” doesn’t work well for us. AFAIK, most people that are overweight are often puzzled by how they got there, attributing it to a “slow metabolism” instead of their own behavior, like eating highly-processed, highly-palatable food that’s high in calories and salt. Personally, I’m under no illusions. Also, whenever I go on a diet, my motivation being really high, I tend to eat too little, until real hunger kicks in, the brain going into panic mode, and then it’s game over, as my brain won’t be satisfied until it regains that weight. Contrary to common wisdom, tracking calories is best for bringing insight into my behavior and for ensuring that I eat enough, and less for restricting calories.

Actual caloric restriction is easier with simpler rules like: eat 2 or 3 meals per day, at approximately the same time; only one plate of food per meal, plus salad; include plenty of protein and vegetables in each meal; if you’re going to snack, snack on fruits.

Note that measurement is all fine and dandy, unless it’s stressful. For this reason, I don’t weigh myself. And it’s probably a bad idea to track calories if you’re recovering from an eating disorder. But if you can take the emotions out of measuring and analyzing data, then it can be a great tool for understanding your behavior and for spotting trends.

Control your environment

For dieting, once you figure out that you snack too much, you can remove those snacks from your vicinity, replacing them with a bowl of fruits. If you feel hungry, and an apple isn’t appeasing, that’s a cue that you’re just bored, and not hungry.

This isn’t to say that you should never eat those delicious snacks you like, but we need to do the right thing easier, and the unhealthy thing harder. This goes for other bad habits, like browsing social media, which can be useful, but maybe you’re doing it too much, and you don’t really need to have social media available on your phone, as you’re not losing anything by delaying it until you’re in front of your computer.

Look for root causes

Unhealthy snacking is a good opportunity to think about other problems we may have, emotional or otherwise. Maybe food is a coping mechanism for stress.

If stress is the prompt for snacking, we’re in trouble. We need to find other ways of coping with stress, like meditation, decluttering our lives, changing our job, etc. The pandemic and war at our borders have been huge sources of stress for me, and I’m only now recovering from their emotional impact. A great way to cope with stress seems to be exercise, and in the future I hope to be the kind of person that picks exercise and meditation over food when stressed.

Stress may be hard to avoid, but understanding that it’s the cause of your bad habits is great, because you can then look for ways of coping with it that doesn’t involve food, or doom-scrolling social media, bad habits that are a feedback loop leading to more stress.

Happy New Year!

For this year, my wish for us is to achieve our resolutions. And we’ll start by making them achievable, by focusing on long-lasting behavior changes 💪

The article New Year Resolutions first appeared on alexn.org.

December Adventure update, thoughts on Rust

Alexandru Nedelcu — Tue, 12 Dec 2023 10:00:00 +0000

This is an update to my december adventure, in which I took it upon myself to learn the Rust programming langauge.

First, I’ve participated in the Advent of Code for the first 10 days, solving problems in Scala, then translating them to Rust. Unfortunately, day 10 got too complicated. I solved part 2 via “ray casting,” an idea that I got early in my thinking, but detecting line intersections is easier said than done. What happened is that I had to detect patterns in the map’s corners, in order to do correct counting, and this happened by debugging on the large input file, which is very impractical. This isn’t unlike other days in which the full specs or corner cases are visible only in the large input file, with the problems being underspecified. Day 10 took 4 hours of my time, and because the problems are increasing in difficulty, I’m finding it harder and harder to find time for a Scala to Rust translation.

My December Adventure, to learn Rust, is progressing well. I’ve made little progress on the books, but solving AoC issues turbocharged my learning for simple tasks related to data structures. Translating AoC solutions made me somewhat familiar with vectors, iterators, hash-maps, plus some useful small libraries, like regex or itertools. I also got a recommendation to use rust-clippy, a linter for Rust that can make some very useful recommendations. I wish we had something like it for Scala, too.

More impressions on Rust

Compared with Scala, Rust has been annoying to work with, the former being more productive for AoC’s problems. However, the point of reference for Rust aren’t garbage collected languages like Scala, but rather C/C++. From that point of view, Rust seems surprisingly productive. It’s not a managed language, and that’s what makes it interesting.

Rust is not an FP language, either. One source of confusion is that Rust’s immutability is a (transitive) property of variables, not of data structures. You can mutate any data structure in Rust by taking ownership. E.g., strings can be mutated, even ones that were previously owned by immutable variables. This is very unlike (impure) FP languages like Scala, F#, OCaml, others, where var or let mutable doesn’t allow you to change the stored data structure. A string in these languages is immutable, regardless of what the variable holding it says. This is required for correctness, as mutability of strings would break the implementation of hash-maps & others.

Rust is a language built for mutability. The borrow checker makes mutation much safer, by disallowing read-only references to exist while the data structure is being mutated. For instance, you’re not allowed to mutate a string while it’s being used as a key in a hash-map. But that’s all there is to its supposed “immutability by default”, a way to control who has read and write access.

Persistent/immutable collections, in Rust, don’t make much sense, for example. Sharing structure, to avoid cloning, is inefficient without a good garbage collector (as you basically have to work with reference counting), and it’s also not in the language’s character.

I do wonder how productive Rust is for real projects. For now, I don’t have any good ideas of where to apply it. I still have much to learn for building simple web services, as that requires async stuff, plus I’d like some problems for which Rust is particularly good at. Maybe I’ll start with some basic CLI tools.

The article December Adventure update, thoughts on Rust first appeared on alexn.org.

December Adventure: Learning Rust

Alexandru Nedelcu — Mon, 04 Dec 2023 11:04:09 +0000

Postcard by publisher Brück & Sohn in Meißen.

This December I’m off to a great personal adventure in programming. Everyone can have their own fun December Adventure. You pick something you want to do, or maybe learn, and you do a little of it everyday, as long as it involves some coding.

The idea is related to the Advent of Code (AoC), which can be a bit much for many. You can, instead, have your own adventure. I am participating in AoC this year, for now, as long as it is easy, mostly because it feels like good fun. And doing it as a group, with the Scala community on Discord, helps. But my real goal for December is to learn Rust.

Why Rust?

Rust is an interesting programming language. It will never replace Scala for me, because it’s not as productive, but the truth is, managed languages (with a heap managed by a garbage collector) will never replace C/C++, and that’s the niche Rust falls into:

Cross-language reusable libraries. C is king here, due to its stable ABI, e.g., libraries for cryptography, a field for which wheel reinvention is a terrible idea. C is also the secret to Python’s popularity, since many libraries have parts implemented with C under the hood (e.g., Numpy/Scipy).
Real-time systems; the JVM is great at throughput, but like all GC platforms, it can have unpredictable pauses that make it unsuitable for real-time systems (despite the awesome low-latency GC implementations, like ZGC); you probably don’t want a GC-managed language for operating the brakes on your car, and even soft-real-time systems, like high-frequency trading or game engines, may be better served by C/C++/Rust.
High-performance CLI utilities. We’ll probably not see projects like ripgrep built on top of the JVM, due to the binary size and the startup time. I also bumped into Helix, a really cool vim replacement. This may change, with GraalVM or Scala/Kotlin Native, but Rust is already used for some pretty kick-ass utilities, and it makes me want to also have some fun.
Projects like the Linux kernel, Firefox, or Servo; these projects are lower-level, and will never accept contributions in your favorite JVM language (or .NET or Go or OCaml) for obvious reasons.
Rust is seemingly embeddable everywhere, because it has the advantage of a small runtime and easy integration with C. For example, it’s the first choice for WebAssembly, you can build Node.js plugins with it, the Godot game engine has unofficial Rust bindings, and it may not shine for building GUIs, but it does have usable GTK and Cocoa bindings (beta quality, but active), which is more than Java can say (in fairness, JavaFX is pretty cool, and Swing is still a workhorse, but it always leaves something to be desired).
Rust has a blog dedicated to building an operating-system, as it’s that kind of language, and I don’t know what other use cases are cooler than that.
The community is seemingly very productive, with many fun projects, resembling that of Scala. For example, if a language doesn’t have its own community-driven game engines, even if immature, it’s not the kind of language that people use for fun. Fortunately, Rust passes (see Amethyst).

This doesn’t mean that I’m giving up on Scala. I predict that it will remain my favorite programming language, by far, but I’ve always been a polyglot, and I prefer not being tied down by the tools that I’m using.

Dec 1-4

On Friday, we had a national holiday, so we had a prolonged weekend. In addition to visiting relatives, or participating in the first days of AoC challenges, I managed to sit down and go through the official Rust book. I’m now at Chapter 7, which gave me the confidence to solve Advent of Code exercises with it. What I’m doing is that I first use Scala, because I think well in it, then translate the code to Rust.

My AoC repository has both Scala and Rust samples, and it’s interesting, because you can compare how my Scala looks and feels, compared with Rust. As an initial impression, Rust is more verbose and awkward. I still don’t fully master its memory ownership system. And this language prefers (safe) mutation. Working with persistent data structures is not idiomatic or comfortable, you’re better off embracing mutation where needed. But for a language with manual memory management, it’s actually quite usable, high-level and safe.

I’ve also started reading “Learn Rust with Entirely Too Many Linked Lists”. I feel confident in saying that if I go through this book, I’ll grok Rust’s memory management. But, it gave me brain damage straight from the first chapter, so it doesn’t look like an easy book for beginners.

On Blogging

I want to write more on this blog, because I have thoughts, and social media is still bad. I want to publish at least once per week, preferably more. For example, I’m going to publish more articles about how my December adventure is going.

This means that I may publish articles that may be shorter, or lower quality.

Let the fun begin~

The article December Adventure: Learning Rust first appeared on alexn.org.

In Scala 3, use 4 Spaces for Indentation

Alexandru Nedelcu — Wed, 08 Nov 2023 18:17:04 +0000

Scala’s coding style advised to use 2 spaces of indentation, but that was before Scala 3’s optional braces, which introduces significant indentation. It’s time for an upgrade of the coding style.

Wisdom

The Linux kernel uses indentation with 8 characters, the reasoning being readability and keeping cyclomatic complexity low. And I quote:

Rationale: The whole idea behind indentation is to clearly define where a block of control starts and ends. Especially when you’ve been looking at your screen for 20 straight hours, you’ll find it a lot easier to see how the indentation works if you have large indentations.

Now, some people will claim that having 8-character indentations makes the code move too far to the right, and makes it hard to read on an 80-character terminal screen. The answer to that is that if you need more than 3 levels of indentation, you’re screwed anyway, and should fix your program.

Using 2-spaces of indentation, in Scala, wasn’t terrible despite the high cyclomatic complexity. This is because Scala is very expression-oriented, very type safe, and very functional. This means that the compiler can catch a lot of errors, and due to tools such as exhaustive pattern matching, tail-recursive or higher-order functions, we rarely miss branches. Using 2-spaces for indentation was already in a gray area, however. And with significant indentation, it definitely moved into the red zone.

You may think this isn’t serious. I’m seeing indentation errors in blog articles, which weren’t there before. And I’ve made a couple of mistakes myself. Sometimes, the compiler catches it, but that may not happen in case of effectful expressions with an irrelevant Unit result. And for it to be human-readable, it has to be in your face, unambiguous, even after 10 hours of looking at your screen.

FAQ

Why not use tabs?

That battle is already lost:

A vast majority of style guides advise against tabs;
For some forsaken reason, people like vertical alignment in Scala, which won’t work with tabs (ASCII-art, goddamn);
People who use spaces make more money;
I’m not even sure if Scalafmt supports tabs, can’t find the setting.

BONUS — using spaces will piss off the tabs-people:

Tabs versus Spaces (open on YouTube.com)

What are similar languages doing?

Here is the style guide for other languages with the off-side rule:

Make: tabs (8 characters);
Python: 4 spaces;
F#: 4 spaces;
Elm: 4 spaces;
Haskell: 2–4 spaces;
YAML: 2 spaces;
CoffeeScript: 2 spaces.

This may be my selection bias, but based on these numbers, I could draw a graph with how much people like these languages 😜

NOTE: Python is different from Scala, as it has the “only one way of doing things” philosophy, it’s statement oriented, and it doesn’t allow significant indentation in the middle of expressions. This is the reason for why Python never got multi-line lambdas. Python is conservative, and the average cyclomatic complexity is lower.

Tooling & Configuration

Add an EditorConfig file, in the root of your project, like this:

# EditorConfig is awesome: https://EditorConfig.org

# top-most EditorConfig file
root = true

# Unix-style newlines with a newline ending every file
[*]
end_of_line = lf
insert_final_newline = true

[*.{scala,sbt,sc}]
indent_style = space
indent_size = 4
max_line_length = 100

Configure the Scalafmt plugin, with the following settings in .scalafmt.conf:

runner.dialect = scala3

maxColumn = 100
indent.main = 4

# Recommended, to not penalize `match` statements
indent.matchSite = 0

newlines.source = keep
rewrite.scala3.convertToNewSyntax = true
rewrite.scala3.removeOptionalBraces = yes
rewrite.scala3.insertEndMarkerMinLines = 5

For your IDE, you should also install extensions for doing syntax coloring for indentation level:

When significant indentation enters your life, add some rainbow coloring 🌈

Sample

This is a sample from one of my personal/small projects, integrating directly with JDBC. Your mileage may vary, and going from 2-spaces to 4-spaces requires adjustments, but your eyes will thank you for it:

import cats.effect.IO
import cats.effect.kernel.Resource
import com.zaxxer.hikari.HikariDataSource
import com.zaxxer.hikari.HikariConfig
import cats.effect.kernel.Resource.ExitCase
import java.sql.Connection
import java.sql.PreparedStatement

final case class Database(
    config: JdbcConnectionConfig,
    pool: HikariDataSource
):
    def connection: Resource[IO, Connection] =
        Resource.make:
            IO.blocking:
                pool.getConnection().nn
            .flatTap: c =>
                IO(c.setAutoCommit(true))
        .apply: c =>
            IO.blocking(c.close())

    def transaction: Resource[IO, Connection] =
        for
            conn <- connection
            _ = conn.setAutoCommit(false)
            _ <- Resource.makeCase(IO.unit):
                case ((), ExitCase.Succeeded) =>
                    IO.blocking:
                        conn.commit()
                        conn.setAutoCommit(true)
                case ((), ExitCase.Canceled | ExitCase.Errored(_)) =>
                    IO.blocking:
                        conn.rollback()
                        conn.setAutoCommit(true)
        yield conn

    def withConnection[A](block: Connection ?=> IO[A]): IO[A] =
        connection.use(ref => block(using ref))

    def withTransaction[A](block: Connection ?=> IO[A]): IO[A] =
        transaction.use(ref => block(using ref))

    def query[A](sql: String)(block: PreparedStatement => A)(using Connection): IO[A] =
        IO.blocking:
            summon[Connection].prepareStatement(sql).nn
        .bracket: stm =>
            IO.interruptible:
                block(stm)
            .cancelable:
                IO.blocking(stm.cancel())
        .apply: stm =>
            IO.blocking(stm.close())

end Database

object Database:
    def connect(config: JdbcConnectionConfig): Resource[IO, Database] =
        for
            pool <- createPool(config)
        yield Database(config, pool)

    def createPool(config: JdbcConnectionConfig): Resource[IO, HikariDataSource] =
        Resource.apply(IO.blocking:
            val cfg = HikariConfig().tap: it =>
                it.setDriverClassName(config.driver)
                it.setJdbcUrl(config.url)
                for
                    u <- config.user
                    p <- config.password
                do
                    it.setUsername(u)
                    it.setPassword(p)
                end for
                it.addDataSourceProperty("cachePrepStmts", "true")
                it.addDataSourceProperty("prepStmtCacheSize", "250")
                it.addDataSourceProperty("prepStmtCacheSqlLimit", "2048")
                // Instructs HikariCP to not throw if the pool cannot be seeded
                // with an initial connection
                it.setInitializationFailTimeout(0)

            val res = HikariDataSource(cfg)
            val cancel = IO.blocking(res.close())
            (res, cancel)
        )
end Database

Go forth and spread the word! 📢 This stuff matters 💪

The article In Scala 3, use 4 Spaces for Indentation first appeared on alexn.org.

OOP classes vs Higher-order Functions (HOFs)

Alexandru Nedelcu — Mon, 25 Sep 2023 07:30:22 +0000

What's the difference between OOP classes and Higher-order Function (HOF)?

Here’s an abstract OOP class:

abstract class MyComponent[I, O] {
  // Abstract methods
  def foo(input: I): I
  def bar(input: I): O

  // Inherited implementation
  final def apply(input: I): O = {
    // Implementation doesn't matter
    val i = foo(input)
    bar(i)
  }
}

Here’s an equivalent “final” OOP class:

final class MyComponent[I, O](
  foo: I => I,
  bar: I => O
) {
  def apply(input: I): O = {
    // Same implementation as above
    val i = foo(input)
    bar(i)
  }
}

And here’s an equivalent Higher-order Function (HOF):

def process[I, O](foo: I => I, bar: I => O)(input: I): O = {
  // Same implementation as above
  val i = foo(input)
  bar(i)
}

There is no difference between these 3 implementations, conceptually they are the same thing, exposing the same complexity. Maybe you heard that OOP inheritance is bad, but if you’re implementing a HOF, or a final class like the above, it’s the same thing.

Well, OK, the abstract class has some gotchas, and all of them are about breaking the Liskov Substitution Principle.

Gotcha 1: Overriding of non-final methods

Let’s say that our process method is non-final. And we override it such that foo is never called:

class Blah[I, O] extends MyComponent[I, O] {
  def foo(input: I): I = ???
  def bar(input: I): O = ???

  override def process(input: I): O =
    bar(i)
}

Well, this would break an implicit, possibly undocumented contract that clients may rely on. Imagine that this class generates pseudo-random numbers, and overriding process would make the output non-random.

This breaks the “Liskov substitution principle” because the derived class no longer behaves like the inherited class. This is why methods should be non-final with intention. So make methods final by default. C++, C# and Kotlin actually got this right, as in these langauges you have to be explicit about which methods can be overriden (e.g., virtual in C++/C#, open in Kotlin).

Gotcha 2: Discrimination based on runtime type (instanceOf)

instanceOf checks and OOP downcasting makes it possible to break encapsulation and discriminate, client-side, based on what “type of component” you have. instanceOf on open OOP classes (non-sealed) is an antipattern that’s going to come back to haunt you. Imagine we have this real-world example:

class ParsingException(message: String) extends Exception(message)

But we need to add information to caught exceptions, like an operationName: String, so we define this:

class ExceptionWithDetails(
  operationName: String,
  cause: Throwable,
) extends Exception(cause.getMessage, cause)

Well, what happens if another part of our code does something like this:

def clasifyException(e: Throwable): ExceptionType =
  e match {
    case _: ParsingException => Input
    case _: TimeoutException => ServiceNotAvailable
    case _                   => Unknown
  }

In this case, wrapping ParsingException in ExceptionWithDetails would break the logic of clasifyException. Using instanceOf checks on Throwable clearly means that you know something about the implementation. And this is an encapsulation violation. I’m not being entirely fair here, because Java’s Exceptions are meant to be used like this, and at least they have a standard cause in their API that you can use for wrapping and unwrapping exceptions.

Here’s another example:

trait ExecutionContext {
  def execute(r: Runnable): Unit
  def reportFailure(e: Throwable): Unit
}

final class SafeExecutionContext(ec: ExecutionContext) {
  def execute(r: Runnable): Unit =
    try ec.execute(r)
    catch { case e: Throwable => ec.reportFailure(e) }

  def reportFailure(e: Throwable): Unit =
    ec.reportFailure(e)
}

This way of adding behavior by wrapping classes is actually a best practice (composition over inheritance!). But clearly, by wrapping any ExecutionContext into a SafeExecutionContext, access to the runtime type of the wrapped implementation becomes inaccessible, so any discrimination you make via instanceOf is error-prone. The “composition over inheritance” principle makes instanceOf checks error-prone.

It would have been great if Scala had a linting option that banned instanceOf checks (pattern matching included) on open types (with the possibility to override, of course).

So, HOFs versus OOP classes?

I haven’t talked about side effects. Classes are great at encapsulating shared mutable state. But capturing mutable state in closures is trivial, so that’s not an argument.

On what to choose … it doesn’t matter.

The problem all of them have is one of complexity. Those functions, taken as parameters, need clearly defined contracts, otherwise the user can unknowingly break such implicit contract. Which is why in code-bases that do this, people basically copy-paste the original call-site. Defining a set of functions with a clearly defined contract can clearly be done … with better types, with a provided TCK, or via documentation. Having some algebraic reasoning in there would be great. But it’s not trivial.

Passing functions as arguments, or implementing them while inheriting from an abstract class, is most often done for implementation reuse. I prefer the reuse of well encapsulated components with APIs that make them usable in multiple scenarios, instead of the reuse of implementation details. Or in other words, if we are talking of OOP, I prefer the reuse of final classes that don’t take functions as parameters. And when functions take functions as parameters, well, they need clearly defined laws. But granted, this isn’t an easy thing to do, as it’s like telling people to design good microservices.

Of course, many times, you just gotta do what you gotta do. Implementation reuse is better than no reuse.

The article OOP classes vs Higher-order Functions (HOFs) first appeared on alexn.org.

Post Once, Syndicate Everywhere (POSSE)

Alexandru Nedelcu — Thu, 21 Sep 2023 09:42:45 +0000

This is my cat, Zuzi. She's completely black with green eyes. A rescued girl. My sweetheart.

I’m a geek, and a software developer. I want to be close to my peers, wherever they meet online. If this means reactivating former social media accounts, so be it. Therefore, I’m implementing POSSE, again 😕

In January, I made the decision to delete my Twitter account. Prior to this, I also deleted my Facebook account. I remained on LinkedIn and on Mastodon. Note that I still have a Twitter/X account, but it’s been there only to prevent username squatting, and all my followers and posts are gone.

Now, 9 months later, I have to reconsider. My Mastodon experience has been good, actually, except for the tiny little problem that many from the Scala community aren’t there, or from the Java or Kotlin community for that mater. And, for example, I really wanted to see Scala Days news. I missed the conference, sadly, but I can still tune online. In fairness, engagement has been going down across the board, communities now being fragmented. The age of centralized social media is over, people going back to Internet silos. Which is probably good, but this means more effort to connect.

I also dislike my Mastodon presence for some of the same reasons I disliked Twitter and Facebook, although at a much smaller scale: I still see triggering posts, making me worry about shit that doesn’t matter. It’s still a waste of time. It’s still dumb and angry. Maybe the real problem is me, and being “extremely online” isn’t good.

What I really want to do is to:

Work on interesting stuff;
Write on my blog about it (which is making me less stupid, not more);
Connect to people that I like, and that share my interests, regardless of their chosen platform;
Always link to my website for anything that I want to communicate into the void.

This strategy is called “POSSE: Publish on your Own Website, Syndicate Everywhere” (archive). I hope it works this time, as it wouldn’t be the first time I try it. Unfortunately, I often have weak moral principles or the attention span and the strong will of a child 🤷‍♂️

Cheers,

The article Post Once, Syndicate Everywhere (POSSE) first appeared on alexn.org.

Java Volatiles

Alexandru Nedelcu — Mon, 19 Jun 2023 10:57:50 +0000

In Java and other JVM languages, “volatile” reads and writes are part of the concurrency toolbox. But adding volatile on variables can be a mistake. This is a poorly understood concept. Let’s gain a better understanding.

Misconceptions

Some common ones:

Volatile variables ensure “visibility” — but the notion is misleading, because it’s not about visibility per se, as all updates to variables are eventually seen by all other threads, and marking a variable as volatile doesn’t necessarily publish changes to that variable any sooner (although in fairness, x86 CPUs have stronger ordering guarantees than ARM CPUs in this regard, so there may be differences of behavior for normal variables, depending on what you’re doing);
Volatile variables are not for “synchronization” — which is false, as volatile reads and writes are a form of synchronization, even if the guarantees are weaker than when using locks or when using compare-and-set (CAS).
Volatile variables prevent CPU caching — depending on the JVM, their use may or may not reduce caching, the problem being that CPU caching is complex, and also the JVM is free to optimize away volatile semantics, the guarantees of the JMM¹ are higher-level than that, and if you’re thinking of CPU caching, you’re probably doing it wrong.

As a somewhat more advanced mental model, volatile variables introduce memory barriers², however:

Memory barriers are only introduced when the JVM writes, and then reads from the same volatile variable (in a pair, just like you must synchronize on the same lock/monitor for synchronization to work);
The JVM is free to optimize away memory barriers, as long as the ordering guarantees are preserved; for example, that’s what the JVM’s escape analysis³ is for.

Guaranteed Ordering

What volatile reads and writes give us are happens-before relationships⁴, AKA guaranteed ordering. Say, for example, we have the following declaration:

class State {
  public String value1 = null;
  public String value2 = null;
  public volatile boolean hasValues = false;
}

The producer could do this:

state.value1 = "Hello";
state.value2 = "World";
state.hasValues = true;

From the point of view of the producer, value1 and value2 are always updated before hasValues. The question is, what happens when we look at the evolution of these variables from another thread? If hasValues wouldn’t be volatile, we could see value1 or value2 as null.

The consumer could do this:

// Waits until the volatile write is seen
while (!state.hasValues) {
  // spin
}
// We are now guaranteed that `value1` and `value2` are set:
System.out.println(
  "Got values: " + state.value1 + " " + state.value2
);

Q: Why does this work?

This works because the JVM guarantees that once the update to our volatile variable is seen on the consumer thread (hasValues == true), then all previous updates are also seen. It’s a happens-before relationship, the updates to value1 and value2 being guaranteed to happen before hasValue = true.

At the hardware level, this means that a “memory barrier” could be introduced, which prevents the CPU from reordering instructions, but thinking of memory barriers in this case would be error-prone.

Q: Could we see those values updated before the volatile update?

Of course. Worth noting that unless we wait for hasValue, we get no guarantees on what we’ll see to all updates prior to it. We could see value1 and/or value2 as null, randomly. This is why thinking about “CPU caching” is entirely misleading.

Q: Don’t we need to mark as `volatile` all variables?

No, making value1 and value2 to be volatile can be worse than useless, at it may lead to poorer performance, due to extra memory barriers (forced orderings). The JVM may try to understand what you’re trying to do, but it’s not that smart. And adding volatile all over the place does not make your code thread-safe. Volatile variables only make your code thread-safe if you can reason about the ordering.

Synchronization sample

You may think that we can’t make code thread-safe just via volatile reads and writes, but you’d be wrong, as there are cases in which we can. Which can be very useful for performance or predictability (lock-free algorithms).

The following is an inefficient (SPSC) queue, see the comments. If you can understand why this sample is correct, you understand volatile variables:

import java.util.Objects;

/**
 * This is a "single-producer, single-consumer" (SPSC) queue.
 * 
 * SPSC means that we can have a single producer (a thread using `push`),
 * working concurrently with a single consumer
 * (another thread using `pop`).
 * 
 * This queue is not thread-safe if we have multiple concurrent
 * producers, or multiple concurrent consumers.
 */
public class SPSCRiskyQueue {
  /**
   * State machine with possible states:
   * wait-for-producer | wait-for-consumer
   *
   * The state is "volatile" because it is used to order updates.
   * Remove "volatile" and witness the test fail.
   */
  private volatile String state = "wait-for-producer";
  private int pushedCount = 0;
  private A value = null;

  public void push(A value) throws InterruptedException {
    while (!Objects.equals(state, "wait-for-producer")) {
      spin();
    }
    this.value = value;
    pushedCount++;
    state = "wait-for-consumer";
  }

  public A pop() throws InterruptedException {
    while (!Objects.equals(state, "wait-for-consumer")) {
      spin();
    }
    // System.out.println("Read elements: " + pushedCount);
    final var value = this.value;
    state = "wait-for-producer";
    return value;
  }

  public long getPushedCount() {
    return pushedCount;
  }

  private void spin() throws InterruptedException {
    // For some reason this piece of logic introduces more ordering,
    // which makes the test more non-deterministic (it takes longer to
    // see it fail).
    Thread.onSpinWait();
    if (Thread.interrupted())
      throw new InterruptedException();
  }
}

Try removing the volatile annotation from hasValues, see how this test works:

import java.util.concurrent.CountDownLatch;

/**
 * Code for TESTING...
 */
class Main {
  public static void runTest() throws InterruptedException {
    final var queue = new SPSCRiskyQueue();
    final var count = 10000;
    final var state = new long[] { 0L };

    final var prepareLatch = new CountDownLatch(2);
    final var concurrentStartLatch = new CountDownLatch(1);

    final var producer = new Thread(() -> {
      try {
        prepareLatch.countDown();
        concurrentStartLatch.await();
        for (int i = 0; i < count; i++) {
          queue.push(i);
        }
      } catch (InterruptedException ignored) {}
    });

    final var consumer = new Thread(() -> {
      try {
        prepareLatch.countDown();
        concurrentStartLatch.await();
        for (int i = 0; i < count; i++) {
          state[0] += queue.pop();
        }
      } catch (InterruptedException ignored) {}
    });

    producer.setDaemon(true);
    producer.start();
    consumer.setDaemon(true);
    consumer.start();
    // Ready, set, go!
    prepareLatch.await();
    concurrentStartLatch.countDown();

    producer.join(3000);
    consumer.join(3000);

    final var sum = state[0];
    final var pushed = queue.getPushedCount();
    System.out.println("Sum: " + sum + "; pushed: " + pushed);
    if (sum != (count * (count - 1)) / 2 || pushed != count) {
      throw new IllegalStateException("Concurrency test failed");
    }
  }

  public static void main(String[] args) throws InterruptedException {
    for (int i = 0; i < 10000; i++) {
      runTest();
    }
  }
}

Java Memory Model is the set of multi-platform guarantees that Java makes for multithreaded code. ↩
Memory barriers are CPU instructions that can force the ordering of updates. The JSR-133 Cookbook can be good for gaining an understanding, but take it with a grain of salt. ↩
Escape analysis is a method for determining the scope of objects, in order to enable subtle optimizations. If the lifetime of an object is limited to the current method or thread, then Java may avoid intrinsic synchronization, or heap allocations. Java’s capabilities have been more limited than what’s possible. ↩
Reasoning about concurrency can be in terms of happens-before relationships, i.e., the order in which events are observed. ↩

The article Java Volatiles first appeared on alexn.org.

Scala 3 Significant Indentation Woes: Sample

Alexandru Nedelcu — Tue, 06 Jun 2023 11:48:10 +0000

Here's a fairly straightforward Scala 3 sample, using significant indentation. Can you spot the compilation error?

//> using scala "3.3.0"

def sequence[A](list: List[Option[A]]): Option[List[A]] =
  list.foldLeft(Option(List.newBuilder[A])):
    (acc, a) =>
      acc.flatMap: xs =>
        a.map: x =>
          xs.addOne(x)
    .map(_.result())

Here’s the compilation error:

[error] ./sample.scala:9:10
[error] Found:    List[A]
[error] Required: scala.collection.mutable.Builder[A, List[A]]
[error]     .map(_.result())
[error]          ^^^^^^^^^^
Error compiling project (Scala 3.3.0, JVM)
Compilation failed

Here’s the corrected code:

//> using scala "3.3.0"

def sequence[A](list: List[Option[A]]): Option[List[A]] =
  list.foldLeft(Option(List.newBuilder[A])):
    (acc, a) =>
      acc.flatMap: xs =>
        a.map: x =>
          xs.addOne(x)
  .map(_.result())

FYI, this cannot happen in Python, because Python does not allow breaking expressions on multiple lines like that:

class MyList:
  def __init__(self, list):
    self.list = list
  def map(self, f):
    return MyList([f(x) for x in self.list])

# Doesn't parse
MyList([1, 2, 3])
  .map(lambda x: x + 1)

The error is:

  File "/tmp/sample.py", line 9
    .map(lambda x: x + 1)
IndentationError: unexpected indent

Maybe no indentation?

# Doesn't parse
MyList([1, 2, 3])
.map(lambda x: x + 1)

  File "/tmp/sample.py", line 9
    .map(lambda x: x + 1)
    ^
SyntaxError: invalid syntax

Yikes! What Python does is to make expressions unambiguous by requiring the escaping of line endings via a backslash:

MyList([1, 2, 3]) \
  .map(lambda x: x + 1)

Scala’s syntax keeps being compared with Python’s, however, they couldn’t be more different, as Python has had a very strict policy to avoid ambiguity, and even rejected multi-line lambdas for this reason.

I’m also begining to feel that 2 spaces for indentation are not enough 🙊

The article Scala 3 Significant Indentation Woes: Sample first appeared on alexn.org.

Scala 3 Enums

Alexandru Nedelcu — Thu, 25 May 2023 11:15:29 +0000

In Scala, how do we model enumerations of values? What are the possible issues? How does Scala compare with Java? What are the changes in Scala 3?

Here’s a simple enumeration in Java:

// Java code
enum Colors {
  Red, Green, Blue
}

Adding keys as strings is also possible:

// Java code
enum Colors {
  Red("RED"), Green("GREEN"), Blue("BLUE");

  private final String value;

  Colors(String value) {
    this.value = value;
  }

  public String getValue() {
    return value;
  }
}

Java should also do exhaustiveness checks in its new switch statement (JEP 354):

//JAVA 17+

Colors myColor = Colors.Green;
String label = switch (myColor) {
  case Red -> Colors.Red.getValue();
}

Which should fail to compile with the following error:

error: the switch expression does not cover all possible input values
  String label = switch (myColor) {

The equivalent in Scala 2.x was supposed to be the Enumeration class:

// Scala 2.x code
object Colors extends Enumeration {
  val Red, Green, Blue = Value
}

// Or you can assign custom keys to each value:
object Colors extends Enumeration {
  val Red = Value("RED")
  val Green = Value("GREEN")
  val Blue = Value("BLUE")
}

// The created type is `Colors.Value`
val myColor: Colors.Value = Colors.Green

The generated type is Colors.Value. This can create issues, because it’s an inner type of Enumeration that’s being erased at runtime, therefore it’s not the equivalent of a Java enum (think of doing serialization with Jackson):

classOf[Colors.Value]
// val res: Class[Colors.Value] = class scala.Enumeration$Value

object Size extends Enumeration {
  val S, M, L, XL, XXL = Value
}

// Yikes!
classOf[Size.Value] == classOf[Colors.Value]
// => true

Importantly, Enumeration provides values and withName as utilities:

Colors.values // List(Red, Green, Blue)
Colors.withName("Red") // Colors.Value = Red

Another big problem with Enumeration is that it’s incapable of doing exhaustiveness checks when pattern matching. Nowadays, this is less safety than what Java provides. The following code compiles without warnings:

myColor match {
  case Colors.Red => println("Red")
}

The replacement that almost everyone used was “sealed” traits/classes, which can encode tagged union types:

// Scala 2.x code

// The `extends Product with Serializable` boilerplate
// is needed to eliminate type-inference junk
sealed abstract class Color(val value: String)
  extends Product with Serializable

object Color {
  case object Red extends Color("RED")
  case object Green extends Color("GREEN")
  case object Blue extends Color("BLUE")
}

This is a bit more verbose, but we can do exhaustiveness checks:

val myColor: Color = Color.Green
myColor match {
  case Color.Red => println("Red")
}
// myColor match {
// ^
// On line 2: warning: match may not be exhaustive.
// It would fail on the following inputs: Blue, Green

This is super useful, especially with -Xfatal-warnings enabled.

One problem here is that we no longer have an enumeration of all available values, and this can make things difficult:

object Color {
  // ...

  // Error-prone, since we need to ensure that
  // we list them all:
  val values: Set[Color] = Set(
    Red,
    Green,
    Blue
  )

  def apply(value: String): Option[Color] =
    values.find(_.value == value)
}

There are libraries that can help, such as Enumeratum.

#!/usr/bin/env -S scala-cli shebang -q

//> using scala "2.13.10"
//> using lib "com.beachape::enumeratum:1.7.2"

import enumeratum._
import enumeratum.values._

sealed abstract class Color(val value: String)
  extends StringEnumEntry

object Color extends StringEnum[Color] {
  case object Red extends Color("RED")
  case object Green extends Color("GREEN")
  case object Blue extends Color("BLUE")

  val values: IndexedSeq[Color] = findValues
}

println(Color.values) // Vector(Red, Green, Blue)
println(Color.withValueOpt("Red")) // Some(Red)

Thankfully, Enumeratum is compatible with Scala 3. However, I found a flaw. The following code compiles without warnings:

import enumeratum.values._

sealed abstract class Color(val value: String)
  extends StringEnumEntry

object Color extends StringEnum[Color] {
  case object Red extends Color("RED")
  case object Green extends Color("GREEN")
  case object Blue extends Color("BLUE")

  // Yikes! This is most likely a bug.
  final case class Other(r: Int, g: Int, b: Int)
    extends Color(s"OTHER($r,$g,$b)")

  val values: IndexedSeq[Color] = findValues
}

// No `Other` in this list:
println(Color.values) // Vector(Red, Green, Blue)

The problem is, of course, that it invalidates our assumptions about how to serialize and deserialize this. Let’s say we’ve been using Circe, and we already had the codecs defined:

//> using lib "io.circe::circe-core:0.14.5"
//> using lib "io.circe::circe-parser:0.14.5"

import io.circe._
import io.circe.parser._
import io.circe.syntax._

implicit val colorEncoder: Encoder[Color] =
  Encoder[String].contramap(_.value)

implicit val colorDecoder: Decoder[Color] =
  Decoder[String].emap(Color.withValueOpt(_).toRight("Invalid color"))

When we add our case class, this obviously doesn’t work:

(Color.Other(1,2,3): Color).asJson() // "OTHER(1,2,3)"

decode[Color]("\"OTHER(1,2,3)\"") // Left(DecodingFailure at : Invalid color)

People don’t necessarily realize that an enumeration is powered by Enumeratum, or what the limitations are.

To fix this, we could find another library, or we could write our own macro. But I dislike macros, I think we’d do just fine with a runtime error:

#!/usr/bin/env -S scala-cli shebang -q

//> using scala "2.13.10"
//> using lib "org.scala-lang:scala-reflect:2.13.10"

import scala.reflect.runtime.{universe => ru}
import scala.reflect.runtime.{currentMirror => cm}

def findValues[T: ru.TypeTag]: Set[T] = {
  val tpe = ru.typeOf[T]
  val clazz = tpe.typeSymbol.asClass
  if (!clazz.isSealed) {
    throw new AssertionError(s"Type $tpe is not sealed")
  }
  clazz.knownDirectSubclasses.map { sym =>
    if (sym.isModule)
      cm.reflectModule(sym.asModule)
        .instance
        .asInstanceOf[T]
    else if (sym.isModuleClass)
      cm.reflectModule(sym.asClass.module.asModule)
        .instance
        .asInstanceOf[T]
    else
      throw new AssertionError(
        s"Direct subtype of $tpe is not an object: $sym"
      )
  }
}

sealed abstract class Color(val value: String)
  extends Product with Serializable

object Color {
  case object Red extends Color("RED")
  case object Green extends Color("GREEN")
  case object Blue extends Color("BLUE")

  val values = findValues[Color]

  def apply(value: String): Option[Color] =
    values.find(_.value == value)
}

println(Color.values) // Set(Red, Green, Blue)
println(Color("RED")) // Some(Red)

Now, if you try to add a new case class, you’ll get a runtime error when trying to access the object Color:

object Color {
  // ...
  case class Other(r: Int, g: Int, b: Int)
    extends Color(s"OTHER($r,$g,$b)")
}
// This will now throw a java.lang.AssertionError:
// 'Direct subtype of Color is not an object: class Other'
Color.values

This is not a macro. You can write a macro, if you want, using mostly the same logic. Like all things with Scala 2’s compile-time reflection, and macros, the knownDirectSubclasses is buggy, and this code breaks in some instances, like when defining Color as an inner class. Also, the macro may be error-prone in other ways. See this StackOverflow answer for a hint on how to do that.

In Scala 3, we can easily define a macro:

#!/usr/bin/env -S scala-cli shebang -q

//> using scala "3.2.2"

inline def findValues[T](using
  m: scala.deriving.Mirror.SumOf[T]
): Set[T] =
  allInstances[m.MirroredElemTypes, m.MirroredType].toSet

inline def allInstances[ET <: Tuple, T]: List[T] =
  import scala.compiletime.*

  inline erasedValue[ET] match
    case _: EmptyTuple => Nil
    case _: (t *: ts)  =>
      summonInline[ValueOf[t]].value.asInstanceOf[T] :: allInstances[ts, T]

//-------------------------------------------------------------------------
//...
  
sealed abstract class Color(val value: String)
  extends Product with Serializable

object Color:
  case object Red extends Color("RED")
  case object Green extends Color("GREEN")
  case object Blue extends Color("BLUE")

  // Uncomment this to get a compile-time error:
  // case class Other(r: Int, g: Int, b: Int)
  //   extends Color(s"OTHER($r,$g,$b)")

  val values = findValues[Color]

  def apply(value: String): Option[Color] =
    values.find(_.value == value)

println(Color.values) // Set(Red, Green, Blue)
println(Color("RED")) // Some(Red)

This is an actual macro, and it will throw a compile-time error if Color is not a sealed trait, or if we try to define a case class Other.

Of course, in Scala 3, we already have the new enums, so the above code is only necessary if we want to port our Scala 2 code to Scala 3. In Scala 3, we can just write:

enum Color(val value: String):
  case Red extends Color("RED")
  case Green extends Color("GREEN")
  case Blue extends Color("BLUE")

object Color:
  def apply(value: String): Option[Color] =
    values.find(_.value == value)

println(Color.values.toSet) // Set(Red, Green, Blue)
Color("RED") // Some(Red)

If we try to add a case class to our enumeration, we can, but values will no longer be available:

// BROKEN CODE
enum Color(val value: String):
  case Red extends Color("RED")
  case Green extends Color("GREEN")
  case Blue extends Color("BLUE")
  case Other(r: Int, g: Int, b: Int)
    extends Color(s"OTHER($r,$g,$b)")

object Color:
  def apply(value: String): Option[Color] =
    values.find(_.value == value)

Our Scala 3 compiler will then throw this error:

-- [E006] Not Found Error: -----------------------------------------------------
11 |    values.find(_.value == value)
   |    ^^^^^^
   |    Not found: values
   |
   | longer explanation available when compiling with `-explain`
1 error found

Furthermore, in Scala 3 we can easily define enums that are compatible with Java, by extending java.lang.Enum, making them compatible with Java libraries:

// Scala 3

enum Color extends java.lang.Enum[Color]:
  case Red, Green, Blue

TLDR:
Scala 2 involves a lot of error-prone boilerplate.
Scala 3 is pretty cool in its handling of enums and macros ❤️

It doesn’t seem like much, but we have A LOT of enumerations in our codebase. The new enum is one of my favorite Scala 3 features, as it removes the error-prone boilerplate.

The article Scala 3 Enums first appeared on alexn.org.

Managing Database Migrations in Kotlin

Alexandru Nedelcu — Tue, 02 May 2023 07:32:06 +0000

The database schema should be described as code, in your repository. And you should be able to semi-automatically update your database schema on new deployments. Now in Kotlin, with Gradle and Flyway.

This article is a rewrite of my previous article on the same topic, that was showing code snippets meant for Scala and the sbt build tool. This article is meant for Kotlin (or Java), with Gradle integration, but also making use of Flyway.

We’re going to use Flyway to manage our database migrations, a Java library that’s useful enough.

Before we start, note that Flyway has a Gradle plugin, just like it has an sbt plugin or a Maven plugin. And with something like Spring Boot or Quarkus, you can get out of the box configuration for Flyway / Liquidbase, possibly using those plugins. We are not going to look at such integrations in this article, because they force you into a rigid configuration, project structure, or deployment possibilities. Here are some potential issues with such integrations:

Reusing your database connection settings, specified somewhere else;
Creating an executable JAR that can execute those migrations;
Executing the migrations at application startup (not recommended for serious™️ apps, but always an option);
Running different migration files for different database types, depending on configuration;
Having subprojects, that may be independent, each with their own set of database migrations.

We gain all of this flexibility with some manual wiring that’s only a couple of lines of code.

This article will depend just on Flyway’s API, or in other words, just on flyway-core. We are not using any available integrations with the build tools.

1. PostgreSQL setup

We are going to use PostgreSQL as our sample database. To start an instance, you could use Docker. Here’s a sample docker-compose.yaml:

version: '3.3'

services:
  postgresdb:
    container_name: postgresdb
    image: 'postgres:15-alpine'
    ports:
      - "5432:5432"
    healthcheck:
      test: ['CMD', 'pg_isready', '-U', 'postgres']
    volumes:
      - 'postgresdb-volume:/var/lib/postgresql/data'
    restart: always
    environment:
      POSTGRES_PASSWORD: pass

volumes:
  postgresdb-volume:

Start this instance:

docker-compose -f ./docker-compose.yaml up -d

And create your initial database named my_sample_db (this step is pretty hard to add as part of your migration files, so might as well not do it):

docker exec -it postgresdb /usr/local/bin/psql \
  -U postgres \
  -c "CREATE DATABASE my_sample_db"

2. Initial project setup

Create a new directory and switch to it from the shell:

mkdir migrations-sample
cd migrations-sample/

To start your new Kotlin project (accept all defaults):

gradle init --type kotlin-application --dsl kotlin

We need to specify a configuration file, and it’s going to be our own configuration file, because why not? One way of doing that is to use the Kotlinx Serialization plugin and library, so we’ll need to add it as a dependency.

Edit the file app/build.gradle.kts, and make sure the plugins section looks like this:

plugins {
  kotlin("jvm") version "1.8.21"
  kotlin("plugin.serialization") version "1.8.21"
  application
}

And we’ll need these library dependencies:

dependencies {
  // For managing our database migrations
  // https://github.com/flyway/flyway
  implementation("org.flywaydb:flyway-core:9.17.0")

  // For parsing CLI arguments
  // https://github.com/Kotlin/kotlinx-cli
  implementation("org.jetbrains.kotlinx:kotlinx-cli:0.3.5")

  // For couroutines support; not strictly needed, but it's nice to
  // indicate when blocking I/O needs the thread-pool meant for blocking stuff.
  // https://github.com/Kotlin/kotlinx.coroutines
  implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.7.0-RC")

  // For parsing our configuration file. Using:
  //  - https://github.com/Kotlin/kotlinx.serialization
  //  - https://github.com/lightbend/config (HOCON as the format)
  implementation("org.jetbrains.kotlinx:kotlinx-serialization-hocon:1.5.0")

  // Database driver (JDBC)
  implementation("org.postgresql:postgresql:42.6.0")

  // Flyway has built-in logging, which we can expose via SLF4J/Logback
  implementation("ch.qos.logback:logback-classic:1.4.7")
}

We’re adding logging (via slf4j/logback), and we might want to silence Flyway’s logging for anything that’s unimportant. Let’s also add a logback.xml file to the app/src/main/resources directory:


  
  
    true
    
      [%date{ISO8601}] [%highlight(%level)] [%boldYellow(%marker)] [%logger] [%thread] %cyan([%mdc]) — %msg%n

3. Configuration

Create a new file app/src/main/resources/database.conf with the following contents:

jdbc-connection.main {
  driver = "org.postgresql.Driver"

  url = "jdbc:postgresql://localhost:5432/my_sample_db"
  url = ${?JDBC_CONNECTION_MAIN_URL}

  username = "sample_user"
  username = ${?JDBC_CONNECTION_MAIN_USERNAME}

  password = ${JDBC_CONNECTION_MAIN_PASSWORD}

  migrationsTable = "main_migrations"
  migrationsLocations = [
    "classpath:db/migrations/main/psql"
  ]
}

There are several things to unpack here:

This is our own format, you can define your own, or reuse whatever configuration file you have; in this case the file is using HOCON (a JSON superset), it’s included as a “resource” in the final artefact, and it is allowing for environment variables to override the values;
migrationsTable and migrationsLocations are needed because we may have multiple sub-projects, each with their own (independent) migrations, and we want to execute them all;
We specify the username and the password, but these are the app’s credentials, and are not the user and password used when migrating the DB; I think it’s a security vulnerability to allow the app’s user to modify tables, or create triggers on its own, so this “MAIN” user should have limited permisisons (but you can ignore this “best practice”);

And then, using the kotlinx-serialization-hocon dependency, we can model this as a type-safe data class, and read this file in our own code. Add this file in app/src/main/kotlin/:

package migrations.sample

import com.typesafe.config.Config
import com.typesafe.config.ConfigFactory
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import kotlinx.serialization.ExperimentalSerializationApi
import kotlinx.serialization.Serializable
import kotlinx.serialization.hocon.Hocon

@Serializable
data class JdbcConnectionConfig(
  val url: String,
  val driver: String,
  val username: String,
  val password: String,
  val migrationsTable: String,
  val migrationsLocations: List,
  val migrationsPlaceholders: Map = emptyMap()
) {
  companion object {
    @OptIn(ExperimentalSerializationApi::class)
    suspend fun loadFromGlobal(
      configNamespace: String,
      config: Config? = null
    ): JdbcConnectionConfig =
      withContext(Dispatchers.IO) {
        val rawCfg = config ?: ConfigFactory.load().resolve()
        val cfg = rawCfg.getConfig(configNamespace)
        Hocon.decodeFromConfig(serializer(), cfg)
      }
  }
}

Modeling your app’s configuration in such a type-safe way isn’t necessarily required, and it’s certainly not a very common practice in Java projects. But it’s a pity, as it makes APIs clearer, being a great way to document your configuration in the code itself.

4. Flyway API library integration

Create a new file RunMigrations.kt in app/src/main/kotlin/:

package migrations.sample

import com.typesafe.config.ConfigFactory
import kotlinx.cli.ArgParser
import kotlinx.cli.ArgType
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withContext
import org.flywaydb.core.Flyway
import org.flywaydb.core.api.configuration.FluentConfiguration
import org.flywaydb.core.api.output.MigrateResult
import org.slf4j.LoggerFactory
import kotlin.system.exitProcess

/**
 * Given a JDBC configuration, run the associated DB migrations.
 *
 * NOTE: `adminUsername` and `adminPassword` are different from the
 * credentials specified in `JdbcConnectionConfig`. That's because the
 * "admin" user may be different from the app's user. So if an
 * `adminUsername` and an `adminPassword` are provided, Flyway will
 * use that admin user to execute migrations.
 *
 * Flyway uses "placeholders" that can be used in the SQL migrations.
 * These can be specified in `JdbcConnectionConfig`, but this code
 * also sets 2 special placeholders to use from the
 * `JdbcConnectionConfig` itself: `dbUsername` and `dbPassword`. These
 * can be used to create the app's user as part of the defined
 * migrations.
 */
suspend fun dbMigrate(
  config: JdbcConnectionConfig,
  adminUsername: String?,
  adminPassword: String?
): MigrateResult =
  withContext(Dispatchers.IO) {
    val m: FluentConfiguration = Flyway.configure()
      .dataSource(
        config.url,
        adminUsername ?: config.username,
        if (adminUsername != null) adminPassword else config.password,
      )
      .group(true)
      .outOfOrder(false)
      .table(config.migrationsTable)
      .locations(*config.migrationsLocations.toTypedArray())
      .baselineOnMigrate(true)
      .loggers("slf4j")
      .placeholders(
        config.migrationsPlaceholders +
            mapOf(
              "dbUsername" to config.username,
              "dbPassword" to config.password
            ).filterValues { it != null }
      )

    val validated = m
      .ignoreMigrationPatterns("*:pending")
      .load()
      .validateWithResult()

    if (!validated.validationSuccessful) {
      val logger = LoggerFactory.getLogger("RunMigrations")
      for (error in validated.invalidMigrations) {
        logger.warn(
          """
            |Failed to validate migration:
            |  - version: ${error.version}
            |  - path: ${error.filepath}
            |  - description: ${error.description}
            |  - error code: ${error.errorDetails.errorCode}
            |  - error message: ${error.errorDetails.errorMessage}
          """.trimMargin("|").trim()
        )
      }
    }
    m.load().migrate()
  }

object RunMigrations {
  private suspend fun migrateNamespace(
    label: String,
    config: JdbcConnectionConfig,
    adminUsername: String,
    adminPassword: String
  ): Unit = withContext(Dispatchers.IO) {
    val result = dbMigrate(
      config,
      adminUsername,
      adminPassword
    )
    println("-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=")
    println("Migrating: $label")
    println("------------------------------------")
    println("Initial schema version: ${result.initialSchemaVersion}")
    println("Target schema version: ${result.targetSchemaVersion}")
    if (result.migrations.isNotEmpty()) {
      println("------------------------------------")
      println("Executed migrations:")
      for (migration in result.migrations) {
        println(" - ${migration.version} ${migration.type} ${migration.description}")
      }
    }
    if (result.warnings.isNotEmpty()) {
      println("------------------------------------")
      System.err.println("WARNINGS:")
      for (warning in result.warnings) {
        System.err.println(" - $warning")
      }
    }
    println("------------------------------------")
    if (result.success) {
      println("Successfully migrated: $label!")
    } else {
      System.err.println("ERROR: Failed to migrate $label!")
      exitProcess(1)
    }
  }

  @JvmStatic
  fun main(args: Array) {
    val parser = ArgParser("RunMigrations")
    val adminUsername by parser.argument(
      ArgType.String,
      fullName = "admin-username",
      description = "Admin username for the database. Example: postgres"
    )
    val adminPassword by parser.argument(
      ArgType.String,
      fullName = "admin-password",
      description = "Admin password for the database."
    )
    parser.parse(args)

    runBlocking {
      val config =
        ConfigFactory.load("database.conf").resolve()
      val mainConfig =
        JdbcConnectionConfig.loadFromGlobal(
          "jdbc-connection.main",
          config
        )
      migrateNamespace(
        "main",
        mainConfig,
        adminUsername,
        adminPassword
      )
    }
  }
}

In this code, we only deal with a single database configuration and its associated migrations. But note that we can have multiple database configurations, each with their own migrations, corresponding to different subprojects. You simply add multiple migrateNamespace calls.

5. Adding the SQL migrations

We are going to create files in app/src/main/resources/db/migrations/main/psql. This matches the migrationsLocations defined in the database.conf above (which gets parsed in JdbcConnectionConfig).

Create a new file named V0010__create-user.sql:

CREATE USER "${dbUsername}" WITH PASSWORD '${dbPassword}';
CREATE SCHEMA IF NOT EXISTS sample
  AUTHORIZATION "${dbUsername}";

GRANT
  CONNECT,
  TEMPORARY
ON DATABASE "my_sample_db"
TO "${dbUsername}";

Then create another file named V0020__create-tables.sql and add some nice tables to it:

CREATE TABLE sample.users
(
  id bigint not null generated always as identity primary key,
  email varchar(255) not null,
  password varchar(255) default null,
  timezone varchar(30) not null,
  created_at timestamp with time zone not null,
  updated_at timestamp with time zone not null
);

CREATE TABLE sample.stuff
(
  id bigint not null generated always as identity primary key,
  user_id bigint not null,
  json_data jsonb not null,
  created_at timestamp with time zone not null,
  updated_at timestamp with time zone not null,
  foreign key (user_id) references sample.users(id)
    on delete cascade
    on update cascade
);

GRANT
  SELECT,
  INSERT,
  UPDATE,
  DELETE,
  TRUNCATE
ON ALL TABLES IN SCHEMA sample
TO "${dbUsername}";

6. Gradle configuration

We need the following in build.gradle.kts:

tasks.register("migrate") {
  group = "Execution"
  description = "Migrates the database to the latest version"
  classpath = sourceSets.getByName("main").runtimeClasspath
  mainClass.set("migrations.sample.RunMigrations")

  val user = System.getenv("POSTGRES_ADMIN_USER")
    ?: "postgres"
  val pass = System.getenv("POSTGRES_ADMIN_PASSWORD")
    ?: throw GradleException(
      "POSTGRES_ADMIN_PASSWORD environment variable must be set"
    )
  args = listOf(user, pass)
}

7. Running the migrations

# Needed by the Gradle task
export POSTGRES_ADMIN_PASSWORD="pass"
# Needed by the application (HOCON) config
export JDBC_CONNECTION_MAIN_PASSWORD="pass"

./gradlew migrate

Which will output:

> Task :app:migrate
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Migrating: main
------------------------------------
Initial schema version: 0020
Target schema version: null
------------------------------------
Successfully migrated: main!

BUILD SUCCESSFUL in 2s
3 actionable tasks: 2 executed, 1 up-to-date

8. Pro-tip: unit-test migrations with HSQLDB

You can use something like HSQLDB to unit-test your JDBC-based code. HSQLDB is an in-memory database perfect for tests.

dependencies {
  //...
  testImplementation("org.hsqldb:hsqldb:2.5.1")
}

In your src/test/resources you could have a test.database.conf with a setup like:

include "database.conf"

// Overrides PostgreSQL connection with in-memory HSQLDB
jdbc-connection.main {
  url = "jdbc:hsqldb:mem:MyTestDB;sql.syntax_pgs=true"
  driver = "org.hsqldb.jdbc.JDBCDriver"
  username = null
  password = null
}

Here we are using the PostgreSQL compatibility mode, which isn’t perfect, as it only supports standard RDBMS stuff. Depending on your SQL code, it might be enough. Note that it probably doesn’t work with jsonb columns 🙂 but for simpler schemas it might be enough. Or you could have code specific for HSQLDB by manipulating the migrationsLocations setting to point to a different path:

include "database.conf"

jdbc-connection.main {
  //...
  migrationsLocations = [
    "classpath:db/migrations/main/hsqldb"
  ]
}

And then in your tests you can run those migrations by simply calling that dbMigrate function, and then profit! 🤑

class MyTest {
  @Test fun something() =
    runBlocking {
      val rawConfig =
        ConfigFactory.load("test.database.conf").resolve()
      val jdbcConfig =
        JdbcConnectionConfig.loadFromGlobal(
          "jdbc-connection.main",
          rawConfig
        )
      // Ta da!
      dbMigrate(jdbcConfig)
      //...
    }
}

Your DB API mocks will never be the same again! 😎

Epilogue

For a sample project, checkout this GitHub repository:

sample-projects/kotlin-db-migrations

Enjoy~

The article Managing Database Migrations in Kotlin first appeared on alexn.org.

Kotlin Coroutines to Cats-Effect

Alexandru Nedelcu — Mon, 24 Apr 2023 00:00:00 +0000

Kotlin Coroutines are usually integrated in Java code via Java’s CompletableFuture, but a tighter integration might be possible with Cats-Effect. I played around to see if I can convert Kotlin’s coroutines, built via suspended functions straight to cats.effect.IO. Turns out I could.

The following snippet is an executable script via Scala-CLI:

//#!/usr/bin/env -S scala-cli shebang -q

//> using scala "2.13.10"
//> using lib "org.typelevel::cats-effect::3.4.9"
//> using lib "org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.4"

import cats.effect._
import kotlin.coroutines.{Continuation, CoroutineContext}
import kotlinx.coroutines.{Deferred, _}

import java.util.Collections
import java.util.concurrent.{AbstractExecutorService, CancellationException, TimeUnit}
import scala.concurrent.ExecutionContextExecutorService
import scala.concurrent.duration._
import scala.util.control.{NoStackTrace, NonFatal}

object Main extends IOApp {
  // Sleeping via Kotlin's coroutines
  def kotlinSleep(duration: FiniteDuration): IO[Unit] =
    KotlinCoroutines.runCancelable_ { (_, cont) =>
      // Kotlin suspended function calls...
      kotlinx.coroutines.DelayKt.delay(duration.toMillis, cont)
    }

  override def run(args: List[String]): IO[ExitCode] =
    for {
      _ <- IO.println("Running...")
      fiber <- kotlinSleep(10.seconds).start
      _ <- IO.sleep(1000.millis)
      _ <- fiber.cancel
      _ <- fiber.joinWithUnit
      _ <- IO.println("Done!")
    } yield ExitCode.Success
}

object KotlinCoroutines {
  def runCancelable_(
    block: (CoroutineScope, Continuation[_ >: kotlin.Unit]) => Any
  ): IO[Unit] = {
    runCancelable(block).void
  }

  def runCancelable[A](
    block: (CoroutineScope, Continuation[_ >: A]) => Any
  ): IO[A] = {
    coroutineToIOFactory[A](block, buildCancelToken)
  }

  private def dispatcher: IO[CoroutineDispatcher] =
    IO.executionContext.map { other =>
      kotlinx.coroutines.ExecutorsKt.from(
        new AbstractExecutorService with ExecutionContextExecutorService {
          override def isShutdown = false
          override def isTerminated = false
          override def shutdown() = ()
          override def shutdownNow() = Collections.emptyList[Runnable]
          override def execute(runnable: Runnable): Unit = other.execute(runnable)
          override def reportFailure(t: Throwable): Unit = other.reportFailure(t)
          override def awaitTermination(length: Long, unit: TimeUnit): Boolean = false
        }
      )
    }

  private def coroutineToIOFactory[A](
    block: (CoroutineScope, Continuation[_ >: A]) => Any,
    buildCancelToken: (Deferred[_], DisposableHandle) => Option[IO[Unit]]
  ): IO[A] = {
    dispatcher.flatMap { dispatcher =>
      IO.async[A] { cb =>
        IO {
          try {
            val context = CoroutineContextKt.newCoroutineContext(
              GlobalScope.INSTANCE,
              dispatcher.asInstanceOf[CoroutineContext],
            )
            val deferred = kotlinx.coroutines.BuildersKt.async(
              GlobalScope.INSTANCE,
              context,
              CoroutineStart.DEFAULT,
              (p1: CoroutineScope, p2: Continuation[_ >: A]) => block(p1, p2)
            )
            try {
              val dispose = deferred.invokeOnCompletion(
                (e: Throwable) => {
                  e match {
                    case e: Throwable => cb(Left(e))
                    case _ => cb(Right(deferred.getCompleted))
                  }
                  kotlin.Unit.INSTANCE
                })
              buildCancelToken(deferred, dispose)
            } catch {
              case NonFatal(e) =>
                deferred.cancel(null)
                throw e
            }
          } catch {
            case NonFatal(e) =>
              cb(Left(e))
              None
          }
        }
      }
    }.recoverWith {
      case PleaseCancel =>
        // This branch actually never happens, but it might
        // prevent leaks in case of a bug
        IO.canceled *> IO.never
    }
  }

  private def buildCancelToken(deferred: Deferred[_], dispose: DisposableHandle): Option[IO[Unit]] =
    Some(IO.defer {
      deferred.cancel(PleaseCancel)
      dispose.dispose()
      // Await for completion or cancellation
      coroutineToIOFactory[kotlin.Unit](
        (_, cont) => deferred.join(cont),
        (_, _) => None
      ).void
    })

  private object PleaseCancel
    extends CancellationException with NoStackTrace
}

The article Kotlin Coroutines to Cats-Effect first appeared on alexn.org.

Integrating Akka with Cats-Effect 3

Alexandru Nedelcu — Mon, 17 Apr 2023 08:05:29 +0000

We are using a combination of Akka and Cats-Effect (ver. 3) for building payment processors. Integrating them isn’t without challenges. This post describes some solutions we’ve discovered.

We’ve been using Akka because Akka Cluster and Akka Persistence fitted our needs for data persistence. We’re also using Akka Stream, because our flows are complicated graphs, even though at times it felt overkill and fs2 might have been a better fit. Note that Akka has gone proprietary, and we’re still on the last FOSS version (2.6.20). We might pay up to upgrade to the proprietary license, although I’m rooting for Apache Pekko to become stable.

Starting Actor Systems as a Resource

Cats-Effect’s Resource is one of the secret weapons of Scala. When you have Cats-Effect in your project, it’s best to manage the lifecycle of resources via Resource. Akka also provides its own coordinated-shutdown mechanism, but I recommend going with Resource, due to the ease of use, and the reasoning capabilities.

First, let’s get the dependencies out of the way:

// sbt syntax
libraryDependencies ++= Seq(
  "com.typesafe.akka" %% "akka-actor" % "2.6.20",
  "com.typesafe.akka" %% "akka-actor-typed" % "2.6.20",
  "ch.qos.logback" % "logback-classic" % "1.4.6",
  // FP awesomeness!!!
  "org.typelevel" %% "cats-effect" % "3.4.9",
)

And the imports:

import akka.Done
import akka.actor.{CoordinatedShutdown, ActorSystem => UntypedActorSystem}
import cats.effect.kernel.Resource
import cats.effect.std.Dispatcher
import cats.effect.{Deferred, IO}
import cats.syntax.all._
import com.typesafe.config.Config
import org.slf4j.LoggerFactory
import scala.concurrent.TimeoutException
import scala.concurrent.duration._

To create an actor system in the context of a Resource, a naive approach would be this:

// Version 1 of 3: DO NOT USE THIS
def startActorSystemUntyped(
  systemName: String,
  config: Option[Config],
): Resource[IO, UntypedActorSystem] =
  Resource(IO {
    val system = UntypedActorSystem(
      systemName.trim.replaceAll("\\W+", "-"),
      config = config
    )
    // Here, system.terminate() returns a `Future[Terminated]`
    val cancel = IO.fromFuture(IO(system.terminate())).void
    (system, cancel)
  })

The first problem with this approach is that both Akka and Cats-Effect 3 are creating their own thread-pool. Having too many thread-pools meant for CPU-bound tasks can decrease performance and make problems harder to investigate. Cats-Effect’s thread-pool is optimized for IO and it would be a pity if we wouldn’t use it. Akka’s thread-pool is the old and reliable ForkJoinPool, that’s also used by Scala’s global, but CE’s thread-pool is perfectly adequate for use with Akka as well, due to properly implementing BlockContext.

Note that it’s best if this would be configurable. So here’s version 2:

// Version 2 out of 3: DO NOT USE THIS
def startActorSystemUntyped(
  systemName: String,
  config: Option[Config],
  useIOExecutionContext: Boolean,
): Resource[IO, UntypedActorSystem] =
  Resource(
    for {
      // Fishing IO's `ExecutionContext`
      ec <- Option
        .when(useIOExecutionContext) { IO.executionContext }
        .sequence
      system <- IO {
        UntypedActorSystem(
          systemName.trim.replaceAll("\\W+", "-"),
          config = config,
          defaultExecutionContext = ec,
        )
      }
    } yield {
      // Here, system.terminate() returns a `Future[Terminated]`
      val cancel = IO.fromFuture(IO(system.terminate())).void
      (system, cancel)
    }
  )

There is one more problem, best described as a setting that you’ll often see in an application.conf file:

akka.coordinated-shutdown.exit-jvm = on

This setting tells Akka to forcefully stop the JVM as part of the coordinated shutdown process. And this setting is good, you should have this on. And that’s because Akka can decide to shut down on its own, and afterward it’s probably best to shut down the process, too. One example is when using Akka Cluster, with a split-brain resolver, in which case Akka should shut down the application in case the node is removed from the cluster. If it doesn’t, then the app could be left in a zombie state.

Both Akka and Cats-Effect can add a shut-down hook to the JVM, via Runtime.addShutdownHook. Cats-Effect can do it via its IOApp implementation. Akka does it as part of its coordinated-shutdown mechanism. This triggers the disposal process on System.exit, no matter the reason it happened. And one problem is that both will start to dispose of resources, concurrently. In other words, Akka can shut down the actor system before Cats-Effect has had a chance to shut down all resources depending on that actor system. Which is usually bad.

Another problem I discovered in testing is that in certain scenarios, Akka, on system.terminate(), seems to return a Future[Done] that never completes. Seems to be some sort of race condition, and it’s bad, as it can indefinitely block Cats-Effect’s resource disposal process. In my testing, this seemed to be alleviated if I first waited on a promise completed via a task registered with Akka’s coordinated-shutdown (see below). I don’t know if this code solves it or not, but a “timeout” on system.whenTerminated seems to be a good idea.

// Version 3 out of 3: USE THIS

/** Starts an (untyped) Akka actor system in the
  * context of a Cats-Effect `Resource`, and integrating
  * with its cancellation abilities.
  *
  * HINT: for apps (in `main`), it's best if
  * `akka.coordinated-shutdown.exit-jvm` is set to `on`,
  * because Akka can decide to shutdown on its own. And
  * having this setting interacts well with Cats-Effect.
  *
  * @param systemName is the identifying name of the system.
  * @param config is an optional, parsed HOCON configuration;
  *        if None, then Akka will read its own, possibly
  *        from `application.conf`; this parameter is
  *        provided in order to control the source of
  *        the application's configuration.
  * @param useIOExecutionContext if true, then Cats-Effect's
  *        default thread-pool will get used by Akka, as well.
  *        This is needed in order to avoid having too many
  *        thread-pools.
  * @param timeoutAwaitCatsEffect is the maximum amount of time
  *        Akka's coordinated-shutdown is allowed to wait for
  *        Cats-Effect to finish. This is needed, as Cats-Effect
  *        could have a faulty stack of disposables, or because
  *        Akka could decide to shutdown on its own.
  * @param timeoutAwaitAkkaTermination is the maximum amount of
  *        time to wait for the actor system to terminate, after
  *        `terminate()` was called. We need the timeout, because
  *        `terminate()` proved to return a `Future` that never
  *        completes in certain scenarios (could be a bug, or a
  *        race condition).
  */
def startActorSystemUntyped(
  systemName: String,
  config: Option[Config],
  useIOExecutionContext: Boolean,
  timeoutAwaitCatsEffect: Duration,
  timeoutAwaitAkkaTermination: Duration,
): Resource[IO, UntypedActorSystem] = {
  // Needed to turn IO into Future
  // https://typelevel.org/cats-effect/docs/std/dispatcher
  Dispatcher.parallel[IO](await = true).flatMap { dispatcher =>
    Resource[IO, UntypedActorSystem](
      for {
        // Fishing IO's `ExecutionContext`
        ec <- Option
          .when(useIOExecutionContext)(IO.executionContext)
          .sequence
        // For synchronizing Cats-Effect with Akka
        awaitCancel <- Deferred[IO, Unit]
        // For awaiting termination via coordinated-shutdown,
        // needed as `terminate()` is unreliable
        awaitTermination <- Deferred[IO, Unit]
        logger = LoggerFactory.getLogger(getClass)
        system <- IO {
          logger.info("Creating actor system...")
          val system = UntypedActorSystem(
            systemName.trim.replaceAll("\\W+", "-"),
            config = config,
            defaultExecutionContext = ec,
          )
          // Registering task in Akka's CoordinatedShutdown
          // that will wait for Cats-Effect to catch up,
          // blocking Akka from terminating, see:
          // https://doc.akka.io/docs/akka/current/coordinated-shutdown.html
          CoordinatedShutdown(system).addTask(
            CoordinatedShutdown.PhaseBeforeServiceUnbind,
            "sync-with-cats-effect",
          ) { () =>
            dispatcher.unsafeToFuture(
              // WARN: this may not happen, if Akka decided
              // to terminate, and `coordinated-shutdown.exit-jvm`
              // isn't `on`, hence the timeout:
              awaitCancel.get
                .timeout(timeoutAwaitCatsEffect)
                .recoverWith {
                  case ex: TimeoutException =>
                    IO(logger.error(
                      "Timed out waiting for Cats-Effect to catch up! " +
                        "This might indicate either a non-terminating " +
                        "cancellation logic, or a misconfiguration of Akka."
                    ))
                }
                .as(Done)
            )
          }
          CoordinatedShutdown(system).addTask(
            CoordinatedShutdown.PhaseActorSystemTerminate,
            "signal-actor-system-terminated",
          ) { () =>
            dispatcher.unsafeToFuture(
              awaitTermination.complete(()).as(Done)
            )
          }
          system
        }
      } yield {
        val cancel =
          for {
            // Signals that Cats-Effect has caught up with Akka
            _ <- awaitCancel.complete(())
            _ <- IO(logger.warn("Shutting down actor system!"))
            // Shuts down Akka, and waits for its termination
            // Here, system.terminate() returns a `Future[Terminated]`,
            // but we are ignoring it, as it could be non-terminating
            _ <- IO(system.terminate())
            // Waiting for Akka to terminate via coordinated-shutdown
            _ <- awaitTermination.get
            // WARN: `whenTerminated` is unreliable, hence the timeout
            _ <- IO.fromFuture(IO(system.whenTerminated))
              .void
              .timeoutAndForget(timeoutAwaitAkkaTermination)
              .handleErrorWith(_ =>
                IO(logger.warn(
                  "Timed-out waiting for Akka to terminate!"
                ))
              )
          } yield ()
        (system, cancel)
      }
    )
  }
}

Note that the correctness of complicated apps relies on akka.coordinated-shutdown.exit-jvm being set to on in application.conf. If you feel uneasy about this, note that this can be accomplished programmatically, by doing a manual System.exit in that CoordinateShutdown task:

// NOT NEEDED, prefer to use `application.conf`
CoordinatedShutdown(system).addTask(
  CoordinatedShutdown.PhaseBeforeServiceUnbind,
  "shutdown-actor-system",
) { () =>
  // System.exit will block the thread, so best to
  // run it async, in a fire-and-forget fashion
  val triggerShutdown =
    IO.blocking(System.exit(255))
      .start
      .void

  dispatcher.unsafeToFuture(
    (triggerShutdown *> awaitCancel.get)
      .timeout(timoutAwaitCatsEffect)
      .recoverWith { ... }
      .as(Done)
  )
}

If you do this, it will work OK, as both Cats-Effect and Akka install shutdown hooks. But I feel that this is duplicating the functionality of akka.coordinated-shutdown.exit-jvm, making behavior unclear for those familiar with Akka and its configuration.

On a final note, you can work with Akka’s typed actor systems, it’s the same thing. For simplicity, you could initialize it as a classic actor system (like we are doing above), and then convert it into a typed one:

import akka.actor.typed.scaladsl.adapter.ClassicActorSystemOps
system.toTyped

Using IO with Akka Stream

For this section we need to depend on Akka Stream:

// sbt syntax
libraryDependencies ++= Seq(
  "com.typesafe.akka" %% "akka-stream" % "2.6.20",
  "com.typesafe.akka" %% "akka-stream-typed" % "2.6.20",
)

Turning an IO into a Flow

An obvious solution for working with IO in Akka Stream is via mapAsync:

import akka.NotUsed
import akka.stream.scaladsl.Flow
import cats.effect.IO
import cats.effect.std.Dispatcher

def uncancelableIOToFlow[A, B](parallelism: Int)(
  f: A => IO[B]
)(implicit d: Dispatcher[IO]): Flow[A, B, NotUsed] =
  Flow[A].mapAsync(parallelism)(a => d.unsafeToFuture(f(a)))

We need a Dispatcher for turning IO values into Future values, as mapAsync works with Future.

There is one glaring problem: Future isn’t cancelable, and these IO tasks may be long-running ones. And the code above will not cancel the running IO when the stream is getting cancelled. Most often this isn’t a problem, and can be in fact desirable.

In the context of Akka Stream, to execute IO tasks as cancelable tasks, we need to work with Publisher from the Reactive Streams specification. Implementation is low-level, as it has to synchronize concurrent calls:

import org.reactivestreams.{Publisher, Subscriber, Subscription}

/** Converts a Cats-Effect `IO` into a Reactive Streams `Publisher`.
  *
  * [[https://github.com/reactive-streams/reactive-streams-jvm]]
  */
def toPublisher[A](io: IO[A])(implicit d: Dispatcher[IO]): Publisher[A] =
  (s: Subscriber[_ >: A]) => s.onSubscribe(new Subscription {
    type CancelToken = () => Future[Unit]

    private[this] val NOT_STARTED: Null = null
    private[this] val LOCKED = Left(false)
    private[this] val COMPLETED = Left(true)

    // State machine for managing the active subscription
    private[this] val ref =
      new AtomicReference[Either[Boolean, CancelToken]](NOT_STARTED)

    override def request(n: Long): Unit =
      ref.get() match {
        case NOT_STARTED =>
          if (n <= 0) {
            if (ref.compareAndSet(NOT_STARTED, COMPLETED))
              s.onError(new IllegalArgumentException(
                "non-positive request signals are illegal"
              ))
          } else {
            if (ref.compareAndSet(NOT_STARTED, LOCKED)) {
              val cancelToken = d.unsafeRunCancelable(
                io.attempt.flatMap { r =>
                  IO {
                    r match {
                      case Right(value) =>
                        s.onNext(value)
                        s.onComplete()
                      case Left(e) =>
                        s.onError(e)
                    }
                    // GC purposes
                    ref.lazySet(COMPLETED)
                  }
                }
              )
              // Race condition with lazySet(COMPLETED), but it's fine
              ref.set(Right(cancelToken))
            }
          }
        case Right(_) | Left(_) =>
          // Already active, or completed
          ()
      }

    @tailrec
    override def cancel(): Unit =
      ref.get() match {
        case NOT_STARTED =>
          if (!ref.compareAndSet(NOT_STARTED, COMPLETED))
            cancel() // retry
        case LOCKED =>
          Thread.onSpinWait()
          cancel() // retry
        case Left(_) =>
          ()
        case current@Right(token) =>
          // No retries necessary; if state changes from Right(token),
          // it means that the stream is already completed or canceled
          if (ref.compareAndSet(current, COMPLETED))
            token()
      }
  })

And we can turn Publisher into a Source:

import akka.stream.scaladsl.Source

def toSource[A](io: IO[A])(implicit d: Dispatcher[IO]): Source[A, NotUsed] =
  Source.fromPublisher(toPublisher(io))

And finally, we can use flatMapMerge to get the desired behavior of having a Flow that executes cancelable IO tasks:

import akka.stream.scaladsl.Flow

def cancelableIOToFlow[A, B](parallelism: Int)(
  f: A => IO[B]
)(implicit d: Dispatcher[IO]): Flow[A, B, NotUsed] =
  Flow[A].flatMapMerge(
    breadth = parallelism,
    a => toSource(f(a))
  )

Warning: mapAsync is much more efficient than flatMapMerge or flatMapConcat. Unfortunately, Akka Stream isn’t optimized for flat-mapping on streams that emit a single event. Also, depending on how your streams are structured, you may actually want uncancelable execution. Apply good judgement!

Repeated execution (fixed delay)

Your challenge, should you choose to accept it, is to turn this into a stream:

def tryPoll: IO[Option[Result]]

So we want to describe a function that turns that into a stream, which is a common pattern:

def poll0[A](
  tryPoll: IO[Option[A]],
  sleepDelay: FiniteDuration,
)(implicit d: Dispatcher[IO]): Source[A, NotUsed] = {
  val logger = LoggerFactory.getLogger(getClass)
  Source.repeat(())
    .via(cancelableIOToFlow(1) { _ =>
      tryPoll.handleError { e =>
        logger.error("Unhandled error in poll", e)
        None
      }.flatTap {
        case None => IO.sleep(sleepDelay)
        case Some(_) => IO.unit
      }
    })
    .collect { case Some(a) => a }
}

The sleep itself could be managed by Akka Stream. At some point, our function looked like this:

def poll1[A](
  tryPoll: IO[Option[A]],
  interval: FiniteDuration,
)(implicit d: Dispatcher[IO]): Source[A, NotUsed] = {
  val logger = LoggerFactory.getLogger(getClass)
  // Notice the `takeWhile`. This is a child stream that gets
  // composed via `flatMapConcat`.
  val drain =
    Source.repeat(())
      .via(uncancelableIOToFlow(1) { _ =>
        tryPoll.handleError { e =>
          logger.error("Unhandled error in poll", e)
          None
        }
      })
      .takeWhile(_.nonEmpty)
      .collect { case Some(a) => a }

  // Main stream, managing the sleep intervals via Akka
  Source
    .tick(initialDelay = Duration.Zero, interval = interval, tick = ())
    .flatMapConcat(_ => drain)
    .mapMaterializedValue(_ => NotUsed)
}

This has the virtue of being fast, since the sleep is managed by Akka, and for draining our queue, we can manage to work with mapAsync (and thus, uncancellable IO tasks). However, this had awkward behavior due to the buffering, and isn’t flexible enough. For instance, we wanted to specify a custom sleep duration for when exceptions are being thrown.

Last, but not least, in this particular case you can simply use fs2. This has the downside of introducing yet another dependency, and thus the area for potential issues is bigger. Extra dependencies on the classpath add risk. On the other hand, fs2 is designed to work with streams of IO tasks, being meant for precisely this use-case. And you can convert an fs2.Stream via the Reactive Streams interoperability.

You’ll need these dependencies:

// sbt syntax
libraryDependencies ++= Seq(
  "co.fs2" %% "fs2-core" % "3.6.1",
  "co.fs2" %% "fs2-reactive-streams" % "3.6.1",
)

And the implementation:

def poll2[A](
  tryPoll: IO[Option[A]],
  interval: FiniteDuration,
)(implicit d: Dispatcher[IO]): Source[A, NotUsed] = {
  val logger = LoggerFactory.getLogger(getClass)
  val repeatedTask = tryPoll.handleError { e =>
    logger.error("Unhandled error in poll", e)
    None
  }.flatTap {
    case None => IO.sleep(interval)
    case Some(_) => IO.unit
  }

  val stream = fs2.Stream
    .repeatEval(repeatedTask)
    .collect { case Some(a) => a }

  import fs2.interop.reactivestreams._
  Source.fromPublisher(new StreamUnicastPublisher(stream, d))
}

Note, however, that using fs2 for use-cases like this isn’t without peril. For example, the cancellation model of Cats-Effect (and that of fs2) is incompatible with the Reactive Streams API when managing resources. You can’t turn a Resource into a Publisher. You can turn a Resource into an fs2.Stream, and if you then try to turn that fs2.Stream into a Publisher, you’ll end up with a Publisher that doesn’t manage the resource correctly. Simply put, an fs2.Stream is more powerful than a Publisher (from the Reactive Streams spec), and so the conversion from fs2 to Reactive Streams can be problematic.

Akka Stream Graphs

When running graphs with Akka Stream, if the processing of individual events is critical (like in our case), the question is what happens when the process is being shut down, as there will be some transactions that will be in-flight, with a process shutdown interrupting them.

We want to wait (with a timeout) for the processing of in-flight transactions, before the process is terminated. To achieve that, there are 2 elements to it:

The use of kill switches, to stop all inputs in your graph;
The detection of the completion signal, giving a chance for in-flight transactions to complete;

A KillSwitch can be managed via Resource, although, as you shall see, we may need to trigger the kill signal outside the context of this Resource:

import akka.stream.{SharedKillSwitch, KillSwitches}

def sharedKillSwitch(name: String): Resource[IO, SharedKillSwitch] =
  Resource(IO {
    val ks = KillSwitches.shared(name)
    (ks, IO(ks.shutdown()))
  })

Another piece of the puzzle is that we may like to add logic to Cats-Effect’s cancellations stack:

def resourceFinalizer(effect: IO[Unit]): Resource[IO, Unit] =
  Resource(IO { ((), effect) })

The method for starting the processing graph can look like this, and note that IO[Done] should signal when the processing is complete (all inputs are completed):

def startProcessor(
  killSwitch: SharedKillSwitch
)(implicit
  system: ActorSystem[_],
): Resource[IO, IO[Done]]

We can then use the KillSwitch to kill all inputs, and then wait for processing to complete:

for {
  ks <- sharedKillSwitch("my-kill-switch")
  // We want to wait for `Done` during cancellation (below)
  awaitDone <- startProcessor(ks)(system)
  // The magic we've been waiting for
  _ <- resourceFinalizer(
    for {
      // Kill all inputs first
      _ <- IO(ks.shutdown())
      // Waits for processor to stop before proceeding;
      // Timeout is required, or this could be non-terminating
      _ <- awaitDone.timeoutAndForget(10.seconds).attempt.void
    } yield ()
  )
} yield awaitDone

If we have this on our hands, we can easily describe a reusable app logic meant for executing Akka Stream graphs, which can take care of most things. And we base it on IOApp:

trait ProcessorApp extends IOApp {
  /** Abstract method to implement... */
  def startProcessor(
    killSwitch: SharedKillSwitch
  )(implicit
    system: ActorSystem[_],
    dispatcher: Dispatcher[IO]
  ): Resource[IO, IO[Done]]

  override final def run(args: List[String]): IO[ExitCode] = {
    val startWithResources = for {
      d <- Dispatcher.parallel[IO]
      system <- startActorSystemTyped(
        systemName = "my-actor-system",
        config = None,
        useIOExecutionContext = true,
        timeoutAwaitCatsEffect = 10.seconds,
        timeoutAwaitAkkaTermination = 10.seconds,
      )
      killSwitch <- sharedKillSwitch("my-kill-switch")
      awaitDone <- startProcessor(killSwitch)(system, d)
      _ <- resourceFinalizer(
        for {
          // Kill all inputs
          _ <- IO(killSwitch.shutdown())
          // Waits the for processor to stop before proceeding;
          // Timeout is required, or this could be non-terminating
          _ <- awaitDone.timeoutAndForget(10.seconds).attempt.void
          _ <- IO(logger.info(
            "Awaited processor to stop, proceeding with shutdown..."
          ))
        } yield ()
      )
    } yield awaitDone

    startWithResources.use { awaitDone =>
      // Blocking on `awaitDone` makes sense, as the processor
      // could finish without the app receiving a termination signal
      awaitDone.as(ExitCode.Success)
    }
  }

  protected lazy val logger =
    LoggerFactory.getLogger(getClass)

  // It's a good idea to set a timeout on shutdown;
  // we need to take faulty cancellation logic into account
  override protected def runtimeConfig =
    super.runtimeConfig.copy(
      shutdownHookTimeout = 30.seconds
    )

  // We want to log uncaught exceptions in the thread pool,
  // via slf4j, otherwise they'll go to STDERR
  override protected def reportFailure(err: Throwable) =
    IO(logger.error("Unexpected error in thread-pool", err))
}

WARNING: when waiting for things to shut down, it’s important to have timeouts all over the place, to avoid non-terminating logic. The last thing you want is a zombie process that can’t shut down without a kill -9, because that requires external monitoring and intervention (e.g., systemd, docker, monit), and you can’t trust that to be correctly configured. The less you trust, the more reliable your process will be.

As mentioned before, that IO[Done] is the completion signal, which we’ll use in our main logic. This is easily accomplished via a Sink:

import akka.stream.scaladsl.Sink

def ignoreSink[A]: Sink[A, IO[Done]] =
  Sink.ignore.mapMaterializedValue(f => IO.fromFuture(IO.pure(f)))

We can now give an actual Main as an example. When running it, try killing it via SIGHUP/SIGINT/SIGTERM, see how it waits for processing to stop before shutting down resources:

object Main extends ProcessorApp {
  override def startProcessor(killSwitch: SharedKillSwitch)(implicit
    system: ActorSystem[_],
    dispatcher: Dispatcher[IO]
  ): Resource[IO, IO[Done]] = {
    Resource.eval {
      for {
        mySource <- IO {
          val counter = new AtomicInteger(0)
          val tryPoll = IO {
            val cnt = counter.incrementAndGet()
            if (cnt % 2 == 0) Some(cnt) else None
          }
          poll0(tryPoll, 1.second)
            // Installing killSwitch on this source
            .via(killSwitch.flow)
        }
        awaitDone <- IO {
          val sink = ignoreSink[Any]
          val graph = RunnableGraph.fromGraph(
            GraphDSL.createGraph(sink) { implicit builder => s =>
              import GraphDSL.Implicits._

              val ints = builder.add(mySource)
              val logEvents = builder.add(
                Flow[Int].map(i => logger.info(s"Received event: $i"))
              )

              // w00t, here's our very complicated graph!
              ints ~> logEvents ~> s
              ClosedShape
            }
          )
          graph.run()
        }
      } yield awaitDone
    }
  }
}

Full Example (Scala CLI)

The full example below can be executed directly via Scala CLI. On macOS, you can install it with:

brew install Virtuslab/scala-cli/scala-cli

And then you can run it:

scala-cli run ./sample.scala

# Or make the script executable; works due to the included 'shebang'
# (https://en.wikipedia.org/wiki/Shebang_(Unix))
chmod +x ./sample.scala

# And then run it directly
./sample.scala

Copy/paste this script into sample.scala:

#!/usr/bin/env -S scala-cli shebang -q

//> using scala "2.13.10"
//> using lib "ch.qos.logback:logback-classic:1.4.6"
//> using lib "co.fs2::fs2-core::3.6.1"
//> using lib "co.fs2::fs2-reactive-streams::3.6.1"
//> using lib "com.typesafe.akka::akka-actor-typed::2.6.20"
//> using lib "com.typesafe.akka::akka-actor::2.6.20"
//> using lib "com.typesafe.akka::akka-stream-typed::2.6.20"
//> using lib "com.typesafe.akka::akka-stream::2.6.20"
//> using lib "org.typelevel::cats-effect::3.4.9"

import akka.actor.typed.ActorSystem
import akka.actor.typed.{ActorSystem => TypedActorSystem}
import akka.actor.{CoordinatedShutdown, ActorSystem => UntypedActorSystem}
import akka.stream.scaladsl.{Flow, GraphDSL, RunnableGraph}
import akka.stream.scaladsl.{Flow, Sink, Source}
import akka.stream.{ClosedShape, KillSwitches, SharedKillSwitch}
import akka.{Done, NotUsed}
import cats.effect.kernel.Resource
import cats.effect.std.Dispatcher
import cats.effect.{Deferred, ExitCode, IO, IOApp}
import cats.syntax.all._
import com.typesafe.config.Config
import java.util.concurrent.atomic.AtomicInteger
import java.util.concurrent.atomic.AtomicReference
import org.reactivestreams.{Publisher, Subscriber, Subscription}
import org.slf4j.LoggerFactory
import scala.annotation.tailrec
import scala.concurrent.duration._
import scala.concurrent.{Future, TimeoutException}

object Main extends ProcessorApp {
  import AkkaUtils._

  override def startProcessor(killSwitch: SharedKillSwitch)(implicit
    system: ActorSystem[_],
    dispatcher: Dispatcher[IO]
  ): Resource[IO, IO[Done]] = {
    Resource.eval {
      for {
        mySource <- IO {
          val counter = new AtomicInteger(0)
          val tryPoll = IO {
            val cnt = counter.incrementAndGet()
            if (cnt % 2 == 0) Some(cnt) else None
          }
          poll0(tryPoll, 1.second)
            // Installing killSwitch on this source
            .via(killSwitch.flow)
        }
        awaitDone <- IO {
          val sink = ignoreSink[Any]
          val graph = RunnableGraph.fromGraph(
            GraphDSL.createGraph(sink) { implicit builder => s =>
              import GraphDSL.Implicits._

              val ints = builder.add(mySource)
              val logEvents = builder.add(
                Flow[Int].map(i => logger.info(s"Received event: $i"))
              )

              // w00t, here's our very complicated graph!
              ints ~> logEvents ~> s
              ClosedShape
            }
          )
          graph.run()
        }
      } yield awaitDone
    }
  }
}

trait ProcessorApp extends IOApp {
  import AkkaUtils._

  /** Abstract method to implement... */
  def startProcessor(killSwitch: SharedKillSwitch)(implicit
    system: ActorSystem[_],
    dispatcher: Dispatcher[IO]
  ): Resource[IO, IO[Done]]

  override final def run(args: List[String]): IO[ExitCode] = {
    val startWithResources = for {
      d <- Dispatcher.parallel[IO]
      system <- startActorSystemTyped(
        systemName = "my-actor-system",
        config = None,
        useIOExecutionContext = true,
        timeoutAwaitCatsEffect = 10.seconds,
        timeoutAwaitAkkaTermination = 10.seconds,
      )
      killSwitch <- sharedKillSwitch("my-kill-switch")
      awaitDone <- startProcessor(killSwitch)(system, d)
      _ <- resourceFinalizer(
        for {
          // Kill all inputs
          _ <- IO(killSwitch.shutdown())
          // Waits the for processor to stop before proceeding;
          // Timeout is required, or this could be non-terminating
          _ <- awaitDone.timeoutAndForget(10.seconds).attempt.void
          _ <- IO(logger.info(
            "Awaited processor to stop, proceeding with shutdown..."
          ))
        } yield ()
      )
    } yield awaitDone

    startWithResources.use { awaitDone =>
      // Blocking on `awaitDone` makes sense, as the processor
      // could finish without the app receiving a termination signal
      awaitDone.as(ExitCode.Success)
    }
  }

  protected lazy val logger =
    LoggerFactory.getLogger(getClass)

  // It's a good idea to set a timeout on shutdown;
  // we need to take faulty cancellation logic into account
  override protected def runtimeConfig =
    super.runtimeConfig.copy(
      shutdownHookTimeout = 30.seconds
    )

  // We want to log uncaught exceptions in the thread pool,
  // via slf4j, otherwise they'll go to STDERR
  override protected def reportFailure(err: Throwable) =
    IO(logger.error("Unexpected error in thread-pool", err))
}

object AkkaUtils {
  /** Starts an (untyped) Akka actor system in the
    * context of a Cats-Effect `Resource`, and integrating
    * with its cancellation abilities.
    *
    * HINT: for apps (in `main`), it's best if
    * `akka.coordinated-shutdown.exit-jvm` is set to `on`,
    * because Akka can decide to shutdown on its own. And
    * having this setting interacts well with Cats-Effect.
    *
    * @param systemName is the identifying name of the system.
    * @param config is an optional, parsed HOCON configuration;
    *        if None, then Akka will read its own, possibly
    *        from `application.conf`; this parameter is
    *        provided in order to control the source of
    *        the application's configuration.
    * @param useIOExecutionContext if true, then Cats-Effect's
    *        default thread-pool will get used by Akka, as well.
    *        This is needed in order to avoid having too many
    *        thread-pools.
    * @param timeoutAwaitCatsEffect is the maximum amount of time
    *        Akka's coordinated-shutdown is allowed to wait for
    *        Cats-Effect to finish. This is needed, as Cats-Effect
    *        could have a faulty stack of disposables, or because
    *        Akka could decide to shutdown on its own.
    * @param timeoutAwaitAkkaTermination is the maximum amount of
    *        time to wait for the actor system to terminate, after
    *        `terminate()` was called. We need the timeout, because
    *        `terminate()` proved to return a `Future` that never
    *        completes in certain scenarios (could be a bug, or a
    *        race condition).
    */
  def startActorSystemUntyped(
    systemName: String,
    config: Option[Config],
    useIOExecutionContext: Boolean,
    timeoutAwaitCatsEffect: Duration,
    timeoutAwaitAkkaTermination: Duration,
  ): Resource[IO, UntypedActorSystem] = {
    // Needed to turn IO into Future
    // https://typelevel.org/cats-effect/docs/std/dispatcher
    Dispatcher.parallel[IO](await = true).flatMap { dispatcher =>
      Resource[IO, UntypedActorSystem](
        for {
          // Fishing IO's `ExecutionContext`
          ec <- Option
            .when(useIOExecutionContext)(IO.executionContext)
            .sequence
          // For synchronizing Cats-Effect with Akka
          awaitCancel <- Deferred[IO, Unit]
          // For awaiting termination, as `terminate()` is unreliable
          awaitTermination <- Deferred[IO, Unit]
          logger = LoggerFactory.getLogger(getClass)
          system <- IO {
            logger.info("Creating actor system...")
            val system = UntypedActorSystem(
              systemName.trim.replaceAll("\\W+", "-"),
              config = config,
              defaultExecutionContext = ec,
            )
            // Registering task in Akka's CoordinatedShutdown
            // that will wait for Cats-Effect to catch up,
            // blocking Akka from terminating, see:
            // https://doc.akka.io/docs/akka/current/coordinated-shutdown.html
            CoordinatedShutdown(system).addTask(
              CoordinatedShutdown.PhaseBeforeServiceUnbind,
              "sync-with-cats-effect",
            ) { () =>
              dispatcher.unsafeToFuture(
                // WARN: this may not happen, if Akka decided
                // to terminate, and `coordinated-shutdown.exit-jvm`
                // isn't `on`, hence the timeout:
                awaitCancel.get
                  .timeout(timeoutAwaitCatsEffect)
                  .recoverWith {
                    case ex: TimeoutException =>
                      IO(logger.error(
                        "Timed out waiting for Cats-Effect to catch up! " +
                          "This might indicate either a non-terminating " +
                          "cancellation logic, or a misconfiguration of Akka."
                      ))
                  }
                  .as(Done)
              )
            }
            CoordinatedShutdown(system).addTask(
              CoordinatedShutdown.PhaseActorSystemTerminate,
              "signal-terminated",
            ) { () =>
              dispatcher.unsafeToFuture(
                awaitTermination.complete(()).as(Done)
              )
            }
            system
          }
        } yield {
          val cancel =
            for {
              // Signals that Cats-Effect has caught up with Akka
              _ <- awaitCancel.complete(())
              _ <- IO(logger.warn("Shutting down actor system!"))
              // Shuts down Akka, and waits for its termination
              // Here, system.terminate() returns a `Future[Terminated]`,
              // but we are ignoring it, as it could be non-terminating
              _ <- IO(system.terminate())
              // Waiting for Akka to terminate via its CoordinatedShutdown
              _ <- awaitTermination.get
              // WARN: `whenTerminated` is unreliable
              _ <- IO.fromFuture(IO(system.whenTerminated)).void
                .timeoutAndForget(timeoutAwaitAkkaTermination)
                .handleErrorWith(_ =>
                  IO(logger.warn(
                    "Timed-out waiting for Akka to terminate!"
                  ))
                )
            } yield ()
          (system, cancel)
        }
      )
    }
  }

  /** Starts a (typed) Akka actor system.
    *
    * @see [[startActorSystemUntyped]] for more details on params.
    */
  def startActorSystemTyped(
    systemName: String,
    config: Option[Config],
    useIOExecutionContext: Boolean,
    timeoutAwaitCatsEffect: FiniteDuration,
    timeoutAwaitAkkaTermination: FiniteDuration,
  ): Resource[IO, TypedActorSystem[Nothing]] =
    startActorSystemUntyped(
      systemName,
      config,
      useIOExecutionContext,
      timeoutAwaitCatsEffect,
      timeoutAwaitAkkaTermination,
    ).map { system =>
      import akka.actor.typed.scaladsl.adapter.ClassicActorSystemOps
      system.toTyped
    }

  /** Converts a Cats-Effect `IO` into a Reactive Streams `Publisher`.
    *
    * [[https://github.com/reactive-streams/reactive-streams-jvm]]
    */
  def toPublisher[A](io: IO[A])(implicit d: Dispatcher[IO]): Publisher[A] =
    (s: Subscriber[_ >: A]) => s.onSubscribe(new Subscription {
      type CancelToken = () => Future[Unit]

      private[this] val NOT_STARTED: Null = null
      private[this] val LOCKED = Left(false)
      private[this] val COMPLETED = Left(true)

      // State machine for managing the active subscription
      private[this] val ref =
        new AtomicReference[Either[Boolean, CancelToken]](NOT_STARTED)

      override def request(n: Long): Unit =
        ref.get() match {
          case NOT_STARTED =>
            if (n <= 0) {
              if (ref.compareAndSet(NOT_STARTED, COMPLETED))
                s.onError(new IllegalArgumentException(
                  "non-positive request signals are illegal"
                ))
            } else {
              if (ref.compareAndSet(NOT_STARTED, LOCKED)) {
                val cancelToken = d.unsafeRunCancelable(
                  io.attempt.flatMap { r =>
                    IO {
                      r match {
                        case Right(value) =>
                          s.onNext(value)
                          s.onComplete()
                        case Left(e) =>
                          s.onError(e)
                      }
                      // GC purposes
                      ref.lazySet(COMPLETED)
                    }
                  }
                )
                // Race condition with lazySet(COMPLETED), but it's fine
                ref.set(Right(cancelToken))
              }
            }
          case Right(_) | Left(_) =>
            // Already active, or completed
            ()
        }

      @tailrec
      override def cancel(): Unit =
        ref.get() match {
          case NOT_STARTED =>
            if (!ref.compareAndSet(NOT_STARTED, COMPLETED))
              cancel() // retry
          case LOCKED =>
            Thread.onSpinWait()
            cancel() // retry
          case Left(_) =>
            ()
          case current@Right(token) =>
            // No retries necessary; if state changes from Right(token),
            // it means that the stream is already completed or canceled
            if (ref.compareAndSet(current, COMPLETED))
              token()
        }
    })

  def toSource[A](io: IO[A])(implicit d: Dispatcher[IO]): Source[A, NotUsed] =
    Source.fromPublisher(toPublisher(io))

  def uncancelableIOToFlow[A, B](parallelism: Int)(
    f: A => IO[B]
  )(implicit d: Dispatcher[IO]): Flow[A, B, NotUsed] =
    Flow[A].mapAsync(parallelism)(a => d.unsafeToFuture(f(a)))


  def cancelableIOToFlow[A, B](parallelism: Int)(
    f: A => IO[B]
  )(implicit d: Dispatcher[IO]): Flow[A, B, NotUsed] =
    Flow[A].flatMapMerge(
      breadth = parallelism,
      a => toSource(f(a))
    )

  def poll0[A](
    tryPoll: IO[Option[A]],
    interval: FiniteDuration,
  )(implicit d: Dispatcher[IO]): Source[A, NotUsed] = {
    val logger = LoggerFactory.getLogger(getClass)
    Source.repeat(())
      .via(cancelableIOToFlow(1) { _ =>
        tryPoll.handleError { e =>
          logger.error("Unhandled error in poll", e)
          None
        }.flatTap {
          case None => IO.sleep(interval)
          case Some(_) => IO.unit
        }
      })
      .collect { case Some(a) => a }
  }

  def poll1[A](
    tryPoll: IO[Option[A]],
    interval: FiniteDuration,
  )(implicit d: Dispatcher[IO]): Source[A, NotUsed] = {
    val logger = LoggerFactory.getLogger(getClass)
    // Notice the `takeWhile`. This is a child stream that gets
    // composed via `flatMapConcat`.
    val drain =
    Source.repeat(())
      .via(uncancelableIOToFlow(1) { _ =>
        tryPoll.handleError { e =>
          logger.error("Unhandled error in poll", e)
          None
        }
      })
      .takeWhile(_.nonEmpty)
      .collect { case Some(a) => a }

    Source
      .tick(initialDelay = Duration.Zero, interval = interval, tick = ())
      .flatMapConcat(_ => drain)
      .mapMaterializedValue(_ => NotUsed)
  }

  def poll2[A](
    tryPoll: IO[Option[A]],
    interval: FiniteDuration,
  )(implicit d: Dispatcher[IO]): Source[A, NotUsed] = {
    val logger = LoggerFactory.getLogger(getClass)
    val repeatedTask = tryPoll.handleError { e =>
      logger.error("Unhandled error in poll", e)
      None
    }.flatTap {
      case None => IO.sleep(interval)
      case Some(_) => IO.unit
    }

    val stream = fs2.Stream
      .repeatEval(repeatedTask)
      .collect { case Some(a) => a }

    import fs2.interop.reactivestreams._
    Source.fromPublisher(new StreamUnicastPublisher(stream, d))
  }

  def ignoreSink[A]: Sink[A, IO[Done]] =
    Sink.ignore.mapMaterializedValue(f => IO.fromFuture(IO.pure(f)))

  def sharedKillSwitch(name: String): Resource[IO, SharedKillSwitch] =
    Resource(IO {
      val ks = KillSwitches.shared(name)
      (ks, IO(ks.shutdown()))
    })

  def resourceFinalizer(effect: IO[Unit]): Resource[IO, Unit] =
    Resource(IO {
      ((), effect)
    })
}

Enjoy!

The article Integrating Akka with Cats-Effect 3 first appeared on alexn.org.

Server Monitoring with Monit

Alexandru Nedelcu — Tue, 17 Jan 2023 11:16:12 +0000

A screenshot of my browser window, showing the 3 monitors that I configured in Monit, all green-lit, indicating that everything is fine.

I self-host my blog, other websites, Matomo, Mastodon, etc. I love self-hosting. But I need monitoring, to be alerted when things go wrong, as my setup is getting more and more complex. So, I recently asked for help online, being in need of a monitoring system for my VPS, as I need simple, common-sense health alerts. I got a recommendation for M/Monit, which seems to work well. This article shows my configuration.

Even though I use Docker, I decided to install it at the OS level. Given I use Ubuntu on my server, this is as simple as:

apt install monit

Its configuration (in /etc/monit/conf.d/my.conf) looks like this:

# Monit configuration

## ----
## Configures server port (proxied via Nginx)
set httpd
    port 2812
        read-only
    unixsocket /run/monit.socket
    allow localhost
    allow monit.alexn.org
    signature disable

## Configures email alerts
set mailserver localhost
set alert user@domain.com not on { instance, action } with reminder on 500 cycles

## Monitors system load
check system $HOST
    if loadavg (1min) per core > 2 for 5 cycles then alert
    if loadavg (5min) per core > 1.5 for 10 cycles then alert
    if cpu usage > 95% for 10 cycles then alert
    if memory usage > 90% then alert
    if swap usage > 25% then alert

## Monitors file-system
check filesystem rootfs with path /
    if space usage > 80% for 5 times within 15 cycles then alert
    if inode usage > 80% for 5 times within 15 cycles then alert

## Monitors Docker instances
check program docker-health with path /opt/bin/vm-docker-check-health
    with timeout 10 seconds
    if status != 0 then alert

My hostname is monit.alexn.org for exposing the HTTP interfaces. I decided for a read-only interface. It’s less to worry about, and I don’t need a remote control. I use Nginx as a proxy:

server {
    server_name monit.alexn.org;
    listen   80;
    listen   [::]:80;
    access_log /var/log/nginx/monit.alexn.org.log combined;

    location / {
        rewrite ^(.*)$ https://monit.alexn.org$1 permanent;
    }
}

server {
    server_name monit.alexn.org;
    listen 443 ssl;
    listen [::]:443 ssl;

    access_log /var/log/nginx/monit.alexn.org.log combined;

    # https://ssl-config.mozilla.org/#server=nginx&version=1.17.7&config=intermediate&openssl=1.1.1k&guideline=5.6
    ssl_certificate /etc/letsencrypt/live/alexn.org/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/alexn.org/privkey.pem;
    ssl_trusted_certificate /etc/letsencrypt/live/alexn.org/fullchain.pem;
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:10m;  # about 40000 sessions
    ssl_session_tickets off;
    ssl_dhparam /etc/nginx/certs/dh2048.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;
    add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
    ssl_stapling on;
    ssl_stapling_verify on;

    location / {
        root   /var/www/monit.alexn.org;
        auth_basic "Monit";
        auth_basic_user_file /etc/secrets/auth/monit-htpasswd;

        proxy_pass http://0.0.0.0:2812;
        proxy_set_header Host $host;
        proxy_connect_timeout 300;
    }
}

Generating a “htpasswd” file can be accomplished with this command:

htpasswd -B -c etc/secrets/auth/monit-htpasswd yourUserName

That configuration alerts me on 3 things:

The system’s load (CPU, RAM, Swap);
The available disk space;
The health of my Docker containers;

For checking the Docker containers, I have a simple script that’s configured above to be executed periodically:


#!/usr/bin/env bash

UNHEALTHY_IDS="$(docker ps -q \
    -f health="none" \
    -f health="unhealthy" \
    -f status="exited" \
    -f status="dead" \
    -f status="paused" \
    )"

if [[ -z "$UNHEALTHY_IDS" ]]; then
    docker ps --format "table {{.Names}}\t{{.Status}}"
    exit 0
fi

echo >&2
echo "WARN: Unhealthy docker instances!" >&2
echo "---------------------------------" >&2
docker ps --format "table {{.Names}}\t{{.State}}\t{{.Status}}" \
    -f health="none" \
    -f health="unhealthy" \
    -f status="exited" \
    -f status="dead" \
    -f status="paused" >&2
exit 1

WARN: you need an email server configured for receiving those alerts!

You need to configure Monit with an email server. I just use my localhost because I can configure postfix to use my Fastmail account as a relay. See my Ubuntu wiki page for details on how to do that.

Now I can sleep well at night 🥱

The article Server Monitoring with Monit first appeared on alexn.org.

#DeleteTwitter

Alexandru Nedelcu — Sun, 15 Jan 2023 08:30:29 +0000

Friday the 13th just passed 😳 and what a great day it was to finally leave Twitter. It’s been a fun ride, I found my programming community on it, I’ve learned a lot, but those days are over 😞

I’m a software developer. I don’t know what Twitter’s future is, it might be a bright one, but the problem for me is that Twitter is no longer the place where I can go to learn about programming. Or to find my peers. Twitter is no longer the place you go to talk of your passions, fruitful discussions being few and far between. Twitter is no longer fun, but rather it’s where you go to get your daily fix of unhinged political drama, and then worry that the world is going to shit.

I had about 6000 followers. Not much, but enough to keep me hooked. However, from my experience, I can tell you that engagement for posts has been going down, for anything but politics. I noticed it not just on my posts, but on the posts of those I followed as well. Having followers feels like an investment, which is one of the ways social networks keep people hooked, but it’s a meaningless number, a proxy for actual reach. We want to learn, to make connections, to promote our work, to be entertained, but these social networks have a real cost, and we have to keep asking ourselves if they help us in achieving our goals.

These days, Twitter may be more alive than ever, but many of the people I care about are no longer active on Twitter. As far as I’m concerned, the elves have left Middle Earth, taking their magic with them, and I doubt that the magic can come back.

Elves leaving Middle Earth (source).

Last year, I deleted my Facebook account, too. These days you can find me on the Fediverse / Mastodon.

The issue with centralized social networks is that their business model is ads-driven, which means they have to steal your attention. The more you engage with their service, the more ads you consume. This creates a perverse incentive, as the timeline of these social networks was optimized for stealing your attention. And it turns out that outrage generates a lot of attention, and engagement.

When you log into Twitter, you get bombarded with political opinions that are counterproductive. Even if you judiciously block, even if you switch to the “latest tweets” timeline, it doesn’t matter, as your connections also get fed outrage, and start sharing. Whenever I see another flame war on Twitter, it’s always another US tantrum, and I have to wait for a couple of days for it to cool off. And US tantrums are digestible because at least I’m not living in the US, whereas Facebook was giving me local news of gruesome local tragedies, many times as “promoted” posts.

People want more content moderation on these platform, this being seemingly in contradiction with freedom of speech. But we are missing the forest from the trees, I think. The problem with these centralized “public squares” is that their algorithms are now adversarial, shoving speech we don’t want in our faces for profit.

I’m actually one of those obnoxious free-speech absolutists, and being a European liberal that believes in capitalism, I consider myself to be aligned more with right-wing values. What you’re witnessing with Musk’s supporters aren’t values rooted in classic liberalism. They aren’t concerned with free speech, given the cheering of journalists or left-wing accounts being banned. This is just tribalism. And I’ve got better things to do than participate in a platform whose new leadership has a mission of “owning the libs”.

Don’t get me wrong, Twitter’s leadership is free to spread election or vaccine misinformation, or to promote hatred against LGBT+, but the beauty of freedom and capitalism is that this works both ways … we are free to stop using their service, and vote with our wallet and/or attention.

And I can’t in good faith continue to contribute to this platform.

I’m not here to tell you what to do, but in case you feel as I do and need some motivation, I recommend either one of these books:

Goodbye, Twitter, and thanks for all the fish 👋

The article #DeleteTwitter first appeared on alexn.org.

Personal Server Backups

Alexandru Nedelcu — Fri, 02 Dec 2022 17:56:39 +0000

Cloud hosting services like Linode or DigitalOcean offer backup services for your VPS. Save your money, you don’t need it. Here’s how to backup your data safely, and with no extra costs…

The plan, for your inspiration:

Place all your server configuration in a personal git repository;
Backup your data via cron jobs, with the help of rclone;
Test how reliable recovery is every time you update your server to a major Linux distribution version;

Server configuration

On my server, I use the latest LTS of Ubuntu Linux, since it’s what I’ve grown accustomed to. I have a personal GitHub repository where I store the server’s configuration. The bulk of it is:

A docker-compose.yaml file;
Nginx configurations for my domains;
Scripts that need to execute periodically via cron.d;
A setup.sh script that configures a server from scratch.

One of these days I’ll try NixOS, as a lot of people love it, and it’s designed for precisely this use-case: to have a reproducible environment described by configuration kept in a repository.

Start small, and think big. Can you reconfigure a server from scratch in less than an hour? If not, why not?

Rclone backups via cron jobs

You can use Dropbox for storing your backups, you can use Amazon’s S3, or as a pretty cheap alternative, you can use Backblaze B2. I use Dropbox, since I’m paying for a Plus account anyway, and I have a lot of unused space. Dropbox also keeps about 1-month worth of version history, so it’s decent.

The Rclone utility supports a lot of backends, including the aforementioned ones.

WARNING: when configuring rclone on your server, you need to think about security. If you have important personal data in your cloud storage, you don’t want to give attackers access to it. Which is why a separation would be better. I used Dropbox because I could set it up such that access is restricted to an “app folder” (see their docs for configuring your own “app”). If you can’t do that, it would be safer to create another account (e.g., Backblaze B2), and give access to buckets that are only used for these backups.

The “remote” that I configure (via rclone config) has the name backup, which are then used in the following scripts.

For your inspiration, as a sample, here’s the script for backing up my MariaDB / MySQL database (named vm-backup-mysql). Note that the database is in a Docker container, and we also need a mysql.env file somewhere with the root password:

#!/usr/bin/env bash

set -e

DIR="$(dirname "$0")"
set -o allexport; source "$DIR/../docker/envs/mysql.env"; set +o allexport

FILENAME="mysql-$(date +"%Y-%m").sql.gz"
FILEPATH="/var/lib/my-backups/$FILENAME"

mkdir -p /var/lib/my-backups

docker exec -i mariadb /bin/mysqldump -u root \
  "-p$MYSQL_ROOT_PASSWORD" \
  --lock-tables=false \
  --all-databases | gzip > "$FILEPATH"

if [ -f "$FILEPATH" ]; then
  rclone copy "$FILEPATH" "backup:MySQL/"
fi

rm -f /var/lib/my-backups/mysql-*.sql.gz

Here’s the script for backing up the data for my Isso comments service, named vm-backup-isso. This script is backing up the configuration (from /etc/isso) and the SQLite database (from /var/lib/isso):

#!/usr/bin/env bash

set -e

FILENAME1="isso-db-$(date +"%Y-%m").tar.gz"
FILEPATH1="/var/lib/my-backups/$FILENAME1"
FILENAME2="isso-cfg.tar.gz"
FILEPATH2="/var/lib/my-backups/$FILENAME2"

mkdir -p /var/lib/my-backups
cd /var/lib/my-backups || exit 1

tar cvzf ./"$FILENAME1" -C / var/lib/isso
tar cvzf ./"$FILENAME2" -C / etc/isso

if [ -f "$FILEPATH1" ]; then
  rclone copy "$FILEPATH1" "backup:Isso/"
fi

if [ -f "$FILEPATH2" ]; then
  rclone copy "$FILEPATH2" "backup:Isso/"
fi

rm -f /var/lib/my-backups/isso-*.tar.gz

Here’s the script for backing up my FreshRSS OPML file, such that I never lose my blog subscriptions 🙂

#!/bin/bash

set -e

FILENAME="freshrss-$(date +"%Y-%m").opml.xml.gz"
FILEPATH="/var/lib/my-backups/$FILENAME"

if [ $(docker inspect -f '{{.State.Running}}' freshrss) ]; then
  echo "[$(date +"%Y-%m-%d %H:%M:%S%z")] Generating $FILEPATH"
  docker exec -t freshrss /bin/bash -c -i "/var/www/FreshRSS/cli/export-opml-for-user.php --user alexandru 2>/dev/null" | gzip >"$FILEPATH"

  if [ -f "$FILEPATH" ]; then
    rclone copy "$FILEPATH" "backup:FreshRSS/"
  fi
fi

rm -f /var/lib/my-backups/freshrss-*.gz

Finally, here’s my cron setup, in a file named /etc/cron.d/vm-cron:

MAILTO=cron-errors@my.address.com

# ----------------
# Backups

10 */6 * * * root  cronic /opt/vm/bin/vm-backup-mysql
20 */6 * * * root  cronic /opt/vm/bin/vm-backup-isso
30 */6 * * * root  cronic /opt/vm/bin/vm-backup-freshrss

As you can see, backups are running on my VPS every 6 hours. I have more fine-grained backups than what services like Linode or DigitalOcean can provide.

WARNING: when doing backups, it’s best if you have a system alterting you when something goes wrong, like when rclone can no longer connect to your remote. Which is why sending emails on error is a good practice.

My server is configured to send emails via Fastmail, as an external relayhost. See my wiki entry for details on the setup. In the cron setup above, I’m using cronic, a small utility that silences the output of those scripts, only outputting to stdout and stderr if the script finishes with an error code. The effect is that the cron service will only send emails when errors happen.

Test your backups

You won’t know that you have working backups, unless you test them periodically.

For me this happens naturally, because I’m staying on Ubuntu LTS releases, and frankly updating major Ubuntu versions is best done from scratch. And I also keep resizing my VPS. This means that I frequently recreate my VPS from scratch.

Nowadays, I can do it in about 1 hour, with no access to the previous VPS setup, as everything I need is already in GitHub or in Dropbox. The more you automate, and the more you test that automation, the more reliable it is when you’ll actually need it.

The article Personal Server Backups first appeared on alexn.org.

Custom Jackson JSON serializer/deserializer from Circe

Alexandru Nedelcu — Tue, 29 Nov 2022 00:00:00 +0000

Snippet for when you’re using Circe and want to define custom Jackson serializers/deserializers from Circe’s codec definitions.

It’s not very efficient, as deserialization seems to parse the JSON in both Jackson and Circe, but I don’t have better ideas.

import com.fasterxml.jackson.core.{ JsonGenerator, JsonParser, TreeNode }
import com.fasterxml.jackson.databind.deser.std.StdDeserializer
import com.fasterxml.jackson.databind.ser.std.StdSerializer
import com.fasterxml.jackson.databind.{ DeserializationContext, SerializerProvider }
import io.circe.parser.decode
import io.circe.syntax._
import io.circe.{ Decoder, Encoder }
import scala.reflect.ClassTag

class JacksonSerializerFromCirce[A: ClassTag: Encoder]
extends StdSerializer[A](
    implicitly[ClassTag[A]].runtimeClass.asInstanceOf[Class[A]]
  ) {
  override def serialize(
    value: A,
    gen: JsonGenerator,
    provider: SerializerProvider
  ): Unit = {
    val json = value.asJson.noSpaces
    gen.writeRawValue(json)
  }
}

class JacksonDeserializerFromCirce[A: ClassTag: Decoder]
  extends StdDeserializer[A](
    implicitly[ClassTag[A]].runtimeClass.asInstanceOf[Class[A]]
  ) {
  override def deserialize(
    p: JsonParser,
    ctxt: DeserializationContext
  ): A = {
    decode[A](p.readValueAsTree[TreeNode]().toString) match {
      case Right(a) => a
      case Left(e) => throw e
    }
  }
}

Usage:

import io.circe.Codec
import io.circe.generic.semiauto.deriveCodec

@JsonSerialize(using = classOf[Sample.Serializer])
@JsonDeserialize(using = classOf[Sample.Deserializer])
final case class Sample(
  name: String,
  isActive: Boolean
)

object Sample {
  implicit val codec: Codec[Sample] =
    deriveCodec

  class Serializer
    extends JacksonSerializerFromCirce[Sample]
  class Deserializer
    extends JacksonDeserializerFromCirce[Sample]
}

The article Custom Jackson JSON serializer/deserializer from Circe first appeared on alexn.org.

I ❤️ Scala's Community

Alexandru Nedelcu — Fri, 28 Oct 2022 07:21:54 +0000

I find Scala’s community to be really nice, welcoming, warm, and productive. I am not talking just of the Typelevel sub-community, which is awesome, but of the whole Scala community. The whole thing.

This may seem odd, because we had, and we may still experience conflicts, that I could personally live without. But in big and heterogeneous groups of people that collaborate, some inevitably get angry with each other.

The official community channels are very civilized, friendly, and welcoming. There, you can easily find very knowledgeable engineers and scientists, willing to help, without having to feel shame for the questions you ask.

This is no accident. Community volunteers, Scala Center, former and previous stewards of the language (Lightbend employees, etc.), worked hard to make these channels nice. I remember my first interaction with Scala, it happened on IRC, on the now defunct Freenode network, where I presented a small implementation, and I immediately got an insult. Fortunately, those days are long gone.

Programmers can be quite vicious. I don’t really understand why. When I got into programming, the biggest joys I experienced in high-school was to implement stuff and then show it off to others. This is how all beginners start. But then we tend to become arrogant, and cynical, forgetting from where we started.

But back to Scala — even the online channels that are sometimes categorized as dumpster fires, are actually nice. For example, Reddit’s /r/scala. If you don’t believe me, venture yourself into the subreddits of other communities, try asking for help, or having opinions™️, see how that works out.

The Scala community is freakishly productive, given its size. It’s quite spectacular how, in GitHub rankings, it’s in the same league as several “10 pound gorillas”. And I sometimes complain about the TIMTOWTDI, but it’s a direct result of the fact that competing ideas aren’t being shut down, but rather actively worked on by people with a passion to follow.

This talent, knowledge, and warmth gives me faith in the future of the Scala language.

The article I ❤️ Scala's Community first appeared on alexn.org.

Immutable Collections should be Your Default

Alexandru Nedelcu — Thu, 27 Oct 2022 12:34:46 +0000

Mutable collection types should only be used strategically, with purpose, otherwise for correctness/safety purposes, the default should be immutable collection types, aka persistent data structures.

Available options

For working with immutable collections:

Java’s standard library has utilities for creating unmodifiable copies, which are better than nothing;
VAVR works well for Java;
Guava is also a standard recommendation for Java;
kotlinx.collections.immutable works well for Kotlin;
In Scala and Clojure, the default collection implementations are immutable by default, a wise design choice for JVM languages that provide their own (maximal) standard library;
Immutable.js works well for JavaScript;

Motivation

Immutable collections are much like String. You don’t need String to be mutable, whenever you build a bigger String, you just do ref += nextLine or you work with a StringBuilder. And String being immutable helps with sharing it safely across threads, or with using it in HashMap implementations. Java’s Strings are surprisingly sane, compared with other implementations, people should take note. So why shouldn’t collections also be immutable by default, just like String?

Mutable collections may have a performance advantage, even when used in a concurrent context. Java’s ConcurrentLinkedQueue for example will perform better than an AtomicReference(immutable.Queue()). Or an ArrayList will perform better than an immutable/persistent List (which is actually a stack). But performance optimizations are often unnecessary, and safety should be the default.

Performance is a currency, and whenever you can afford it (most of the time), what better way to spend such a currency than on correctness?

Sharing of mutable data structures is bad, and demonstrating it is easy:

// Java code

record class ProjectConfig(
  List availableTimezones,
  //...
) {}

//... later ...
config.availableTimezones().add("Mars/SpaceXFirst");

This is bad because it modifies that configuration for all call sites. Even if a project-wide configuration change was your intent, this is a pretty bad way to model configuration updates in your code, because then your components need to be notified of such configuration changes. For the call-site, this is bad too, because the user doesn’t really know if doing this is safe or not. The users may end up thinking that they received their own copy, so this is safe.

And note, it’s acceptable to use Java’s way of handling this — by wrapping that list into something that throws an UnsupportedOperationException when you try to add() or remove(). But it would be even better if you used some type that makes the immutability clear, e.g., ImmutableList.

Sample: Concurrent Data Structures

Just this week I stumbled on the following Java declaration:

// Java code
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

//...
Map> dns = new ConcurrentHashMap<>();

You should immediately notice that something is wrong, because the type of ConcurrentHashMap is hidden, which means that its methods specifically meant for dealing with concurrent updates aren’t used. Then, I stumbled on gems like:

// All unsynchronized of course
var value = "another value"
var list = dns.get("key");
if (list != null) {
  list.add(value);
} else {
  list = new ArrayList<>();
  list.add(value);
  dns.put("key", newList);
}

I mean, yikes! The developer that wrote this code felt that multi-threading synchronization was needed. But rubbing some ConcurrentHashMap on your code won’t help, if you don’t treat those values as being immutable. By extracting a List from that ConcurrentHashMap, then modifying it, that code is thread unsafe. Also, that get followed by put is not atomic. Such concurrency issues lead to overridden updates (lost values), or even to corrupted data structures. This could have all been avoided with proper use of an immutable List in those values, combined with good use of ConcurrentHashMap’s API (e.g., compute).

So, the result was a hard to trace bug in our project, coming from a (proprietary) library that we can’t fix.

In Scala, even if we use ConcurrentHashMap, we’d use it in combination with a Scala immutable List, which is available by default, no import necessary:

// Scala code
import java.util.concurrent.ConcurrentHashMap

//..
val dns = new ConcurrentHashMap[String, List[String]]()

//..
dns.compute("key", (k, v) => {
  val current = if (v != null) v else Nil
  current.prepended("new-value")
})

You can do this with Java’s mutable ArrayList, of course, but:

you need to remember to not modify the extracted values — this means creating a clone, and adding values to that clone;
creating clones of mutable data structures can cost more than using specialized implementations of persistent/immutable collection types (e.g., Scala’s List, Queue, Vector, Map, Set);

Immutable collection types can also be transformed into concurrent collections, very cheaply, by wrapping them into an AtomicReference. Which is actually a neat trick used by many libraries handling concurrency.

// Scala 3 code, making use of "Explicit Nulls"
// Needs the -Yexplicit-nulls compiler option to see it in action
import java.util.concurrent.atomic.AtomicReference
import scala.collection.immutable.Queue
import scala.collection.mutable.ListBuffer

final class ConcurrentQueue[V]() {
  private val ref = new AtomicReference(Queue.empty[V])

  def enqueue(value: V): Unit = {
    var continue = true
    while (continue) {
      val current = ref.get().nn
      val update = current.enqueue(value)
      continue = !ref.compareAndSet(current, update)
    }
  }

  def dequeue(): Option[V] = {
    // A legitimate case when Option's None doesn't mean Null 🙂
    var result: Option[V] | Null = null
    while (result == null) {
      val current = ref.get().nn
      current.dequeueOption match {
        case None =>
          result = None
        case Some((v, update)) =>
          if (ref.compareAndSet(current, update)) {
            result = Some(v)
          }
      }
    }
    result.nn
  }

  def dequeueAll(): List[V] = {
    val buffer = ListBuffer.empty[V]
    var continue = true
    while (continue) {
      dequeue() match {
        case None =>
          continue = false
        case Some(v) =>
          buffer.append(v)
      }
    }
    buffer.toList
  }
}

And usage:

// Scala code
val queue = new ConcurrentQueue[String]()
queue.enqueue("Hello")
queue.enqueue("World")

for (value <- queue.dequeueAll()) {
  println(value)
}

WARN: your AtomicReference or your ConcurrentHashMap references won’t work for managing concurrent access, if the values you put in them aren’t immutable (or at least treated as such)!

If you’re ever tempted to do this, remember, THIS IS A BUG:

// Java code
final AtomicReference> ref =
  new AtomicReference<>(new ArrayList<>());

// BUG!!!
ref.get().add("value")

Note that instead of atomic references, or other concurrent and mutable data structures, you could use intrinsic locks via synchronize blocks and a mutable.Queue, of course, but by using atomic references the algorithm is lock-free, and for example, with virtual threads in Java 19 (Project Loom) by using synchronize blocks you can still have OS/platform threads blocked (pinned), whereas if you work this way, you can avoid that.

As a fair warning, the algorithm may be non-blocking, but it is not “wait-free”, and might not do well on high contention. So for performance reasons, when you need concurrent queues, you’re better off using Java’s ConcurrentLinkedQueue or specialized implementations, such as those in JCTools (which are awesome!🌞). However, you can put any immutable data structure in an AtomicReference, and then claim that you’re working with STM (or at least something close) 😎.

Epilogue

Immutable collections being used by default is one of the biggest wins people get when adopting “functional programming”, or when adopting alternative JVM languages, like Scala or Clojure. Nothing else comes close.

Even when working with Java, you can adopt immutable collection types by default, as a best practice. You can do this either by:

Wrapping them in “unmodifiable” collections, using the standard library — but this won’t bring all possible benefits;
Adopting libraries such as Vavr or Guava.

Now go forth and convince your team to adopt this best practice 🎤🎶😎

The article Immutable Collections should be Your Default first appeared on alexn.org.

On Scala 3's Optional Braces

Alexandru Nedelcu — Mon, 24 Oct 2022 09:07:21 +0000

I dislike Scala 3's significant whitespace syntax. At this point it's safe to say that I hate it 🤷‍♂️, being (IMO) an unfortunate evolution of the language.

As a disclaimer, this may well be a subjective opinion, so full disclosure: I have never liked working with languages that have significant whitespace syntax. On the list of languages I dislike most, CoffeeScript is in the top 3, with YAML being a close second, and I had hoped that CoffeeScript’s failure will finally make the notion of significant whitespace unpopular. But significant whitespace is like a siren song that keeps coming back in language design, possibly propelled by the popularity of Python and of YAML, and I don’t understand why.

This is a strongly worded article, using words such as “hate”. I’m criticising ideas, not people, and I’m only criticising Scala’s new developments because it’s a language that I love. Since we don’t do science, expressing feelings is perfectly adequate 😛

Virtues of indentation-based syntax

There are some virtues of a syntax based on significant whitespace. For example, code like this is sometimes a bug:

if (x > 0)
  foo(x)
  bar(x)

But you can have code linters, or the compiler, to force that {} when if is used as a statement, and once you add the braces, it’s much clearer what the code is supposed to do, no longer depending on whitespace. In my opinion, significant whitespace makes these instances harder to detect and solve by tooling. Think of all the copy/paste issues you can have.

if (x > 0) {
  foo(x)
}
bar(x)

// .. vs ..

if (x > 0) {
  foo(x)
  bar(x)
}

Also, which formatting style should you pick?

if (x > 0) {
  ...
} else if (x < 0) {
} else {
  ...
}
// ..vs ...
if (x > 0) {
  ...
}
else if (x < 0) {
  ...
}
else {
  ...
}

But with tools such as Scalafmt, or gofmt, this is a nonissue. It’s certainly not the kind of choice that has any impact on code quality, and it’s not enough to require the changing of an entire language. Scala could have an official coding style, enforced via Scalafmt, and this conversation would be over.

Less boilerplate you say?

I’m one of those people that doesn’t mind the ; at the end of lines in Java. I don’t miss it either, but it’s trivial to automate via the IDE, and sometimes it can serve as a useful visual delimiter. Because in fact ; separates imperative statements that are sequenced. And for {}, what I notice from my peers is that they often want more braces, not less. Don’t you have colleagues that tend to do this?

something match {
  case Something => { // <- unnecessary, yet, desired

  }
}

Even though in this case the syntax is not ambiguous, some people would prefer those extra braces as a visual delimiter. It takes Scalafmt to enforce a common style, although I can never complain about extra chars meant to make the code less ambiguous.

Scala is not Python

In Scala 3, I don’t know what the motivation was, but the word is that the new syntax is supposed to make Scala 3 more appealing to Python developers. I’m going to focus on Python, since I’m assuming that few people actually like YAML. So, at the risk of building a straw-man, I want to dispel this notion that Scala can be attractive to Python developers.

First, Python is popular in spite of its syntax, because it’s an interactive/dynamic language that’s easy to learn, it comes installed by default on all Linux distributions, and it comes with useful libraries such as Numpy, Scipy, Matplotlib, and others, which makes it the de facto standard for certain domains. To try to copy Python’s recipe for success, by making the syntax to have significant indentation, is shortsighted at best. I worked as a Python developer, and I can tell you that its syntax was my least favorite part.

Such cosmetic changes may look appealing, but any copied success recipe should start with Python striving to NOT be a TIMTOWTDI language. And Scala 3 did, in fact, introduce even more ways to express yourself. The language that proudly allowed many ways to express yourself, such as programming in the Klingon language, is Perl, which is Python’s nemesis 😎. Even more, Python historically rejected multi-line anonymous functions. In Language Design Is Not Just Solving Puzzles, Guido van Rossum says about a proposal for multi-line lambdas:

But such solutions often lack “Pythonicity” – that elusive trait of a good Python feature. It’s impossible to express Pythonicity as a hard constraint. Even the Zen of Python doesn’t translate into a simple test of Pythonicity.

… And still that’s not why I rejected this proposal. If the double colon is unpythonic, perhaps a solution could be found that uses a single colon and is still backwards compatible (the other big constraint looming big for Pythonic Puzzle solvers). I actually have one in mind: if there’s text after the colon, it’s a backwards-compatible expression lambda; if there’s a newline, it’s a multi-line lambda; the rest of the proposal can remain unchanged. Presto, QED, voilà, etcetera.

But I’m rejecting that too, because in the end (and this is where I admit to unintentionally misleading the submitter) I find any solution unacceptable that embeds an indentation-based block in the middle of an expression. Since I find alternative syntax for statement grouping (e.g. braces or begin/end keywords) equally unacceptable, this pretty much makes a multi-line lambda an unsolvable puzzle.

I find this quote very interesting, as it says that in Python significant indentation is reserved for grouping statements, and is not for describing expressions. Guido certainly finds the distinction between groups of statements and expressions to be an important one. I think it’s pretty clear that Scala’s new (fewer-braces) syntax is not “pythonic”:

xs.map: x =>
  val y = x - 1
  y * y

And I seriously fail to see how it improves on:

xs.map { x =>
  val y = x - 1
  y * y
}

Reading the Zen of Python should make it clear that Scala is, at this point, a very unpythonic language. Note that I always found it odd that Python has the “only one way to do it” mantra, since in practice that’s very far from true, but at least it tries.

New syntax is unclear

One thing that I really don’t get is the end marker:

def largeMethod(...) =
  ...
  if ... then ...
  else
    ... // a large block
  end if
  ... // more code
end largeMethod

Python does not have an end marker, Ruby or Pascal do. You could say that Python has the virtue of forcing you to keep your functions short, since obviously, indentation-based syntax is problematic for big blocks of code. I never bought that, which is why an “end marker” makes sense, except that Scala has already had a perfectly usable syntax that made use of {} braces. And no matter how much more readable you find this new end marker to be, the ensuing TIMTOWTDI is just not worth it.

Braces were already optional in Scala, in the case of expressions. For example, method definitions could omit braces, in case the implementation was a single expression. If you needed multiple statements, or in case you needed a lexical scope, you added the braces. Braces are super useful for hiding implementation details in the local scope:

val y = something()
val x = {
  // `y` and `z` can shadow values in the enclosing scope
  // and are no longer visible after this scope ends
  val y = foo()
  val z = bar()
  y + z
}

This is such a beautiful syntax. IMO, lexical scopes need visual delimiters that are more significant than indentation. It may be important to mention that scoping in Python is at the “enclosing-function” level. You can’t do what I just did here with Scala, unless you create a closure, and then execute it. Which kind of makes sense, since establishing the scope by the indentation level seems to be pretty ambiguous.

# Python code
y = something()
# creating new lexical scope
def createX():
  y = foo() # shadowing
  z = bar()
  return y + z
x = createX()

In Scala 2, groups of statements needed braces, expressions didn’t. What those braces meant (a group of statements, also creating a lexical scope) was simple to explain. In Scala 3 this now changes.

Tooling is problematic

Another problem we have is one of tooling. Scala 3 is a language that’s harder to parse than Scala 2. Syntax based on indentation is context-sensitive. IDEs, like IntelliJ IDEA or Metals, or tools like Scalafmt, now have to do more work. My own website isn’t able to correctly do syntax highlighting for Scala 3 yet.

And simple things, like copy/pasting a piece of code, are a problem because the IDE now has to guess the correct indentation level.

On backwards compatibility

Languages evolve, but there is such a thing as too much evolution, for the simple reason that backwards compatibility has to be provided, otherwise you’re effectively talking about a new language, and nobody is going to adopt the new version. This is why Scala will have to support both the old and the new indentation-based syntaxes for a very, very long time. Which means that projects will have to depend on compiler options (e.g., -no-indent), or on tools, such as Scalafmt, to impose the blessed syntax rules. And, don’t get me wrong, Scalafmt is great, but I considered it optional, whereas now it becomes mandatory. This is similar to introducing the using/given syntax, which is nice, but the old implicit keyword is still there, so it leads to more complexity, not less.

Java’s slow evolution makes a lot of sense. Love it or hate it, you can probably take a JAR compiled with Java 1.1, and it would still run on the latest JVM, and that Java 1.1 code probably compiles as well. There are some exceptions, but those are very few and far between. Java’s devotion to backwards compatibility is what propelled it to be considered a platform you can depend on, being in the same league as POSIX. Developers may not like its generics, its boilerplate, or its culture of libraries doing runtime introspection to workaround language issues. But at least its syntax won’t dramatically change overnight, and those Python developers probably learned Java in school, so they can always get back to it.

Scala 3 is succeeding in sending a message that it’s not Java++, but it’s not doing so in a way that I find appealing.

In closing

What I expected from Scala 3 was a simplification of Scala 2. It does simplify in some ways, e.g., macros are better, the type system fixes some holes, I love untagged union types, etc. But it also introduces complexity of its own, and for no good reason that I can see.

I think it’s too late to backtrack on these changes, significant-indentation syntax is probably here to stay (not in my projects), but one can hope.

The article On Scala 3's Optional Braces first appeared on alexn.org.

Building a Queue for Delayed Messages via a RDBMS (1): Design

Alexandru Nedelcu — Fri, 21 Oct 2022 06:46:07 +0000

Ever had the need to deliver messages on your queue at a certain timestamp in the future? Look no further, because your RDBMS can do it. This is part 1 of a series that builds a solution from scratch.

In our $work project we had the need to push messages on a queue, but delayed, for delivery at a certain timestamp in the future. Our existing MQ servers aren’t adequate for that, and we can’t introduce more technology in our stack.

Turns out, a relational database is perfectly adequate (such as MySQL or PostgreSQL). Here’s how…

First, establish the requirements:

Multiple consumers can read from the queue, at the same time, however only one consumer can gain access to a message for processing;
When a consumer pulls a message from the queue, that message becomes invisible for other consumers — but if, after a timeout, the consumer fails to process the message and then “acknowledge” that the processing is done (commit), then the message becomes visible again for other consumers (standard MQ functionality);
We need a message key, such that we can avoid duplicate messages (functionality usually not provided by standard queues);
The payload is usually JSON, but the datatype should be encoded in the table row, such that we know how to parse it, also doubling as an extra validation;

Run-loop

Your code should look more or less like this pseudocode, you can insert your favorite libraries and abstractions, as this could be built via reactive streams or what not:

while (true) {
  val message = selectNextMessageAvailable()
  if (message == null) {
    sleep(15.seconds)
    continue // next please
  }

  val lockAcquired = updateAcquireLock(message)
  if (!lockAcquired) {
    continue // next please
  }

  try {
    // do whatever it is we want with our message
    process(message)
    // if everything went well, mark it as processed
    delete(message)
  } catch (e: Exception) {
    logger.error(e)
  }
}

Creating the table

The RDBMS table might look like the following — note we are using MySQL/MariaDB as our chosen dialect:

CREATE TABLE DelayedQueue
(
    pKey VARCHAR(200) NOT NULL,
    pKind VARCHAR(100) NOT NULL,
    payload BLOB NOT NULL,
    scheduledAt BIGINT NOT NULL,
    scheduledAtInitially BIGINT NOT NULL,
    createdAt BIGINT NOT NULL,
    PRIMARY KEY (pKey, pKind)
);

CREATE INDEX DelayedQueue__KindPlusScheduledAtIndex
ON DelayedQueue (pKind, scheduledAt);

Thus, we have these fields:

The primary key is the tuple (pKey, pKind) — the pKey can uniquely identify the message, but pKind is needed to indicate the stored data type, as we need to validate that we can parse it;
The payload can be our JSON document;
scheduledAt is a Unix timestamp (in seconds) indicating when the message is scheduled for delivery — note that this gets used in the SELECT, but also doubles as the “lock” we acquire on messages that are in processing, being UPDATED on each pull — which is why we also need scheduledAtInitially, meant for debugging;

Pushing new messages

Pushing messages for new keys is easy:

INSERT IGNORE INTO DelayedQueue
  (pKey, pKind, payload, scheduledAt, scheduledAtInitially, createdAt)
VALUES (
  "c71de6b4-510f-11ed-9d4d-0242ac120002",
  'Contact',
  '{ "name": "Alex", "emailAddress": "noreply@alexn.org" }',
  UNIX_TIMESTAMP('2022-10-22 10:00:00'),
  UNIX_TIMESTAMP('2022-10-22 10:00:00'),
  UNIX_TIMESTAMP() -- now
);

Note, I am using INSERT IGNORE, because we may deal with duplicates. MySQL/MariaDB allows us to do that, which is pretty nice. Some databases don’t have this syntax, and in Java, primary key violations turn into exceptions. Pattern matching java.sql.SQLException is something that should be avoided at all costs, as the error you get depends on the database vendor and the context, and you need to find those by trial and error. For example, for Microsoft’s SQL Server, you have to look for error code 2627, or error code 2601, possibly in combination with sql state 23000. Whereas for HSQLDB, you have to look for error code -104 in combination with sql state 23505.

Polling the queue

We do a SELECT to see if there are any messages where scheduledAt <= NOW.

And for as long as there are no messages available, we repeat the query after a configurable delay. The time interval depends on your latency requirements, but for delayed messages this is not an issue, so you could repeat the query every 15 seconds or so. Repeating it more often could have a negative impact on the database, so be careful with this configuration.

  pKey, payload, scheduledAt, createdAt
FROM DelayedQueue
WHERE
  pKind = 'Contact' AND scheduledAt <= UNIX_TIMESTAMP()
ORDER BY scheduledAt
LIMIT 1;

Note that this query is optimized by the index that we already created.

Acquiring the lock

Once we have a message available, we have to make it invisible for other consumer, such that there is at most one consumer processing it at the same time. So we need a “lock” per message.

UPDATE DelayedQueue
SET
  -- acquires the lock, sets the timeout in 5 minutes
  scheduledAt = UNIX_TIMESTAMP() + 60 * 5
WHERE
  pKey = 'c71de6b4-510f-11ed-9d4d-0242ac120002' AND
  pKind = 'Contact' AND
  scheduledAt = 1666422000 -- concurrency check ;-)
;

Whatever database client you’re using (e.g., JDBC), it will return the number of updated rows. If the update suceeds, it should return 1, if the update fails (due to another consumer winning this race), then it should return 0. If updatedRows > 0, then you have successfully acquired the lock on this message, otherwise, you cannot proceed, instead you need to retry the transaction (SELECT + UPDATE).

NOTE: scheduledAt is updated to be in the future. THIS here is what makes it invisible to other consumers, with a 5 minutes timeout (after which it becomes visible again).

Transactional commit (acknowledge)

Once a consumer processes the message, it needs to be marked as being processed. We can do that by deletion, but we need to be careful:

DELETE FROM DelayedQueue
WHERE
  pKey = 'c71de6b4-510f-11ed-9d4d-0242ac120002' AND
  pKind = 'Contact' AND
  -- Race-condition check (1) — value should be set from user code:
  createdAt = 1666340050 AND
  -- Race-condition check (2) — value should be set from user code:
  scheduledAt = 1666422300
;

We can’t just delete anything that has that key, because:

The initial timeout might have passed, and we might now have a concurrent execution (which is inevitable);
We might have a new key that was reused;

As such, we need to check the combination of createdAt (to check that we still have the same entry, instead of a new one), and our updated scheduledAt (to ensure that we are not dealing with a concurrent execution after timeout).

Coming up next …

I’m going to follow up with an article that actually builds this, as a Java/Scala/Kotlin library.

The article Building a Queue for Delayed Messages via a RDBMS (1): Design first appeared on alexn.org.

Cut the Technobabble

Alexandru Nedelcu — Wed, 19 Oct 2022 12:00:00 +0000

"Sir, we have a bug!"; "But we used algebraic reasoning!"

The marketing for Functional Programming is made of technobabble. Technobabble was used in Star Trek. Those long discussions are what Star Trek was loved for, but technobabble isn't good for sharing knowledge or advancing our field.

Technobabble: a type of nonsense that consists of buzzwords, esoteric language, or technical jargon.

This is a follow-up to my last article, “The case against Effect Systems (IO)”.

Composition

Composition is about taking pieces and combining them in a bigger piece. Functions are automatically composable, and in FP, when people talk about composition, they usually mean this:

It’s not just functions, that’s why we have “category theory”, but to put it plainly in code:

def functionAtoB: A => B = ???
def functionBtoC: B => C = ???

// We get this for free
def functionAtoC: A => C =
  a => functionBtoC(functionAtoB(a))

This works if you have an F[_] monadic type as well, which is nice, and we can say that these functions provide us with a simple and established protocol to compose smaller pieces into bigger pieces:

def functionAtoB: A => F[B] = ???
def functionBtoC: B => F[C] = ???

// We get this for free
def functionAtoC[F[_]: Monad]: A => F[C] =
  a => functionAtoB(a).flatMap(functionBtoC)

Different monadic types don’t compose well. So, for example, if you have 2 types, F[_] and G[_], you can’t automatically combine them into an F[G[_]] or G[F[_]] (think IO[Option[_]] and Option[IO[_]]). Knowledge of their monadic nature isn’t enough for you to do that, as you need more. Hence, we have a need for “monad transformers”, e.g. OptionT, EitherT, ReaderT. Or we need another type-class, like Traverse, which allows us to transform an F[G[_]] into G[F[_]] (e.g. (list: List[IO]).sequence). That, or you can basically take the monad transformers and combine everything into a bigger type. Obviously something like this can be hard-coded, to be more ergonomic and/or efficient, moving the costs around:

type ZIO[-Env, +Err, +R] = Kleisli[EitherT[IO, Err, ?], Env, R]
// type ZIO[-Env, +Err, +R] = Env => IO[Either[Err, R]]

But folks, from where I’m sitting, I don’t see that much automatic “composition” happening for monads, in general, compared to plain old functions. The “composition” happening in Haskell’s ecosystem, via the prevalence of monad transformers, is for me a turnoff, alongside ReaderT used for dependency injection, even when encoded into something more ergonomic. I’ll take Java’s Spring over that, thanks. But that’s just a personal opinion.

“Composition” is usually technobabble because …

Objects (from OOP) compose — that’s their whole purpose actually, that’s why we care about subtype polymorphism, or encapsulation, because it’s all about their composition. We may need design patterns to compose, but they compose well. And maybe we have a hard time coming up with an automatic protocol for it, i.e., drawing those arrows from category theory, but it’s composition nonetheless. And structured/imperative programming constructs also compose. It’s why you’re able to build anything at all.

In the context of IO, what people mean by “composition” is basically “reuse” via abstract interfaces (type classes). For example, it’s nice being able to transform a List[IO[A]] into an IO[List[A]], or a List[EitherT[IO, E, A]] into an EitherT[IO, E, List[A]], by using very generic functions that make use of type classes. For example, I love doing stuff like this:

// Scala code
def processInBatches[A, B](
  batchSize: Int,
  list: List[A],
  job: A => IO[B]
): IO[List[B]] =
  list
    .sliding(batchSize, batchSize)
    .map(batch => batch.map(job).parSequence)
    .sequence
    .map(_.flatten)

Functional programming is expression-oriented, we process stuff by transforming input into output via function composition, essentially assembling a pipeline. Working with IO here allows us to remain within this paradigm, and it’s awesome for it. Expressions are awesome.

But as far as composition is concerned, this argument isn’t as strong as you’d think, because in the context of blocking I/O and side-effecting functions, going from () => A to A or from List[() => A] to List[A] is trivial, not to mention you can always use plain-old foreach loops, which also compose 🤷‍♂️

I don’t see any monadic IO in the following code:

// Kotlin code — only used because Java currently needs boilerplate
suspend fun  processInBatches(
  batchSize: Int,
  list: List,
  job: suspend (A) -> B
): List = coroutineScope {
  list.windowed(size = batchSize, step = batchSize)
    .map { batch ->
      batch
        .map { async { job(it) } }
        .awaitAll()
    }
    .flatten()
}

And, say, if you want “compositional resource-safety”, nothing beats this in terms of how clear it is, or the contract that the JVM runtime gives you:

// Java code / try-with-resources try (final var res1 = new Resource1()) { try (final var res2 = new Resource2(res1)) { //... } }
Cats-Effect’s Resource (or similar abstractions) is a super-power, being a presence in all of my Scala projects. However, it would be a mistake to think that more classic OOP alternatives aren’t available, or that it wouldn’t work with blocking I/O. The problem with resource management in Java is linked to bad design related to Closeable, or to the Java EE flavored dependency injection. C++ developers have a much better time with RAII, possibly because they have to do that for memory management, too, so it’s everywhere. I also find Scala’s own Using.Manager interesting, being a simple and effective way to cope with multiple Closeable resources in imperative code. And better options can happen for imperative code too. Solutions like Resource aren’t perfect either, because we can leak references, to be used after they are disposed.

In short: expression-oriented programming is awesome, and IO implementing common type-classes keeps you into the right paradigm. But “composition”, when used to justify the path we took (e.g., IO), is often just technobabble.

Algebraic reasoning

What “algebraic reasoning” means is that math gets used.

But this spans multiple topics, so let’s see …

Design from first principles

“First principles” are axioms, a word that I like more, used in the Romanian language as well — these are statements we consider to be true with no evidence (or implementation) available, or “intrinsic”, and then every other theorem or operation or principle gets derived from those statements. “Principled” simply means that the software system was started from a bunch of primitive/abstract operations and everything else was implemented in terms of those. Being principled is useful for demonstrating correctness. When adding new operations that get built on top of already available ones (e.g., non-abstract functions/methods), you don’t need to demonstrate again the correctness of your entire design.

Designing “from first principles” is a choice for the library authors in order to cut costs, and … that’s about all there is to it. As it does nothing for the user, and can be a hindrance as well. Because obviously, the real world isn’t principled. Even choosing the principles you’re building on is a matter of design (aka, ideology). There are really complex systems that you’re using every day, that aren’t based on first principles. Designs based on science and engineering are very often not based on first principles.

Type class laws

Defining a “technology compatibility kit” (TCK) (aka “the laws”) … is important in case users can implement abstract interfaces / protocols that need a certain behavior that isn’t well expressed just via the exposed types. And these “laws” are defined with math language, with equivalence tests between expressions, but this isn’t a requirement for defining a TCK, and quite often math is not enough when testing the side effects.

When you see a flatMap, you know it should have a certain behavior (i.e., left+right identity, and associativity). Monad’s definition is described by “laws” expressed with math. And yet IO should not cache its result, a potential side effect that can’t really be described with math. Which is fine, I mean, the Reactive Streams specification + TCK isn’t algebraic, but it’s useful nonetheless.

I’d also argue that you don’t need algebraic properties in order to be able to operate things, to have a good mental model for it (that only has to be useful, not real). I keep recommending the book The Design of Everyday Things for that reason. Algebraic reasoning is useful in the design process, but if you have to look at math expressions in order to predict the behavior of the API you’re operating, that design is error-prone and hard to use.

People, programmers, in general, don’t care about algebraic reasoning. What people care about is the User Experience™️, and the harsh reality is that UX usually trumps theorem proofs. Just look at Go/Python versus Haskell. This is why statically typed languages aren’t winning the markets dominated by dynamically typed languages, and will never do, for as long as the UX doesn’t improve, as dynamic languages are great at UX.

Good UX is about exposing a user interface that, after some training, can allow the user to go in autopilot mode. Think about driving a car, as programming isn’t very different. In that sense, the laws of type classes can help due to having tests for common protocols, implemented by different types, such that you can rely on those protocols no matter what the types represent. I think the innovation is in having a TCK in the first place, and less about that TCK being algebraic.

Local reasoning

“Local reasoning” means that:

You can take a piece of code, and understand what it does, without the wider context in which it gets used;

In the context of FP, it means that you can assess the correctness of a function call without depending on the history of its invocations (i.e., functions are deterministic);

For one, code makes sense locally, without a wider context. This isn’t a property of FP, necessarily. The Linux kernel is famous for rejecting C++ for that reason — C doesn’t have classes, and its subroutines need to have any state passed in as parameters. This makes it easier for code reviewers to judge commit diffs. Don’t believe me, see Linus Torvalds’ thoughts on C++:

For example, I personally don’t even write much code any more, and haven’t for years. I mainly merge…

One of the absolute worst features of C++ is how it makes a lot of things so context-dependent - which just means that when you look at the code, a local view simply seldom gives enough context to know what is going on.

That is a huge problem for communication. It immediately makes it much harder to describe things, because you have to give a much bigger context. It’s one big reason why I detest things like overloading - not only can you not grep for things, but it makes it much harder to see what a snippet of code really does.

Put another way: when you communicate in fragments (think “patches”), it’s always better to see “sctp_connect()” than to see just “connect()” where some unseen context is what makes the compiler know that it is in the sctp module.

And C is a largely context-free language. When you see a C expression, you know what it does. A function call does one thing, and one thing only - there will not be some subtle issue about “which version” of a function it calls.

So there are particular reasons why I think C is “as simple as possible, but no simpler” for the particular case of an OS kernel, or system programming in particular.

I think C is as far from functional programming as you can get. The language in which you pass stuff as void*, only to reinterpret it as anything, depending on context, is as mutable, as dirty and as unsafe as it can get. And here is Linus Torvalds, talking about being able to locally reason about its subroutines, better than he’d be able to do with C++.

Going back to FP, it’s nice when the correctness of a computation does not depend on the history of invocations. Eliminating non-determinism is the ideal in functional programming. The dirty secret, however, is that’s not happening with IO.

class Counter private (ref: AtomicInt) { // Looks like it depends on the history of invocations to me 🤷‍♂️ // I mean, technically, the function call is deterministic, but it's // not returning the data that we crave for, this being codata (computations); def increment: IO[Int] = IO(ref.incrementAndGet) }

The ideal in functional programming is for functions to return data, the output depending entirely on the call’s (explicit) input parameters, such that the output doesn’t depend on prior invocations of that function. But that’s when you avoid IO, because IO is modeling access to shared mutable state by definition.

You heard it here first, “local algebraic reasoning” is usually technobabble.

Is FP good at all? Is IO?

Of course!

“Referential transparency”, driving the “substitution model” of evaluation, isn’t technobabble, although I have my doubts when we are talking of “codata” (suspended computations). But in fairness, with IO, even if the program is still modeling the access to shared mutable state, and still describes a step-by-step computation, fact of the matter is that IO keeps people honest.

For example:

Access to heavy resources, that can only be built via side effects, has to be modelled via IO;

This forces those resources to be passed as function or constructor parameters;

Shared global state, even when it exists, becomes more local — ideally, still depends on code reviews;

The state of the world (e.g., the current time, or randomness), passed as function parameters, forces saner data modelling (e.g., maybe you don’t need that timestamp there), and makes the code easier to test — ideally, or you can just have IO everywhere, much like having side effects everywhere;

All of these are best practices, but best practices are best enforced by a compiler 😈

Also, functional programming is expression-oriented programming, and expressions are awesome, as it puts you into the mindset of transforming data via pipelines (aka function composition). It’s all about the UX, frankly. It’s always about the UX.

I’m starting to have some doubts about using monads for modelling side effects, though, but I’m still digesting my feelings 🤷‍♂️

Type R? (open on YouTube.com)

The article Cut the Technobabble first appeared on alexn.org.

The case against Effect Systems (e.g., the IO data type)

Alexandru Nedelcu — Thu, 13 Oct 2022 09:00:00 +0000

As Scala developers and fans of the functional programming promoted by Haskell, how do we justify the use of IO to newbies coming from Java? It’s been a fun ride, but the truth is that Java 19 is changing everything.

IO has been great, with libraries like Cats-Effect and fs2 taking functional programming on the JVM to new levels, and using these libraries has been a super-power. But here I’m going to argue that the problem hasn’t been imperative programming, and this IO-driven style of static FP has an identity crisis on its hands, due to the introduction of Virtual Threads (JEP 425), and the adoption of “structured concurrency” concepts (JEP 428), first popularized in Kotlin.

How IO shines

When you have the time, you should watch Daniel Spiewak’s presentation:

The Case For Effect Systems — Daniel Spiewak (open on YouTube.com)

I loved it, and I’ve long been a proponent of IO, having exactly these arguments. And yet …

A history of asynchrony

I once wrote an article on asynchronous programming, in which I explained the progression from callbacks to monads. Here’s the TL;DR …

If we are to describe asynchronous computations with a type, it would go something like this:

type Callback[-A] = (A) => Unit type Async[+A] = (Callback[A]) => Unit

So, an async computation is something that executes somewhere else, other than the current thread, so it could be on another thread, or another process, or on another machine on the network.

Now, imagine a classic sequence of synchronous function calls:

val x = foo(); val y = bar(x); baz(); val z = qux(x, y); return z

We can turn these into a callback-driven sequence, which is what would happen in JavaScript/Node.js land, before the advent of Promise:

def foo(): A = ??? // ...becomes... def foo(): Async[A] = ??? //... // Sequence becomes: foo()(x => bar(x)(y => baz()(_ => qux(x, y)(z => ???) ) ) )

There are consequences for working like this:

callback hell, due to all the nesting;

the implementation is often stack-unsafe, so it can be hard and error-prone to express loops;

it invalidates standard language features (e.g., try/catch/finally);

it’s very low level, so managing concurrency is tricky (e.g., waiting on 2 or more jobs running in parallel to finish);

In Scala, the answer has been the introduction of Future/Promise, which is a “monadic” type, implementing a flatMap:

// Scala code def foo(): Future[A] = ??? //... foo().flatMap { x => bar(x).flatMap { y => baz().flatMap(_ => qux(x, y)) } } // ...or... for { x <- foo() y <- bar(x) _ <- baz() r <- qux(x, y) } yield r

Which kind of looks like our original, imperative, synchronous program. But, here be dragons — what happens if we invoke Future-driven functions outside that for-expression?

// Scala code val b = baz() for { x <- foo() y <- bar(x) _ <- b r <- qux(x, y) } yield r

The non-obvious answer is concurrent execution happens — this is no longer a sequence of steps, and it wasn’t necessarily what we wanted, leading to bugs, as the logic may access shared mutable state that may be broken by concurrent access. This is invalidating our common sense, because when we see val b = baz() in code, we do not expect concurrent execution, as it’s not how our brains have been trained in these imperative programming languages. This here invalidates our mental model for how things behave.

Note that we were careful waiting on the result of baz(). What happens if we forget about it?

// Scala code val b = baz() for { x <- foo() y <- bar(x) r <- qux(x, y) } yield r

Well, this is triggering a “fire and forget” job, and if it wasn’t intended, we may now have a leak 🙀

Imperative programming, the devil we know

“Imperative programming” is a paradigm that focuses on describing a program step by step, via a sequence of instructions that modify state. And I’m going to make this statement:

Imperative programming is extraordinarily intuitive!

In imperative programming ; is a separator between instructions, or statements. Imperative programming tends to be statement-oriented. But ; simply denotes sequencing of steps. We can think of it as being an operator:

A ; B ; C

The above simply means statement A gets executed before statement B, which gets executed before statement C.

We can follow sequential steps towards achieving something. For example a vast majority of humans can cook, including many children. We may be bad cooks, we may only know simple recipes, given access to a stove we may hurt ourselves, but give people the raw ingredients and the cookware, and they won’t starve. Children understand cooking too, being a sequence of steps, not rocket science.

My CS teacher from high school introduced us to algorithms with a recipe for pancakes. I don’t remember the specifics, and it’s been years since I cooked pancakes, so my memory is hazy, but I imagine the recipe went something like this:

// As an introductory lesson, he was completely ignoring resource safety, ofc: val fryingPan = takeFryingPan(); // 1 val batter = mix(eggs, milk, flour, sugar, bakingPowder, salt); // 2 fryingPan.pour(oil); // 3 fryingPan.preHeat(2.minutes); // 4 fryingPan.pour(batter); // 5 while (!fryingPan.check(isContentsBrown)) { // 6 sleep(30.seconds); // 7 fryingPan.scoop(); // 8 } fryingPan.pull(); // 9

Our IO-driven programming is still a sequence of steps. We may try to describe pure data structures that get interpreted later, but that seldom happens. In our FP programs, what actually happens in practice is still imperative in nature:

// Scala code for { fryingPan <- takeFryingPan() batter <- mix(eggs, milk, flour, sugar, bakingPowder, salt) _ <- fryingPan.pour(oil) _ <- fryingPan.preHeat(2.minutes) _ <- fryingPan.pour(batter) _ <- { def loop(): IO[Unit] = fryingPan.check(isBrown).flatMap { case true => IO.unit case false => IO.sleep(30.seconds).flatMap(_ => loop()) } loop() } r <- fryingPan.pull() } yield r

In Haskell, and in the Scala FP community, the semicolon (;) gets replaced with flatMap (AKA bind, >>=, SelectMany, etc.).

A ; B ; C; // becomes... A.flatMap(_ => B).flatMap(_ => C)

This is made digestible via syntactic sugar. In Haskell that’s the do-notation, in Scala we have the for comprehensions, and in F# we have computation expressions.

So our Scala programs can look like this:

// Scala code for { x <- foo() y <- bar(x) _ <- baz() z <- qux(x, y) } yield z

This is similar to what we did with Future. In Scala this syntactic sugar is more generic, driven by monadic types, because in languages such as C# or TypeScript/JavaScript, we have special async/await syntax that only works with Future/Promise/Task data types and that looks like this:

// JavaScript code async function doStuff() { const x = await foo() const y = await bar(x) await baz() // better not forget the `await` 😉 const z = await qux(x, y) return z }

Call-sites of functions returning IO, however, are referentially transparent. And it’s important to contrast with Future/Promise. Take for example this program:

// Scala code for { x <- fireRocketsToMars() y <- fireRocketsToMars() } yield x + y // ... versus ... val r = fireRocketsToMars() for { x <- r y <- r } yield x + y // ... versus ... val rx = fireRocketsToMars() val ry = fireRocketsToMars() for { x <- rx y <- ry } yield x + y

What’s the difference of behavior between IO and Future in this case?

// Scala code def fireRocketsToMars(): IO[Int] // ... versus ... def fireRocketsToMars(): Future[Int]

Well, with IO the behavior of the program is the same, in all 3 cases, whereas with Future the behavior changes. With Future we can talk of 3 different programs with wildly different behavior. And that’s not good, as it can be counter-intuitive, being a source of bugs. IO in this case behaves as it should, although in the context of Scala, IO isn’t without fault either.

// Scala code // No concurrent execution here, but we never use this value, // so this is a hard to trace no-op: val bazJob = baz() for { x <- foo() y <- bar(x) z <- qux(x, y) } yield z

Speaking of, I am asking questions on this difference of behavior in interviews. Surprisingly, many people get this wrong, in spite of having experience with real-world IO usage. And that’s not their fault. The fault lies with the Scala language, because:

Future is a broken abstraction, completely beyond redemption, not much better than the callback hell it improves on, and the async/await syntactic sugar is only an ineffective band-aid — async doesn’t help, as the default evaluation model should never be concurrent execution, you shouldn’t need to mention an explicitawait to force sequencing;

Scala, here, suffers from a severe case of TIMTOWTDI, because it’s a strict language, and has multiple ways of expressing the sequencing of instructions, and it’s no wonder that beginners are getting confused;

More on that later, but first a rant on math 😎

Math is not intuitive

Math is abstraction, and it takes maturity to learn math abstraction. Some people never do. My high-school teacher used to say that there are two kinds of students, those that understand the formal definitions of limits (with the epsilon notation), and those that don’t. I always thought that’s just a language problem, use better communication and more people will understand, but it’s without doubt that children need to develop the necessary cognitive abilities before understanding abstraction.

This is important to realize, because, while IO-driven programs are very much imperative in nature, laziness brings us closer to math. It’s why languages like Haskell may never be in top 5, or why Scala can have a lot of accidental complexity. In Curse of the Excluded Middle, Erik Meijer argues just that … in imperative, strictly-evaluated languages, lazy behavior is surprising, and that’s bad.

I’d argue that laziness can be surprising in general, even if you’re working in a non-strict language, such as Haskell. Haskell’s non-strict evaluation keeps people honest. You can always call unsafePerformIO to trigger side effects that aren’t tracked by the type system, but it’s tricky getting the runtime to actually evaluate it, esp if you don’t need the returned result. This means that shortcuts meant for debugging are hard (e.g., logging), and this can surprise people. And no sufficiently smart compiler or runtime has been invented yet to solve efficiency issues, which are a problem, because performance is hard to reason about when thunks get lazily evaluated, even in terms of big-O complexity. Data structures, at least, are meant to be already evaluated and inspectable. When that doesn’t happen (e.g., streams), that’s “codata”, it takes (runtime) effort to inspect such values, and Haskell seriously blurs the lines between them.

For Haskell developers out there that disagree, I have a question — in your programs, are you using String, or are you using Text?

There are many things I like about doing FP in Scala, and having strict evaluation as the default is one of them.

Future is bad because asynchrony is bad

When I say that Future is broken, the reason is that its usage is prone to accidents. Future is an honest representation of asynchronous computations, and that’s not the fault of imperative programming.

To wit, we started from this:

// Scala code def fireRocketsToMars(): IO[Int] // ... versus ... def fireRocketsToMars(): Future[Int]

But there is a third option that we did not take into account, which is blocking I/O:

def fireRocketsToMars(): Int

With blocking I/O expressed like this, we have no need for flatMap or for special sequencing syntax, as it’s pretty damn clear what’s going on — at least in terms of sequencing of steps. It’s not all rosy, my problem being that the signature is lying, as it doesn’t make it clear that there are dangerous side effects going on (but in Scala this could be fixed).

val x = fireRocketsToMars(); val y = fireRocketsToMars(); x + y

It’s important to realize the virtues of doing this:

if everything becomes synchronous/blocking by default, there are no accidents related to accidental concurrent execution;

the distinction between data and codata (i.e., data structures versus computations) becomes clear as day;

standard language constructs still work (e.g., for/while loops, try-catch-finally, try-with-resources or Java’s checked exceptions);

Here’s the other sample, again:

val x = foo(); val y = bar(x); baz(); val z = qux(x, y); return z

I’d argue that there is no meaningful difference between this, and the equivalent IO-driven program, at least in terms of accidents that can happen. If this were Future-driven, a lot of things could go wrong because all of those invocations could be concurrent, by accident. But we are not using Future here.

There can be no accidents here, because once the execution returns from a function invocation, that function is done. And having this mental model is awesome.

Java: what’s old is new again

Blocking I/O has always been the norm in Java land. Java was built for using threads, and for blocking those threads. Java’s memory model, the ease of working with threads, was one of its main innovations.

This is why Java’s standard library is filled with concurrency primitives that block, such as BlockingQueue, Semaphore, ReadWriteLock or ReentrantLock, with no async equivalents. It’s why Java has a Future interface whose only means of getting its result is a blocking .get() call.

The newer CompletableFuture happened in Java 8. Java 8 also introduced lambda expressions, so you can feel that was about the time asynchronous programming APIs took off, as a sort of detour from the official way of doing things.

And this is because Java does “1:1 kernel-level threading”, meaning that all Java threads are OS/platform threads, and platform threads are super expensive. This is because:

each thread has its own call-stack, thus consuming memory;

the kernel does “preemptive multithreading”, so it does its best to execute many threads on few CPU cores — to do this, the kernel assigns time slots, pausing running threads, resuming previously paused threads, in a process called “context switching” — during which the memory used by a thread needs to be reloaded in a CPU’s cache hierarchy, which consumes a lot of CPU;

due to consumption of both memory and CPU, there’s a low limit on how many threads you can use;

The answer in Java land has been to work with thread-pools, to reuse available threads as much as possible, and to limit the maximum number of threads that can be started. Thread-pools are problematic as well, because:

ThreadLocal values can now leak, and thread interruption is very unsafe if you don’t own the thread;

Complicated libraries tend to start their own thread-pool, and you can assess the maturity of Java projects by the number of thread-pools active at the same time;

Blocking I/O makes limiting threads hard, because you can end up with thread-starvation, a type of deadlock, a situation in which threads are unable to make progress due to hard limits on the thread-pool;

And managing threads and thread-pools being low level, the community evolved towards using “reactive”/fluent APIs, such as RxJava, Project Reactor, Akka, and others. Which are essentially libraries meant to recreate “M:N threading” on top of the JVM, i.e., multiplexing many jobs on few OS/platform threads.

Java 19 introduces “virtual threads”, bringing M:N threading support at the runtime level, which makes threads, and blocking I/O cheap. It’s not perfect, as the JVM can still block OS threads, generating “pinned” events in “flight recorder”, which I’m sure will be the new bread and butter of profilers everywhere. But everything in the language and the standard library starts making sense again. All those APIs built for blocking suddenly become much cheaper to use, effectively obsoleting the Future data types.

Structured concurrency

Thus far we’ve seen that sequential / synchronous execution is intuitive and should probably be the default, in order to prevent concurrency accidents (in our general purpose programming languages, not talking of domain-specific ones). But what if we need concurrency?

“Structured concurrency”, as a concept, is similar to that of structured programming. Back in the day, when GOTO-driven languages were still used, introductory CS lessons included an incursion into structured programming and why it is needed.

Old-school structured programming diagram, showing Euclid's algorithm for the "greatest common divisor".

The problem with GOTO statements is that they create a fork in the road, the program’s flow becoming very hard to follow, leading to unmaintainable code. Edsger Dijkstra called GOTO statements harmful, because it complicates program analysis, verifying the correctness of algorithms becoming difficult, particularly the correctness of loops.

In the context of concurrency, this should sound very familiar. With classical Java, concurrent execution would look like this:

// Java code final var mixJob = ec.submit(() -> mix(eggs, milk, flour, sugar, bakingPowder, salt) ); final var prepareFryingPanJob = ec.submit(() -> { final var fryingPan = takeFryingPan(); fryingPan.pour(oil); fryingPan.preHeat(Duration.ofMinutes(2)); return fryingPan; }); final var fryingPan = prepareFryingPanJob.get(); final var batter = mixJob.get(); //...

We now have concurrent execution, making more efficient use of our resources. But therein lie problems:

if prepareFryingPanJob.get() throws an exception, the execution of mixJob won’t get cancelled, thus creating a leak;

if the current thread gets interrupted, the interruption signal doesn’t propagate to the started concurrent tasks;

if prepareFryingPanJob takes a long time to execute, but mixJob fails immediately, we won’t see that failure until prepareFryingPanJob finishes;

Kotlin’s coroutines did not introduce the notion of “structured concurrency”, but I think it popularized it. The basic idea is this:

Concurrent jobs should be cancellable;

Concurrent jobs get started in a “scope”, and that scope can’t finish until all started concurrent jobs finish or get cancelled;

On error, all running concurrent jobs get cancelled;

This is similar to the idea behind C++’s RAII.

// Kotlin code coroutineScope { val mixJob = async { mix(eggs, milk, flour, sugar, bakingPowder, salt) } val prepareFryingPanJob = async { val fryingPan = takeFryingPan() fryingPan.pour(oil) fryingPan.preHeat(2.minutes) fryingPan } val fryingPan = prepareFryingPanJob.await() val mix = mixJob.await() //... }

In Kotlin, if the code of such a coroutineScope throws an error, all its concurrent jobs gets cancelled. The scope also awaits all concurrent jobs to finish, before it can finish, so there can be no accidental “fire and forget” jobs. And if any of the concurrent jobs throws an exception, then the other concurrent job gets cancelled.

Java 19 also introduced very experimental extensions for doing the same, in JEP 428:

// Java code try (var scope = new StructuredTaskScope.ShutdownOnFailure()) { final var mixJob = scope.submit(() -> mix(eggs, milk, flour, sugar, bakingPowder, salt) ); final var prepareFryingPanJob = scope.submit(() -> { final var fryingPan = takeFryingPan(); fryingPan.pour(oil); fryingPan.preHeat(Duration.ofMinutes(2)); return fryingPan; }); scope.join(); scope.throwIfFailed(); final var fryingPan = prepareFryingPanJob.resultNow(); final var batter = mixJob.resultNow(); //... }

This API is currently “incubating” and looks clumsy, but the concept is the same, and its efficient use is made possible due to blocking I/O becoming cheap.

What would we do in Scala with Cats-Effect’s IO?

// Scala code // No-op val mixJob = mix(eggs, milk, flour, sugar, bakingPowder, salt) // No-op val prepareFryingPanJob = for { fryingPan <- takeFryingPan _ <- fryingPan.pour(oil) _ <- fryingPan.preHeat(2.minutes) } yield fryingPan (mixJob, prepareFryingPanJob).parMapN { (mx, fryingPan) => //... }

With Cats-Effect IO concurrency/parallel execution must be made explicit. Here we are using the parMapN operator from the Parallel type class. The creation of those tasks is lazy. Nothing gets executed then and there. This makes it a little clumsy, so you’d better turn your linter to warn against unused values.

But this raises important questions — in this context, what does IO buy us? I’m finding this question increasingly difficult to answer. I used to say that IO is very explicit about how things get evaluated (e.g., in parallel or sequential), so there can be no accidents, but IO isn’t the only way for achieving that. And due to its laziness, it introduces some accidental complexity of its own.

IO is very composable. You can, for example, combine it with Either, via EitherT. Or you can bake EitherT in, like what ZIO did.

But with blocking I/O, in Java, you can make use of checked exceptions again. And for Kotlin, checkout Arrow, see their article on why suspend () -> A instead of IO. If typed exceptions is your cup of team, here’s how that sample would look like:

// Kotlin code suspend fun makePancakes(): Either = either { val fryingPan = takeFryingPan().bind() val batter = mix(eggs, milk, flour, sugar, bakingPowder, salt).bind() fryingPan.pour(oil).bind() fryingPan.preHeat(2.minutes).bind() fryingPan.pour(batter).bind() //... }

For another more real use-case, here’s a snippet from my own personal project:

// Kotlin code either { val project = Either .fromNullable(config.projects[projectKey]) .mapLeft { RequestError.NotFound("Project `$projectKey` does not exist") } .bind() val signature = call.request.header("X-Hub-Signature-256") ?: call.request.header("X-Hub-Signature") val body = call.receiveText() EventPayload .authenticateRequest(body, project.secret, signature) .bind() val parsed = EventPayload.parse(call.request.contentType(), body).bind() val result = if (parsed.shouldProcess(project)) { commandTriggerService.triggerCommand(projectKey) } else { RequestError.Skipped("Nothing to do for project `$projectKey`").left() } result.bind() }

This isn’t different from what was tried in Scala via attempts like Monadless, a use-case that Kotlin’s coroutines makes comfortable. In my opinion, if Scala continues to embrace flatMap, then it should expand the syntax of “for comprehensions” to be more ergonomic. F#’s “computation expressions” look more like imperative programming, going beyond flatMap and blending better within F#’s syntax, and that’s good.

In closing

Functions working with monadic types, such as IO, are referentially transparent, following the substitution model of evaluation, and facilitate algebraic reasoning. But showing what that is good for can be a challenge, especially in light of the TIMTOWTDI.

IO is awesome, but its existence in strictly-evaluated languages is increasingly questionable. Even in Scala land, many developers don’t believe that monads are the only or the best way to deal with effects, even if (I bet) the community’s opinion is increasingly biased, due to the natural churn that happens (people that like monads may stick around more than those that don’t). I’m happy that proposals for alternatives still happen (even if they may go nowhere), checkout: PRE-SIP: Suspended functions and continuations.

IO (and monads in general) on the JVM can have a bright future, but need better stories to tell.

The article The case against Effect Systems (e.g., the IO data type) first appeared on alexn.org.

Execute Shell Commands in Java/Scala/Kotlin

Alexandru Nedelcu — Mon, 03 Oct 2022 05:13:13 +0000

The following describes snippets for executing shell commands, in Java, Scala, and Kotlin, using standard functionality. It’s also useful to compare Java vs Scala vs Kotlin for this particular problem.

These snippets make use of Runtime.getRuntime().exec. You may find libraries that already do this, but I find dependencies to be a risk, and wheel reinvention to be fun. I apologize for the NIH in advance.

UPDATE (2022-10-05): all code samples were updated to concurrently collect the output from the input streams. This makes the code more foolproof, as programs with a lot of output can overflow a stream’s buffer.

This article is a follow-up to: execute shell commands in F#.

Java

This has been developed with Java 17, so please excuse the use of newer syntax additions, such as “record”, or “var”. The only dependency that this declares is Apache’s commons-text, because we need to do proper escaping of shell arguments:

///usr/bin/env jbang "$0" "$@" ; exit $? //JAVA 17+ //DEPS org.apache.commons:commons-text:1.9 import org.apache.commons.text.StringEscapeUtils; import java.io.IOException; import java.nio.charset.StandardCharsets; import java.nio.file.Path; import java.time.Duration; import java.util.Arrays; import java.util.Objects; import java.util.concurrent.*; import java.util.stream.Collectors; record CommandResult( int exitCode, String stdout, String stderr ) {} class OSUtils { /** * Executes a program. This needs to be a valid path on the * file system. * * See {@link #executeShellCommand(ExecutorService, String, String...)} * for the version that executes `/bin/sh` commands. */ public static CommandResult executeCommand( ExecutorService es, Path executable, String...args ) throws IOException, InterruptedException { Objects.requireNonNull(executable); Objects.requireNonNull(args); final var commandArgs = prepend( executable.toAbsolutePath().toString(), args ); final var proc = Runtime.getRuntime().exec(commandArgs); Future stdout = null; Future stderr = null; try { // Reading output streams in parallel stdout = es.submit(() -> proc.getInputStream().readAllBytes()); stderr = es.submit(() -> proc.getErrorStream().readAllBytes()); return new CommandResult( proc.waitFor(), // Should both be ready after `waitFor` new String(stdout.get(), StandardCharsets.UTF_8), new String(stderr.get(), StandardCharsets.UTF_8) ); } catch (ExecutionException e) { throw new RuntimeException(e); } finally { // Should close streams as well: proc.destroy(); // Idempotent — it's fine if already complete: TaskUtils.cancelAll(stdout, stderr); } } /** * Executes shell commands. *
* WARN: command arguments need be given explicitly because * they need to be properly escaped. */ public static CommandResult executeShellCommand( ExecutorService es, String command, String... args ) throws IOException, InterruptedException { Objects.requireNonNull(command); Objects.requireNonNull(args); final String shellCommand = Arrays .stream(prepend(command, args)) .map(StringEscapeUtils::escapeXSI) .collect(Collectors.joining(" ")); return executeCommand( es, Path.of("/bin/sh"), "-c", shellCommand ); } private static String[] prepend(String elem, String[] array) { final var newArray = new String[array.length+1]; newArray[0] = elem; System.arraycopy(array, 0, newArray, 1, array.length); return newArray; } } class TaskUtils { public static A withTimeout( ExecutorService es, Duration timeout, Callable task ) throws InterruptedException, TimeoutException { final var ft = new FutureTask<>(task); try { es.submit(ft); return ft.get(timeout.toMillis(), TimeUnit.MILLISECONDS); } catch (ExecutionException e) { ft.cancel(true); throw new RuntimeException(e); } catch (Exception e) { ft.cancel(true); throw e; } } public static void cancelAll(Future...futures) { for (final var f : futures) if (f != null) { f.cancel(true); } } } class Main { public static void main(String[] args) throws Exception { final var es = Executors.newCachedThreadPool(); try { final var r = TaskUtils.withTimeout(es, Duration.ofSeconds(2), () -> OSUtils.executeShellCommand(es, "ls", "-alh") ); System.out.print(r.stdout()); System.out.print(r.stderr()); System.exit(r.exitCode()); } finally { es.shutdown(); } } }

Scripting with Java

The above sample is an executable script, you can play with it directly via JBang. On macOS this can be easily installed via:

brew install jbang # You might want this too: brew install openjdk@17

Save the above script as runCommand.java. You can then execute the script above:

jbang ./runCommand.java # Or make the script executable; works due to the included 'shebang' # (https://en.wikipedia.org/wiki/Shebang_(Unix)) chmod +x ./runCommand.java # And then run it directly ./runCommand.java

Notes on concurrency in Java

The code needs an explicit ExecutorService because we need concurrent execution for:

reading the STDOUT and STDERR input streams;

triggering a timeout with interruption of the running process;

I don’t like passing explicit ExecutorService references, because blocking I/O is best executed on top of unbounded thread-pools, so the configuration is error-prone. On the other hand, platform threads are expensive, and I also don’t like libraries that initiate their own thread-pools for blocking I/O. This is why I’m very happy about the upcoming Virtual Threads from Java 19.

To collect the input streams in parallel, we submit 2 jobs in this thread-pool:

Future stdout = es.submit(() -> proc.getInputStream().readAllBytes()); Future stderr = es.submit(() -> proc.getErrorStream().readAllBytes());

In Java, cancelling blocking I/O tasks is done via thread interruption, however it’s a low-level protocol that’s very error-prone (especially if you don’t own the threads you’re interrupting). This is why it’s best to leave interruption to higher level abstractions, like Future references initialized via ExecutorService#submit. We use ExecutorService.submit in order to create Future references that can be cancelled. In this case it doesn’t really help, as that InputStream#read doesn’t listen to thread interruption signals, but it’s a good practice anyway, as a future implementation might be interruptible.

And to have blocking I/O timing out after a timespan, we can use the same mechanism, although for such use-case you’re better off finding a library that does this better. And I am hopeful for what will come out of the additions for structured concurrency.

public static A withTimeout( ExecutorService es, Duration timeout, Callable task ) throws InterruptedException, TimeoutException { final var ft = new FutureTask<>(task); try { es.submit(ft); return ft.get(timeout.toMillis(), TimeUnit.MILLISECONDS); } catch (ExecutionException e) { ft.cancel(true); throw new RuntimeException(e); } catch (Exception e) { ft.cancel(true); throw e; } }

Note how I’m just using blocking I/O, and not bothering with any async abstractions here. That’s because Java 19 moves Java’s paradigm back to blocking I/O, and the underlying API (Runtime#exec) is based on blocking I/O.

Scala

For Scala, we’re going to introduce a Cats-Effect dependency to describe the above as an interruptible IO data type. It’s not in the standard library, but all my Scala projects have an IO data type 😎. Besides the cool factor of working with this legendary monadic type, IO can handle the timeout under the hood via thread interruption:

#!/usr/bin/env -S scala-cli shebang -q //> using scala "2.13.9" //> using lib "org.typelevel::cats-effect::3.3.12" //> using lib "org.apache.commons:commons-text:1.9" import cats.effect.{ExitCode, IO, IOApp} import cats.syntax.all._ import org.apache.commons.text.StringEscapeUtils import java.nio.charset.StandardCharsets.UTF_8 import java.nio.file.Path import scala.concurrent.duration._ final case class CommandResult( exitCode: Int, stdout: String, stderr: String, ) object OSUtils { def executeCommand(executable: Path, args: String*): IO[CommandResult] = IO.blocking { val commandArgs = executable.toAbsolutePath.toString +: args Runtime.getRuntime.exec(commandArgs.toArray) } // A `bracket` works like `try-with-resources` or `try-finally` .bracket { proc => // These aren't "interruptible", what actually interrupts them // is proc.destroy(); and due to how they are used, it's better // to not declare them as interruptible, as to not mislead: val collectStdout = IO.blocking { new String(proc.getInputStream.readAllBytes(), UTF_8) } val collectStderr = IO.blocking { new String(proc.getErrorStream.readAllBytes(), UTF_8) } // This is actually cancellable via thread interruption val awaitReturnCode = IO.interruptible { proc.waitFor() } for { // Starts jobs asynchronously stdoutFiber <- collectStdout.start stderrFiber <- collectStderr.start // Waits for process to complete code <- awaitReturnCode // Reads output stdout <- stdoutFiber.joinWithNever stderr <- stderrFiber.joinWithNever } yield { CommandResult(code, stdout, stderr) } } { proc => IO.blocking { println("Destroying process") proc.destroy() } } def executeShellCommand(command: String, args: String*): IO[CommandResult] = executeCommand( Path.of("/bin/sh"), "-c", (command +: args).map(StringEscapeUtils.escapeXSI).mkString(" ") ) } object Main extends IOApp { def run(args: List[String]): IO[ExitCode] = for { r <- OSUtils.executeShellCommand("ls", "-alh").timeout(3.seconds) _ <- IO.print(r.stdout) _ <- IO.print(r.stderr) } yield ExitCode(r.exitCode) }

Scripting with Scala

The above Scala sample is an executable script, you can play with it directly via Scala CLI. On macOS this can be easily installed via:

brew install Virtuslab/scala-cli/scala-cli

Save the above script as runCommand.scala. You can then execute it:

scala-cli run ./runCommand.scala # Or make the script executable; works due to the included 'shebang' # (https://en.wikipedia.org/wiki/Shebang_(Unix)) chmod +x ./runCommand.scala # And then run it directly ./runCommand.scala

Notes on concurrency in Scala

IO can handle this automatically via its timeout method:

import scala.concurrent.duration._ //... for { r <- OSUtil .executeShellCommand("sleep", "30") .timeout(3.seconds) _ <- IO.print(r.stdout) _ <- IO.print(r.stderr) } yield ExitCode(r.exitCode)

This works because we’ve used IO.bracket and the IO.interruptible builder in executeCommand, which knows how to cancel the running task via actual thread interruption. And timeout creates a concurrent race condition, cancelling the running process after the given timespan. The Cats-Effect library is designed for safe resource acquisition and release, having cancellation baked in.

Forking concurrent “fibers” happens, in this sample, via .start. At the moment of writing, forked fibers aren’t getting cancelled when the main fiber is cancelled. This is because Cats-Effect has other mechanisms for dealing with scopes, and .start is considered to be lower-level. But it was appropriate for this sample.

Kotlin

For Kotlin, we are going to use its coroutines support (with the kotlinx.coroutines dependency):

///usr/bin/env jbang "$0" "$@" ; exit $? //JAVA 17+ //KOTLIN 1.7.20 //DEPS org.apache.commons:commons-text:1.9 //DEPS org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.4 import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.async import kotlinx.coroutines.runBlocking import kotlinx.coroutines.runInterruptible import kotlinx.coroutines.withContext import kotlinx.coroutines.withTimeout import org.apache.commons.text.StringEscapeUtils import java.nio.charset.StandardCharsets.UTF_8 import java.nio.file.Path import kotlin.time.Duration.Companion.seconds data class CommandResult( val exitCode: Int, val stdout: String, val stderr: String, ) /** * Executes a program. This needs to be a valid path on the * file system. * * See [executeShellCommand] for the version that executes * `/bin/sh` commands. */ suspend fun executeCommand( executable: Path, vararg args: String ): CommandResult = // Blocking I/O should use threads designated for I/O withContext(Dispatchers.IO) { val cmdArgs = listOf(executable.toAbsolutePath().toString()) + args val proc = Runtime.getRuntime().exec(cmdArgs.toTypedArray()) try { // Concurrent execution ensures the stream's buffer doesn't // block processing when overflowing val stdout = async { runInterruptible { // That `InputStream.read` doesn't listen to thread interruption // signals; but for future development it doesn't hurt String(proc.inputStream.readAllBytes(), UTF_8) } } val stderr = async { runInterruptible { String(proc.errorStream.readAllBytes(), UTF_8) } } CommandResult( exitCode = runInterruptible { proc.waitFor() }, stdout = stdout.await(), stderr = stderr.await() ) } finally { // This interrupts the streams as well, so it terminates // async execution, even if thread interruption for that // InputStream doesn't work proc.destroy() } } /** * Executes shell commands. * * WARN: command arguments need be given explicitly because * they need to be properly escaped. */ suspend fun executeShellCommand( command: String, vararg args: String ): CommandResult = executeCommand( Path.of("/bin/sh"), "-c", (listOf(command) + args) .map(StringEscapeUtils::escapeXSI) .joinToString(" ") ) fun main(vararg args: String) = runBlocking { // Dealing with timeouts val r = withTimeout(3.seconds) { executeShellCommand("ls", "-alh") } System.out.print(r.stdout) System.err.print(r.stderr) System.exit(r.exitCode) }

Scripting with Kotlin

JBang has experimental Kotlin support. Save the above script as runCommand.kt. You can then execute the script like so:

jbang ./runCommand.kt # Or make the script executable; works due to the included 'shebang' # (https://en.wikipedia.org/wiki/Shebang_(Unix)) chmod +x ./runCommand.kt # And then run it directly ./runCommand.kt

Notes on concurrency in Kotlin

Kotlin’s coroutine jobs are cancellable. The runInterruptible function transforms regular blocking code into a suspending function that can be cancelled, with the cancellation signal being converted into thread interruption. And by explicitly specifying Dispatchers.IO as the “coroutine context”, we also require the execution to happen on the thread-pool designated for blocking I/O tasks.

Due to use of the API, and due to the structured concurrency design, installing timeouts works as expected:

import kotlinx.coroutines.withTimeout import kotlin.time.Duration.Companion.seconds //... val r = withTimeout(3.seconds) { executeShellCommand("sleep", "30") }

This is equivalent with the Scala sample. Kotlin’s “suspended” functions are slick, for many purposes being equivalent with IO (also see Arrow).

An IO data type can be better due to reusability and compositionality, but it relies on flatMap handling the sequencing of effects (instead of Java’s ;), which is at the same time a strength and a weakness — for one, it can be awkward to learn and use without syntactic sugar, it leads to more TIMTOWTDI, and it encourages reuse via a lot of type-level programming that may be too complex. In Scala having type classes and Cats is pretty awesome, though.

The article Execute Shell Commands in Java/Scala/Kotlin first appeared on alexn.org.

The Trouble with Checked Exceptions: Part 2

Alexandru Nedelcu — Wed, 28 Sep 2022 22:41:48 +0000

Java’s Checked Exceptions are problematic, and it’s not only due to their ergonomics. The bigger problem is that they are in conflict with abstraction and OOP. Also, few people care about typed exceptions (unless they are happy path results, not errors).

And Scala solutions have a tendency to reinvent Java’s checked exceptions, due to the allure of static typing, I guess. You should read my previous articles on this matter:

Bifunctor IO and Java’s Checked Exceptions;

Scala OOFP Design Sample, in which I argue for “designing errors out of existence”;

In support, I stumbled on the perfect example from Java’s standard library. It’s a classic one, it turns out, I’m not the first to point at it … Java’s StringBuilder, which implements the following method:

Appendable append(CharSequence csq) throws IOException;

Which is why code using StringBuilder can end up with ridiculous try-catch statements like this:

try { buffer.append(string) } catch (IOException ignored) { // never happens! }

Java does have a redeeming quality here. Due to the covariance of the return type, you can make the return type more specific, which means you can skip from the throws clause in implementations (the equivalent of throws Nothing).

class StringBuilder implements Appendable { // ... @Override public StringBuilder append(Sequence s) { //... } }

StringBuilder works in-memory, it can never throw IOException. The reason for that signature is because StringBuilder implements Appendable, and that IOException is there just in case Appendable gets implemented by something that does side effects. Note that this isn’t a case of ergonomics, but rather a fundamental problem. It wouldn’t matter if we used your favorite bifunctor IO or monad transformer, replacing checked-exceptions with Either:

trait Appendable { // Same issue, although EitherT is worse due to // having issues with covariance; def append(csq: CharSequence): EitherT[IO, IOException, Unit] }

Covariance issues of EitherT aside, introducing an Either data type makes this even more awkward:

trait Appendable { def append(csq: CharSequence): Either[IOException, Unit] } class StringBuilder extends Appendable { // ... // Awkward, because the `Either` is now completely unneeded, // and because it's lying, since on top of the JVM, // `Throwable` can always happen... override def append(Sequence s): Either = ??? }

Scala users already suffer from flatMap chains replacing ; and from unneeded type wrappers, so I’m guessing that if we squint enough, we can make this work.

Regardless of how you model this, fact of the matter is that StringBuilder will never throw IOException. Like all Scala things, we could try introducing type parameters, or abstract type members, but that’s ridiculous, because the interface becomes unusable, as you can’t treat the error if you don’t know what it is, so you’re either going to work with Any, or you’re going to have an extra type parameter at all call sites:

// Ridiculous! trait Appendable { type Error def append(csq: CharSequence): EitherT[IO, Error, Unit] } // Even more ridiculous! (ver.2) trait Appendable[+E] { def append(csq: CharSequence): EitherT[IO, E, Unit] }

We have the issues mentioned in the original article:

IOException is often an irrelevant exception, equivalent with Throwable, as it doesn’t provide anything more specific than a signal for “you can probably retry”, something which Throwable does too;

The error type is an encapsulation leak, which forces a tax on all implementations;

It pushes complexity to the user;

Ironically, in Java’s world, input errors (i.e., those thrown by tasks that you can’t retry) are often not checked exceptions. E.g., InvalidArgumentException is a RuntimeException, and this one gets thrown in constructors, being what libraries like Jackson expect for validating input when parsing, a contract that you have to find by reading the documentation. You can see here another issue with checked exceptions: importance gets decided by the library author instead of the downstream user, but this classification is often wrong, as it depends on the use-case at the call-site.

Going further than the issue of abstract methods leaking implementation details, this is also an issue of changes breaking compatibility. Turns out, IOException isn’t a good super-type for I/O related exceptions, and the library’s authors might want to change that signature to this:

Appendable append(CharSequence csq) throws IOException, SQLException;

The implementation leak should be even more obvious, it would be absurd to deal with SQLException when using a StringBuilder. But even more problematic is that this breaks both source and binary compatibility, hurting all downstream users of the API, and Mima will complain, leading to interesting compromises. Turns out, people care about correctness only as long the fix is cheap 😉 so imagine, if you will, big repositories making liberal use of this function, alongside an entire ecosystem of libraries built on top of it. And remember, in this case, the exception type is absolutely useless, as people care only about these things:

closing resources safely — important to remember that a vast majority of call-sites care about the finally more than the catch;

retrying locally (in which case this information is insufficient);

short-circuiting the happy-path, and logging the message and the call stack trace (globally, as locally this is done only when the outcome gets ignored);

BTW, this widening of the exception type isn’t unheard of. In Java’s standard library we now have:

Closeable, which has a close() throws IOException, inheriting from AutoCloseable;

AutoCloseable, a newer interface (introduced by try-with-resources) that has a close() throws Exception, a trick in order to make Closeable compatible with it (covariance FTW), and note the total lack of information in throws Exception;

I don’t know why AutoCloseable was introduced, but if I were to take a guess, it’s probably because having that IOException in the interface of Closeable sucks. Java designers carrying about compatibility, however, chose this path instead of just modifying Closeable, which is what a Scala FP developer would have done, to force correctness down on everyone 😎

There is no loss of information, of course, if we just used IO, which is what Java meant to do, but lacked the right type to do it:

trait Appendable { // "throws IOException" is Java's `IO`, but it didn't quite work out def append(csq: CharSequence): IO[Unit] }

The throws Throwable part is implicit in I/O tasks. And now the interface is abstract enough to be useful, yet more specific, in the sense that we can see that it has side effects, but not because of the returned exceptions. And evaluating an IO can throw.

Scala is an OOP language, and I’ve never worked on any major codebase that did not contain OOP interfaces. It’s a take it or leave it sort of deal. And this is a taste of the different ideologies in OOP versus static FP. What FP developers call “parametricity” is actually in opposition to well encapsulated abstractions. It’s white-box versus black-box design.

For me, not carrying about implementation details is a powerful coping mechanism. And you also see such design choices in the design of everyday things that you use, meaning that user interfaces often get designed with the minimum amount of info and control knobs exposed for the users to be able to do their job. Think about the complexity of your car, in stark contrast to its simple controls (2 pedals, a steering wheel, and rearview mirrors), the ultimate black box. And all cars get steered mostly in the same way, standard interface, no reason to expose controls for implementation-specific errors.

Exposing stuff “just in case” is not a good recipe, as it shifts responsibility on the user, and the user will do stupid things. Can’t emphasize this enough: errors should be designed out of existence!

For your reading pleasure, here are some references talking about these issues:

The Trouble with Checked Exceptions, a conversation with Anders Hejlsberg;

Java’s checked exceptions where a mistake — Rod Waldhoff;

Or hear Bruce Eckel, a quote from “Thinking in Java”:

Examination of small programs leads to the conclusion that requiring exception specifications could both enhance developer productivity and enhance code quality, but experience with large software projects suggests a different result – decreased productivity and little or no increase in code quality.

It’s difficult, however, to convince people that their carefully designed, super explicit return types are actually a smell of bad design. Therefore, the legend and allure of checked exceptions will persist, but hey, placebos are valid treatments too 😛

The article The Trouble with Checked Exceptions: Part 2 first appeared on alexn.org.

Proprietary Environments are a Trap

Alexandru Nedelcu — Thu, 22 Sep 2022 07:00:00 +0000

Me, with my son, learning the Lua programming language on proprietary macOS laptops.

What to learn in order to not become obsolete as a software developer, and then being forced into early retirement from programming? What platforms to prefer for building products that last with minimal maintenance?

Jeremy Allison (bio) is a computer programmer, contributor to Samba, and Free Software proponent that I have long admired. In Working for The Man, he has a paragraph on proprietary platforms that stuck in my head…

Proprietary environments are a trap

I used to be a Microsoft Windows programmer, as well as a UNIX/POSIX programmer.

The knowledge I’ve gained about programming POSIX is still useful, even though I learned a lot of it over twenty years ago. My Windows knowledge is now rather out of date, and getting more so over the years. It isn’t worth my time anymore to keep up with each increasingly baroque change to the Windows environment. Just as an example, over this time the latest “hot” communication paradigm that Microsoft recommended developers use in Windows changed from NetBEUI, to NetDDE, then OLE, followed by OLE2, then COM, DCE/RPC, DCOM, and now currently seems to be Web Services (SOAP and the like).

Meanwhile, in the UNIX world the Berkeley socket API was useful in the 1980s and is still the core of all communications frameworks in the open standards world. All the UNIX RPC, object and Web Service environments are built on that stable base. Learn as much about the proprietary environments as you need to be able to help people port programs over to open standards. You’ll never be lost for work. The same is true of any proprietary environment, not just Windows. Windows just happens to be the one I know best.

What will get very interesting in the future is the effect a fully open Java platform will have on the software environment in the next ten years. After initially ignoring Java due to its proprietary restrictions, I now believe Java and it’s associated libraries have the potential to be the next POSIX.

This was written in 2007, so 15 years ago. Was Jeremy Allison right?

I know people that left the software industry due to obsoletion, not able to keep up with the latest and greatest. Learning proprietary platforms is planned obsolesce for your career. Windows, a shadow of its former self, is now just an ads-delivery vehicle and a GUI toolkit for MS Office. Java, propelled by its openness and availability on all platforms, grew and is now more popular than ever, although it’s not alone in being popular, or a POSIX-like platform, others being JavaScript, .NET, or LLVM, and all are FOSS.

There is one dimension that this misses – learning the latest and greatest can also be a losing bet. What young programmers should focus on is standards. And de facto standards happen with age, solutions becoming more entrenched and better as time passes, much like good wine.

The Atom¹ text editor is dead. GitHub killed it, because it was acquired by Microsoft, and Microsoft has its own editor, VS Code². These editors were inspired by TextMate³, a once popular, but proprietary editor for macOS that died, and the resurrection attempt via open sourcing failed. VS Code is right now the most popular editor. I don’t know if it will be around in another 20 years. But I do know that Emacs and Vi, both released in 1976 (46 years ago), will still be around in another 20 years, and the skills you acquire while using them won’t be obsolete any time soon.

Programming languages become entrenched faster, because programs get built with them and then those programs need to be maintained. But Borland’s Pascal⁴ or FoxPro are dead, and C isn’t. Although, oddly, at the time of writing the TIOBE Index⁵ has Object Pascal / Delphi in 13th place 🤦‍♂️

Embrace open platforms, open standards, build on top of Open Source. But there’s also wisdom in “embracing boring technology”⁶, as boring is simply a signal for older, more entrenched, more stable, that survived fashion trends. And boring technology is usually Open Source / Free Software, because FOSS survives for much longer 😉

In addition to preferring boring FOSS technology, I’d also add … learn math and algorithms. Math is the ultimate language and open standard for what we do, and it will never be obsolete. And not much progress happened in CS algorithms, except for machine learning. These are the fundamentals, which you may shun, but if all you’re doing is to call library functions, one of these days that job will get automated.

Also, and perhaps this is the most important advice — engage in building software that helps people. If you’re not proud of your work, like for example if you work on an unscrupulous ads-delivery network or some online bets platform, or maybe if you work on military drones, quit your job and go work on software that doesn’t make the world more miserable than it is. You may find your job technically challenging and stimulating, but working on immoral products is just not worth it.

I grew up as a software developer with such words, with essays instilling ideas of software freedom, and I see such advice less and less these days.

What changed is that Open Source won. It won the hearts and minds of software developers. Most of us build on top of FOSS libraries, and deploy on top of FOSS runtimes and operating systems, using many FOSS tools in the process. But it seems to me like the trend is reversing⁷.

“Life swings like a pendulum backward and forward between pain and boredom.”

— Arthur Schopenhauer

When freedom is abundant, we begin taking it for granted, forgetting why it is needed, forgetting to contribute. Oblivion is how freedom dies.

Atom.io (archive), also see Wikipedia — is a text editor developed by GitHub, based on Electron, scriptable via JavaScript/CoffeeScript; ↩

VS Code, also see Wikipedia — is a text editor developed by Microsoft, also based on Electron, scriptable in TypeScript; ↩

TextMate was a proprietary editor built for macOS, made popular by the screencasts of DHH and others; ↩

Turbo Pascal was the programming language and environment I learned in high school; ↩

Tiobe Index (archive) is a piece of shit that people take way too seriously — the GitHub language stats (archive) are IMO far better at assessing a language’s popularity, as even if biased, it shows the FOSS output of the language’s community, and that’s a much stronger signal than Google searches, or whatever crap the TIOBE Index does; ↩

Choose boring technology (archive) is an essay by Dan McKinley that became an instant hit; ↩

What triggered these thoughts is Akka’s license change, following other “open source companies” flipping, such as Elasticsearch or MongoDB, and others, alongside the general sentiment I’m seeing among my peers; ↩

The article Proprietary Environments are a Trap first appeared on alexn.org.

Java 19

Alexandru Nedelcu — Wed, 21 Sep 2022 16:00:00 +0000

Honestly, Java's mascot is pretty cool, Scala needs a mascot too.

Java 19 is now released, and it’s a big deal, as it ships with Virtual Threads (JEP 425), among other goodies, like structured concurrency (JEP 428), or improvements to pattern matching (JEPs 405, 427). I haven’t been as excited about a Java release in a long time.

I once claimed that this function signature needs special platform support, on top of the JVM this needs to block threads, and on top of some other platforms this gets implemented with continuations/coroutines. Not all platforms can provide support for it, and blocking (platform) threads is very expensive, therefore it should be avoided:

Future[A] => A // aka `Await.result`

The Java ecosystem has been increasingly moving to reactive APIs, with projects like Project Reactor, RxJava, or Vert.x becoming really popular. The introduction of Virtual Threads, however, has the potential to move the ecosystem back to usage of blocking APIs for I/O or structured concurrency.

Blocking threads plays nice with Java’s features in a way that reactive APIs can’t. For instance: for loops, try/catch, try-with-resources, or checked exceptions. Java was built to synchronize on concurrent tasks by blocking threads, and the incursion into reactive APIs was only temporary. Other languages, such as Kotlin, implemented coroutines for mostly the same effect. But note that Virtual Threads improves everything, including all blocking APIs from the java.io package, and it does so for all languages running on top of the JVM, whereas Kotlin’s coroutines can’t work around the platform’s limitations (blocking I/O is still blocking I/O). This is what having control over the entire stack gives you.

“With only a tiny code change to use JDK19 “virtual threads” JRuby can now spin up over 100k fibers at once! Launching and completing them takes about 0.7s after warmup. This will make async IO servers and thousands of concurrent users possible on JRuby. 🤯”

— Charles Oliver Nutter (@headius)

Mind you, I don’t think this invalidates the available projects built with reactive APIs. I think Virtual Threads will make everything better, as these projects will simply adapt to provide the best API depending on use-case. We use reactive APIs due to the strong guarantees they provide, and due to their declarative approach, and that won’t change. But users will no longer be forced to use reactive APIs if all they want is efficiency.

Virtual Threads also has the potential to change APIs for Scala, too. If blocking threads is now cheap, this should no longer be a big problem in terms of efficiency:

IO[A] => A // aka `unsafeRunSync`

And neither is something like this:

Stream[IO, A] => ??? => A // Iterator#next() ftw 😛

This is interesting, as we can now do some stuff afforded with Kotlin’s coroutines, see Arrow-fx. Meaning that, instead of using flatMap to compose IO values, we could use a more imperative approach, and have Scala help with the safety. Many people will object to replacing flatMap, as it’s about composition, but other Scala devs could prefer such an API. Even without any API changes, calling Java APIs will be less like walking on a landmine field. The big problem that I’m seeing for evolving Scala APIs is that this is a runtime feature, not a language one, meaning that the portability to Scala.js and Scala Native will suffer.

Did you know that Cats-Effect’s IO now supports Scala Native? 😱

And do you remember that practice of having 2 thread-pools in your project, one for CPU-bound tasks and another for blocking I/O tasks? Oracle just ended it in Java 19. You no longer need to shift between thread-pools in order to avoid thread starvation, while efficiently using your CPU cores.

Interesting times ahead — and I’d like to predict the future, but truth is, I have no idea how our Scala effect systems will change to take advantage of it. In the meantime:

sdk install java 19-oracle

Here’s a good video explaining why Java got virtual threads, and how it compares with alternatives:

Ron Pressler — Why user-mode threads are (often) the right answer (open on YouTube.com)

The article Java 19 first appeared on alexn.org.

Akka Fork FUD

Alexandru Nedelcu — Wed, 21 Sep 2022 09:00:00 +0000

Lightbend made Akka proprietary from version 2.7.x onward. This left the community wondering about the possibility of a fork, and unfortunately, I see some FUD that needs to be addressed.

Fear: Lightbend can sue

There is a high likelihood that bugs get discovered and will have to be fixed in both the BSL-licensed Akka, and in the community fork. As such, this leaves people wondering … what if Lightbend sues for copyright infringement? What if the bug fixes are so similar that a court will decide such bug fixes are derivate works of the BSL-licensed code?

Speaking of a fork, I’d license all new development of a fork in a copyleft license, like LGPLv3, precisely so that the BSL-licensed Akka can’t use it, and it would be compatible with the current Apache 2.0 license. But IANAL, and I don’t know how practical a copyleft license is for such libraries. This inability to share future bug fixes can go both ways 😉

First, of how many lawsuits targeting Open Source forks have you heard of? Do your research and name them. I’d bet that your list will be mostly blank. Because, it turns out, suing and winning in cases of copyright or patents infringement is actually hard.

Anyone can sue for anything, and due to the high costs of such lawsuits, settlements are quite common. But for the few lawsuits against FOSS products that happened, it didn’t go so well for the plaintiff. For example, Oracle, with its patents war chest, sued Google for using pieces from a 3rd party Java implementation in Android, built via reverse engineering. They also sued for copyright infringement. And Oracle lost. I remember the anti-Java FUD campaigns from back then, people being happy they picked some alternative, like JavaScript, even if Oracle’s patents war chest probably impacted them, too, since modern JS engines are also derivates of Smalltalk and Java’s Hotspot VM.

As a similar issue, for years people have spread FUD about Mono, the clean room dotNET-compatible implementation (I was part of that choir). The likelihood of Microsoft suing was high, actually, as this was unfolding in the age when Microsoft was threatening to sue Linux distributions (they never stopped the patents racketeering BTW). Microsoft never sued for Mono, probably because they would’ve lost. Their puppet company did lose. If the Mono contributors could manage to not look at .NET’s disassembled code, or at Rotor for that matter, I think projects smaller than that can be fine. Note, lawsuits may not work, but many in the Linux, or Mono, or BSD camps can probably tell you: FUD works, as big organizations are risk-averse.

In our case, we aren’t even talking of clean room reverse engineering, which is a much bigger problem. In this instance a fork is perfectly legal due to the Apache 2.0 license, which is perfect for forking, because it contains an explicit patents grant too. Worth mentioning, as this explicit patents grant does not favor any company, and covers derivate works too. This in contrast with what Facebook and Microsoft have been doing 😉 and why Apache 2.0 is better than the MIT or BSD licenses.

The whole value proposition of Open Source is that you can fork. That’s all there is to it. There’s nothing else worthwhile about Open Source that’s worth mentioning.

If that’s not true, then might as well drop all Open Source libraries and tools from your project right now, especially those projects built by companies that require copyright agreements or assignments, legally binding documents granting them the right to re-license everything you contribute as proprietary software.

For instance, one of the contenders for Akka’s market share is ZIO. Ziverge, the company behind it, does require a grant of rights (archive) on all contributions, giving them the right to distribute your contributions under a proprietary license. Discussions on copyleft aside, they can do precisely what Lightbend is doing. Personally, I don’t trust such CLAs and companies that require them, unless we’re speaking of the Apache Foundation. If I’d fear the companies behind FOSS projects, at the very least I’d use libraries that don’t require CLAs for contributions.

NOTE: not all companies do this, not all FOSS projects do this. Typelevel in particular does not require the signing of any copyright agreement, and the only agreement required of you is for your work to get distributed strictly under Apache 2.0 (see contributing), mentioned here as I was there, complaining. I am not the first to have this opinion, I won’t be the last either. And as the most successful FOSS project in history, now getting a majority of contributions from companies, the Linux kernel should be enough to give you pause.

The only protection we get, in all such cases, is the Apache 2.0 license, due to the ability for forking. Open Source licenses exist because “trust” isn’t enough, and if you rely on trust, and you’re not paying, then “you’re the product”. If the ability to fork is off the table, then all Open Source software built by companies is a liability. Might as well drop such dependencies, as they are all landmines that will keep you hostage when the company behind them flips.

Thankfully, that’s not how Open Source works.

And I am mentioning Ziverge explicitly, as they are the ones spreading these fears right now. Not cool. Especially because Akka can’t be replaced. There are a lot of projects building on top of Akka, including projects built in Java or in .NET, that can’t just switch to something else, no matter how much you like your monads and dislike actors. And all claims otherwise are either shortsighted or disingenuous, given how software development at scale works.

I’ve always been a fan of “growing the whole pie” in terms of market share. Competition does in fact grow the market for everyone. Which is why I never understood “fear, uncertainty and doubt” (FUD) as a marketing tactic, because it shrinks the pie for everyone, and the salesmen using FUD end up getting a bigger piece from a smaller pie.

Akka has been a central part of Scala, the gravitational force that attracted a lot of developers, and quite frankly, if Akka users get thrown under the bus like this, in order to pump up the numbers for some company or another, I will view the entire Scala ecosystem as a liability, for any project, in spite of all my Scala investments, and I’m sure that I won’t be alone. It’s not personal, it’s just business.

Don’t get me wrong, nobody can demand free labor, a fork may not be possible, but let us all leave our crystal balls at home, and just leave it to the market. Because Scala is small, compared to the juggernauts in this space, and unfortunately how this Akka situation is being handled shows the world what Scala is made of. It’s unfair, and I wish this wasn’t so, but it is what it is.

Fear: Akka is too complex

The idea is that Akka is built with VC money, a lot of work went into it, and a community fork isn’t sustainable, as it requires man-hours and knowledge about distributed systems.

All I can say is that forks like Akka .NET and OpenSearch exist (🎤 drop).

Yes, it will take resources, but if there really are big companies with Akka investments, such a fork can happen. And if a fork doesn’t happen, that’s life, and maybe Akka isn’t so popular or valuable after all.

For our project, I’d also argue that we have everything needed in Akka already, and what we require (beside bug fixing) is for Akka to not hold us back when upgrading dependencies. I’d like to update the libraries that Akka depends on, I’d like us to upgrade to Scala 3 at some point. Updating such dependencies in an Akka fork may take a lot of effort, but it probably doesn’t require a PhD in distributed computing.

Therefore, personally, I remain hopeful.

The article Akka Fork FUD first appeared on alexn.org.

Java's Cultural Problem

Alexandru Nedelcu — Mon, 19 Sep 2022 09:00:00 +0000

Java is good by modern standards, from a technical perspective, the platform having received a lot of improvements from Java 8 to 17. Unfortunately, it still stinks, and the problem is its "enterprise" culture.

Let me illustrate the problem via examples …

Quarkus is a very promising framework, being a lightweight replacement for Spring, promising compatibility with GraalVM’s Native Image. Full of hope, I enthusiastically opened its documentation, and started with how to configure an app, expecting something with common sense, like Dropwizard’s quick-start guide.

Quarkus depends on SmallRye Config. And if you want to map your configuration to objects, the documentation has this to say:

@ConfigMapping(prefix = "server") interface ServerConfig { String host(); int port(); }

In Java’s world, interfaces and abstract classes get used by dependency injection libraries, with methods left abstract in order for their implementation to be filled-in later. Any FP developer should scream when seeing this, because:

This should be a pure data structure;

These abstract methods signal the possibility of side effects — in general, it is the possibility of side effects that drives the demand for abstract methods, as pure data structures rarely need it;

Do you know what the library does in this instance? I sure don’t. It could be reading from a file and block a thread on every access, it could be thread unsafe, I wouldn’t know, since whatever it does is magic™️, and this isn’t my data structure. Even if it generates a pure data structure, for all I know its implementation can always change in future versions to also launch rockets to Mars.

Since Java 14 we have records. The more common-sense definition doesn’t work, the library being (currently) unable to work with it:

// // java.lang.IllegalStateException: SRCFG00043: // The @ConfigMapping annotation can only be placed in interfaces... // @ConfigMapping(prefix = "server") final record ServerConfig( String host, int port, ) {}

UPDATE: to drive this point home, let me make it clear that I don’t care from where the configuration is being read (the actual I/O side effect), but rather what happens afterwards. This configuration has an implicit usage protocol that isn’t properly expressed by an abstract interface:

The host and port values should be read from the same configuration source;

These values shouldn’t change during the application’s lifecycle, otherwise the interface should provide the ability to register a listener;

There’s no point in doing the side effect more than once, at the application’s start;

In other words, this is not just bad FP design, this is bad OOP design. A better abstract interface would be this, which makes the behavior crystal clear:

interface ServerConfigReader { ServerConfig read() throws IOException; }

Dropwizard has a more common-sense approach, as it leaves you in charge of defining a type safe configuration object. But it, too, was infected by the Java EE culture (aka Jakarta EE), preferring Bean Validation via annotations, with the help of hibernate-validator.

public record ServerConfig( @NotNull @NotEmpty String host, @NotNull Integer port, @NotNull @Email String contact ) {}

I understand the @NotNull annotation. Java has null in it, all object references can be null, and it’s too late for that to change. What’s odd is that contact should always be an email address, no matter the context. Not even if you consider this “raw input”, because actual “raw input” is a plain string or an array of bytes, and if you ever reach this stage of having a structured record, then you should already have an email address.

Java’s culture eschews common sense approaches, like defining new types. Defining new types would make illegal states unrepresentable. Note how, with the following definition, there is no way to get an EmailAddress that doesn’t pass the “validation”, and you don’t need a freaking annotations-driven library to validate your values:

import java.util.Objects; public record EmailAddress(String value) { public EmailAddress { // We could've used an Either data type, ofc; Objects.requireNonNull(value); // regexp could be better if (!value.matches("^[^@\\s]+@\\S+$")) throw new IllegalArgumentException( String.format("'%s' is not a valid email address", value)); } }

Back to Quarkus, usage of a DI container is required (via CDI annotations), and it’s not easy to keep the Java EE crap out of your classes. Reading their documentation, by default you’d end up with something like this:

@ConfigMapping(prefix = "greetings") public interface GreetingServiceConfig { String name(); } // Not a final class @RequestScoped public class GreetingService { // Abstract, and it cannot be private 😱 @Inject GreetingServiceConfig config; public String greeting() { return "Hello, " + name + "!"; } // @PreDestroy is required for "closeable" resources; // I would have expected the framework to work with AutoCloseable, // but ALAS it doesn't; @PreDestroy public void close() { LoggerFactory.getLogger(getClass).info("Destroying GreetingService!"); } }

In the sample above, adding a scope (e.g., @RequestScoped) makes the framework automatically initialize this “bean” when needed. And @PreDestroy marks methods that have to be called when the “bean” is disposed. Note that my “bean” should implement Closeable, but the framework completely ignores it, this being another instance in which a Java EE implementation ignores the Java language. You need that @PreDestroy, or otherwise you’ll have a leak.

Of note is how this approach will infect your entire codebase, forcing all your downstream users to forgo Java language constructs, such as easily building an instance with new, or safe disposal of resources via try-with-resources.

With this approach, not working with final classes jumps at me, because “final” is a best practice. This isn’t related to Quarkus in any way, but rather with DI containers in general. For instance, Kotlin’s classes are final by default, yet if you want to build Spring apps, the recommended way would be to import the kotling-spring plugin, which automatically “opens” your classes that have certain DI-related annotations. Whether you agree with “final by default” as a best practice or not, you’re getting a bad deal if the framework makes that choice for you.

Quarkus ships with a CDI 2.0 implementation, like Spring before it, although its implementation isn’t fully compliant, since all those annotations and runtime behavior can’t be fully supported on top of GraalVM’s Native Image. It’s odd seeing such a framework forcing the use of Java EE’s CDI.

Quarkus makes it hard to have a common-sense composition root. Thankfully, I discovered how, via trial and error, meaning half-baked Stack Overflow answers and using the right keywords to appease the search gods. One of these days I’ll find out what the heck is a “bean”.

// ------------------------------------------------ // No Java EE crap public record GreetingServiceConfig(String name) {} // ------------------------------------------------ // No Java EE crap public final class GreetingService implements Closeable { private final GreetingServiceConfig config; public GreetingService(GreetingServiceConfig cfg) { this.config = cfg; } public String greeting() { return "Hello, " + config.name() + "!"; } @Override public void close() { LoggerFactory.getLogger(getClass()).info("Destroying GreetingService!"); } } // ------------------------------------------------ // Implements the "composition root" pattern... // // All of these imports are already a code smell, but at // least it's localized, and does help with managing DI. import javax.enterprise.context.RequestScoped; import javax.enterprise.inject.Disposes; import javax.ws.rs.Produces; import javax.enterprise.context.ApplicationScoped; import javax.enterprise.context.RequestScoped; import javax.enterprise.inject.Disposes; import javax.ws.rs.Produces; public class AppConfiguration { @Produces @ApplicationScoped public GreetingServiceConfig gsConfig() { return new GreetingServiceConfig( ConfigProvider.getConfig().getValue("greetings.name", String.class) ); } @Produces @RequestScoped public GreetingService greetingService(GreetingServiceConfig config) { return new GreetingService(config); } // Closeable resource needs to be destroyed, and // framework won't do it automatically; void disposesGreetingService(@Disposes GreetingService ref) { ref.close(); } }

This way you can limit the impact of Java EE on your codebase. But it does need restraint, and you still have to learn a domain-specific language that has few things in common with Java, the language.

Newcomers to Java have to deal with this nonsense, and it’s a pity given that Java 17 is a decent language and awesome platform. Java’s ecosystem still hasn’t learned enough from its competition, but hope never dies.

The article Java's Cultural Problem first appeared on alexn.org.

Scripting with Scala

Alexandru Nedelcu — Tue, 13 Sep 2022 09:00:00 +0000

Unix has a long tradition with scripting — small programs represented as text files that can be inspected, modified, and executed. Scala can be used for scripting too.

Scala is a compiled language, your average project has quite a complicated build setup, but we have 2 tools that makes scripting possible and quite pleasant:

Ammonite;

Scala CLI, which is newer, does more, and can embed Ammonite;

For this sample, I’m going to use Scala CLI.

First step, install scala-cli. I’m on macOS, using Homebrew, for other operating systems refer to its documentation:

brew install Virtuslab/scala-cli/scala-cli

We are going to create a command-line utility that tells us how much time has passed since some timestamp in the past. It’s usage and output will look like this:

$ time-since.sc 2022-01-01 Since: Sat, 1 Jan 2022 00:00:00 +0200 Until: Tue, 13 Sep 2022 14:06:15 +0300 Elapsed: 255 days, 13 hours, 6 minutes, 15 seconds Years: 0.70 Months: 8.40 Weeks: 36.51 Days: 255.55 Hours: 6133.10 Minutes: 367986.26

Create a text file in $HOME/bin/time-since.sc (or somewhere else that’s on your system $PATH):

#!/usr/bin/env -S scala-cli shebang -q //> using scala "2.13.8" //> using lib "com.github.scopt::scopt::4.1.0" import scopt.{OParser, Read} import java.time.format.DateTimeFormatter import java.time._ import java.util.concurrent.TimeUnit import scala.util.Try case class Args( since: LocalDateTime, until: Option[LocalDateTime], zoneId: ZoneId, ) val parsedArgs = { val builder = OParser.builder[Args] import builder._ implicit val readsTime: Read[LocalDateTime] = implicitly[Read[String]].map { dt => Try(LocalDateTime.parse(dt, DateTimeFormatter.ISO_LOCAL_DATE_TIME)) .orElse { Try(LocalDate.parse(dt, DateTimeFormatter.ISO_LOCAL_DATE)) .map(_.atTime(LocalTime.of(0, 0, 0, 0))) } .getOrElse( throw new IllegalArgumentException( s"Not a valid timestamp, correct format is `yyyy-mm-dd` OR `yyyy-mm-ddTHH:MM:SS`." )) } implicit val readsZoneId: Read[ZoneId] = implicitly[Read[String]].map { id => Try(ZoneId.of(id)) .getOrElse(throw new IllegalArgumentException(s"'$id' is not a valid timezone id")) } val parser = OParser.sequence( programName("time-since.sc"), head("time-since", "1.x"), arg[LocalDateTime]("") .text("Format: `yyyy-mm-dd` or `yyyy-mm-ddTHH:MM:SS`.") .action((ts, args) => args.copy(since = ts)), opt[LocalDateTime]('u', "until") .text("Format: `yyyy-mm-dd` or `yyyy-mm-ddTHH:MM:SS`. Defaults to NOW.") .action { (ts, args) => args.copy(until = Some(ts)) }, opt[ZoneId]('z', "zone-id") .text("Example: Europe/Bucharest") .action { (id, args) => args.copy(zoneId = id) }, ) OParser .parse(parser, args, Args(null, None, ZoneId.systemDefault())) .getOrElse { System.exit(1) throw new RuntimeException() } } val since = parsedArgs.since.atZone(parsedArgs.zoneId) val until = parsedArgs.until.fold(ZonedDateTime.now(parsedArgs.zoneId))(_.atZone(parsedArgs.zoneId)) val sinceTs = since.toInstant.toEpochMilli val untilTs = until.toInstant.toEpochMilli println() println(s"Since: ${since.format(DateTimeFormatter.RFC_1123_DATE_TIME)}") println(s"Until: ${until.format(DateTimeFormatter.RFC_1123_DATE_TIME)}") val totalMs = untilTs - sinceTs val days = TimeUnit.MILLISECONDS.toDays(totalMs) val rem1 = totalMs - TimeUnit.DAYS.toMillis(days) val hours = TimeUnit.MILLISECONDS.toHours(rem1) val rem2 = rem1 - TimeUnit.HOURS.toMillis(hours) val minutes = TimeUnit.MILLISECONDS.toMinutes(rem2) val rem3 = rem2 - TimeUnit.MINUTES.toMillis(minutes) val seconds = TimeUnit.MILLISECONDS.toSeconds(rem3) println() println(s"Elapsed: $days days, $hours hours, $minutes minutes, $seconds seconds") println() println(f"Years: ${(untilTs - sinceTs) / (1000.0 * 60 * 60 * 24 * 365.24)}%11.2f") println(f"Months: ${(untilTs - sinceTs) / (1000.0 * 60 * 60 * 24 * 30.417)}%11.2f") println(f"Weeks: ${(untilTs - sinceTs) / (1000.0 * 60 * 60 * 24 * 7)}%11.2f") println(f"Days: ${(untilTs - sinceTs) / (1000.0 * 60 * 60 * 24)}%11.2f") println(f"Hours: ${(untilTs - sinceTs) / (1000.0 * 60 * 60)}%11.2f") println(f"Minutes: ${(untilTs - sinceTs) / (1000.0 * 60)}%11.2f") println()

This script has what’s called a shebang declaration, which tells your shell (Bash, Zsh) how to interpret your script:

#!/usr/bin/env -S scala-cli shebang -q

Also, note the dependencies — we can specify the Scala version, and any library dependencies we require, in this case scopt, a library that helps us parse command line arguments:

//> using scala "2.13.8" //> using lib "com.github.scopt::scopt::4.1.0"

I usually build such scripts in Python or Ruby. These are good choices because they can be installed everywhere easily, and have “batteries included”, but extra functionality is hard to import. Scala-CLI (Ammonite too) allows us to import any dependency from Maven Central, and that’s awesome! ❤️

Make this script executable:

$ chmod +x ~/bin/time-since.sc

Let’s execute it, and see what happens:

$ time-since.sc Error: Missing argument time-since 1.x Usage: time-since.sc [options] Format: `yyyy-mm-dd` or `yyyy-mm-ddTHH:MM:SS`. -u, --until Format: `yyyy-mm-dd` or `yyyy-mm-ddTHH:MM:SS`. Defaults to NOW. -z, --zone-id Example: Europe/Bucharest $ time-since.sc 2022-03-01 Since: Tue, 1 Mar 2022 00:00:00 +0200 Until: Tue, 13 Sep 2022 14:06:37 +0300 Elapsed: 196 days, 13 hours, 6 minutes, 37 seconds Years: 0.54 Months: 6.46 Weeks: 28.08 Days: 196.55 Hours: 4717.11 Minutes: 283026.63

Note that you can edit this script in IntelliJ IDEA, with auto-completion and everything. See the documentation.

A disadvantage of working with Scala-CLI is that it’s not available everywhere, as for example, on Ubuntu/Debian I prefer .deb packages via official repositories. But Java is available everywhere, and you can always package scripts in JARs.

The startup time is not ideal either, being that of any Java app:

$ time time-since.sc ... time-since.sc 0.51s user 0.13s system 99% cpu 0.645 total

So, the startup cost is half a second. It’s not that bad though. ~~And I hope some out-of-the-box integration with GraalVM’s Native Image will happen.~~

You can package the script using GraalVM’s Native Image to speed things up, see the documentation:

cd $HOME/bin scala-cli package --native-image ./time-since.sc -o ./time-since -- --no-fallback

GraalVM’s native image only has partial support for libraries using reflection, and needs to know about them ahead of time. This sample was easy, but depending on your dependencies, other scripts may require some tweaking.

The startup time is now much better:

$ time ./time-since 2022-01-01 1>/dev/null ... 0.01s user 0.01s system 36% cpu 0.034 total

I’m in love! 😍

The article Scripting with Scala first appeared on alexn.org.

#DeleteFacebook

Alexandru Nedelcu — Mon, 12 Sep 2022 09:00:00 +0000

I finally managed to delete my Facebook account. This was a long time coming. Being disconnected feels good, it feels liberating.

On Saturday I turned 40, and this was the ultimate test — not being on Facebook meant that my acquaintances did not know about my birthday, which meant I only received Happy Birthday wishes from family and very close friends. I received very few birthday-wishes this year. And it felt … weird, but good.

On Facebook you “Like”, “Share”, or provide shallow comments, as a substitute for having meaningful face-to-face conversations. The comments are shallow because, when online, people no longer have the patience to listen, and start talking past each other, becoming adversarial, the goal being to be right, not to learn. Complex human interactions are replaced with mere bits of information (like / dislike), and those bits are enough to satisfy our inner urges. Yet society is more and more depressed, especially the teenagers.

Humans are social beings, we crave human connection, and Facebook is the junk food of social interaction. Facebook pushes people to wish you Happy Birthday, which is how you get (impersonal) birthday wishes from people you haven’t talked with in 20 years, or from complete strangers. Those birthday wishes don’t mean much, though. We like to think that long-lost acquaintances, from school, from former jobs, are still thinking of us, but that ain’t true. It’s just a Facebook-driven automation. If anything, given that people tend to look stupid on social media, Facebook can and does severe ties, instead of strengthening them. If you connect with your work colleagues on Facebook, you’re not wise.

I first thought of deleting Facebook on reading “Ten arguments for deleting your social media accounts right now”. Actually, I thought about it, then tried to find a book that affirmed my decision. Well, it’s a poor book actually, as the arguments are almost religious. What the book argues is that social media companies invest in behavior modification, because they are betting on humans being just automatons, in order to get us to buy more stuff. I don’t doubt that social media companies try doing just that, except that I no longer believe humans are so easy to manipulate. It’s like with smoking. Tobacco companies may have engaged in despicable practices, but in the end, human nature is to blame, as we often like to engage in toxic vices for instant gratification. Nowadays, people smoke a lot less only due to increased costs (extremely high taxes, and having fewer smoker-friendly places). In other words, smokers exist, not because they’ve been manipulated, but because they like it and because cigarettes were made easily accessible.

Even if the environment plays a huge role in our behavior, the argument that we’re manipulated robs us of personal responsibility, of agency, an argument that I won’t buy. Red flags go off in my head whenever I hear of profit-driven puppeteers. When I quit smoking, more than a decade ago, I managed to do it solely due to economic reasons, as it has gotten simply too expensive to smoke. This argument also overestimates Facebook’s power, playing into their marketing. After all, nobody would pay for those ad-placements, if Facebook would fail to influence its users. Truth is, mentalities are really hard to change, and the only people buying the crap that FB sells are the people that would’ve bought that crap anyway.

Facebook isn’t any more guilty than your favorite messenger application (SMS included), or the smartphone that you keep near you 24/7. Technology, even when neutral, has unintended consequences. Internet-connected smartphone applications are behaving like slot machines, triggering dopamine releases, pushing us to check for notifications and refresh the page. Stories of profit-driven automatic algorithms may be good for your self-esteem, or for your motivation to quit, but such stories don’t tell the whole picture. And the full picture is that many apps are slot machines, especially those designed for portable devices, fueling our addiction for shallow social interaction. Which would be OK, except that by stealing so much time away from our lives, it detracts us from having meaningful social and learning experiences.

Slot machines in a Las Vegas casino.

And so, I’ve decided to delete Facebook. Not because I believe Facebook is evil, or special. But because it provides no value, it’s a source of stress (due to all the politics), and the time I lose on social media is time that I’d rather spend elsewhere. Time isn’t unlimited and it needs budgeting. For me the question is less about what I’m doing on Facebook, and more about what I’m not doing in general. I’m not calling my friends, for example. Or, I’m not learning as much as I’d like, or engaging in meaningful projects. Or, I’m reading less and less fiction books, because my time and patience is in short supply. Admit it, it takes great effort to just sit down with a book without checking any status updates on your smartphone 😉

Deleting Facebook is easier said than done. These platforms make it hard to delete your account by merely “deactivating” your account for 30 days, giving you plenty of time to reconsider. You initiate the deletion process, and then a couple of days later guilt starts setting in, or fear that you’re going to miss out. You invested in your online identity, and there’s that lingering thought that you still have time to recover it, and maybe you can impose usage limits, etc. This dark pattern (the 30 days deactivation) is, of course, not accidental.

To finally delete it, after a couple of failed attempts, I had to make recovery impossible:

I changed the email address to a “temporary” one (e.g., Fastmail’s masked emails);

I removed the phone number from my account;

I changed the username to something random;

I changed the password to something random;

I deleted the account (starting the 30 days process);

I deleted all traces of the temporary email address, and of the new username, or the new password;

(Take that, Facebook puppeteers)

I managed to do it, as I realized that I’m not missing anything. If I want pictures from my family, I’ll ask for them. If I want to keep in touch with people, I’ll simply call them on the phone, and then maybe go out for coffee or beer, if possible; although just having phone conversations is nice, too. For telling my distant family how my life is going, if phone calls or visits aren’t practical, a plain-old newsletter might be better. I don’t need the events, or the politics, or the support groups. Everything I got from Facebook was either not productive, or had better options elsewhere.

And not just Facebook. I deleted multiple social media accounts, actually. For example Instagram, and TikTok. And almost 2 years ago I gave up on my Hacker News account. Giving up on HN, with my hard-earned karma, was actually more painful than Facebook. I’m still on Twitter, as it can be useful for learning, or for notifying others of my blog posts. But only if kept in check, the rules being: no social interactions on it whatsoever, absolutely all notifications off, and time limited. I’m pretty sure that, given my current mentality, I’m no longer the ideal Twitter user, having turned it into a glorified RSS feed. Ditto Reddit.

So here I am, a 40-year-old with no Facebook. Now, you kids get off my lawn!

The article #DeleteFacebook first appeared on alexn.org.

Scala isn't fun anymore

Alexandru Nedelcu — Fri, 09 Sep 2022 05:00:37 +0000

I’ve just spent over a day, going to sleep at 1:00 am, to upgrade dependencies and fix eviction warnings in the build of a Scala project. Had to fix the usual suspects:

Slf4j-api 1.7.x evicted by 2.0.0. They are binary backwards compatible, but the linking to the backend happens at runtime, so it can fail at runtime. This was a mix and match job. I had to downgrade Doobie from 0.13.4 to 0.13.3, because that one patch version also bumped HikariCP from 3.4.5 to 4.0.3, which in turn bumped slf4j-api.

HikariCP, also a mix and match job, because akka-persistence-jdbc also depends on it. I would have stayed on 4.0.3, but Akka wants 3.4.x, and I have no idea if they are compatible.

Scala-xml 1.2.0 evicted by 2.1.0; unfortunately we rely on Twirl templates, that require both a sbt plugin and a library dependency, and sbt plugins still rely on Scala 2.12, which require Scala-xml 1.2.0. The fix was to forcefully upgrade to 2.1.0 and hope for the best.

Jackson (JSON) 2.11.4 evicted by 2.12.7 — normally, in Java world, this would be fine, except that jackson-module-scala is sensitive to minor version bumps, and it fails at runtime (thank God we have unit tests); and we wouldn’t use jackson-module-scala, but ALAS, akka-serialization-jackson depends on it. This was hard to fix, because it manifests as a weird runtime exception, and I had no idea where it came from or who’s complaining.

In case you’re wondering, we use Circe too, but Circe has been fine this round, as nobody updated its minor version (early-semver).

You can guess the maturity of any Scala project by the number of libraries it uses to do JSON parsing.

org.apache.kafka:kafka-clients version 3.0.1 was also evicted by 7.0.5-css; the former is used by akka-streams-kafka, whereas the latter is required by kafka-avro-serializer; no worries, I consulted this HTML table of compatibility and I think they were compatible.

Couple of months ago this list was bigger, but I prefer dropping dependencies causing issues like hot potatoes.

This was before I went in to fix the compilation and the test errors; at least Scala fails a lot at compile time, instead of runtime. Getting sbt evicted to stay silent can be a chore. In fairness, sbt is really sweet for the ability to turn eviction warnings to errors, by setting the versioning schemes for the required dependencies. Did I mention that I absolutely love sbt? (except for how freaking slow it is)

Anyway, this wasn’t fun:

// https://www.scala-lang.org/blog/2021/02/16/preventing-version-conflicts-with-versionscheme.html libraryDependencySchemes ++= ourLibraryVersioningSchemes, // Configures eviction reports evicted / evictionWarningOptions := EvictionWarningOptions.default .withWarnDirectEvictions(true) .withWarnEvictionSummary(true) .withWarnScalaVersionEviction(true) .withWarnTransitiveEvictions(true) .withShowCallers(true), // Checks evictions on resolving dependencies update := update.dependsOn(evicted).value,

PRO-TIP: to hunt down from where transitive dependencies come from, sbt dependencyBrowseTreeHTML is your friend 😉😍

Many Scala libraries are really well maintained and stable (e.g., Cats), and we’ve got the tools to do it, such as sbt being awesome at conditional/cross compilation, checking versioning schemes, or the availability of Mima, a plugin meant to check for breakages of binary compatibility. It’s a useful case study in communities adapting to their sins. But being the user, and having to deal with all the breakage, is still painful as hell, and I feel that in general many libraries have no respect for downstream users suffering from breakage.

Speaking of versioning schemes, I’ve set Akka’s stuff to pvp. This means that minor versions break compatibility, which should trigger eviction errors. Not sure if it will work. I hope we can pin it down to 2.6.20 (the last Open Source version) 🤷‍♂️

This new development is such a shame, as in my experience Akka has been the gravitational force attracting a vast majority of the Java programmers (the ones into programming, not data science 😛). That will be over soon. I wonder what will happen to the libraries depending on Akka. You may think of libraries meant for the enterprise, with corporate sponsorship, but what about libraries made by volunteers, like Twitter4s? Oh, well.

We’re left with the Scala FP communities, which yield awesome libraries and are awesome people, but the ecosystem is essentially a microcosm of the US political landscape. I’m guessing all programming communities are turning to this nowadays. As an Eastern European, however, it kind of pisses me off.

Maybe this is just me getting older (going to turn 40 soon). Maybe all programming is terrible. I’ve actually started to read Java newsletters, maybe Spring isn’t that bad, I’m also having affectionate memories of JavaScript and Python, and thinking about having a plan B in case this programming thing doesn’t work out 🤷‍♂️

Oh, how the mighty have fallen!

Bye, going to reboot my MacBook, since IntelliJ wanted to reload the project, and nothing works anymore.

The article Scala isn't fun anymore first appeared on alexn.org.

Akka is moving away from Open Source

Alexandru Nedelcu — Wed, 07 Sep 2022 16:31:21 +0000

According to today’s announcement, Lightbend is changing Akka’s licensing to “Business Source License (BSL)”. This is not an Open Source, or a Free Software license. This is a proprietary license.

UPDATE: The license change arrived in the repository.

Freedom

Read their licensing FAQ. Here’s what to take away from it:

It restricts free usage to non-commercial purposes only, otherwise for any production use you’ll need to have a commercial license:

This violates Open Source’s rule 6: “no discrimination against fields of endeavor”;

Or Free Software’s freedom zero: “the freedom to run the program as you wish, for any purpose”;

It adds a clause that says the software will eventually be licensed as Open Source after 3-4 years.

The commercial license might be free of charge for small companies. That is, if you’re comfortable building your company on freeware that’s always subject to change.

Note that such licenses are similar to what Microsoft tried doing with their shared source initiative, back in the day when they were comparing copyleft licenses to a virus. It never took off — access to source code is NOT the point of Open Source / Free Software, but rather the freedom to use that source code, even if you’re making money off of it, even if you’re an evil genius that wants to take over the world with your sharks with freaking laser beams attached to their heads, all powered by FOSS.

I understand the reasoning given in the blog post, but I disagree with it. Here’s why …

Akka, like other products that pulled a bait-and-switch (e.g., MongoDB and others), is popular because it was marketed as being Open Source / Free Software, as developers such as myself would never touch proprietary libraries with a ten-foot pole. Such libraries for me simply don’t exist. There are 2 reasons for it:

I need control of whatever runs in my program, I need the ability to fix it myself, or to have other people fix it for me, people that may not be affiliated with the maker of those tools;

Software licenses are expensive, add up, and even in big companies that can afford it, going through the endless bureaucracy of having such expenses approved is freaking painful, which is why FOSS may be even more popular in corporations than it is in startups;

Open Source is free as in “free market economy.”

Market for Open Source

Selling support or extra tooling in FOSS sometimes works, because it’s complementary — employees can introduce a FOSS library or tool, without any kind of expense approval from upper management, and then the contract for extra stuff can come later, after it has proven its value.

Smart companies try to commoditize their products’ complements, which is what Lightbend tried, but apparently it didn’t work for them. Probably because a software library makes for a poor complement.

The blog post mentions that MariaDB also adopted this proprietary license. The claim is misleading, because MariaDB (the product) is licensed under GPL2 / LGPL2, just like MySQL before it. It couldn’t be otherwise as, to my knowledge, Oracle hasn’t donated the copyright of MySQL. What MariaDB (the company) is actually doing is to offer complementary products, such as MaxScale, otherwise the core server and clients are still Open Source, under GPL2 and LGPL2 respectively. See their own documentation. The difference couldn’t be more striking. In embracing this license, MariaDB became more open compared to other companies that are doing the open-core model. Whereas Lightbend is embracing this license for their core, on which infrastructure is already built. MariaDB took nothing away from what we already had, a rock-solid FOSS database. Not to mention that MariaDB is server software, not a library.

It’s morally wrong to make the product popular, by advertising it as Open Source / Free Software, and then doing a reversal later. Don’t get me wrong, I am sympathetic to the issue that Open Source contributors aren’t getting paid. But in the Java community nobody wants to pay licenses for libraries. If that model ever worked, it was in other ecosystems, such as that of .NET, and that model has been dying there as well. Turns out, trying to monetize software libraries is a losing proposition.

Contributor’s agreements

UPDATED (2022-09-08 09:32:18): This section was modified to correct some missunderstandings. The previous version of this section is available on archive.org.

Lightbend developed most of Akka, but due to its popularity, Akka definitely received contributions from the community. I’m fairly sure that the people that have contributed to Akka will not get compensated right now. This is similar to other products that made similar moves, such as MongoDB. And I’d love to be proven wrong, BTW, although I’m not sure what compensation would be appropriate, given that FOSS contributions are often made in the faith that the project will keep being FOSS.

What makes such license changes possible, even if the project used a copyleft license, is the contributor’s license aggreement (CLA), which is very similar to Apache’s CLA, saying:

“Grant of Copyright License. Subject to the terms and conditions of this Agreement, You hereby grant to the Company and to recipients of software distributed by the Company a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works.” (source)

IANAL, but contributors grant them the right to “sublicense”, meaning that they can redistribute your work under a different license. In other words, the project could have used a copyleft license (GPL), and a license change would still have been possible. Granted, it could have been worse, like the FSF requiring copyright assignments.

Would a copyleft license help, such as the LGPL?

It depends. Copyleft licenses would prevent the code from being incorporated into proprietary code. However, the license change is still possible, depending on the contributor’s aggreement being signed.

I think copyright assignment in Open Source, or aggreements that grant relicensing rights, are EVIL, in spite of all good reasons for it. I do not like to contribute to FOSS projects that ask for such permissions, and you should avoid it too. A license change should require the explicit approval of all contributors, no matter how hard that is.

Will a fork happen?

The silver lining is that the code will be made Open Source in 3-4 years time. However, consider that:

Once a license change happened, it can happen again; license changes destroy trust in the future of the project;

3-4 years is an eternity in Scala’s ecosystem (or for libraries in general), due to all the binary backwards compatibility breakage — consider how Akka took a long time to upgrade to Scala 3, or that whatever bug fixes are coming for the current Akka, we’ll probably need those fixes;

Also, older versions are still FOSS, still licensed under APL 2.0. This means that a fork is possible. That’s the primary value of FOSS actually — if you’re not happy with the direction of the project, you can always fork. I guess we shall see. Note that forks are hard, primarily due to the resources needed and the branding. People don’t switch easily. But regardless, the notion that Akka is a FOSS project is now over, which means any potential for outside contributions to Akka proper is gone.

I do hope for a fork to happen. Will FOSS libraries depending on Akka stick to the old FOSS versions, break compatibility by upgrading to the new and non-FOSS versions, or maintain multiple versions? In my opinion, an unintentional fork will happen regardless, simply because many projects will refuse to upgrade to newer versions. I have no popularity metrics though, my assumption here being that Akka is popular enough.

Feelings

Akka is a great project, and has been one of the major reasons for why people chose Scala, or even Java. We’ve been using it at work, with great success. However, going forward I can no longer recommend any kind of investment in it.

Again, I understand and have known the struggle that FOSS developers and companies go through. FOSS is just not a good business model. But when making decisions about what libraries to use, or where to invest my time, such concerns are simply not my problem. Consider that before this license change I have recommended Akka, and I may have contributed to Akka in my free time, if I ever found the need for it. But after this change, I can no longer do so without getting paid 🤷‍♂️

I do want to thank Lightbend, from the bottom of my heart ❤️, for all they have contributed. I always loved their work. And I just wish that whatever this is wouldn’t have happened. But it did, and now we have to deal with it.

The article Akka is moving away from Open Source first appeared on alexn.org.

Open Source vs Free Software

Alexandru Nedelcu — Wed, 07 Sep 2022 16:28:56 +0000

You may think that Open Source is about having “access to source code”, whereas Free Software is about freedom. Kids, gather around, let me tell you why that’s wrong.

Open Source is based on the Debian Free Software Guidelines. The DFSG was first published in July 1997, the primary author being Bruce Perens. This was Debian Linux’s social contract. And in 1998 the Open Source Initiative was born, with “Open Source” being officially coined, although the term happened earlier.

Interestingly, DSFG and its successor, the Open Source definition, where not based on the Free Software definition. And the Free Software definition did not contain “freedom zero” (aka Open Source’s rule 6), which was added later, being inspired by OSS’s definition.

However, the definitions of Free Software and that of Open Source are equivalent. I know of no license that is compatible with Free Software, and not with Open Source, or vice-versa. For all intents and purposes, the 2 definitions are the same, except that the Open Source one seems to be more precise (to this uninformed citizen).

So what’s the difference?

The difference is one of ideology … while both emphasize the freedom to do what you wish with a program and its source-code, Free Software is an online movement that believes proprietary software is immoral. And while certain proprietary software may be acceptable for now, it is only acceptable on the path towards it being made Free Software. In other words, Free Software proponents are working towards an ideal world in which all software is Free Software.

Open Source, on the other hand, leaves these politics behind and focuses on the economics of it — Open Source software should exist because it’s good from an economic standpoint, on its own merits, because it can be better than proprietary software (cheaper to build, more secure, etc.). This mentality is shunned by Free Software advocates, who’d prefer Free Software even if there wouldn’t be any economic advantage to it.

The Open Source definition exists because businesses wanted to release products as Free Software, but either wanted a more clear social contract, or wanted nothing to do with its politics, or the organization behind it. It’s still about freedom though, but from a slightly different perspective.

When I say that I prefer Open Source software, I say so because I want complete control over it. I want the ability to inspect it, to fix it (or pay others to fix it), to enhance it, and to freely distribute those changes. Some restrictions on redistribution may apply (e.g., GPL), but “redistribution” is clearly defined by copyright law, and those aren’t restrictions on usage, or on the ability to share the derived work. Freedom is about having control, with the ensuing responsibility.

Many times I prefer proprietary software; I’m typing this on a MacBook after all. Sometimes I don’t want freedom, I just want to get the job done. But, if I am to build a program, I’d rather build on top of Open Source infrastructure, because it’s a pretty bad idea to depend on the whims of some other company that may not even answer phone calls.

The difference between Free Software and Open Source does manifest in what people choose to work on. It’s the difference between GCC and LLVM. For those unfamiliar, GCC is (was?) a monolith that made it hard for companies, such as Apple, to build proprietary plugins and tools that interact with it, such as Xcode. It wasn’t just the license, as allegedly the code was by design made to interact poorly with tooling such that tooling would have to be distributed under a GPL-compatible license. Or so the saying goes. This is precisely how LLVM grew, being funded by Apple, and I might say that LLVM is now a wildly successful project. You might say that the difference is in the copyright license: copyleft vs liberal, but that would be wrong. For another example, Linux’s ethos is closer to Open Source, which is why some people still hoped for GNU Hurd to arrive, in spite of Linux’s dominance (and no, it’s not GNU/Linux 😜).

Ever since the movements were born, it attracted many followers amongst computer programmers and sysadmins. As Linux started to dominate Internet servers, and as former junior programmers and sysadmins became seniors with buying power, Open Source gained market-appeal. It’s why we had attempts to redefine it, like Microsoft’s Shared Source, which granted access to the source code, but only for non-commercial purposes. You’ll be forgiven if you don’t remember Microsoft’s Rotor, as nobody cared, and Mono developers were banned from reading that source code, since Microsoft was threatening the Linux ecosystem with patents lawsuits. This was back when Microsoft compared copyleft licenses to a virus and did everything in their power to stop its spread.

Keeping to the official Open Source definition is more than just pedantry. If a license is Open Source, you know exactly what you can do with that software, lawyers rarely needed. When a piece of software is not Open Source, the primary cost is your (economic) freedom, and that’s something you need to take into account when calculating your risk and budget.

Open Source is free as in “free market economy” 😉

The article Open Source vs Free Software first appeared on alexn.org.

Limiting Toxic Technology

Alexandru Nedelcu — Tue, 06 Sep 2022 00:00:00 +0000

Apuseni Moutains, Romania, near the Scărișoara Ice Cave

Toxic usage patterns of technology, begone — inspired by a self-help book book 🤷‍♂️ I changed my philosophy and habits regarding the use of technology.

I’m doing this because it felt like the use of online services, driven by my smartphone, has been consuming me. I might say that I’m a smartphone or an online addict, craving shallow online socializing over meaningful face to face conversations, and wasting time on social media, over productive work. The Covid-19 pandemic has made things worse.

Social Media use

In regard to social media use, the only winning move is to not play!

There are advantages to using services like Twitter or Facebook, but overall social media is a toxic environment. I still want to follow some people, or find out about cool technologies, but drastic changes in my usage patterns are needed.

On Twitter:

Only follow people talking exclusively of programming (via a limited client, e.g., RSS Bridge).

No likes, no retweets, no comments, no private messages, no engagement whatsoever.

Some people will fall off my social radar and that’s fine.

All of my blog posts will get published automatically (via IFTTT or similar).

No engagement analytics checked, ever; as a commitment to this rule, if I ever end up checking the engagement on my blog posts, I’ll give up blogging.

On Facebook:

Only follow my family and very close friends.

Excluding those that talk of politics.

No likes, no shares, no comments, no engagement.

Or I might just go through with its deletion.

On Reddit:

Only follow it via an RSS/Atom feed reader, and only programming channels;

No likes, comments or other engagement;

Use it primarily as a search engine 😉

If I want to send a comment, I’ll pick up the phone, or write an email.

PRO TIP — if you want to delete your social media account right now, to work around the 30 days deletion period in which you can change your mind: change your email address to a “masked” one, using your service of choice (e.g., FastMail), change the password to something random, change your username to something random, remove your phone or other contact details from your social account, delete your account, delete your masked email address, then delete all traces of your new username, and password. This should make it next to impossible to recover.

Smartphone use

The smartphone is a slot-machine. Even if you go off Twitter, the smartphone provides an endless stream of apps that are behaving like slot machines, giving us that dopamine hit we crave for. I actually contemplated switching to a dumb phone, but I still need some smartphone functions that would be hard to replace (e.g., maps, internet hotspot, etc.).

If I pick up my smartphone, I shouldn’t have any statuses to check, or anything on it to fix my boredom. Therefore, the rules are:

No social media;

No video entertainment (Netflix, TikTok, YouTube);

No games;

Browser allowed only for 15 minutes per day;

Messaging restricted to essential (team members, close family members);

Audiobooks and podcasts allowed, but only if fiction, or about programming;

Laptop use

The laptop is primarily a work device, and it should remain a work device.

Block all social media, or news websites;

No games allowed;

Restrict all noisy apps;

Restrict app choices (e.g., web browser, text editor, etc.).

The only way I stay in the loop is via a highly curated list of feeds, to which I subscribe via my self-hosted FreshRSS instance, in combination with Reeder.

Only subscribe to technical blogs, no politics allowed.

Save interesting articles for later;

Read those saved articles on the weekend;

WARN: the feed reader is a slot machine, too, the reason for why it’s added in the “laptop” section, being banned from my smartphone (via DNS-level blocking, to serve as a useful reminder).

Video entertainment

My use of Netflix/Disney+/HBO Max tends to be pretty unhealthy too, as I end up watching alone, and during late hours.

Rules:

No binge-watching, I’m allowed only a single episode per day;

Only watch on big screens, no personal devices allowed;

Prefer to watch with another family member, as watching movies can be a social activity;

Audio entertainment

For programming, listening to music is useful, as it plays the role of white noise, something to help with concentration. Unfortunately I ended up wearing headphones everywhere else, e.g., while walking on the street, during my commute, during house chores, or while eating.

That’s not very productive, as it’s anti-social, and it deprives you of just thinking, aka meditation or solitude. Due to in-ear headphones I also ended up with an in-ear irritation that was hard to treat. And the habit eventually evolved to listening podcasts, many times with political topics too. Listening to podcasts or audiobooks is actually tiring, and should be done after work, not before, or during.

I still like listening to music, or to podcasts, however this habit needs limitations too. I’m yet to figure out some good rules for it, but these days I find myself increasingly without headphones.

Analog entertainment

A common advice given to addicts of technology is to prefer manual chores as leisure activities. Thankfully, we now live in a house with a yard, which provides an endless stream of things to do.

Reading books, having meaningful face to face conversations, or just thinking are always good options. I’ve also started running again.

Boredom is good, actually.

The article Limiting Toxic Technology first appeared on alexn.org.

Tracking Side Effects in Scala

Alexandru Nedelcu — Mon, 23 May 2022 00:00:00 +0000

What if we’d use Scala’s type system for tracking side-effects in impure code, too? In the Scala/FP community we use and love effect systems, such as Cats Effect with its IO data type. “Suspending side-effects” in IO is great, but in Scala it’s either IO or forgoing any kind of type-safety for side-effects, and that’s bad.

In spite of our wishes to the contrary, integration code or low-level code in Scala is inevitable. Also, there are instances in which impure/imperative code is actually clearer. I still have the feeling that Cats-Effect’s Ref is harder to understand in practice than AtomicReference, for example, in spite of all benefits of referential transparency or the decoupling that happens between the declarations of expressions and their evaluation. And software-transactional memory can be worse than Ref, which is probably why in Scala it never took off, in spite of decent implementations available.

For impure code, I’d still like to notice the side effects, visually, via types, and I still want the compiler to protect me from leaks. Scala 3 makes this easier, via Context Functions.

// SCALA 3 import scala.io.StdIn // https://dotty.epfl.ch/docs/reference/experimental/erased-defs // import scala.language.experimental.erasedDefinitions /*erased*/ class CanSideEffect /*erased*/ class CanBlockThreads extends CanSideEffect type UnsafeIO[+A] = CanSideEffect ?=> A type UnsafeBlockingIO[+A] = CanBlockThreads ?=> A def unsafePerformIO[A](f: UnsafeIO[A]): A = f(using new CanSideEffect) def unsafePerformBlockingIO[A](f: UnsafeBlockingIO[A]): A = f(using new CanBlockThreads) //------- object Console { def writeLine(msg: String): UnsafeBlockingIO[Unit] = println(msg) def readLine: UnsafeBlockingIO[String] = StdIn.readLine } @main def main(): Unit = { unsafePerformBlockingIO { Console.writeLine("Enter your name:") val name = Console.readLine Console.writeLine(s"Hello, $name!") } }

We might also make use of erased definitions, to eliminate any overhead, but this is an experimental feature that’s only available in nightly builds.

Note that this is possible to do in Scala 2, but less nice, although in libraries we might think of developing for the future, without leaving users of Scala 2 in the dust.

// SCALA 2 class CanSideEffect class CanBlockThreads extends CanSideEffect type UnsafeIO[+A] = CanSideEffect => A type UnsafeBlockingIO[+A] = CanBlockThreads => A def unsafePerformIO[A](f: UnsafeIO[A]): A = f(new CanSideEffect) def unsafePerformBlockingIO[A](f: UnsafeBlockingIO[A]): A = f(new CanBlockThreads) //------- object Console { def writeLine(msg: String)(implicit permit: CanBlockThreads): Unit = println(msg) def readLine(implicit permit: CanBlockThreads): String = StdIn.readLine } object Main extends App { unsafePerformBlockingIO { implicit permit => Console.writeLine("Enter your name:") val name = Console.readLine Console.writeLine(s"Hello, $name!") } }

Scala 3 also introduced an experimental feature for dealing with “checked exceptions” that makes use of this mechanism, see the “safer exceptions” PR

There is precedent in Scala 2 for using implicits like this. For example, the Await utilities that we use for Future:

import scala.concurrent._ import scala.concurrent.duration._ //... Await.result(future, 10.seconds)

The implementation of result is this one:

object Await { // ... def result[T](awaitable: Awaitable[T], atMost: Duration): T = blocking(awaitable.result(atMost)(AwaitPermission)) }

This is making use of Scala’s “blocking context”, being decoupled from Scala’s Future implementation via this interface:

trait Awaitable[+T] { //... def result(atMost: Duration)(implicit permit: CanAwait): T }

In other words, the implementation forces the use of Await.result via a CanAwait implicit (which can’t be instantiated from user code). This is also how Monix disallows blocking I/O methods to run on top of JavaScript (such as Task.runSyncUnsafe) 😉

Another API that I can remember is Scala-STM, a project that I used back in the day, and that looks like this:

def addLast(elem: Int): Unit = atomic { implicit txn => val p = header.prev() val newNode = new Node(elem, p, header) p.next() = newNode header.prev() = newNode }

The implementation is keeping track of the current transaction, at compile-time, via that implicit parameter. Although, in this case the implicit isn’t just a “compile-time proof”, but rather something that affects the runtime too.

I think I’m being inspired by Martin Odersky’s presentation:

Plain Functional Programming by Martin Odersky (open on YouTube.com)

Do you think tracking side effects like this would be useful in libraries? Monix, for example, also contains impure parts, and could make use of it.

Would you like it? Any prior art that I’m not aware of?

The article Tracking Side Effects in Scala first appeared on alexn.org.

OOP vs Type Classes: Ideology

Alexandru Nedelcu — Fri, 13 May 2022 00:00:00 +0000

Jacques-Louis David, The Intervention of the Sabine Women, 1799. Oil on canvas. Paris, musée du Louvre.

This article explores the difference between OOP design, and parametric polymorphism with Type Classes, as both are possible in Scala.

Motivation

Scala is a hybrid OOP+FP language. If you love OOP, Scala is one of the best static OOP languages. But Scala also exposes parametric polymorphism and can encode type classes.

Thus, developers can also choose to use parametric polymorphism restricted by type classes (aka ad hoc polymorphism). As if choosing when to use immutability versus object identity wasn’t bad enough, developers are also faced with a difficult choice when expressing abstractions. Such choices create tension in teams, with the code style depending on the team leader or whoever does the code reviews.

Let’s go through what is OOP, what is ad hoc polymorphism via type classes, how to design type classes, go through the pros and cons, and establish guidelines for what to pick, depending on the use case.

Video Presentation (YouTube)

I also gave a speech in 2021 at Scala Love in the City on this same topic. If you’re more into videos, rather than reading words, you can watch this as an alternative, although note this article series has more details that couldn’t fit in video form.

Head over to YouTube if you want to watch it:

Video link (open on YouTube.com)

Abstraction

We are gifted with a brain that’s great at recognizing patterns. Take a look at these shapes, which one doesn’t belong?

What’s cool is that there is no right answer, you can point at any one of them, thus grouping the other 3 with some common characteristic. We have above:

3 triangles;

2 triangles with a right angle;

2 isosceles triangles;

3 shapes containing a right angle;

Here’s another one:

We have:

3 squares, and 1 regular quadrilateral that’s not a square;

1 red square, 3 blue;

1 big square, 3 small regular quadrilaterals;

1 square with a different orientation than the other 3;

What about standard Scala types, how can you group these?

Some clues:

Types that have a type parameter;

Types that implement map and flatMap;

Types that implement foreach;

Data collections;

Collection types that can be indexed efficiently;

Collections that have an insertion order;

Types meant for signaling errors;

Types for managing side effects;

Types with a sum/concatenation operation;

Types with an “empty” value;

We’re pretty good at observing similarities, right?

For these images, I’m plagiarizing Julie Moronuki’s The Unreasonable Effectiveness of Metaphor, a keynote and an article that I recommend.

So what is abstraction?

“To draw away, withdraw, remove”, from the Latin “abstractus”;

“To consider as a general object or idea without regard to matter”;

“The act of focusing on one characteristic of an object rather than the object as a whole group of characteristics; the act of separating said qualities from the object or ideas” (late 16th century);

“A member of an idealized subgroup when contemplated according to the abstracted quality which defines the subgroup”;

In the context of software development, abstraction can mean:

idealization, removing details that aren’t relevant, working with idealized models that focus on what’s important;

generalization, looking at what objects or systems have in common that’s of interest, such that we can transfer knowledge, recipes, proofs;

We do this in order to manage complexity because abstractions allow us to map the problem domain better, by focusing on the essential, and it also helps us to reuse code.

The complexity of software projects only grows over time and there’s only so much we can juggle with in our heads.

Black Box Abstraction

A black box is a device, system or object that can be viewed in terms of its inputs and outputs. This means that the input and output are well specified, such that we can form a useful mental model for how it works. Note that the mental model doesn’t have to be correct, it just has to be useful, such that we can operate the system without breaking it open and taking a look at the implementation:

The engineering black box ideal

These can be simple functions, associating one output value to one input, but not necessarily. Examples of complicated black boxes:

Web services;

Automobiles;

The input for an automobile is going to be the steering wheel, the gas and brake pedals. Automobiles are complex machines, but we don’t need to know how they work under the hood in order to drive a car from point A to point B.

And in software development, a common strategy is to build bigger and bigger systems out of black boxes:

No paradigm has a monopoly on composition. FP developers can talk about functions composing, OOP developers can talk about objects composing. FP has the advantage that composition is more automatic, being governed by common protocols that have laws:

If f and g are functions, it’s easy to compose them in a bigger function, g ∘ f. Things get more complicated when the types involved don’t align. For example when that B type is a Future[_] or an IO[_]. But we can come up with protocols such that we can keep this automatic composition.

So what are we talking about?

OOP-driven design is best for building black boxes, connected via design patterns for ensuring decoupling (the arrows between the boxes);

FP-driven design is usually more about white-boxes; and for composing boxes, static FP gives us features such as Type Classes, for describing those design patterns via the type system, maximizing correctness and reusability;

🎤 drop — just kidding 😄

What is OOP?

I like asking this question in interviews, as it’s a little confusing. Ask 10 people, and you’ll probably get 10 different answers, being a question that can be attacked on multiple axes: theory, philosophy, implementation details, best practices and design patterns.

OOP is a paradigm based on the concept of “objects” and their interactions, objects that contain both data and the code for manipulating that data, objects that communicate via messages. And in terms of provided features, we can talk of:

Subtype polymorphism, via single (dynamic) dispatch;

Encapsulation (hiding implementation details);

Inheritance of classes or prototypes;

If there’s one defining feature that defines OOP, that’s subtype polymorphism, everything else flowing from it. Subtype polymorphism gives us the Liskov subtitution principle:

if S is a subtype of T, then objects of type T may be replaced with objects of type S

And note that OOP can be seen in the actor model; what are actors if not objects capable of async communication? And of course, web services behave like OOP objects, because OOP does a great job at modeling black boxes.

A great example of an OOP interface:

trait Iterable[+A] { def iterator: Iterator[A] } trait Iterator[+A] { def hasNext: Boolean def next(): A }

Implemented by most collection types. We’ll come back to it in a future article, to contrast and compare with a Type Class approach.

Are OOP and FP orthogonal? Can they mix?

FP is about working with mathematical functions. Nothing stops objects from being “pure”, with their methods being math functions. For one, objects can perfectly describe immutable data structures, although these are less interesting, since they are just “product types” with names, or records:

case class Customer( name: FullName, emailAddress: EmailAddress, pass: PasswordHash )

When contrasted with FP, the notion of object identity (aka mutability) comes up a lot:

“FP removes one important dimension of complexity — To understand a program part (a function), you need no longer account for the possible executions that can lead to that program part” — Martin Odersky in Simple Functional Programming.

However, this is just an ideal, in practice being often a false statement. Let’s take a mutable interface:

trait Iterator[+A] { def hasNext: Boolean def next(): A }

We can of course “purify” it via IO:

import cats.effect.IO trait Iterator[+A] { def hasNext: IO[Boolean] def next: IO[A] }

Is this not compatible with FP? How about this one?

final class Metrics(counter: AtomicLong) { def touch(): Long = counter.incrementAndGet() } // FP version final class Metrics private (ref: Ref[IO, Long]) { def touch: IO[Long] = ??? }

What’s the difference between that, and this totally pure function?

object Metrics { // Note the internals are now exposed, but function output // depends entirely on function input FTW /s def touch(ref: Ref[IO, Long]): IO[Long] = ??? }

Does that not have “identity” too? Of course, it does. Suspending side effects in IO is cool and useful, but in terms of divorcing a function from its context and history of invocations, that’s easier said than done, and FP won’t save you.

Classes are, after all, just closures with names.

In Scala almost every type definition is a class, and every value is an object, functions included. Interacting with objects happens only via method calls. Turtles all the way down. Scala's OOP facilities and type system are so powerful that they allow us to encode whatever we want, such as type classes or ML modules.

OOP may be orthogonal with FP, but the ideologies behind them are often at odds! So the answer to the question is: yes, they are orthogonal, but with caveats.

What are Type Classes?

In static FP we have parametric polymorphism. Witness the simplest function possible:

def identity[A](a: A): A = a // Compare and contrast with this one — how // many implementations can this have? def foo(a: String): String

This identity function can only have one implementation (unless you use reflection or other runtime tricks). But it would be useful to specify capabilities for that A type:

def sum[A](list: List[A]): A = ??? // Should work for integers sum(List(1, 2, 3)) //= 6 // Should work for BigDecimal sum(List(BigDecimal(1.5), BigDecimal(2.5))) //= 4.0 // Should work for strings sum(List("Hello, ", "World")) //= Hello, World // Should work for empty lists sum(List.empty[Int]) //= 0 sum(List.empty[String]) //= ""

Can Scala’s or Java’s standard library provide this OOP interface?

First try:

trait Combine { def combine(other: Combine): Combine }

Yikes, that’s not good. We can’t combine any two objects inheriting from Combine. We can’t sum up an Int and a String, as this isn’t JavaScript 😊 Liskov’s substitution principle actually sucks here, as we care about the type, and we don’t want to lose type safety 🙂

trait Combine[Self] { self: Self => def combine(other: Self): Self } class String extends Combine[String] { ... } def sum[A <: Combine[A]](list: List[A]): A = ???

We are making use of Scala’s self types, but as you can see, this barely works for combine, and we are missing an empty, the neutral element, “zero” for integers, or the empty string. Java/Scala OOP developers would give up at some point, and those stubborn enough would define this dictionary:

trait Combinable[A] { def combine(x1: A, x2: A): A def empty: A } def sum[A](list: List[A], fns: Combinable[A]): A = list.foldLeft(fns.empty)(fns.combine)

Notice that Combinable is more or less the shape of that foldLeft operation.

These dictionaries are defined per type, and not per object instance. But wouldn’t it be cool if we also had automatic discovery? That’s what implicit parameters in Scala are for:

// Oops, the jig is up trait Monoid[A] { def combine(x1: A, x2: A): A def empty: A } object Monoid { // Visible globally. // WARN: multiple monoids are possible for integers ;-) implicit object intSumInstance extends Monoid[Int] { def combine(x1: Int, x2: Int) = x1 + x2 def empty = 0 } } ///... def sum[A](list: List[A])(implicit m: Monoid[A]): A = list.foldLeft(m.empty)(m.combine) // Or with some syntactic sugar def sum[A : Monoid](list: List[A]): A = ???

Behold Type Classes:

Dictionaries of functions, defined per type, plus…

A mechanism for the global discovery of defined instances, provided by the language.

Type-classes can have laws too. The Monoid here has the following laws:

combine(x, combine(y, z)) == combine(combine(x, y), z)

combine(x, empty) == combine(empty, x) == x

These are like a TCK for testing the compatibility of implementations. Some type classes are lawless, because the laws are hard to specify, or because the signature says it all. That’s fine.

Let’s go back to this signature:

def sum[A : Monoid](list: List[A]): A

Can you see how it describes precisely what the function does? It takes a Monoid and List. What can it do with those, other than to sum up the list? There aren’t that many implementations possible.

With parametric polymorphism, coupled with Type Classes, the types dictate the implementation — and this intuition, that the signature describes precisely what the implementation does, is what static FP developers call “parametricity”

And as we shall see, this gives rise to one of the biggest ideological clashes in computer programming, being right up there with Vim vs Emacs, or tabs vs spaces 😎

Ideological clash

OOP and FP are ideologies — sets of beliefs on how it’s best to build programs and manage the ever-growing complexity. This is much like the political ideologies, like the Left vs Right, Progressives vs Conservatives, Anarchism vs Statism, etc., different sets of beliefs for solving the same problem, promoted by social movements.

Yin and yang

I’m not using this word lightly, computer science is not advanced as a science (via the scientific method), because computer science is also about communication and collaboration, therefore computer science also involves social problems, much like mathematics, but worse, because we can’t rely only on idealized models and logic. And investigating social issues via the scientific method is hard, therefore we basically rely a lot on philosophy, intuition, experience, and fashion.

Computer science is only self-correcting based on what fails or succeeds in the market, thus relying on free market competition. If a solution is popular, that means that it found product/market fit. The downside is that going against the tide is difficult, due to lack of credible scientific evidence. For example “it’s based on math” doesn’t cut it.

Two sides of the same metaphorical coin, both ideologies attack the problem of local reasoning and ever-growing complexity, and both have been succeeding, but from two slightly different perspectives.

OOP is an evolution on procedural programming by encapsulating side effects, whereas FP shuns side effects.

We should learn from both, and we should have some fun while doing it, preferably without anyone getting hurt in the process.

OOP values

flexibility of implementation, meaning that the provider of an API only exposes what they can promise to support, leaving the door open for multiple implementations;

backwards compatibility, think web services — if protocols often change, this is disruptive for teams having to do integration, new development often happens by feature accretion and not breakage, and in case of breakage, developers think of versioning, migrations and in-place updates of components such that disruption is minimal (on the downside, this also means web services exposing APIs with JSON documents having a lot of nullable fields 🤕);

black boxes, components being described in terms of their inputs and outputs, data being coupled with the interpretation of that data (see this exchange between Alan Kay and Rich Hickey);

resource management, in contrast with FP, which makes resource management the job of the runtime, in OOP languages like Scala we implement our own IO, our own HashMap and ConcurrentQueue, etc., and that’s because we have good encapsulation capabilities, access to low-level platform intrinsics, and we don’t shy away from doing optimizations via all sorts of nasty side effects;

Static FP values

I’m saying static FP because NOT ALL of these are values of the FP ideology as practiced in dynamic languages. I recommend watching Spec-ulation (YouTube), a keynote by Rich Hickey.

flexibility at the call site, meaning that an implementation, like a function, should be able to work with as many types as possible, and this is often at odds with flexibility of the implementation, because it means restricting what the implementation can do via type signatures (a very subtle point, being a case of pick your poison);

correctness, API signatures in static FP being very precise, correctness is awesome, but unfortunately this is at odds with backwards compatibility; correctness often means breakage in the API; static FP devs pride themselves with their language’s ability for refactoring, but refactoring often means breakage by definition;

dumb data structures — based on the creed that data lives more than the functions operating on it, and that data structures can be more reusable when they are dumb, with interpretation being a matter of the functions operating on it, which can add or remove restrictions as they like; but this is often at odds with encapsulation and flexibility of implementation;

OOP prefers black boxes, FP prefers white boxes;

dealing with data — FP shines within the I/O boundaries set by the runtime (or by OOP), there’s nothing simpler than a function that manipulates immutable data structures, and such functions can be easily divorced from their underlying context;

derivation of laws and implementations — static typing is about proving correctness, and static FP languages give us tools for automatically deriving proofs and implementations from available ones; e.g. if you have an Ordering[Int] you can quickly derive an Ordering[List[Int]], which isn’t possible in classic OOP;

Degenerate cases

A "black hole", depiction of the "Degenerate Era", taken from "TIMELAPS OF THE FUTURE" (YouTube)

Compare the degenerate signature for OOP classes:

trait Actor { def send(message: Any): Unit }

Versus the degenerate signature for parametric functions in static FP:

def identity[A](a: A): A

This is a spectrum of course. An Actor can implement literally anything, but dealing with it is very unsafe, that contract is useless, and all reasoning enabled by static typing goes out the window. And identity works for any type, maximally usable and composable, but it does absolutely nothing, in practice its utility being only to play “Type Tetris”, i.e. succeeding in the invocations of other functions by making the types match. It’s such a sad little function, in spite of being a star at FP conferences.

What do you want?

These are the questions you have to ask yourself, and they are equivalent:

Whom do you want to constrain, the provider or the client of an API?

What do you prioritize, backwards compatibility or correctness?

What we want is everything, but we can only get some balance, depending on use-case. “It depends” is a boring answer, but the only correct one.

Everything I just said is the conflict of visions that also applies whenever we talk of the merits of dynamically typed versus statically typed languages 😎 It's a conflict of visions born from the problem domains and the compromises that people have had to make, which are then modeling their opinions. This conflict is always present, and people have to be aware of it when picking tools, techniques or languages, and when designing their programs, as these visions transcend the tools used or the programming languages and their features.

Coming up next…

I hope you liked my 1^st article on this topic, as more is coming. Watch this space 😎

The article OOP vs Type Classes: Ideology first appeared on alexn.org.

Implicit vs Scala 3's Given

Alexandru Nedelcu — Wed, 11 May 2022 00:00:00 +0000

I don’t like given, as an alternative to implicit in Scala 3. The more I try working with it, the more it annoys me; and my understanding may be superficial, but I don’t like this direction. Here’s a comparisson between given and implicit, that I hope is fair…

This article uses Scala 3.1.2, which is the most recent version at the time of writing. I just played around, and may have an incomplete understanding — so instead of doing thorough research, I prefer to complain, in order to have others jump in and correct my misunderstandings 😜 Take this with a grain of salt.

Evaluation

Let’s define a type-safe alias for representing email addresses, and we’d like to implement cats.Show and cats.Eq.

In Scala 2.13 we’d define the “type-class instances” in the class’s companion object, like so:

package data import cats._ import cats.syntax.all._ final case class EmailAddress(value: String) object EmailAddress { implicit val show: Show[EmailAddress] = { println("Initializing Show") v => v.value } implicit val equals: Eq[EmailAddress] = { println("Initializing Eq") (x, y) => x.value === y.value } }

If we’d like to use these instances, they are available in the “global scope”, so we can do:

package main // another package import cats.syntax.all._ import data.EmailAddress object Main extends App { val email = EmailAddress("noreply@alexn.org") println("------") println(s"Show: ${email.show}") println(s"Is equal to itself: ${email === email}") println("\nMemoized?\n-------") println(s"show: ${implicitly[Show[EmailAddress]] == implicitly[Show[EmailAddress]]}") println(s"equals: ${implicitly[Eq[EmailAddress]] == implicitly[Eq[EmailAddress]]}") println() }

Which would print:

Initializing Show Initializing Eq ------ Show: noreply@alexn.org Is equal to itself: true Memoized? ------- show: true equals: true

Let’s switch to given, which should replace the implicit flag on these instances. In the official Scala 3 documentation, the given definitions are given outside the companion object, like so:

//... final case class EmailAddress(value: String) given show: Show[EmailAddress] = { println("Initializing Show") v => v.value } given equals: Eq[EmailAddress] = { println("Initializing Eq") (x, y) => x.value === y.value }

This is because in Scala 3 the “package objects” don’t need syntax, so you can just dump such definitions in a file. There’s just one problem — these instances are no longer global, so when we try compiling our project, the compiler now says:

[error] |no implicit argument of type cats.Show[example.EmailAddress] was found for parameter e of method implicitly in object Predef [error] | [error] |The following import might fix the problem: [error] | [error] | import example.show

Well, there is one thing about Scala 3 that I love here: the errors on missing implicits are awesome, because they suggest the possible imports ❤️

But if you want global visibility, and you should, you still have to place them in a companion object; so the official documentation is a little confusing right now.

//... final case class EmailAddress(value: String) object EmailAddress { given show: Show[EmailAddress] = { println("Initializing Show") v => v.value } given equals: Eq[EmailAddress] = { println("Initializing Eq") (x, y) => x.value === y.value } }

Now it works, but there’s a difference:

------ Initializing Show Initialized: noreply@alexn.org Initializing Eq Equal to itself: true Memoized? ------- show: true equals: true

These given instances are defined as lazy val, in fact. The documentation even has this sample:

given global: ExecutionContext = ForkJoinPool()

Kind of makes sense for this to be lazily evaluated, but consider how you’d define this value in Scala 2.x:

implicit lazy val global: ExecutionContext = ForkJoinPool()

To me, in a strictly evaluated language like Scala, this definition is much clearer, whereas the given definition “complects” storage / evaluation, which to me is a separate concern. In a language like Scala, how our values get initialized is a pretty big problem that we always care about.

But wait, is a given always a lazy val? What if we add a type parameter?

final case class EmailAddress(value: String) object EmailAddress { //... // Forcing a type parameter that doesn't do anything: given equals[T <: EmailAddress]: Eq[T] = { println("Initializing Eq") (x, y) => x.value === y.value } }

As you’re probably going to guess, the answer is no — adding a simple type parameter turns this given definition into a def, so now we get this output:

------ Initializing Show Initialized: noreply@alexn.org Initializing Eq Equal to itself: true Memoized? ------- show: true Initializing Eq Initializing Eq equals: false

This isn’t obvious at all, because in fact the object reference for this Eq instance could be reused, this being perfectly safe:

object EmailAddress { //... given equals[T <: EmailAddress]: Eq[T] = // This cast is perfectly safe due to // contra-variance of function parameters 😉 eqRef.asInstanceOf[Eq[T]] private val eqRef: Eq[EmailAddress] = { println("Initializing Eq") (x, y) => x.value === y.value } }

Well, compare and contrast with usage of implicit:

implicit def equals[T <: EmailAddress]: Eq[T]

Which signature is easier to read?

But what if we don’t want the lazy val? What if we always want the behavior of a def? The lazy val implies synchronization behavior that we may want to avoid.

Scala 3 supports an inline keyword, which is pretty cool, and does what you’d expect:

//... object EmailAddress { //... inline given equals: Eq[EmailAddress] = { println("Initializing Eq") (x, y) => x.value === y.value } }

But that’s not quite the same as a def. Inlining functions is cool, sometimes, for performance reasons, but other times the effects are unpredictable and should be used with care.

Compare and contrast:

implicit def equals: Eq[EmailAddress]

In this context, the only advantage of using given is in eliminating the name of those values:

object EmailAddress { given Show[EmailAddress] = v => v.value given Eq[EmailAddress] = (x, y) => x.value === y.value }

Sure, this looks cool. Kind of a high price to pay though. And there’s more …

Final

Consider this example:

final case class TypeInfo[T]( typeName: String, packageName: String, ) trait Newtype[Src] { opaque type Type = Src implicit val typeInfo: TypeInfo[Type] = { val raw = getClass TypeInfo( typeName = raw.getSimpleName.replaceFirst("[$]$", ""), packageName = raw.getPackageName, ) } } object FullName extends Newtype[String] { // Override, as the default logic may not be suitable; override implicit val typeInfo = TypeInfo( typeName = "FullName", packageName = "example", ) }

I’m using this same pattern in monix/newtypes, this being an instance in which an implicit value is provided by a trait, but we leave the possibility of overriding it, and for good reasons.

What if we’d use given?

trait Newtype[Src] { opaque type Type = Src given typeInfo: TypeInfo[Type] = { val raw = getClass TypeInfo( typeName = raw.getSimpleName.replaceFirst("[$]$", ""), packageName = raw.getPackageName, ) } }

Well, that doesn’t work, because the declared given is a final member, therefore we can no longer override it:

[error] -- [E164] Declaration Error: ... [error] 22 | override implicit val typeInfo = [error] | ^ [error] |error overriding given instance typeInfo in trait Newtype of type example.TypeInfo[example.FullName.Type]; [error] | value typeInfo of type example.TypeInfo[example.FullName.Type] cannot override final member given instance typeInfo in trait Newtype

So finally, this:

given typeInfo: TypeInfo[Type]

Is actually equivalent with this:

implicit final lazy val typeInfo: TypeInfo[Type]

Yikes; that’s not obvious 😏

Vocabulary

The documentation can definitely improve, but I don’t think the documentation is the problem.

As a vocabulary preference, I don’t see how given makes things easier to understand, especially for beginners, but this is a subjective opinion, and it may be a matter of learned taste. It’s exactly the same concept, though, using given and using does not make things easier to understand, unless the power of implicits is limited, implicits being hard to understand due to their power, not due to their naming. And I don’t see how beginners could be given an explanation that doesn’t use the term “implicit parameters”, something that “givens” obscures. But I concede that I may lack imagination for teaching 🤷‍♂️

I will argue that the existence of both implicit and given introduces more Perl-like TIMTOWTDI, but in a bad way (I’m saying this as a guy that loves TIMTOWTDI, usually). If given is the future, I’d like to say that implicit should be deprecated, but given the current behavior of given, I hope implicit stays 🤨

Does given have any redeeming quality that I’m not seeing? Maybe any good design reasons for why it forces the current behavior?

The article Implicit vs Scala 3's Given first appeared on alexn.org.

Scala OOFP Design Sample

Alexandru Nedelcu — Mon, 18 Apr 2022 00:00:00 +0000

Scala is considered a multi-paradigm language, for better or worse, being one of the best OOP languages, which is why it's so versatile. Let's do a design exercise, going from OOP to static FP, and back. Let's understand the various techniques promoted in the community, and understand why the OOP design isn't just “idiomatic” for Scala, but can be superior to alternatives.

The design problem

Design a “queue of delayed messages.”

Should you choose to accept this challenge, the queue should be similar to using a message broker like IBM-MQ, or a Kafka topic, but with the ability to schedule messages to be delivered at exact timestamps in the future. It also has to support an acknowledgement mechanism, with messages becoming invisible when pulled from the queue, and if the acknowledgement doesn’t happen, then messages have to reappear on the queue after a timeout.

As an underlying implementation, we could use a RDBMS database via JDBC — in our project (at $work) we use MS-SQL. But we could also have an in-memory implementation, which is actually super useful, and not just as a mock in tests. And we could migrate to a NoSQL database, such as MongoDB, or Redis, or maybe even Cassandra — anything that can do atomic compare-and-swap operations (on keys) should work.

Using a real-world example can be boring, due to all the concerns involved, bear with me…

Classic and impure OOP design

In classic OOP (e.g. Java) we could work with an interface like this:

import java.util.UUID import java.time.Instant import scala.concurrent.Future trait DelayedQueue[A] { /** Schedules new messages on the queue. */ def offer(m: OfferedMessage[A]): Future[OfferOutcome] /** Returns messages ready to be processed. */ def tryPoll: Future[Option[ReceivedMessage[A]]] /** Deletes messages from the queue. */ def discard(key: String): Future[Unit] } enum OfferOutcome { // Scala 3's tagged unions case Created case Updated case Ignored } final case class OfferedMessage[+A]( key: String, payload: A, scheduleAt: Instant, canUpdate: Boolean, ) final case class ReceivedMessage[+A]( message: A, messageID: UUID, receivedAt: Instant, acknowledge: () => Future[Unit], )

Using this API, to illustrate its usage for building consumers:

import scala.concurrent.ExecutionContext def drain[A](queue: DelayedQueue[A])(using ExecutionContext): Future[Unit] = queue.tryPoll.flatMap { case None => Future.unit case Some(msg) => println(s"Received: ${msg.message}") // Acknowledge it, or it will reappear after a timeout; msg.acknowledge() .flatMap(_ => drain(queue)) // process next }

There are many subtle design choices exposed by this API, which make it an acceptable one. Let’s enumerate …

1) Given the OOP context, we should already have an intuition that this interface does I/O, because there aren’t many good reasons for why open-world OOP interfaces are needed, the primary reason being that we need to abstract away implementation details that involve I/O;

2) We are using Future, so the I/O can be asynchronous. And Future can be completed with exceptions, so code should install generic exception handlers that do logging or whatever it is that you do with unexpected exceptions;

3) Messages get persisted based on a key: String, because when scheduling messages in advance, we may want to also get rid of them in advance; this “delayed queue” could then be used to trigger timeout exceptions for asynchronous operations that fail to yield a result; it’s why we have a discard(key) operation, which may seem redundant with acknowledge from ReceivedMessage, but isn’t;

4) The generic parameter in DelayedQueue[A] means we are forced to have a queue per message-type, which is a best practice, and avoids implementation leaks (more on that below);

5) All actionable “error” conditions are expressed in the returned types — in the case of tryPoll via the returned Option, and in the case of offer via the OfferOutcome union type; they don’t seem like errors, because they are “designed away”; in this sample we don’t need Either or Java’s checked exceptions (more on that below);

The interface does have flaws — for example, tryPoll is an operation that has to be repeated, in order to keep the channel open and ready to process messages. If we sleep between successive tryPoll operations, this might be an implementation detail as well. And it can be improved, but how we handle streaming isn’t the purpose of this article.

Beware: constraints can be implementation leaks

Leaky abstractions are common when designing abstract APIs. If you don’t have at least 2 very different implementations, the API isn’t really abstract. And in these abstract OOP interfaces, the constraints placed on parameter types lead to leaky abstractions. But let’s put this to the test…

Let’s say we don’t want the A type parameter on the interface, so we could have it per method. But having it per method requires introducing a constraint for serialization/deserialization. Let’s design the type-class:

import monix.newtypes.TypeInfo class ParsingException(message: String) extends RuntimeException(message) /** * Type-class for serializing and deserializing to and from `String`. */ trait ValueCodec[A] { def kind: TypeInfo[A] def serialize(a: A): String def deserialize(s: String): Either[ParsingException, A] }

Notes:

TypeInfo is from monix-newtypes, but you can replace it with a plain String — we are making the name of the type be part of the persisted key, otherwise we can end up with conflicts when reusing the same database table / queue for multiple data types;

Serialization should never return an error; but deserialization can end in error because we are losing information in the serialization process, thus we cannot force validations via the type system, and this error needs to be very explicitly modeled via the type system, as it’s not just an “exception”;

Based on this codec, our interface could become:

trait DelayedQueue { def offer[A: ValueCodec](m: OfferedMessage[A]): Future[OfferOutcome] def tryPoll[A: ValueCodec]: Future[Option[ReceivedMessage[A]]] def discard[A: ValueCodec](key: String): Future[Unit] }

You can see how this sucks:

We are encouraging the usage of a single queue for multiple message types;

The call-sites of those methods are awkward;

Serialization method is an implementation leak;

Serializing the values to string makes no sense for an in-memory implementation. It’s also restrictive, as your database might support a binary serialization format that’s more efficient than string (e.g. protocol buffers).

NOTE — our ValueCodec isn’t useless. We can still use it for our RDBMS implementation:

final class DelayedQueueJDBC[A: ValueCodec] private ( connection: DataSource, //... ) extends DelayedQueue[A] { //... }

Notice how the constraint is now part of the class’s constructor, and by being a constructor parameter, it doesn’t have to be part of the abstract interface.

Suspending side effects

We can improve our interface by working with an IO datatype, such as that of Cats Effect, replacing our usage of Future:

import cats.effect.IO trait DelayedQueue[A] { def offer(m: OfferedMessage[A]): IO[OfferOutcome] def tryPoll: IO[Option[ReceivedMessage[A]]] def discard(key: String): IO[Unit] } final case class ReceivedMessage[+A]( //... acknowledge: IO[Unit], )

Our IO-driven API is definitely an improvement. Due to the usage of IO, it’s even more clear that we are modelling I/O operations with side effects involved. And the applications of these functions are referentially transparent. We are now proudly making use of FP. For more details on how IO is better than Future, I recommend this presentation:

Video link (open on YouTube.com)

And, I would argue that usage of IO in this API can be the best you can do in Scala 😉

Beware: IO in our API is a constraint!

By specifying IO we mean that “implementations are free to launch rockets to Mars.” And that’s good in the OOP sense, as we allow for flexibility of implementation, while keeping the protocol abstract enough that we can understand what’s going on.

Static FP proponents usually don’t like this (more details below), but I disagree.

Tagless Final

IO can become a type parameter:

trait DelayedQueue[F[_], A] { def offer(m: OfferedMessage[A]): F[OfferOutcome] def tryPoll: F[Option[ReceivedMessage[F, A]]] def discard(key: String): F[Unit] } final case class ReceivedMessage[F[_], +A]( //... acknowledge: F[Unit], )

F[_] is a placeholder for what we call “an effect”. An effect in the context of FP isn’t a side effect — think of “effects” as anything other than returning the result of a computation.

Option and Either are modeling effects too. NOT side effects, but the main effects. In the English language, an effect is something that has a cause, and in our programming language we model causality via flatMap. So, as an incomplete and wrong intuition, effects in Scala are modeled via monadic data-types (types that implement a lawful flatMap and constructor pair, aka Monad).

By making DelayedQueue work with any F[_]:

we could use alternative IO data type implementations;

we could use monad stacks, e.g. EitherT[IO, *];

we could use cats.Id for really simple and pure tests;

And “tagless final” is a technique that’s equivalent to usage of OOP interfaces, except that the interfaces we get are parametrized by F[_], aka the “effect type”.

There are better places on the web that explain tagless final, and I’m likely doing a bad job. For those wanting to learn more about tagless final and its correspondence to object algebras, I recommend “Extensibility for the Masses: Practical Extensibility with Object Algebras” and “From Object Algebras to Finally Tagless Interpreters”.

In “tagless final” we define capabilities via OOP-like interfaces (also called “algebras”):

trait Logger[F[_]] { def info(message: String): F[Unit] }

Then to model our consumer, we can combine multiple algebras:

import cats.Monad import cats.syntax.all._ def drain[F[_], A](using // Scala 3's `implicit` Monad[F], Logger[F], DelayedQueue[F, A] ): F[Unit] = summon[DelayedQueue[F, A]].tryPoll.flatMap { case None => Applicative[F].unit case Some(msg) => summon[Logger[F]].info(s"Received: ${msg.message}") .flatMap(_ => msg.acknowledge) // deletes the message .flatMap(_ => drain[F, A]) // next please }

Let’s ignore the implementation and zoom in on this function’s signature:

def drain[F[_], A](using Monad[F], Logger[F], DelayedQueue[F, A] ): F[Unit]

We are saying that we want F[_] to implement:

Monad: which means a sequencing/chaining of steps is involved, via flatMap;

Logger: which means we want to log something;

DelayedQueue: our queuing capability;

This is what’s called “parametricity” in static FP circles: by looking at the function’s signature, we have a pretty good idea of what it does, because the function is so generic that it’s hard for it to do anything else.

And yet there’s something amiss for any seasoned OOP architect — let’s zoom back on the API:

trait DelayedQueue[F[_], A] { def offer(m: OfferedMessage[A]): F[OfferOutcome] def tryPoll: F[Option[ReceivedMessage[F, A]]] def discard(key: String): F[Unit] }

In this example, we aren’t talking of any F[_] effect type. It’s ridiculous to think that we can use Either or Option here. If we use Id, we only use it for demonstrative purposes, with absolutely no value.

We say F[_] in this API, but as its authors, we know that we are talking of a data type that encapsulates / suspends side effects, but without saying so. This is the opposite of parametricity, and in this context usage of this style is not good API design, unless you really, really want tagless final in your project, and thus rely on conventions (e.g. whenever you see F[_], you see IO). The danger being that readers of this code might not get a good mental model of using this API, because there’s information missing, and that’s bad.

To be clear — this isn’t good API design because F[_] is left unspecified.

To be fair, we could restrict F[_] to the cats.effect.Concurrent type-class, but at that point we might as well work with IO directly. And if we’d revert to the IO-driven API, we could have:

def drain[A]( logger: Logger, queue: DelayedQueue[A] ): IO[Unit]

The big difference, in terms of “parametricity”, is that executing an IO could launch rockets to Mars for all we know, so we can’t really restrict the implementation of this function to depend entirely on its input. But this is Scala for you. On top of the JVM we can do anything, anywhere, anyway, relying somewhat on good practices and conventions for not triggering side effects, or using shared mutable state. And while using F[_] can make violations of parametricity harder, it doesn’t make it impossible, and those violations have the potential to be worse, since without IO, you definitely end up with unsuspended side effects.

There is also another difference … implicit parameters mean that we are using only one instance in the whole project. We have only one Logger instance, and only one DelayedQueue[A] instance. This is a best practice. As another example, instances of type-classes need to be unique, because type-classes need coherence, i.e. if you can’t rely on a single instance being available in the whole project, then the code can lead to bugs, due to function calls yielding different results for the same parameters, depending on the call-site.

Nothing stops you from converting to implicit parameters, even without F[_]. I think people abuse implicit parameters, I prefered to reserve use of implicit parameters for type-class instances, but that battle was lost, as with tagless final these “algebras” aren’t necessarily type-classes anyway:

def drain[A](using logger: Logger, queue: DelayedQueue[A] ): IO[Unit]

For me (personal opinion warning!), this version is pretty damn explicit about what it does, I don’t see much of a difference in my understanding of it, certainly not a difference that’s big enough to outweigh the cost of introducing higher-kinded types and type-classes. If you’ve been on any medium-sized project, a project that’s regularly looking for promising beginners, I’m pretty sure the concern did come up repeatedly – how in the world are beginners going to cope with learning the use of F[_] with type-classes and everything related? This on top of everything else, which can increase the barrier to entry significantly. And I think the experience of the job market varries based on the available talent.

Monad Transformers, ZIO … are useless here

If we’d like to get fancy, we could work with a type that handles “dependency injection” and explicit errors, equivalent to Java’s “checked exceptions”:

type ZIO[-Env, +Error, +Result] = Env => IO[Either[Error, Result]]

We could work with this, but we have to be careful because the JVM doesn’t do tail-calls optimizations (TCO), so this could be memory unsafe. Also, we need Monad and other type-classes and utilities defined for it, so if using Cats, we could work with:

import cats.data.{EitherT, Kleisli} type ZIO[Env, Error, Result] = Kleisli[[A] =>> EitherT[IO, Error, A], Env, Result]

We are combining multiple monad transformers, and so we have a “monad transformers stack”. And to express this type, we are using type lambdas.

NOTE:

the Env is the “environment”, which specifies what’s needed for the execution of the operation — for example, we might need a JDBC DataSource or a Logger;

the Error type parameter is for signaling errors that must be handled;

This type is equivalent to ZIO[R, E, A]. A data-type like ZIO is nicer if you use it everywhere, however I prefer the monad transformers from Cats, because they are more composable, and we only pay this price where needed … which in my case is almost never 🤷‍♂️

Can we have …

An explicit environment that makes sense in an abstract API?

An explicit error type that signals currently unspecified error conditions?

The answer is NO and NO, because in both cases we are talking of implementation leaks. It doesn’t prevent us from trying, though:

trait DelayedQueue[A] { def offer(m: OfferedMessage[A]): ZIO[DataSource, SQLException, OfferOutcome] def tryPoll: ZIO[DataSource, SQLException, Option[ReceivedMessage[F, A]]] def discard(key: String): ZIO[DataSource, SQLException, Unit] }

DataSource or SQLException are only relevant for the JDBC-powered implementation, otherwise they are implementation leaks.

For one, I’m always amazed by the people’s struggle to introduce various automated dependency injection techniques in their project, given we have OOP encapsulation and OOP constructors. Haskell’s monad transformer stacks have always felt off, and it isn’t just Scala’s more limited type inference. Haskell is just not very good at encapsulation, as it’s not a very good OOP language, most features and most of its culture conspiring against OOP-like encapsulation. In Scala, if you have dependencies that you’d like to avoid as explicit parameters, the first strategy is to give them as parameters in a class constructor that’s only called once, then pass that object around. Anything else feels like a hack in Scala, IMO, even if constructor-based dependency injection has its drawbacks.

As for errors, one can argue that SQLException is important, and not declaring it in the output type is simply an unspecified error condition. But I’d argue that if it is important, then your API and your application’s design is probably screwed (with exceptions).

I’m repeating myself — we have no important error type that wasn’t already captured in the return types of our API:

Option is an effect type that signals None in case there are no messages available; in which case you can stop, or you can retry later;

OfferOutcome is a union data type that tells us if our offering has succeeded or not, taking into account other concurrent offerings;

But in most cases, it doesn’t matter what this result even is, because after this call is executed, we know for a fact that there is a message scheduled with its corresponding key; so the result itself could be ignored;

In this context the usage of ZIO, to enhance the return types, would violate an important OOP principle:

“Program to an interface, not to an implementation.”

This is the equivalent of “parametricity”, except it refers to encapsulating implementation details, and use flexible interfaces in your logic. Abstraction, as a general concept, is not just about observing groups of data types that can be operated in the same way. Abstraction is also about ignoring the non-essential, which is what OOP’s encapsulation and subtype polymorphism are doing.

There is nothing about this ZIO data type that improves our API in any way, quite the contrary, what it brings to the table is only going to obscure what we’re trying to express, as now we are forced to fill in those type parameters with something:

trait DelayedQueue[-Env, A] { def offer(m: OfferedMessage[A]): ZIO[Env, Nothing, OfferOutcome] def tryPoll: ZIO[Env, Nothing, Option[ReceivedMessage[F, A]]] def discard(key: String): ZIO[Env, Nothing, Unit] }

Note that IO[A] from Cats Effect is the equivalent of ZIO[Any, Nothing, A] (no, the explicit error type isn’t Throwable). So we have to parametrize the environment and ignore the error type. Or use both as parameters, but that’s just awful.

Error Handling

As said multiple times, the important error conditions are already baked into the API. But that’s not entirely accurate — what we did was to design them away. Remember this good principle of software design:

“Design the errors out of existence.”

An “error”, in the English language, means a mistake, an action which is inaccurate or incorrect. This can happen in our programming due to:

incomplete or incorrect input;

runtime conditions that are outside of our program’s control;

bugs;

Our programs mostly care about the happy path, but we have to treat errors too. “Designing errors out of existence” means that, if you care about an error, then it shouldn’t be an error. This means:

Model an output data type that expresses the “error” in a way that is hard to ignore — e.g. the result of a parsing function is either the parsed result, or some message that needs to be displayed; but this only works for input errors of functions whose result we depend on (i.e. we have a data dependency), otherwise …

Make the errors impossible, such that the caller of your API no longer has the responsibility of dealing with those errors;

Question — what can you do with an SQLException?

Answer: mostly nothing! The right place to treat an SQLException is in the encapsulated implementation of your API. The lower-level it is, the better. Once you stream that to the client, it’s already too late to do anything about it.

Both Cats’ EitherT and ZIO exist in order to make errors hard to ignore in instances in which they’d be easy to ignore:

def offer(m: OfferedMessage[A]): IO[Either[SQLException, OfferOutcome]]

In this case that SQLException can be easy to ignore, as we may not care about the OfferOutcome, so you can easily end up with this:

for { _ <- queue.offer(message) // oops! _ <- log("done") } yield ()

As a band-aid you can make it more difficult to ignore such exceptions by usage of monad transformers, or related data types. The purpose here being to make that flatMap less forgiving:

// cats.EitherT def offer(m: OfferedMessage[A]): EitherT[IO, SQLException, OfferOutcome] // ZIO def offer(m: OfferedMessage[A]): ZIO[Any, SQLException, OfferOutcome]

Except this is bad software design. Not only is SQLException an implementation leak, the bigger problem is that there are only 2 courses of action:

you can shrug and re-throw it like any other Throwable, such that the process can log it and do whatever it does for unknown exceptions;

you can retry the operation — and here’s the rub — a retry is best left as an implementation detail, because the retry logic depends on the resource type;

And to make matters worse, it’s really hard to parse an SQLException. Anyone that has ever done it knows that these exceptions aren’t standard at all, and sometimes we end up parsing its message in order to ascertain whether the transaction can be retried, or not. And there are multiple policies possible:

in case a concurrent update happened, with the transaction failing, it’s often best for the retry to happen immediately, although it depends on the contention, and it’s good to keep in mind that RDBMS databases don’t do well with many concurrent updates hitting the same table;

in case the connection is down, or in case of timeouts, using an exponential backoff strategy might be wise;

in case of an SQL syntax error, no retry is possible, of course, so you need to bail out as fast as possible;

Note that we probably have to parse SQLException in our implementation, in order to return a usable OfferOutcome. As an API author, if you take the time to parse SQLException such that you can discriminate between such cases, then why not take care of the retry logic yourself? Designing errors out of existence, in this case, means that the user of the API shouldn’t care.

If you’re designing a light-weight API that just wraps JDBC, by all means, expose SQLException, although I’ll be eternally grateful for any JDBC wrapper that avoids exposing SQLException and gives me something more usable, because it is goddamn awful.

Epilogue

In application code, in a Scala project, I’d argue that nothing beats the clarity and flexibility of interfaces like this:

import cats.effect.IO trait DelayedQueue[A] { def offer(m: OfferedMessage[A]): IO[OfferOutcome] def tryPoll: IO[Option[ReceivedMessage[A]]] def discard(key: String): IO[Unit] }

For library code, using an F[_] type parameter for the effect type might be better, as it allows people to come up with their monad transformers stack, or a different IO implementation. Such library code allows for a “tagless final” approach, like we’ve seen here, and I admit, people have some good reasons for liking it. However, this doesn’t come without cost, a cost that needs to be balanced by great documentation.

Scala isn’t Haskell, and for me the tagless final approach (powered by implicit parameters / type-classes, or even monad transformers), or ZIO, both proved to be anti-OOP. An “idiomatic” Scala style should use its OOP features for OOP polymorphism, as that’s the “path of least resistance”. And tagless final isn’t incompatible with this approach, but I fear that usage of F[_] or of advanced language features end up obscuring the APIs exposed — compared to just passing IO-driven OOP interfaces around, as simple parameters.

These techniques and advanced features have their place in our toolbox, for sure, and the stronger the types, the better, but nothing beats good taste, or design that focuses on ease of use and the user’s mental model, and such skills have less to do with static typing, or the application of math.

Designing good APIs requires empathy for the users of your API. Don’t get me wrong, when you inflict pain on your users, it’s better if that pain is strongly typed, but it would be much better to not inflict that pain at all.

The article Scala OOFP Design Sample first appeared on alexn.org.

Scala's Gitter to Discord migration mistake

Alexandru Nedelcu — Sat, 09 Apr 2022 00:00:00 +0000

The Scala community is increasingly using Discord for “real-time chat”. This was announced in December, and the community page lists several Discord “servers”, one of them belonging to Typelevel. I think this move from Gitter to Discord is probably a mistake.

I participated in Typelevel’s migration as well. The migration was done out of genuine concern for the health of the community, and I considered Discord as a good experiment to have. One can also argue that the community simply went to where the users wanted to be, although in our case it was definitely a decision pushed forward by a committee that helped the migration. Can’t speak for others, but I went after a couple of contributors to convince them to join Discord.

When comparing Gitter to Discord, the main reasons for why I now believe Gitter is better is because:

Information on Gitter is public, it does not require an account to read;

Information on Gitter can be linked from across the web;

Information on Gitter being public, is indexed by search engines and is being archived;

By contrast, Discord is an information black hole, Discord is a closed ecosystem, Discord is anti-web. It doesn’t matter how much Gitter’s client used to annoy you, these reasons alone make Gitter better than Discord, with Discord unable to have any redeeming quality that make it better for Open Source contributors that want to provide support or to spread knowledge.

If engagement of new people is a concern, Discord can’t be better because it can’t be read without having an account. If the slowed growth of the community is a concern, I don’t think Gitter was to blame, or that Discord made things better. I could be convinced with numbers, but I don’t think those numbers exist.

If per-repo silos are a concern, Discord can’t be better because its servers and channels can’t be linked from across the web. Compare the discoverability of https://gitter.im/typelevel/cats or https://gitter.im/http4s/http4s with https://discord.gg/XF3CXcMzqD. On Gitter, people in the scala/scala Gitter chat room could be redirected to the http4s/http4s chat room, via web links (an outstanding technology). On Discord, however, that’s not how it works, because Scala-lang and Typelevel have different servers. You must convince people to jump between servers, and to join a server you need an invitation. Discord servers are islands, which makes them silos in a way that Gitter chat rooms have never been.

As an Open Source contributor, to open a support channel, your project has to be either very popular, or be part of / related to Scalameta, or Typelevel, or ZIO, or Play Framework. Otherwise, you’ll have an uphill battle to convince people in joining Discord, or if they already are Discord users, to convince them to join your server. And creating a support channel on a server that you don’t own means giving up control of that channel to the server’s admins, which has pros and cons. Giving up control means there’s better enforcement of a code of conduct, making people feel safer however, I’m not convinced that the chat rooms are nicer/safer than they were on Gitter.

Maybe I’m wrong, and moderation is indeed a soul-wrecking job made by unsung heroes. I will also add that there is value in having a Gitter channel per repo, with separate moderators, because in big communities the moderation isn’t a job that can be done by one or a couple of individuals. Not without it leading to burnout. We are burning out our most valuable contributors by pushing them to pick up administrative tasks.

If inclusivity is a concern, usage of Discord discriminates against those that cannot or will not use Discord, either because Discord blocks them, or because of privacy concerns. Discord’s Terms of Service has been controversial, and they are known for having banned people for discussing cheats in games. AFAIK, they once banned a server that discussed anti-cheating solutions. It also asks for phone numbers if it detects VPNs or Tor, and their app does not work from behind our corporate VPN/proxy, which I believe is a deliberate strategy, not a bug. If you live in a country that’s problematic for the US, which of Discord or Gitter do you think is more likely to ban you?

I was triggered by the news that Thunderbird now supports Matrix. Matrix is an open standard for decentralized chat. Gitter has joined Element, and it speaks Matrix, which means you can use the Element app to connect to it. It’s not perfect, but it works.

Open Source needs open platforms for teaching. We need the web to stay decentralized. And decentralization is always inconvenient at first, solutions evolve more slowly, but think of how pervasive and awesome email or the web are. I’m not even advertising for Gitter, I’m just anti-Discord. Personally, I’d wish for Discourse.org servers to be more common in the community. Shout-out to users.scala-lang.org, contributors.scala-lang.org or discuss.lightbend.com. Real-time chat isn’t good for providing support or for spreading knowledge, IMO, but that’s a personal preference that wouldn’t exclude people out of the conversation or lock information behind proprietary walls.

If you’re an Open Source contributor, or user, what I ask of you is to not give up on the open web. That ship may have sailed, but one can hope.

Update:

To answer one concern — Gitter may not be good for finding information in it, however we are comparing a technical problem with a system that is “defective by design”, just like DRM is. There is nothing you can do to fix DRM’s issues, just like there is nothing that can be done to fix Discord.

I remember legendary conversations that happened on Gitter, and they are still accessible. The web is filled, for example, with bits of wisdom by Fabio Labella (SystemFw) — here’s where he exposed his ideas for what became Cats-Effect 3’s new interruption model:

gitter.im/typelevel/cats-effect?at=5c5f1a2fef98455ea4096756

Such bits may be hard to find, but there, I just pointed you to a useful conversation on Gitter, which is accessible, and you don’t have to have a Discord account in order for you to read it. When comparing how much they suck, they aren’t in the same league.

The article Scala's Gitter to Discord migration mistake first appeared on alexn.org.

On Typelevel and Monix

Alexandru Nedelcu — Tue, 05 Apr 2022 00:00:00 +0000

Planning the future is difficult, but can bring clarity and purpose. See Finding Focus in Harsh Times for context.

On Typelevel

Typelevel is a great community of builders that want to practice FP in Scala. Its “steering committee” (link / archive) is a group of brilliant and kind people that are doing great work in keeping the community welcoming and inclusive.

I’m stepping down from the Typelevel “Steering Committee”.

Moderating and leading a community is gruesome work made by unsung heroes, and I can’t be a part of it anymore. For some time now I’ve been absent, with my only contributions to steering having been rants, and frankly I’d rather get back to coding or other contributions to Open Source that I can manage.

Typelevel is growing, and you can make a difference. If you feel you’re a fit, get involved, as there’s a call for ‘steering committee’ members.

The Future of Monix

Monix has been my love project, but due to events unfolding since 2019, with life and the world going mad, I’ve been on an unplanned hiatus from Open Source contributions. I did contribute monix-newtypes, as contributing a new project, scratching an immediate itch, felt easier 🙂

I’ll be forever grateful to Piotr Gawryś, who helped in maintaining and developing Monix, but eventually development stalled. Development of Monix was stalled primarily because small, incremental improvements are no longer possible. And this happened due to the release of Cats-Effect 3.

Cats Effect 3 is an awesome new version of an already good library. But while being a necessary upgrade, it fundamentally changes the concurrency model it was based on. The changes are so profound that it’s arguably an entirely new library, and upgrading Monix isn’t easy, because compatibility means updating everything. This means not just Task, but also Cancelable, CancelableFuture, Observable, Iterant, and I mean everything.

When Monix started, it had the goal of having “zero dependencies”. Having no dependencies is a virtue, precisely because those dependencies can seriously break compatibility. There is no way for Monix and CE 3 to currently coexist in the same project, due to JVM’s limitations and the decision for Monix to depend directly on Cats-Effect. If Monix were independent of such base dependencies, it could coexist while its maintainers could afford a hiatus.

I’m always reminded of Rich Hickey’s thoughts from his Spec-ulation Keynote. TLDR — when you break compatibility, maybe it’s better to change the namespace too. Given in static FP we care about correctness a lot, how to evolve APIs is a really tough problem. I’m still thinking that Monix’s major versions should be allowed to coexist by changing the package name (e.g. monix.v4), but few other projects are doing it.

I will be resuming the work on Monix, and will be calling for volunteers. And I hope I won’t let people down again. My current plan is:

Monix will be upgraded to the Cats-Effect 3 model, which will include project-wide changes;

The dependency on Cats and Cats-Effect 3, however, will probably be separated in different subprojects; while this involves “orphaned instances”, this decision is made easier by tooling (modularity, ftw):

the Scala compiler supports custom implicit imports via -Yimports; I don’t recommend it, but the option is there;

Scala 3 automatically suggests possible imports for missing implicits;

IntelliJ IDEA too automatically suggests imports;

I’d like to make monix-bio be part of the main project;

I have some new functionality in mind that will make Monix an unbeatable replacement for scala.concurrent, RxJava, Akka Streams, and the middle-ground that people need in adopting FP in Scala;

Looking forward to having fun, I’m very excited about it, actually 🤩

If you’d like to help in building the next version of Monix, contact me.

The article On Typelevel and Monix first appeared on alexn.org.

A Return to Blogging

Alexandru Nedelcu — Fri, 01 Apr 2022 00:00:00 +0000

I like writing about my personal projects and about programming in general. Helping others learn programming seems to be my calling. I rarely write on this blog, however, but I'm trying to change that.

When writing, perfect is the enemy of good. I’ve noticed this time and time again: the need for an article to be perfect, the need to go in depth, is what’s preventing me from creating more content. And social media is antithetical to good writing, pushing us to regurgitate shallow thoughts that have no lasting value. It’s back to blogging for me.

So expect more … words, words about programming, personal projects and the life of a software developer, published on this blog.

This blog is undergoing a re-organization/simplification. For those that follow my RSS feeds, I apologize if your feed reader is seeing older items as unread.

If you’re a software developer, you should have a technical blog. Technical writing improves code documentation and all your communications with colleagues or customers.

Write on your own domain name. My domain is 13 years old by now, and online platforms have risen and fallen during this time, while my blog is still here and will still be here in another decade, happily receiving visitors that find it via search engines.

And remember, cool URIs don’t change! Don’t let your website die, or its articles vanish. Treat your website as a long term personal project that pays dividends. Choose cheap and long-lasting hosting — you want your website to wait for you while you’re on a break. For this reason alone, websites compiled with a static website generator (e.g. Jekyll, Hugo) are the best for us.

The article A Return to Blogging first appeared on alexn.org.

Finding Focus in Harsh Times

Alexandru Nedelcu — Wed, 30 Mar 2022 14:08:01 +0000

Current events have obliterated my ability to focus, both at work, or in my personal projects. Since the beginning of 2020, I've had bouts of helplessness, anger, or feelings of isolation. When the war started at our borders, I felt brain fog that lasted for days, my heart sinking for all those innocent people that are under fire. On top of the pandemic, we're now dealing with war, and the related anxiety. Yikes!

“Looking for tips on how to keep focus at work while the world is burning” — @alexelcu / March 18, 2022

I should feel privileged. My close family and friends are well and healthy, Covid-19 has spared us for now, financially we’ve been thriving. Life has been good for us, yet this privilege seems fleeting.

I’m probably depressed, because the world around us seems to be burning, and the suffering of others makes me suffer. The symptoms of depression may be subtle for the uninitiated — it’s not just feeling sad, as that comes and goes in short episodes. It’s also boredom, inability to focus, itching for the next news cycle, lack of energy, changes in appetite, loss of joy.

I’m not qualified to give advice, I’m not even over the current events. This is mostly advice given to myself…

Is it time to talk with a mental health professional? I don’t know. If you’re in the same situation as I am, maybe you should. Post-2020, maybe we all need to explore our feelings, and even treat any chemical imbalances if given a prescription. Right now, though, I’m exploring changes in my behavior, as some things are under our control.

The first step in any recovery is admitting the problem, and focusing on why it needs to be fixed.

While my standing is secure, I’m my family’s provider, my family is depending on me, I can’t afford to disappoint them, a persistent thought that’s always with me. Having responsibilities towards others is good, as loving oneself is easier said than done. I always said that my responsibilities as a husband and father have kept me on my toes. In general, helping others helps us thrive, not just because of karma, but because it makes us feel needed, and being needed makes us happier. I am privileged to have a family that depends on me. And maybe I need to do more.

No News Is Good News (archive). TLDR — the author of this article advocates for disconnecting from news, because nothing important will happen without you finding out about it. Political discourse and virtue signaling on social media doesn’t help, righteous fury doesn’t help, it only contributes to the noise.

Donations help, punditry doesn’t. I’ve committed to a recurring donation to UNICEF until this war is over. Put your money where your mouth and mind are. If there’s going to be any physical or mental effort, that effort needs to count.

Worry about things that are in our control. We can’t stop the war, we can’t stop the pandemic, these aren’t in our control. It’s counterproductive to worry about things that we can’t control. I’m trying to embrace stoicism, as you can see 🙂 Like all philosophies, this comes with practice.

“ Things in our control are opinion, pursuit, desire, aversion, and, in a word, whatever are our own actions. Things not in our control are body, property, reputation, command, and, in a word, whatever are not our own actions.” — opening paragraph of Enchiridion, a handbook of Stoicism.

Find your sense of spirituality. Did I ever mention that I’m an Agnostic Theist? I don’t know if there is a God, but I’m not an atheist either. Like many millennials, I have my own spirituality. I certainly believe in a soul, and in the need to take good care of it. Humans crave religion, because science is limited in scope and doesn’t have answers or even questions for our spirit. There are multiple types of atheists in our society, and most believe in a religion that can be worse than Christianity with its ethics. And in theory nihilism can be optimistic and constructive, but I don’t believe it. This may be a good time to search for your god — which is another way of saying to go find meaning in life.

In Romania, we are going through Orthodox Lent, which is a 40 days period of fasting before Easter. This is a period of soul cleansing. And so I’m implementing a great fast that includes abstinence from all animal products, abstinence from absolutely all news, and abstinence from politics. Fasting from news is not going great, my fingers are twitching to get my phone, and see the live stream of recent events, or political commentary. But I’m pulling through, and it’s immensely refreshing to start the day without a doomscroll each morning.

To implement this I’ve blocked (via DNS and browser extensions): Twitter, Facebook, Reddit, and all news outlets I could think of. This ensures that I stop and think whenever I get a craving for news, instead of mindlessly pulling my phone and starting to doomscroll. The only news I get is from low-traffic blogs, mostly on programming, that I follow via my RSS/Atom feed reader.

The best therapy in dark times is probably work and routine. We are designed to be active, to build and to think, so we can try (in no particular order):

Being more present at our $job, helping our colleagues, creating value;

Writing in one’s blog or personal diary;

Contributing to Open Source;

Gardening, other hobbies;

Exercise;

Meditation;

More social interactions, preferably face to face;

To my shame, my social interactions are low, I haven’t been doing exercises during winter, I haven’t been doing any meditation, I rarely write on my blog or in a diary nowadays, I have a long hiatus from Open Source, and I don’t really have hobbies, not ones that involve physical activity anyway. Hopefully I can do something about it.

Being around other people helps. Forget old grudges, especially with friends and family. This is the time to strengthen our relationships, to give and forgive, as there’s strength in numbers. A primary effect of this pandemic has been dehumanization. Companies asking employees to get back to the office can be a good thing, if all this remote work has led to our isolation.

Seasons always change. During summer, it feels like winter will never come back, and during winter, it feels like summer will never come back.

My neighbour's magnolia tree.

Spring and summer always come after winter, and the world tends to change for the better if we wait long enough, so trust that all will be well in the end, doing our part in the meantime. We deserve our spring, though, it’s been a long time coming.

The article Finding Focus in Harsh Times first appeared on alexn.org.

Cleanup Scala/Java project

Alexandru Nedelcu — Sat, 22 Jan 2022 00:00:00 +0000

#!/usr/bin/env bash set -e DIR_TO_CLEAN="$1" if [ -z "$DIR_TO_CLEAN" ]; then echo "ERROR: path was not given!" exit 1 elif ! [[ -d "$DIR_TO_CLEAN" ]]; then echo "ERROR: path given is not a directory!" exit 2 fi cd "$DIR_TO_CLEAN" || exit 1 # Deletes the project's "target" directories, # except for stuff in ./project or in dot dirs (.bloop) find . -name target -type d \ -not $ -path './project*' -o -path './.*' $ \ -print \ -prune \ -exec rm -rf "{}" \; # Deletes all empty directories, except for stuff in dot dirs (.git) find . -type d -empty \ -not $ -path './.*' -o -path './project*' $ \ -print \ -prune \ -exec rm -r "{}" \;

The article Cleanup Scala/Java project first appeared on alexn.org.

VSCode workplace recommended extensions

Alexandru Nedelcu — Sun, 29 Aug 2021 00:00:00 +0000

You can recommend the required VSCode extensions per repository to your fellow programmers. This is what VSCode calls “workspace recommended extensions”.

For example, if your project is a Scala project, maybe you want to recommend the extension for syntax highlighting, Metals, and others.

Add .vscode/extensions.json to your repository:

// .vscode/extensions.json { // See http://go.microsoft.com/fwlink/?LinkId=827846 // for the documentation about the extensions.json format "recommendations": [ // Scala IDE: https://scalameta.org/metals/ "scalameta.metals", // This package is already included by Metals (above), but if people // don't like Metals, then at least recommend syntax highlighting "scala-lang.scala", // Scaladex search for software dependencies "baccata.scaladex-search", ] }

On opening the workspace, VSCode will recognize the file and ask if you want to install the recommended extensions:

VSCode asking you if you want to install the recommended extensions.

The article VSCode workplace recommended extensions first appeared on alexn.org.

Unsafe Lazy Resource.scala

Alexandru Nedelcu — Mon, 02 Aug 2021 00:00:00 +0000

I need an impure way (no IO) to create a resource atomically only once and later be able to know if it was created or not, so I can close this resource safely. 🤔

Jules Ivanic (@guizmaii) ⬇️ August 2, 2021

import scala.util.control.NonFatal /** Builds a "closeable" resource that's initialized on-demand. * * Works like a `lazy val`, except that the logic for closing * the resource only happens in case the resource was initialized. * * NOTE: it's called "unsafe" because it is side-effecting. * See homework. */ final class UnsafeLazyResource[A]( initRef: () => A, closeRef: A => Unit, ) extends AutoCloseable { /** Internal state that works like a FSM: * - `null` is for pre-initialization * - `Some(_)` is an active resource * - `None` is the final state, a closed resource */ @volatile private[this] var ref: Option[A] = null /** * Returns the active resources. Initializes it if necessary. * * @return `Some(resource)` in case the resource is available, * or `None` in case [[close]] was triggered. */ def get(): Option[A] = ref match { case null => // https://en.wikipedia.org/wiki/Double-checked_locking this.synchronized { if (ref == null) { try { ref = Some(initRef()) ref } catch { case NonFatal(e) => ref = None throw e } } else { ref } } case other => other } override def close(): Unit = if (ref ne None) { val res = this.synchronized { val old = ref ref = None old } res match { case null | None => () case Some(a) => closeRef(a) } } }

Example:

import java.io._ def openFile(path: File): UnsafeLazyResource[InputStream] = new UnsafeLazyResource( () => new FileInputStream(path), in => in.close() ) val lazyInput = openFile(new File("/tmp/file")) // .. later try { val in = lazyInput.get().getOrElse( throw new IllegalStateException("File already closed") ) //... } finally { lazyInput.close() }

Homework

Try using an AtomicReference instead of synchronizing a var — not as obvious as you’d think — initialization needs protection, you’ll need an indirection 😉

Try designing a pure API with Cats Effect’s Resource (you might need Ref and Deferred for your internals too)

The article Unsafe Lazy Resource.scala first appeared on alexn.org.

Combining the terminal (iTerm) with the IDE

Alexandru Nedelcu — Sun, 18 Jul 2021 12:23:50 +0000

⌘+Click on a file path triggers my terminal to open that file inside my IDE, and that helps a lot:

Demo of iTerm's semantic history

I’m a terminal-first, IDE-second guy. Whatever I can get quickly done in the terminal, I do in the terminal.

When doing software development, if you compile and test the code in the terminal, going back and forth between the terminal and your IDE has friction. A lot of people just open a terminal from within their IDE, but personally I don’t like that, I think the terminal with the build tool should run separately from the IDE, because you then have the flexibility to restart them independently, or switch IDEs when in the mood.

I’m using iTerm, the macOS terminal emulator. And I do a lot of Scala programming, spending a lot of time in sbt, its build tool, for compiling at testing the project. As IDEs I use IntelliJ IDEA or VS Code + Metals interchangeably.

iTerm can be configured to open files in your editor or IDE of choice via its Semantic History feature. My difficulty is that I want to use two IDEs, not just one. IntelliJ IDEA is very heavy, and I want files opened in it only when it’s running. Another problem is that IntelliJ requires a problematic command parameter if you want to specify the line number. So here’s how to workaround that …

Create a file in ~/bin/iterm-goto and copy/paste this script:

#!/usr/bin/env bash GOTO_FILE="$1" GOTO_LINE="$2" GOTO_CHAR="$3" if ! [ -f "$GOTO_FILE" ]; then echo "ERROR: file path missing or invalid!" >&2 exit 1 fi pgrep -x "idea" > /dev/null IDEA_RUNNING=$? if [[ "$GOTO_FILE" =~ ^.*\.(scala|sbt)$ ]] && [ $IDEA_RUNNING -eq 0 ]; then EDITOR_PATH="$(which idea)" if [ -z "$IDEA_PATH" ]; then EDITOR_PATH="/usr/local/bin/idea" fi if ! [ -z "$GOTO_LINE" ]; then exec "$EDITOR_PATH" --line "$GOTO_LINE" "$GOTO_FILE" else exec "$EDITOR_PATH" "$GOTO_FILE" fi else EDITOR_PATH="$(which code)" if [ -z "$EDITOR_PATH" ]; then EDITOR_PATH="/usr/local/bin/code" fi if ! [ -z "$GOTO_CHAR" ]; then exec "$EDITOR_PATH" --goto "$GOTO_FILE:$GOTO_LINE:$GOTO_CHAR" elif ! [ -z "$GOTO_LINE" ]; then exec "$EDITOR_PATH" --goto "$GOTO_FILE:$GOTO_LINE" else exec "$EDITOR_PATH" "$GOTO_FILE" fi fi

Make sure to make it executable:

chmod +x ~/bin/iterm-goto

Then configure iTerm by:

Open its settings (Cmd+,)

Go into Profiles > Advanced > Semantic History

Select Run command...

Copy/paste: $HOME/bin/iterm-goto \1 \2

It should look like this:

Picture showing iTerm's Semantic History setting.

Enjoy~

The article Combining the terminal (iTerm) with the IDE first appeared on alexn.org.

Tolerance

Alexandru Nedelcu — Tue, 30 Mar 2021 20:12:35 +0000

I just refactored a piece of code. I deleted 6 source code files, and rebuilt the functionality with a bunch of dirty OOP classes shoved in a single file 😱

I used OOP for simplification, and in the process I got rid of type parameters, type classes, and implicit parameters. Plain-old OOP design saved the day, i.e. shove those side effects behind an interface, and pass those objects around, blissfully unaware as to what they do (launching rockets A-OK). It was glorious! I can’t remember the last time it felt this good.

Then it dawned on me that I wrote that code to begin with. This piece already underwent a major refactoring, having introduced extra complexity, without catching the subtleties, or smelling the stench of my own doing. What’s going on?

I started blaming the COVID-19 pandemic, it’s been a year since I’ve been lethargic (aka mild to moderate depression), missing the creative energy that I remember since before the world went mad. But then I started giving myself some credit, because the truth is … as I grow old, I tolerate people more, while tolerating BS in programming less.

Here’s some science¹:

The normal distribution of BS tolerance, it's all downhill after 30, yikes!

This is another way of saying that you should never forget YAGNI², KISS³, the Principle of Least Surprise⁴, or whatever kids call it these days. Just because you can, doesn’t mean you should, with great power comes great responsibility, learn from my mistakes, etc, etc. Most importantly, sometimes DRY⁵ is a bad philosophy, monomorphic code is sometimes the best code, copy/paste that shit.

Babies have no tolerance obviously, when they are hungry or sleepy they won’t take no for an answer. Babies eventually get suckered into accepting life’s compromises. When looking at what helps us fulfil our potential, the ability to eat shit is an obvious confounding factor. ↩

You Ain’t Gonna Need It is what agile developers crave for. ↩

Keep It Simple Stupid was a hip phrase back in my Ruby days, but searching “KISS Ruby” on Google yielded some strange results, so unfortunately I couldn’t get some of that groove back. ↩

OOP and FP developers actually understand different things by the rule of least power 🤷‍♂️ ↩

Don’t Repeat Yourself is not the mantra of Go developers. ↩

The article Tolerance first appeared on alexn.org.

Atom/RSS Feeds are the Best Way to Consume the Web

Alexandru Nedelcu — Mon, 29 Mar 2021 19:58:29 +0000

I stay connected to websites I care about via an RSS/Atom feed reader. It’s better than social media for finding out what’s new because it’s clutter-free. By following RSS/Atom feeds, I discover wonderful gems that otherwise would be lost in the noise.

RSS is not dead

Web syndication is alive and kicking. A vast majority of websites and online services expose RSS/Atom feeds, and you can find workarounds for those that don’t.

When I like individuals on the web, I tend to like pretty much everything that they write. Even when I connect to a fire hose, the RSS/Atom feeds are all content. Even when following pundits, you get long-form writing on websites instead of shallow opinions that must fit in 280 chars. Websites or even YouTube channels, compared with social media, are outlets in which people usually put more thought when expressing themselves.

My RSS/Atom feed reader is the best way to follow:

Hacker News, via hnapp.com

Twitter accounts that are important, via a self-hosted rss-bridge

Library releases, via GitHub’s atom feeds

Podcasts

Reddit (a fire hose, but I can filter by subjects I care about)

Programming communities (Scala, Haskell, etc)

YouTube

News aggregators (RSS readers)

Web hosted:

Newsblur (managed, the best IMO)

FreshRSS (easy to self-host)

Native:

Reeder (macOS, iOS)

FeedMe (Android)

For your blog

Own your audience!

If you’re cool enough to have your own blog, maybe you’ve built your website with a static website generator, such as Jekyll, but please don’t forget to expose an RSS/Atom feed. Those of us using a news aggregator will thank you ❤️ And if you expose an RSS feed, you can then build an automated newsletter from it too 😉

Here’s some cool stuff to get you going:

jekyll-feed

Mailchimp RSS to Email

Enjoy 😎

The article Atom/RSS Feeds are the Best Way to Consume the Web first appeared on alexn.org.

ING Scala Meetup on Scala 3 (live coding session)

Alexandru Nedelcu — Fri, 26 Feb 2021 06:30:07 +0000

We’ve had an ING Scala Meetup last night.

In the first half of the meetup, Ayush Mittal did a live presentation on the newly introduced types of Scala 3 — enums, intersection types, untagged union types.

The types of Scala 3 — Ayush Mittal (open on YouTube.com)

In the second half of the meetup, I did a live coding session on implementing Type Classes in Scala 3, and what this major version of the compiler brings new to the table for helping us to encode type classes — presentation starts from beginner level but then progresses to advanced 😅

Type Classes in Scala 3 — Alexandru Nedelcu (open on YouTube.com)

The article ING Scala Meetup on Scala 3 (live coding session) first appeared on alexn.org.

Implementing a CountDownLatch (async and dirty)

Alexandru Nedelcu — Mon, 22 Feb 2021 13:39:35 +0000

Video link (open on YouTube.com)

Yo dawg, I heard you liked concurrency primitives. Let’s implement our own asynchronous, dirty CountDownLatch.

Just to be clear, what we are implementing here is probably not production ready (please do some testing first, as I didn’t 😅). Also this is not functional programming, and it’s not using Monix or Cats-Effect, or any other libraries that might have helped. This is implemented using just the standard library.

Also, you might want the classic java.util.concurrent.CountDownLatch, which is pretty awesome, and it might be exactly what you need, although this one is useful too.

Why implement one? Because we can 💪

Watch the YouTube video for full explanations

Checkout the following code for digesting it at your own pace

See the usage examples at the end

The Code

import java.util.concurrent.{Executors, RejectedExecutionException} import java.util.concurrent.atomic.AtomicReference import scala.annotation.tailrec import scala.concurrent.duration.{Duration, FiniteDuration} import scala.concurrent.{ExecutionContext, Future, Promise, TimeoutException} final class AsyncCountDownLatch(count: Int)(implicit ec: ExecutionContext) extends AutoCloseable { // Shared mutable state, managed by my favorite // synchronization primitive private[this] val state = new AtomicReference(State(count, Set.empty)) private final case class State( count: Int, tasks: Set[Promise[Unit]] ) /** * INTERNAL — scheduler for managing timeouts, must be destroyed * internally too. Note that this isn't the only possible design, * we could of course take this as a parameter in the class's * constructor, or use a globally shared one 🤷‍ * * One cool alternative is to use Monix's `Scheduler` instead of * `ExecutionContext` (`monix.execution.Scheduler`). */ private[this] val scheduler = Executors.newSingleThreadScheduledExecutor((r: Runnable) => { val th = new Thread(r) th.setDaemon(true) th.setName(s"countdownlatch-${hashCode()}") th }) /** * INTERNAL — Given a `Promise`, install a timeout for it. */ private def installTimeout(p: Promise[Unit], timeout: FiniteDuration): Unit = { // Removing the promise from our internal `state` is necessary to avoid leaks. @tailrec def removePromise(): Unit = state.get() match { case current @ State(_, tasks) => val update = current.copy(tasks = tasks - p) if (!state.compareAndSet(current, update)) removePromise() case null => () // countDown reached zero } // Keeping this as a value in order to avoid duplicate work val ex = new TimeoutException( s"AsyncCountDownLatch.await($timeout)" ) val timeoutTask: Runnable = () => { // Timeout won, get rid of promise from our state removePromise() p.tryFailure(ex) } try { // Finally, installing our timeout, w00t! val cancelToken = scheduler.schedule( timeoutTask, timeout.length, timeout.unit ) // Canceling our timeout task, if primary completes first p.future.onComplete { r => // Avoiding duplicate work if (r.fold(_ != ex, _ => true)) cancelToken.cancel(false) } } catch { case _: RejectedExecutionException => // This exception can happen due to a race condition: // When countDown() reaches zero, the scheduler is being // shut down, however while that happens we may be in // the process of installing a timeout — but this is no // longer necessary, since the promise will be completed; // NOTE — for extra-safety, we might want to check the // promise's state, to ensure that it is complete, and // a happens-before relationship probably exists (TBD) () } } override def close(): Unit = { state.lazySet(null) // GC purposes scheduler.shutdown() } /** * Decrements the count of the latch, releasing all waiting * consumers if the count reaches zero. * * If the current count is already zero, then nothing happens. */ @tailrec def countDown(): Unit = state.get() match { case current @ State(count, tasks) if count > 0 => val update = State(count - 1, tasks) if (!state.compareAndSet(current, update)) countDown() // retry else if (update.count == 0) { // Deferring execution to another thread, as it might // be expensive (TBD if this is a good idea or not) ec.execute(() => { for (r <- tasks) r.trySuccess(()) // Releasing resources close() }) } case _ => () } /** * Causes the consumer to wait until the latch has counted down * to zero, or the specified waiting time elapses. * * If the timeout gets triggered, then the returned `Future` * will complete with a `TimeoutException`. */ @tailrec def await(timeout: Duration): Future[Unit] = state.get() match { case current @ State(count, tasks) if count > 0 => val p = Promise[Unit]() val update = State(count, tasks + p) timeout match { case d: FiniteDuration => installTimeout(p, d) case _ => () } if (!state.compareAndSet(current, update)) await(timeout) // retry else p.future case _ => Future.unit } }

Usage sample of the classic java.util.concurrent.CountDownLatch that we’re trying to mimic:

import java.util.concurrent.CountDownLatch import scala.concurrent.ExecutionContext.Implicits.global val threadCount = 4 val iterations = 10000 var results = Map.empty[Int, Int] for (_ <- 0 until iterations) { val consumersStarted = new CountDownLatch(threadCount) val mainThreadProceed = new CountDownLatch(1) val consumersFinished = new CountDownLatch(threadCount) var sharedState = 0 for (_ <- 0 until threadCount) global.execute(() => { consumersStarted.countDown() mainThreadProceed.await() sharedState += 1 consumersFinished.countDown() }) consumersStarted.await() mainThreadProceed.countDown() consumersFinished.await() results = results.updated(sharedState, results.getOrElse(sharedState, 0) + 1) } println("Done ... ") println(results)

Usage sample of our AsyncCountDownLatch, again, testing faulty concurrent update:

import scala.concurrent.ExecutionContext.Implicits.global import scala.concurrent.{Await, Future, TimeoutException} import scala.concurrent.duration._ val threadCount = 4 val iterations = 100000 var results = Map.empty[Int, Int] for (_ <- 0 until iterations) { val consumersStarted = new AsyncCountDownLatch(threadCount) val mainThreadProceed = new AsyncCountDownLatch(1) val consumersFinished = new AsyncCountDownLatch(threadCount) var sharedState = 0 for (_ <- 0 until threadCount) { for { _ <- Future.unit _ = consumersStarted.countDown() _ <- mainThreadProceed.await(Duration.Inf) } yield { sharedState += 1 consumersFinished.countDown() } } val await = for { _ <- consumersStarted.await(Duration.Inf) _ = mainThreadProceed.countDown() _ <- consumersFinished.await(1.millis) } yield { results = results.updated(sharedState, results.getOrElse(sharedState, 0) + 1) } Await.result(await, Duration.Inf) } println("Done ... ") println(results)

The article Implementing a CountDownLatch (async and dirty) first appeared on alexn.org.

Matomo (Analytics) Hosting via Docker

Alexandru Nedelcu — Mon, 22 Feb 2021 00:00:00 +0000

For self-hosting Matomo, a FOSS alternative to Google Analytics — via docker-compose:

version: '3.3' services: db: container_name: mariadb image: mariadb:10.5 ports: - "3306:3306" labels: - "com.centurylinklabs.watchtower.enable=true" command: --max-allowed-packet=64MB restart: unless-stopped volumes: - db:/var/lib/mysql env_file: - ./envs/mysql.env networks: - main redis: container_name: redis image: redis:6-alpine restart: unless-stopped command: redis-server --appendonly yes labels: - "com.centurylinklabs.watchtower.enable=true" volumes: - redis:/data networks: - main sysctls: # https://github.com/docker-library/redis/issues/191#issuecomment-528693994 - net.core.somaxconn=511 matomo: container_name: matomo image: matomo:4-fpm-alpine links: - db - redis restart: unless-stopped expose: - "9000" ports: - "9000:9000" labels: - "com.centurylinklabs.watchtower.enable=true" volumes: - /var/www/matomo:/var/www/html networks: - main environment: - MATOMO_DATABASE_HOST=db env_file: - ./envs/mysql.env - ./envs/matomo.env user: "$WWW_UID:$WWW_GID" networks: main: external: name: main volumes: db: redis:

The article Matomo (Analytics) Hosting via Docker first appeared on alexn.org.

Scala's List has a Secret

Alexandru Nedelcu — Fri, 12 Feb 2021 11:32:33 +0000

OOP couples the “data” with the methods operating on it, and that’s considered bad in FP circles, because supposedly data outlives the functions operating on it. Also in static FP circles, dumb data structures are reusable, so it’s a good idea to make them generic, and add restrictions on the functions themselves.

Few data structures could be simpler than an immutable List definition, right? At least as far as recursive data structures go 🙂 For the standard List you’d expect the following:

sealed abstract class List[+A] final case class :: [+A](head: A, tail: List[A]) extends List[A] case object Nil extends List[Nothing]

Oh boy, I’ve got news for you — this is the actual definition from Scala’s standard library:

sealed abstract class List[+A] final case class :: [+A]( head: A, // mutable var 😱 private[scala] var next: List[A @uncheckedVariance]) extends List[A] { // 😱 memory barrier releaseFence() } case object Nil extends List[Nothing]

Yikes, that private next value is a var. They added it as a var such that ListBuffer can build a list more efficiently, because an immutable List is in essence a Stack, so to build a new list from an existing one, you’d need to do an O(n) reversal at the end.

With the pure definition, we’d build List values like this:

def map[A, B](self: List[A])(f: A => B): List[B] = { var buffer = List.empty[B] for (elem <- self) { buffer = f(elem) :: buffer } // Extra O(n) tax if that list would be pure, no way around it // (legend has it that this is a computer science issue related to stacks) buffer.reverse }

But with ListBuffer, due to making use of that var:

def map[A, B](self: List[A])(f: A => B): List[B] = { val buffer = ListBuffer.empty[B] for (elem <- self) { buffer += f(elem) } // O(1), no inefficiency buffer.toList }

Contrary to popular opinion, this means List does not benefit from final (val in Scala) visibility guarantees by the Java Memory Model. So it might have visibility issues in a multi-threaded context (e.g. you might end up with a tail being null when it shouldn’t be). Which is probably why we see this in both the class constructor and in ListBuffer#toList:

override def toList: List[A] = { aliased = nonEmpty // We've accumulated a number of mutations to `List.tail` by this stage. // Make sure they are visible to threads that the client of this ListBuffer might be about // to share this List with. releaseFence() first }

Yikes, they are adding manual memory barriers everywhere 😲 I guess it beats an added O(n) penalty when building new lists. But this goes to show the necessity of coupling data structures with the methods operating on them.

FP developers don’t care about resources, because of the expectation that resources should be handled by the runtime, but sometimes that isn’t possible or optimal — even dumb data structures are resources and sometimes need special resource management, for efficiency reasons. In which case coupling the data with the methods operating on it is healthy 😉

I don't like manual memory barriers BTW. They are expensive and a design based on proper acquisition and release (reads and writes in volatiles) would be better, as it lets the JVM do its magic. Don't copy their design here without knowing what you're doing.

The article Scala's List has a Secret first appeared on alexn.org.

Tail Recursive Functions (in Scala)

Alexandru Nedelcu — Tue, 26 Jan 2021 19:49:35 +0000

Video link (open on YouTube.com)

Turning imperative algorithms to tail-recursive functions isn’t necessarily obvious. In this article (and video) I’m showing you the trick you need, and in doing so, we’ll discover the Zen of Functional Programming.

Choose between watching the video on YouTube (linked above), or reading the article (below), or both.

The Trick

Let’s start with a simple function that calculates the length of a list:

def len(l: List[_]): Int = l match { case Nil => 0 case _ :: tail => len(tail) + 1 }

It’s a recursive function with a definition that is mathematically correct. However, if we try to test it, this will fail with a StackOverflowError:

len(List.fill(100000)(1))

The problem is that the input list is too big. And because the VM still has work to do after that recursive call, needing to do a + 1, the call isn’t in “tail position”, so the call-stack must be used. A StackOverflowError is a memory error, and in this case it’s a correctness issue, because the function will fail on reasonable input.

First let’s describe it as a dirty while loop instead:

def len(l: List[_]): Int = { var count = 0 var cursor = l while (cursor != Nil) { count += 1 cursor = cursor.tail } count }

THE TRICK for turning such functions into tail-recursions is to turn those variables, holding state, into function parameters.

def len(l: List[_]): Int = { // Using an inner function to encapsulate this implementation @tailrec def loop(cursor: List[_], count: Int): Int = cursor match { // Our end condition, copied after that while case Nil => count case _ :: tail => // Copying the same logic from that while statement loop(cursor = tail, count = count + 1) } // Go, go, go loop(l, 0) }

Now this version is fine. Note the use of the @tailrec annotation — all this annotation does is to make the compiler throw an error in case the function is not actually tail-recursive. That’s because that call is error-prone, and it needs repeating, this is an issue of correctness.

Let’s do a more complex example to really internalize this. Let’s calculate the N-th number in the Fibonacci sequence — here’s the memory unsafe recursive version:

def fib(n: Int): BigInt = if (n <= 0) 0 else if (n == 1) 1 else fib(n - 1) + fib(n - 2) fib(0) // 0 fib(1) // 1 fib(2) // 1 fib(3) // 2 fib(4) // 3 fib(5) // 5 fib(100000) // StackOverflowError (also, really slow)

First turn this into a dirty while loop:

def fib(n: Int): BigInt = { // Kids, don't do this at home 😅 if (n <= 0) return 0 // Going from 0 to n, instead of vice-versa var a: BigInt = 0 // instead of fib(n - 2) var b: BigInt = 1 // instead of fib(n - 1) var i = n while (i > 1) { val tmp = a a = b b = tmp + b i -= 1 } b }

Then turn its 3 variables into function parameters:

def fib(n: Int): BigInt = { @tailrec def loop(a: BigInt, b: BigInt, i: Int): BigInt = // first condition if (i <= 0) 0 // end of while loop else if (i == 1) b // logic inside while loop statement else loop(a = b, b = a + b, i = i - 1) loop(0, 1, n) }

(Actual) Recursion

Tail-recursions are just loops. But some algorithms are actually recursive, and can’t be described via a while loop that uses constant memory. What makes an algorithm actually recursive is usage of a stack. In imperative programming, for low-level implementations, that’s how you can tell if recursion is required … does it use a manually managed stack or not?

But even in such cases we can use a while loop, or a @tailrec function. Doing so has some advantages. Let’s start with a Tree data-structure:

sealed trait Tree[+A] case class Node[+A](value: A, left: Tree[A], right: Tree[A]) extends Tree[A] case object Empty extends Tree[Nothing]

Defining a fold, which we could use to sum-up all values for example, will be challenging:

def foldTree[A, R](tree: Tree[A], seed: R)(f: (R, A) => R): R = tree match { case Empty => seed case Node(value, left, right) => // Recursive call for the left child val leftR = foldTree(left, f(seed, value))(f) // Recursive call for the right child foldTree(right, leftR)(f) }

This is the simple version. And it should be clear that the size of the call-stack will be directly proportional to the height of the tree. And turning it into a @tailrec version means we need to manually manage a stack:

def foldTree[A, R](tree: Tree[A], seed: R)(f: (R, A) => R): R = { @tailrec def loop(stack: List[Tree[A]], state: R): R = stack match { // End condition, nothing left to do case Nil => state // Ignore empty elements case Empty :: tail => loop(tail, state) // Step in our loop case Node(value, left, right) :: tail => // Adds left and right nodes to stack, evolves the state loop(left :: right :: tail, f(state, value)) } // Go, go, go! loop(List(tree), seed) }

If you want to internalize this notion — recursion == usage of a stack — a great exercise is the backtracking algorithm. Implement it with recursive functions, or with dirty loops and a manually managed stack, and compare. The plot thickens for backtracking solutions using 2 stacks 🙂

Does this manually managed stack buy us anything?

Well yes, if you need such recursive algorithms, such a stack can take up your whole heap memory, which means it can handle a bigger input. But note that with the right input, your process can still blow up, this time with an out-of-memory error (OOM).

NOTE — in real life, shining examples of algorithms using manually managed stacks are Cats-Effect’s IO and Monix’s Task, since they literally replace the JVM’s call-stack 😄

Zen of Functional Programming?

In FP, you turn variables into (immutable) function parameters. And state gets evolved via function calls 💡

That’s it, that’s all there is to FP (plus the design patterns, and the pain of dealing with I/O 🙂).

Enjoy!

The article Tail Recursive Functions (in Scala) first appeared on alexn.org.

Snippet: Tagless Final vs OOP

Alexandru Nedelcu — Wed, 20 Jan 2021 00:00:00 +0000

These may as well be interview questions for Scala developers:

Which signature do you prefer?

Just looking at the signature, what can potentially be wrong with the “tagless final” version?

Also discuss the 2 type-class approaches, what are the potential design problems there?

// ----------------- // Tagless final def registerUser[F[_]: UserDB: EmailService: Monad](user: User): F[Unit] // ----------------- // ReaderT (Kleisli) & OOP def registerUser[F[_]: Monad]( user: User): Kleisli[F, (UserDB[F], EmailService[F]), Unit] // ----------------- // Plain function parameters & OOP def registerUser[F[_]: Monad]( db: UserDB[F], es: EmailService[F], user: User): F[Unit] // ----------------- // OOP class final class RegistrationService[F[_]: Monad]( db: UserDB[F], es: EmailService[F]) { def registerUser(user: User): F[Unit] } // ----------------- // OOP interface trait RegistrationService[F[_]] { def registerUser(user: User): F[Unit] } object RegistrationService { def apply[F[_]: Monad]( db: UserDB[F], es: EmailService[F]): RegistrationService[F] = ??? } // ----------------- // Type Class (1) — two type params trait RegistrationService[F[_], Env] { def registerUser(env: Env, user: User): F[Unit] } object RegistrationService { implicit def instance[F[_]: Monad] : RegistrationService[F, (UserDB[F], EmailService[F])] = ??? } // ----------------- // Type Class (2) — single type param trait RegistrationService[Env[_[_]]] { def registerUser[F[_]: Monad](env: Env[F], user: User): F[Unit] } object RegistrationService { type Env[F[_]] = (UserDB[F], EmailService[F]) implicit val instance: RegistrationService[Env] = ??? }

The article Snippet: Tagless Final vs OOP first appeared on alexn.org.

Snippet: Remove blank lines from text

Alexandru Nedelcu — Thu, 14 Jan 2021 00:00:00 +0000

text.replaceAll("(?m)(^\\s*$\\r?\\n)+", "")

Or via Pattern objects:

import java.util.regex.Pattern Pattern .compile("(^\\s*$\\r?\\n)+", Pattern.MULTILINE) .matcher(text) .replaceAll("")

The article Snippet: Remove blank lines from text first appeared on alexn.org.

Execute shell commands in F#

Alexandru Nedelcu — Sun, 06 Dec 2020 00:00:00 +0000

#!/usr/bin/env -S dotnet fsi open System open System.Diagnostics open System.Threading.Tasks type CommandResult = { ExitCode: int StandardOutput: string StandardError: string } let executeCommand executable args = async { let! ct = Async.CancellationToken let startInfo = ProcessStartInfo() startInfo.FileName <- executable startInfo.RedirectStandardOutput <- true startInfo.RedirectStandardError <- true startInfo.UseShellExecute <- false startInfo.CreateNoWindow <- true for a in args do startInfo.ArgumentList.Add(a) use p = new Process() p.StartInfo <- startInfo p.Start() |> ignore let outTask = Task.WhenAll([| p.StandardOutput.ReadToEndAsync(ct); p.StandardError.ReadToEndAsync(ct) |]) do! p.WaitForExitAsync(ct) |> Async.AwaitTask let! out = outTask |> Async.AwaitTask return { ExitCode = p.ExitCode StandardOutput = out.[0] StandardError = out.[1] } } let executeShellCommand command = executeCommand "/usr/bin/env" [ "-S"; "bash"; "-c"; command ] // Invocation sample let r = executeShellCommand "ls -alh" |> Async.RunSynchronously if r.ExitCode = 0 then printfn "%s" r.StandardOutput else eprintfn "%s" r.StandardError Environment.Exit(r.ExitCode)

If you create the file test.fsi with the code above, under Linux you can make it executable:

# Make it executable chmod +x ./test.fsi # Execute it ./test.fsi

Also, see: Execute Shell Commands in Java/Scala/Kotlin.

The article Execute shell commands in F# first appeared on alexn.org.

Using ScalaTest for Effects

Alexandru Nedelcu — Sun, 22 Nov 2020 00:00:00 +0000

ScalaTest helpers for testing effects (e.g. cats.effect.IO, monix.eval.Task). This is similar to PureApp.

Requirement: EffectRuntime (snippet)

Usage:

object Fns { def fireMissiles[F[_]: Sync]: F[Int] = Sync[F].delay { println("Firing missiles...") Random.nextInt(100) + 1 } } class MyTestSuite extends EffectTestSuite.ForIO { testEffect("fire missiles") { for { count <- Fns.fireMissiles } yield { assert(count > 0) } } }

Implementation:

import cats.effect._ import org.scalactic.source import org.scalatest.compatible.Assertion import org.scalatest.funsuite.AsyncFunSuiteLike import org.scalatest.{ BeforeAndAfterAll, Tag } import scala.concurrent.Promise import scala.concurrent.duration._ trait EffectTestSuite[F[_]] extends AsyncFunSuiteLike { protected def effectTimeout: FiniteDuration = 30.seconds protected def testEffect(testName: String, testTags: Tag*)( testFun: => F[Assertion] )(implicit F: ConcurrentEffect[F], timer: Timer[F], pos: source.Position): Unit = { test(testName, testTags: _*) { val task = Concurrent.timeout(testFun, effectTimeout) val p = Promise[Assertion]() F.runAsync(task)(r => F.delay { p.complete(r.toTry); () } p.future } } } object EffectTestSuite { /** For working with `cats.effect.IO` */ trait ForIO extends EffectTestSuite[IO] with BeforeAndAfterAll { protected def runtimeFactory: Resource[SyncIO, EffectRuntime[IO]] = { // This should be a concrete implementation ??? } protected final lazy val (runtimeRef, runtimeCancel) = { runtimeFactory .allocated .unsafeRunSync() } override protected def beforeAll(): Unit = { super.beforeAll() // forces initialization runtimeRef; () } override protected def afterAll(): Unit = { super.afterAll() runtimeCancel.unsafeRunSync() } protected implicit lazy val contextShift: EffectRuntime[IO] = runtimeRef protected implicit lazy val F: ConcurrentEffect[IO] = IO.ioConcurrentEffect(contextShift) protected implicit lazy val timer: Timer[IO] = runtimeRef.timer(F) } }

The article Using ScalaTest for Effects first appeared on alexn.org.

Best practice for natural Ordering

Alexandru Nedelcu — Tue, 17 Nov 2020 09:02:13 +0000

Scala has scala.math.Ordering, Java has java.lang.Comparable. These are interfaces used for defining a natural order, which can then be used to sort lists of elements, or in data structures implemented via binary search trees, such as SortedSet.

This is what it looks like:

trait Ordering[A] { def compare(x: A, y: A): Int }

That function is supposed to define a “total order”, by returning a number that’s:

strictly negative if x < y

strictly positive if x > y

0 in case x == y

What’s often overlooked is that — ordering has to be consistent with equals, as a law, or in other words:

compare(x, y) == 0 <-> x == y

Example:

import scala.math.Ordering final case class Contact( lastName: String, firstName: String, phoneNumber: String ) object Contact { // WRONG — never do this! implicit val ordering: Ordering[Contact] = (x: Contact, y: Contact) => x.lastName.compareTo(y.lastName) }

This might seem reasonable if you were building some sort of contacts agenda, but isn’t. Some reasons:

data-structures backed by binary-search trees can use just compare when searching for keys

the result of sorting a list should not depend on its initial ordering

Example of what can happen when sorting lists:

val agenda1 = List( Contact("Nedelcu", "Alexandru", "0738293904"), Contact("Nedelcu", "Amelia", "0745029304"), ) agenda1.sorted //=> List(Contact(Nedelcu,Alexandru,0738293904), Contact(Nedelcu,Amelia,0745029304)) val agenda2 = List( Contact("Nedelcu", "Amelia", "0745029304"), Contact("Nedelcu", "Alexandru", "0738293904"), ) agenda2.sorted //=> List(Contact(Nedelcu,Amelia,0745029304), Contact(Nedelcu,Alexandru,0738293904)) agenda1.sorted == agenda2.sorted //=> false

Example of what happens when using SortedSet:

import scala.collection.immutable.SortedSet val set1 = SortedSet(agenda1:_*) //=> TreeSet(Contact(Nedelcu,Alexandru,0738293904)) val set2 = SortedSet(agenda2:_*) //=> TreeSet(Contact(Nedelcu,Amelia,0745029304))

SortedSet, being a Set implementation, is not duplicating items, and as you can see here, it basically eliminates all contacts with the same lastName, except for the first one it saw.

If you’ve ever defined your ordering like this, don’t feel bad, as it happens to many of us. I was bitten by this just last week, in spite of knowing what I was doing … I defined a private ordering and figured that it won’t get used improperly. Except that after a refactoring, by yours truly, it did end up being used improperly. Which is why definitions have to always be correct, as their correctness has to survive refactorings, even when it’s just you operating on that codebase.

The correct definition for our Ordering would be:

object Contact { implicit val ordering: Ordering[Contact] = (x: Contact, y: Contact) => { x.lastName.compareTo(y.lastName) <||> x.firstName.compareTo(y.firstName) <||> x.phoneNumber.compareTo(y.phoneNumber) } // Just to help us out, this time private implicit class IntExtensions(val num: Int) extends AnyVal { def `<||>`(other: => Int): Int = if (num == 0) other else num } }

Don’t take shortcuts, don’t do anything less than this, even if you think that you currently don’t need it. Because one of your colleagues, or even your future self, might reuse this definition without knowing that it’s broken.

The article Best practice for natural Ordering first appeared on alexn.org.

Managing Database Migrations in Scala

Alexandru Nedelcu — Sun, 15 Nov 2020 16:17:25 +0000

The database schema should be described as code, in your repository. And you should be able to semi-automatically update your database schema on new deployments.

A very popular Java library for handling migrations is Flyway. We’ll combine that with Typesafe Config (aka HOCON) for configuration, along with PureConfig for parsing it. And Cats Effect for describing our effects, because we love FP, right? 😎

This is a complete solution, that’s customizable and easy to implement (couple of lines of code), that can be used in multi-project builds, and that doesn’t tie you to a particular framework, or a particular build tool, or an enterprise solution solving problems that you don’t have.

1. Setup MySQL on localhost

We’re going to use MariaDB (MySQL) as our database, but this works with any relational database.

To start a MariaDB instance on your localhost, you could use Docker:

# NOTE: --rm means the container gets deleted after shutdown; # if you want to keep the container around, then remove it docker run --rm \ --name mariadb \ -e MYSQL_ROOT_PASSWORD=pass \ -p 3306:3306 \ mariadb:10.4

Then to create our initial database:

docker exec -it mariadb mysql -uroot -ppass \ -e "CREATE DATABASE MyAwesomeApp"

WARN: as a best practice, the CREATE DATABASE statement should not be in your database migrations. The database, from production at least, must be created manually. Do not create or refer to specific databases in your migrations!

2. Initial project setup

Start a new project with:

sbt new scala/scala-seed.g8

Then add these library dependencies to your build.sbt:

lazy val root = (project in file(".")) .settings( //... add these ... libraryDependencies ++= Seq( "org.typelevel" %% "cats-effect" % "2.2.0", "com.github.pureconfig" %% "pureconfig" % "0.14.0", "org.flywaydb" % "flyway-core" % "7.2.0", "mysql" % "mysql-connector-java" % "8.0.22", "com.typesafe.scala-logging" %% "scala-logging" % "3.9.2", ) )

3. Configuration

The app’s configuration is best described as concrete types in your project. Add something like this for the JDBC connection parameters, in src/main/resources/application.conf:

example.jdbc { driver = "com.mysql.cj.jdbc.Driver" host = "127.0.0.1" host = ${?DB_HOST} port = "3306" port = ${?DB_PORT} dbName = "MyAwesomeApp" dbName = ${?DB_NAME} url = "jdbc:mysql://"${example.jdbc.host}":"${example.jdbc.port}"/"${example.jdbc.dbName} url = ${?DB_CONNECTION_URL} user = "root" user = ${?DB_USER} password = "pass" password = ${?DB_PASS} migrations-table = "FlywaySchemaHistory" migrations-locations = [ "classpath:example/jdbc" ] }

Which is modelled by a type like this on the Scala side:

package example package jdbc import cats.effect.Sync import com.typesafe.config.{ Config, ConfigFactory } import pureconfig.{ ConfigConvert, ConfigSource } import pureconfig.generic.semiauto._ final case class JdbcDatabaseConfig( url: String, driver: String, user: Option[String], password: Option[String], migrationsTable: String, migrationsLocations: List[String] ) object JdbcDatabaseConfig { def loadFromGlobal[F[_]: Sync](configNamespace: String): F[JdbcDatabaseConfig] = Sync[F].suspend { val config = ConfigFactory.load() load(config.getConfig(configNamespace)) } def load[F[_]: Sync](config: Config): F[JdbcDatabaseConfig] = Sync[F].delay { ConfigSource.fromConfig(config).loadOrThrow } // Integration with PureConfig implicit val configConvert: ConfigConvert[JdbcDatabaseConfig] = deriveConvert }

NOTES:

migrations-table specifies the name of the table auto-created by Flyway, used to keep track of what migrations have been executed thus far;

migrations-locations is a list of locations for our migrations, most often a list of packages available on the classpath;

F[_] here is a datatype for managing I/O, such as cats.effect.IO or monix.eval.Task, but there’s no reason to not make these functions generic, such that you can later use whatever you want; see the documentation for Cats Effect’s Sync;

I chose this design because we may have a multi-project build, and we may want to run database migrations for all projects in one go.

4. Flyway library integration

We can now make use of Flyway, coupled with our config above:

package example package jdbc import cats.effect.Sync import cats.implicits._ import com.typesafe.scalalogging.LazyLogging import org.flywaydb.core.api.configuration.FluentConfiguration import org.flywaydb.core.api.Location import org.flywaydb.core.Flyway import scala.jdk.CollectionConverters._ object DBMigrations extends LazyLogging { def migrate[F[_]: Sync](config: JdbcDatabaseConfig): F[Int] = Sync[F].delay { logger.info( "Running migrations from locations: " + config.migrationsLocations.mkString(", ") ) val count = unsafeMigrate(config) logger.info(s"Executed $count migrations") count } private def unsafeMigrate(config: JdbcDatabaseConfig): Int = { val m: FluentConfiguration = Flyway.configure .dataSource( config.url, config.user.orNull, config.password.orNull ) .group(true) .outOfOrder(false) .table(config.migrationsTable) .locations( config.migrationsLocations .map(new Location(_)) .toList: _* ) .baselineOnMigrate(true) logValidationErrorsIfAny(m) m.load().migrate().migrationsExecuted } private def logValidationErrorsIfAny(m: FluentConfiguration): Unit = { val validated = m.ignorePendingMigrations(true) .load() .validateWithResult() if (!validated.validationSuccessful) for (error <- validated.invalidMigrations.asScala) logger.warn(s""" |Failed validation: | - version: ${error.version} | - path: ${error.filepath} | - description: ${error.description} | - errorCode: ${error.errorDetails.errorCode} | - errorMessage: ${error.errorDetails.errorMessage} """.stripMargin.strip) } }

5. Create your first DB migration

Add this in src/main/resources/example/jdbc/V0010__CreateMyFirstTable.sql:

CREATE TABLE MyFirstTable( id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY, uKey VARCHAR(200) NOT NULL, uValue TEXT NOT NULL, createdAt TIMESTAMP NOT NULL, updatedAt TIMESTAMP NOT NULL ); CREATE UNIQUE INDEX MyFirstTable__Key ON MyFirstTable(uKey);

6. Add sbt command

Define a command, such that we can apply all migrations like:

// TBD, does not work yet sbt run-db-migrations

First, define an “app” that can run all migrations:

package example package jdbc import cats.implicits._ import cats.effect.IOApp import cats.effect.{ExitCode, IO} import com.typesafe.scalalogging.LazyLogging object DBMigrationsCommand extends IOApp with LazyLogging { /** * Lists all JDBC data-sources, defined in `application.conf` */ val dbConfigNamespaces = List( "example.jdbc" ) def run(args: List[String]): IO[ExitCode] = { val migrate = dbConfigNamespaces.traverse_ { namespace => for { _ <- IO(logger.info(s"Migrating database configuration: $namespace")) cfg <- JdbcDatabaseConfig.loadFromGlobal[IO](namespace) _ <- DBMigrations.migrate[IO](cfg) } yield () } migrate.as(ExitCode.Success) } }

NOTE: you can unify multiple migrations locations from multiple projects, in case you have a multi-project build. This is the reason for why dbConfigNamespaces is a List 😉

Then add this in build.sbt:

lazy val runMigrate = taskKey[Unit]("Migrates the database schema.") addCommandAlias("run-db-migrations", "runMigrate") lazy val root = (project in file(".")) .settings( //... fullRunTask(runMigrate, Compile, "example.jdbc.DBMigrationsCommand"), fork in runMigrate := true, )

Now test that it works:

sbt run-db-migrations

Then see what tables were created. Execute this in your shell:

docker exec -it mariadb mysql -uroot -ppass MyAwesomeApp \ -e "SHOW TABLES"

You should get this output:

+------------------------+ | Tables_in_MyAwesomeApp | +------------------------+ | FlywaySchemaHistory | | MyFirstTable | +------------------------+

7. Pro-tip: Unit-test migrations with HSQLDB

You can run these migrations in your tests too. And you can use an in-memory database, such as HSQLDB, if your SQL does not use any specific MySQL features.

Add this to your build.sbt:

lazy val root = (project in file(".")) .settings( //... libraryDependencies ++= Seq( //... // Needed for testing "org.hsqldb" % "hsqldb" % "2.5.1" % Test, scalaTest % Test, ), // Recommended fork in Test := true, )

Add src/test/resources/application.test.conf:

include "application.conf" // Overrides MySQL connection with in-memory HSQLDB example.jdbc { url = "jdbc:hsqldb:mem:MyAwesomeApp;sql.syntax_mys=true" driver = "org.hsqldb.jdbc.JDBCDriver" }

Then add your test (in src/test/scala/example/jdbc/DBTestSuite.scala):

package example package jdbc import cats.effect._ import org.scalatest.funsuite.AnyFunSuite import com.typesafe.config.ConfigFactory class DBTestSuite extends AnyFunSuite { test("database migrations") { val conf = ConfigFactory .load(getClass.getClassLoader, "application.test.conf") .resolve() val jdbcConf = JdbcDatabaseConfig .load[SyncIO](conf.getConfig("example.jdbc")) .unsafeRunSync() val m = DBMigrations.migrate[SyncIO](jdbcConf).unsafeRunSync() assert(m == 1) } }

Run with:

sbt test

Enjoy~

The article Managing Database Migrations in Scala first appeared on alexn.org.

List static blog tags in folder (Jekyll, etc)

Alexandru Nedelcu — Fri, 13 Nov 2020 00:00:00 +0000

I maintain a Jekyll blog, with tagged articles (tags specified in the Markdown front-matter), but maintaining tags isn’t easy, because I often forget what tags I used. E.g. is it Functional, or FP? I can never remember.

Here’s a simple script that traverses all the files in a directory, parsing their YAML front-matter, gathers all the tags, then displays them sorted by popularity:

#!/usr/bin/env ruby require 'optparse' require "json" require "yaml" options = {} OptionParser.new do |opt| opt.on('--folder PATH') { |o| options[:path] = File.expand_path(o) } end.parse! if !options[:path] && ARGV[0] && File.directory?(ARGV[0]) options[:path] = File.expand_path(ARGV[0]) end raise OptionParser::MissingArgument.new("--folder") if options[:path].nil? filters = [ File.join(options[:path], "*.md"), File.join(options[:path], "*.markdown"), File.join(options[:path], "*.html"), File.join(options[:path], "*.htm"), ] all_tags = {} filters.each do |filter| Dir[filter].each do |path| contents = File.read(path) if contents =~ /\A---\s*\r?\n(.*?)^---/m yaml_str = ($1).strip yaml = YAML.load(yaml_str) tags = yaml['tags'] || [] tags.each do |tag| all_tags[tag] ||= 0 all_tags[tag] += 1 end end end end all_tags.to_a.sort_by{|x| -x[1]}.each do |tag| printf("%20s %3d\n", tag[0], tag[1]) end

E.g. invoking it on my current _snippets folder reveals:

$ list-tags _snippets Scala 15 Cats Effect 7 Akka 4 Reactive Streams 4 Async 3 Python 2 sbt 2 TypeScript 2 CLI 2 FP 2 Bash 1 Ruby 1 Jekyll 1 Testing 1 Monix 1 Fun 1 Haskell 1 JVM 1 JavaScript 1

The article List static blog tags in folder (Jekyll, etc) first appeared on alexn.org.

I like Option.get

Alexandru Nedelcu — Thu, 12 Nov 2020 18:02:46 +0000

In strong, static, expressive FP languages, such as Scala, or Haskell, there’s the ongoing drive to “capture invariants in the type system” and “to make illegal states unrepresentable”. For a nice introduction, see Parse, don’t validate by Alexis King.

Option.get or List.head get such bad reps because these functions aren’t total. To wit:

val value: Option[String] = None value.get //=> java.util.NoSuchElementException: None.get // at scala.None$.get(Option.scala:627)

This is indeed bad, because it fails at runtime. And if it fails at runtime, this means it can fail in production, in spite of all our unit tests and fancy CI setup.

In fairness, even with the current status quo, option.get is still better than usage of null, because developers are still aware that the value might be missing (by seeing the type, then having to call .get, at the very least), and even in absence of such mindfulness, at least the exception is clearer, as NullPointerException is often thrown due to faulty internals, and JVM initialization timing issues. At least you know it’s your own fault 🙂

If get wasn’t available on Option, then you’d be forced to do this:

value match { case Some(v) => v case None => "unknown" } // Or this ... value.getOrElse("unknown")

And if you’d miss a case, you’d get a warning (or an error, if you work with fatal warnings):

value match { case Some(v) => v } //=> warning: match may not be exhaustive. // It would fail on the following input: None

The compiler can thus force you to deal with None explicitly.

All of that aside however, in the following piece of code, what’s wrong isn’t the presence of Option.get, but rather the compiler’s inability of seeing the if expression:

// This is correct and will never trigger runtime exceptions if (option.nonEmpty) option.get // Like a boss 😎 else "unknown"

In other words, I’ll blame Scala and Haskell, and not the availability of Option.get. I learned to expect more from my tools. It’s not me, it’s you, Scala.

We could say that in absence of compiler features to cope with this, then .get shouldn’t exist. However, programming languages are general purpose, and often get used in contexts in which strong static guarantees are not only useless, but get in the way. I still build my scripts in Ruby, because the static languages that I love are really bad for scripting. I’d like to disable some static guarantees, whenever brevity is important, and not correctness. E.g. for my own throwaway scripts I couldn’t care less that Option.get throws exceptions.

TypeScript has untagged unions, and under --strict this throws an error:

const sample: number | null = null sample + 10 //=> error TS2531: Object is possibly 'null'. // This works if (sample !== null) { sample + 10 }

Option being boxed (tagged) provides us with benefits, like the ability to express Option> (without auto-flattening), or the ability to define monadic operations for it. Here’s one way to express Option in TypeScript:

type None = { nonEmpty: false } type Some = { nonEmpty: true value: A } type Option = Some | None // ---------------------------------- const sample: Option = { nonEmpty: false } // None sample.value //=> error TS2339: Property 'value' does not exist on type 'Option'. // Property 'value' does not exist on type 'None'. // Compiles just fine if (sample.nonEmpty) { console.log( sample.value ) }

This is called Flow-sensitive typing. And minus some limitations and gotchas, it works just fine.

I hope Scala will evolve to do it too, because TypeScript, and Kotlin can already do this 🙂 and it would be a shame for Scala to not evolve such abilities, to go along with its brand new untagged union types.

The article I like Option.get first appeared on alexn.org.

Organize and Index Your Screenshots (OCR) on macOS

Alexandru Nedelcu — Wed, 11 Nov 2020 12:43:46 +0000

I maintain a growing Screenshots folder. Screenshots contain text, text that should be searchable, as finding a screenshot later is the whole point of creating it.

My folder is stored in Dropbox, and unfortunately they are not indexing images on the “Plus” plan, at the moment of writing. And OneDrive currently has this functionality suspended for personal accounts, due to some technical issues they are having. Having to depend on a cloud service for searching your screenshots sucks, and I don’t want the lock-in of online services.

Here’s how to organize and index your screenshots locally, using open source stuff, and save some money …

1. Install: Tesseract OCR Engine

Tesseract OCR is a neat OSS project, available in Homebrew:

brew install tesseract

Given a PNG image file, you can convert it into a properly annotated PDF with:

tesseract /input/path/to/file.png /output/path/to/file -l eng pdf

This will process /input/path/to/file.png and create a properly annotated /output/path/to/file.pdf from it, that can then be indexed by macOS’s Spotlight, or Dropbox.

2. Create folder structure

In Dropbox I have the following folders:

Screenshots

Processing

Raw

OCR

New screenshots go in Processing. Indexable PDF files generated by Tesseract go in OCR. And after processing, the raw files are moved in Raw, possibly with some renaming, which helps in sorting the files by their timestamp.

3. Change the default screenshots folder

Instruct your macOS to save screenshots directly in your Processing folder:

defaults write com.apple.screencapture location ~/Dropbox/Screenshots/Processing

4. Add the synchronization script

I’ve built my script with Ruby. In case you don’t have Ruby installed:

brew install ruby

Create the following script somewhere on your PATH and make it executable. I prefer ~/bin/screenshots-sync:

#!/usr/bin/env ruby require 'optparse' require 'pp' USAGE = < -o -r [-d] ENDUSAGE options = {} OptionParser.new do |opts| opts.banner = USAGE opts.on("-i", "--input-dir INPUT_DIR", "Path to the input directory.") {|v| raise OptionParser::InvalidArgument unless File.directory?(v) options[:processingDir] = File.expand_path(v) } opts.on("-o", "--output-ocr-dir OUTPUT_OCR_DIR", "Path to the output directory for OCR-ed PDF files.") {|v| raise OptionParser::InvalidArgument unless File.directory?(v) options[:ocrDir] = File.expand_path(v) } opts.on("-r", "--output-raw-dir OUTPUT_RAW_DIR", "Path to the output directory for the raw image files.") {|v| raise OptionParser::InvalidArgument unless File.directory?(v) options[:rawDir] = File.expand_path(v) } opts.on("-f", "--filter FILTER", "File name filter (LIST), defaults to *.jpg, *.jpeg, *.png") {|v| options[:filter] ||= [] options[:filter].push(v) } opts.on("-v", "--[no-]verbose", "Run verbosely") {|v| options[:verbose] = v } end.parse! options[:tesseract] = `which tesseract` options[:tesseract] = "/usr/local/bin/tesseract" unless File.exists? options[:tesseract] raise "Missing tesseract executable from PATH" unless File.executable?(options[:tesseract]) options[:filter] ||= ["*.png", "*.jpeg", "*.jpg"] unless options[:filter] raise OptionParser::MissingArgument.new("--input-dir") if options[:processingDir].nil? raise OptionParser::MissingArgument.new("--output-ocr-dir") if options[:ocrDir].nil? raise OptionParser::MissingArgument.new("--output-raw-dir") if options[:rawDir].nil? if options[:verbose] puts "\nRunning with options:\n\n" pp options puts end def execute(cmd, options) puts cmd if options[:verbose] out = if options[:verbose] then "" else "1>/dev/null 2>&1" end r = system("#{cmd} #{out}") unless r $stderr.puts "ERROR — command exited with error code (#{r}):\n #{cmd}" exit 1 end end options[:filter].each do |filter| Dir["#{options[:processingDir]}/#{filter}"].each do |f| # Filename format generated by macOS if f =~ /Screenshot\s+(\d{4}-\d{2}-\d{2})\s+at\s+(\d{2}\.\d{2}\.\d{2})/ fname = "Screenshot #{$1} #{$2}#{File.extname(f)}" # Filename format generated by my Galaxy Tab (Android) elsif f =~ /Screenshot[_\s-]+(\d{4})(\d{2})(\d{2})[_\s-]+(\d{2})(\d{2})(\d{2})(?:[_\s-]+([^.]*))?/ details = if $7 then " #{$7}" else "" end fname = "Screenshot #{$1}-#{$2}-#{$3} #{$4}.#{$5}.#{$6}#{details}#{File.extname(f)}" else fname = File.basename(f) end raw_output = File.join(options[:rawDir], fname) ocr_output = File.join(options[:ocrDir], fname) source = File.expand_path(f) if source != raw_output execute("mv \"#{source}\" \"#{raw_output}\"", options) end execute("#{options[:tesseract]} \"#{raw_output}\" \"#{ocr_output}\" -l eng pdf", options) end end

Don’t forget to make it executable:

chmod +x ~/bin/screenshots-sync

Note that this script can be used standalone, for importing existing screenshots archives:

$ screenshots-sync -h Usage: screenshots-sync [-h] -i -o -r [-d] -i, --input-dir INPUT_DIR Path to the input directory. -o OUTPUT_OCR_DIR, Path to the output directory for OCR-ed PDF files. --output-ocr-dir -r OUTPUT_RAW_DIR, Path to the output directory for the raw image files. --output-raw-dir -f, --filter FILTER File name filter (LIST), defaults to *.jpg, *.jpeg, *.png -v, --[no-]verbose Run verbosely

5. Load a Launch Agent

Installing a launch agent for running our synchronization script, whenever new files appear in Processing, is easy.

Create the ~/Library/LaunchAgents/my.sync.ocr.plist file:

Label my.screenshot.ocr ProgramArguments /Users/alex/bin/screenshots-sync --input-dir /Users/alex/Dropbox/Screenshots/Processing --output-ocr-dir /Users/alex/Dropbox/Screenshots/OCR --output-raw-dir /Users/alex/Dropbox/Screenshots/Raw WatchPaths /Users/alex/Dropbox/Screenshots/Processing ThrottleInterval 5 StandardOutPath /Users/alex/Library/Logs/screenshots-sync.log StandardErrorPath /Users/alex/Library/Logs/screenshots-sync.log

Then install it:

launchctl load -w ~/Library/LaunchAgents/my.sync.ocr.plist

Enjoy

Your screenshots will now be indexed by macOS’s Finder, searchable via Spotlight.

They’ll also get indexed by Dropbox, because the Plus plan supports searching in PDF files.

The article Organize and Index Your Screenshots (OCR) on macOS first appeared on alexn.org.

Block comments on the web

Alexandru Nedelcu — Tue, 20 Oct 2020 19:51:45 +0000

Comments on the web can be toxic, and a waste of time. Here's how to block them ...

In Firefox/Chrome, and Android

I assume you already use uBlock Origin, and if not, then throw away whatever alternative you have, and switch to it.

On Android, uBlock Origin is supported by Firefox. If Firefox isn’t an option, there are others. At the moment of writing Vivaldi has backed-in content blocking, and supports custom filtering rules, but you might have to publish these rules, in a text file, somewhere online. Adguard might also support custom content filtering rules, although I have issues with trusting them MITM my traffic.

In uBlock Origin, add these to “My Filters” in its dashboard:

news.ycombinator.com##table.comment-tree reddit.com##div.commentarea youtube.com###comments ||disqus.com/embed/comments/ ||disqus.com/embed.js ###disqus_thread ##.dsq-brlink ##.disqus-container

Here’s how it should look like:

These rules block comments from:

Hacker News

Reddit

YouTube

Disqus (on all websites)

Optionally, for extra sanity and street cred, also add these rules 😎

||twitter.com ||facebook.com

A short primer on the syntax …

Prefix CSS selectors with ##; so ##.dsq-brlink will block HTML elements with a .dsq-brlink class, and ###disqus_thread will block HTML elements with a #disqus_thread ID

These CSS selectors can be “generic”, meaning they apply to all websites (in the case of Disqus), or they can apply to a specific domain: e.g. ###disqus_thread is a cosmetic filter that applies to all websites, whereas youtube.com###comments filters HTML elements with a #comments ID that are shown on youtube.com

See relevant cheat sheet section

You can block specific resources from loading; a || prefix means that a domain name follows, until a separator such as /

A rule such as ||disqus.com/embed.js will block /embed.js on all subdomains of disqus.com

See relevant cheat sheet section

For more details on these rules, see this cheat sheet:
Adblock filters explained

Publish it

You can publish this list, as a text file, somewhere, anywhere, and reuse it wherever you have uBlock Origin installed. For example, I published mine at:

alexn.org/assets/misc/block-lists/no-comments.txt

You can then import it in uBlock Origin easily:

Safari / iOS (iPhone, iPad)

Safari, on macOS and iOS, has content blockers too. Not as capable, but they do the job. An excellent one is Wipr, but it doesn’t allow setting custom rules.

A prepackaged solution that can block comments is Quiet, but that’s less fun

1Blocker appears to allow for creating custom rules, but I haven’t tried it yet

Building your own content blocker could be a fun project

If you know of a better solution, write about it in the comments section 😅😂

The article Block comments on the web first appeared on alexn.org.

Generic IOApp alternative

Alexandru Nedelcu — Thu, 15 Oct 2020 00:00:00 +0000

This is a simple and generic IOApp and TaskApp replacement. For those instances in which you want to work with F[_] and not IO or Task, even in main apps.

Requirements:

EffectRuntime (snippet); or if not in the mood for that, replace it with the basic cats.effect.ContextShift

Monix Catnap

import cats.effect._ import monix.catnap.SchedulerEffect abstract class PureApp[F[_]](implicit protected val F: Async[F]) { self => protected def runtime: Resource[SyncIO, EffectRuntime[F]] protected def effect(implicit r: EffectRuntime[F]): ConcurrentEffect[F] protected implicit def timer(implicit F: Concurrent[F], r: EffectRuntime[F]): Timer[F] = r.timer def run(args: List[String])(implicit F: ConcurrentEffect[F], r: EffectRuntime[F] ): F[ExitCode] final def main(args: Array[String]): Unit = { val (res, cancel) = runtime.allocated.unsafeRunSync() implicit val r = res try { new CustomIOApp()(effect(r), r).main(args) } finally { cancel.unsafeRunSync() } } private final class CustomIOApp(implicit F: ConcurrentEffect[F], runtime: EffectRuntime[F]) extends IOApp { override protected implicit def contextShift: ContextShift[IO] = SchedulerEffect.contextShift[IO](runtime.unsafe.scheduler)(IO.ioEffect) override protected implicit def timer: Timer[IO] = SchedulerEffect.timerLiftIO[IO](runtime.unsafe.scheduler)(IO.ioEffect) override def run(args: List[String]): IO[ExitCode] = F.toIO(self.run(args)) } } object PureApp { abstract class ForIO extends PureApp[IO] { override final def effect(implicit r: EffectRuntime[IO]): ConcurrentEffect[IO] = IO.ioConcurrentEffect(r) } }

Sample:

abstract class MyGenericApp[F[_]: Async] extends PureApp[F] { def run(args: List[String])(implicit F: ConcurrentEffect[F], r: EffectRuntime[F] ): F[ExitCode] = { for { name <- Sync[F].delay(scala.io.StdIn.readLine()) _ <- Timer[F].sleep(1.second) _ <- Sync[F].delay(println(s"Hello, $name")) } yield { ExitCode.Success } } } /** * Actual main implementation that the JVM can recognize. */ object MyApp extends MyGenericApp[IO] { override def runtime: Resource[SyncIO, EffectRuntime[F]] = ??? override def effect(implicit r: EffectRuntime[F]) = IO.ioConcurrentEffect(r) }

The article Generic IOApp alternative first appeared on alexn.org.

Effect Runtime

Alexandru Nedelcu — Mon, 12 Oct 2020 00:00:00 +0000

Defining an EffectRuntime that is used to build IO effects. This would be a replacement for ContextShift (Cats Effect 2.x), with an integrated Logger, Scheduler (thus having access to Timer too), and utilities for monitoring.

The UnsafeLogger / SafeLogger and UnsafeMonitoring interfaces are left as an exercise for the reader.

import cats.effect._ import cats.implicits._ import monix.execution._ // ... /** * Slice of [[EffectRuntime]], to be used only when a dependency on * [[UnsafeLogger]] is needed. */ trait UnsafeRuntimeLogger { def unsafe: UnsafeLoggerRef trait UnsafeLoggerRef { def logger: UnsafeLogger } } /** * Slice of [[EffectRuntime]], to be used only when a dependency * on `ExecutionContext` / `monix.execution.Scheduler` is needed. */ trait UnsafeRuntimeScheduler { def unsafe: UnsafeSchedulerRef trait UnsafeSchedulerRef { def scheduler: Scheduler } } /** * Slice of [[EffectRuntime]], to be used only when a reference * to [[UnsafeMonitoring]] is needed. */ trait UnsafeRuntimeMonitoring { def unsafe: UnsafeMonitoringRef trait UnsafeMonitoringRef { def monitoring: UnsafeMonitoring } } /** * Slice of [[EffectRuntime]], to be used only in an "unsafe"context * (where side effects are not suspended in `F[_]`). */ trait UnsafeRuntime extends UnsafeRuntimeLogger with UnsafeRuntimeScheduler with UnsafeRuntimeMonitoring { def unsafe: Unsafe trait Unsafe extends UnsafeLoggerRef with UnsafeSchedulerRef with UnsafeMonitoringRef } /** * Slice of [[EffectRuntime]], to be used only when a reference * to [[SafeLogger]] is needed. */ trait EffectRuntimeLogger[F[_]] extends UnsafeRuntimeLogger { protected implicit def F: Sync[F] def logger: SafeLogger[F] } /** * Our evolved `cats.effect.ContextShift` that has everything * we need in it to build IO effects. */ abstract class EffectRuntime[F[_]] extends ContextShift[F] with EffectRuntimeLogger[F] with UnsafeRuntime { self => protected implicit def F: Async[F] def logger: SafeLogger[F] def scheduler: F[Scheduler] def blocker: Blocker def timer(implicit F: Concurrent[F]): Timer[F] def deferAction[A](f: Scheduler => F[A]): F[A] = self.scheduler.flatMap(f) }

Sample for wrapping a Future-based API:

import scala.concurrent._ import scala.concurrent.duration._ def unsafeGetRequest(req: Request)( implicit ec: ExecutionContext ): Future[Response] = { ??? } def getRequest[F[_]: Concurrent](req: Request)( implicit r: EffectRuntime[F] ): F[Response] = r.deferAction { implicit ec => for { _ <- r.logger.debug(s"Triggering request: $req") resp <- Async .fromFuture(Sync[F].delay(unsafeGetRequest(req))) .handleErrorWith { err => r.logger.error("Unexpected error, retrying", err) >> r.timer[F].sleep(1.second) >> getRequest(req) } } yield { resp } }

WARN: the retry logic isn’t a good one. For a better implementation, see Retry Failing Tasks with Cats and Scala.

The article Effect Runtime first appeared on alexn.org.

When My World Vanishes

Alexandru Nedelcu — Sat, 10 Oct 2020 00:00:00 +0000

It’s me, facing a hard to solve problem. It’s a difficult one, and I’m having problems focusing. I make some coffee, I move to another room, and I’m already thinking of running to some coffee shop, forgetting that we’re still in a pandemic. I sometimes find it hard to focus, it can be hard to start, I procrastinate a lot, or I’m just too damn lazy.

The problems I’m dealing with can require an extreme attention to details, as you have to load all of those fine-grained interactions between components, all of those side effects, all the system’s responsibilities and limitations; you go over the problem repeatedly, maybe you can understand it, maybe you can simplify it, maybe you don’t need to solve it at all. When things don’t work, you try a change in perspective.

My 10-year-old son is often imitating my computer posture. He wants to grow up to be just like me, but the jury is still out on whether that’s a good thing 🤷‍♂️

People can’t solve multiple problems in parallel. When we do, we only do it for superficial problems. Yes, we can drive a car while talking on the phone, or thinking about some argument with a colleague. But that’s only because of the autonomic nervous system, which can be trained to make us do things out of reflex, putting us in auto-pilot. It’s like when breathing, we don’t think about breathing in or out.

When I drive alone, I’m so deep in my own thoughts, that I’m in full auto-pilot, changing gears, making turns, stopping at the red light, avoiding obstacles. I sometimes awaken while driving towards the office, even if we are still during the pandemic, it’s weekend, and my trip was to the supermarket, in the opposite direction. Funny enough I get the same effect when ridding my bike, or my kick scooter, in spite of the increased effort, the constant motion of my legs, and the increased awareness needed to avoid traffic accidents.

Hard problems demand focus. A single word from a colleague, or from my wife or kid, a single runaway thought could destroy my focus. Working from home can be hard, especially when you have a son that wants to play, and you can’t say no to his puppy eyes.

To help focus I often play some tune on repeat, over and over again. Or maybe there’s some inertia, the problem is exciting, and that magic moment finally comes …

And the world around me vanishes …

All other problems, my hopes, my dreams, my worries about tomorrow, the world’s hum, everything becomes irrelevant and silent, feeling as if I’ve migrated to another reality, being just me and the algorithm I’m working on. A reality in which the problems are solvable, and the solutions are born out of thin air. It’s like a vice; if I ever had one that I can’t quit, this is it.

It’s the reason for why I never liked alcohol, as it prevents me from going there. I remember being junior year in high-school, and getting drunk with my gang. It’s a memory that got imprinted on me, because the national “Olympiad in Informatics” was in a couple of days, and drunk as I was, I kept repeating the recipe for “killed backtracking”, out of fear I’d forget it. Fun times, got third prize.

I remembered this intense feeling of focus while browsing my music library, and finding a gem. I give you the sound-track of Monix, a tune I played over and over again during some of the most focused days I’ve ever had:

Best Mamgu Ever — Underworld (open on YouTube.com)

Tonight I worked on a hard problem, playing this tune on auto-repeat.

I’ll probably fail to go to sleep too soon, as this dumb tune is going to keep ringing in my ears. But it’s not really the music that’s keeping me awake, but the problem I’ve been working on, with the tune being just its echo. Oh well, my brain will have to calm down eventually.

The article When My World Vanishes first appeared on alexn.org.

Scala Snippet: Unlawful Effects

Alexandru Nedelcu — Thu, 08 Oct 2020 00:00:00 +0000

Unlawful/independent version of cats.effect.Effect from Cats Effect v2. Allows for converting (and executing) IO-like values to scala.concurrent.Future, being also good for a graceful migration to Cats Effect v3:

import cats.ApplicativeError import cats.effect.{ Effect, IO } import cats.implicits._ import simulacrum.typeclass import scala.concurrent.{ Future, Promise } /** * Type class defining an "unlawful" variant of `cats.effect.Effect`. * * This allows it to work with plain `Future`, which cannot implement `Effect`. * It also allows for a graceful migration to Cats Effect v3. * * NOTE: if the requirement is `F[_] : Sync : UnlawfulEffect` then this * is de facto equivalent with `Effect`, therefore `UnlawfulEffect` shouldn't * be used. */ @typeclass trait UnlawfulEffect[F[_]] { def unsafeToFuture[A](fa: F[A]): Future[A] } object UnlawfulEffect extends UnlawfulEffectLowLevelImplicits { /** * Standard `Future` instance, which couldn't be a lawful `Effect`. */ implicit val forFuture: UnlawfulEffect[Future] = new UnlawfulEffect[Future] { override def unsafeToFuture[A](fa: Future[A]): Future[A] = fa } /** * Optimization for `cats.effect.IO`, even if this should be handled by * [[forAnyEffect]]. */ implicit val forIO: UnlawfulEffect[IO] = new UnlawfulEffect[IO] { override def unsafeToFuture[A](fa: IO[A]): Future[A] = fa.unsafeToFuture() } } trait UnlawfulEffectLowLevelImplicits { self: UnlawfulEffect.type => /** * Converts from: * [[https://typelevel.org/cats-effect/typeclasses/effect.html]] */ implicit def forAnyEffect[F[_]](implicit F: Effect[F]): UnlawfulEffect[F] = new UnlawfulEffect[F] { override def unsafeToFuture[A](fa: F[A]): Future[A] = { val p = Promise[A]() F.runAsync(fa) { result => // Not really cool to not suspend side-effects here, but // we know the context in which we are in, and it's fine, this time; // Don't try this at home! p.complete(result.toTry) IO.unit }.unsafeRunSync() p.future } } }

We can then interact with more impure APIs, that aren’t IO-driven:

import akka.stream.scaladsl.Flow implicit class FlowOps[In, Out, Mat](flow: Flow[In, Out, Mat]) extends AnyVal { def mapEffect[F[_]: UnlawfulEffect, Out2]( parallelism: Int )(fa: Out => F[Out2]): Flow[In, Out2, Mat] = { flow.mapAsync(parallelism)(out => UnlawfulEffect.unsafeToFuture(fa(out))) } }

Sample:

import cats.effect.IO Flow[Int].mapEffect { num => IO { println(s"Received: $num") num.toString } }

This is similar to usage of TaskLike in Monix.

The article Scala Snippet: Unlawful Effects first appeared on alexn.org.

Running integration tests, with Scala + sbt

Alexandru Nedelcu — Mon, 05 Oct 2020 00:00:00 +0000

For separating integration tests, from unit tests, in build.sbt:

// sbt command-line shortcut addCommandAlias("ci-integration", "Integration/testOnly -- -n integrationTest") lazy val IntegrationTest = config("integration").extend(Test) //... lazy val root = Project(base = file(".")) .configs(IntegrationTest) .settings( // Exclude integration tests by default (in ScalaTest) Test / testOptions += Tests.Argument(TestFrameworks.ScalaTest, "-l", "integrationTest"), // Include integration tests, by nullifying the above option IntegrationTest / testOptions := Seq.empty, ) .settings( // Enable integration tests inConfig(IntegrationTest)(Defaults.testTasks) )

Then in your tests:

import org.scalatest.Tag import org.scalatest.funsuite.AsyncFunSuite object IntegrationTest extends Tag("integrationTest") class MyTestSuite extends AsyncFunSuite { // Tagging this test as an integration test test("integration works", IntegrationTest) { ??? } }

Then run:

sbt ci-integration

The article Running integration tests, with Scala + sbt first appeared on alexn.org.

Scala Snippet: Flow/Processor to Effect

Alexandru Nedelcu — Fri, 25 Sep 2020 00:00:00 +0000

Converts an Akka Streams Flow into an In => IO[Out] method, thus wrapping Flow into Cats-Effect’s IO.

import akka.actor.ActorSystem import akka.stream.Materializer import akka.stream.scaladsl.Flow import cats.effect.{ ContextShift, IO, Resource } import cats.implicits._ import monix.execution.AsyncSemaphore import monix.execution.atomic.Atomic import org.reactivestreams.{ Processor, Publisher, Subscriber, Subscription } import scala.annotation.{ nowarn, tailrec } import scala.concurrent.duration.Duration import scala.concurrent.{ Await, ExecutionContext, Future, Promise } final class FlowToEffect[In, Out] private ( processor: Processor[In, Out] )(implicit cs: ContextShift[IO], ec: ExecutionContext ) { private[this] val (producer: Publisher[Out], subscriber: Subscriber[In]) = (processor, processor) private[this] val awaitCallLatch = AsyncSemaphore(1) private[this] var publisherResponsePromise: Promise[Out] = _ private[this] val requested = Atomic(Left(Promise()): Either[Promise[Unit], Long]) private[this] val connectionClosed = Promise[Unit]() subscriber.onSubscribe(new Subscription { // Called by Akka Streams @tailrec override def request(n: Long): Unit = { assert(n >= 0) if (n == 0) return val update = if (n - 1 > 0) Right(n - 1) else Left(Promise[Unit]()) requested.get() match { case current @ Left(promise) => if (!requested.compareAndSet(current, update)) request(n) else promise.success(()) case current @ Right(n0) => if (!requested.compareAndSet(current, Right(n + n0))) request(n) } } // Called by Akka Streams override def cancel(): Unit = throw new IllegalStateException() }) producer.subscribe(new Subscriber[Out] { private[this] var sub: Subscription = _ override def onSubscribe(s: Subscription): Unit = { sub = s connectionClosed.future.onComplete { _ => println("Cancelling connection") sub.cancel() } sub.request(1) } override def onNext(t: Out): Unit = { publisherResponsePromise.success(t) sub.request(1) awaitCallLatch.release() } override def onError(t: Throwable): Unit = { publisherResponsePromise.failure(t) // TODO: signal future requests that stream ended in error awaitCallLatch.release() } override def onComplete(): Unit = () }) private val cancelIO: IO[Unit] = IO { connectionClosed.success(()) () } @nowarn("cat=deprecation") def pushEvent(in: In): IO[Out] = IO.fromFuture(IO { awaitCallLatch.acquire().flatMap { _ => val promise = Promise[Out]() publisherResponsePromise = promise val backpressurePermission = requested.transformAndExtract { case Right(n) => if (n > 1) (Future.successful(()), Right(n - 1)) else (Future.successful(()), Left(Promise())) case current @ Left(promise) => (promise.future, current) } backpressurePermission.flatMap { _ => subscriber.onNext(in) promise.future } } }) } object FlowToEffect { def apply[I, O]( f: IO[Processor[I, O]] )(implicit cs: ContextShift[IO], ec: ExecutionContext ): Resource[IO, FlowToEffect[I, O]] = { Resource(f.map { processor => val ref = new FlowToEffect(processor) (ref, ref.cancelIO) }) } def apply[I, O, Mat]( flow: Flow[I, O, Mat] )(implicit cs: ContextShift[IO], m: Materializer, ec: ExecutionContext ): Resource[IO, FlowToEffect[I, O]] = { val res = Resource.liftF(IO { flow.toProcessor.run() }) res.flatMap(proc => apply(IO.pure(proc))) } def main(args: Array[String]): Unit = { import ExecutionContext.Implicits.global implicit val as = ActorSystem("test") implicit val cs = IO.contextShift(global) val flow = Flow.fromFunction[Int, String] { int => show"Received number: $int" } val (res, cancelRes) = FlowToEffect(flow).allocated.unsafeRunSync() try { val f1 = res.pushEvent(1).unsafeToFuture() val f2 = res.pushEvent(2).unsafeToFuture() val f3 = res.pushEvent(3).unsafeToFuture() val r1 = Await.result(f1, Duration.Inf) println(show"Received: $r1") val r2 = Await.result(f2, Duration.Inf) println(show"Received: $r2") val r3 = Await.result(f3, Duration.Inf) println(show"Received: $r3") } finally { cancelRes.unsafeRunSync() Await.result(as.terminate(), Duration.Inf) () } } }

The article Scala Snippet: Flow/Processor to Effect first appeared on alexn.org.

Privilege

Alexandru Nedelcu — Thu, 10 Sep 2020 00:00:00 +0000

My son ❤️

Today it's my 38ᵗʰ birthday. I was born into privilege. Other people aren't as lucky.

I’m privileged to have a wonderful wife and son, to have a family.

I’m privileged that I’m middle-class, that I have a comfortable and stable job, that I have more job security during a crisis; I’m privileged that my worries for tomorrow are superficial; I’m lucky not to have debt.

I’m privileged to have found my vocation; to have a profession that I would practice even if I’d win the lottery tomorrow.

I’m privileged to have enough to eat and enjoy anything I’d like, such as cooking with extra-virgin olive oil and Irish grass-fed butter. Privileged to eat turkey, pork, bream, or salmon on every meal; to have whatever dessert I want, and fresh fruits, even when out of season and expensive. I’m privileged to eat delicious home-cooked meals every day.

I’m privileged to afford to go on vacation, and that in this Covid-19 pandemic, my biggest problem was that I got bored while working from home and that I’ve got unspent vacation days left.

I’m privileged to have gone to school, attending the Romanian public school system, which for all its problems, it’s extraordinary. We can only see how an incredible privilege this is when looking at the issues faced by poor children in Romanian villages or the US. And I have no life long debt from that education.

I’m privileged to be a European heterosexual white male, born into an official religion. Everywhere I go, all doors are open for me. I’m not judged for where I was born. I don’t have have to keep the receipt when going to the store to prove that I haven’t stolen anything. If I am rejected, I’m not dismissed because of my skin color, sexual orientation, being a Jew, or a woman.

I’m privileged to wear a size M or L for my tee-shirt. Even though I tend to get fat, I gravitate between overweight and XL, but no more than that. I’m privileged not to be “morbidly obese” and couldn’t get blamed for having a flaw of character because of my body constitution. I’m not condemned for gluttony, which is what society does against fat people.

I’m privileged to be healthy. But even if I had a problem, I have access to a public healthcare system, which has its issues, but it’s available. I wouldn’t have to sell my apartment to pay for my medical bills. As a middle-class white male (with money for tips and an attitude), I am prioritized. And if I’m not in the mood for public institutions, for staying in lines, I can always go to that private clinic that I’m subscribed to.

I was born in privilege. Other people aren’t as lucky.

I’m not writing this to feel good, or grateful for what I have, due to other people being less fortunate, even though that’s a thing. It’s essential to be mindful that many of us have a social privilege that placed us at the front of the pack, which gave us an advantage. That as much as we’d like to believe that we’ve made it due to hard work, that opportunities at birth are equal, that’s not true. And this is unfair, it’s unjust.

We work hard for what we have, for sure. But in our society, hard work isn’t everything. And the privilege we’re born into matters more.

Systemic racism, misogyny, antisemitism, fatphobia, xenophobia—these are issues even in 2020. And when people end up hating other human beings, based on class or category, most often than not, it is privilege they are protecting, not their hard work.

My son is and will be privileged. I’m glad that he is because he won’t be discriminated against. It’s not shameful to be born into privilege, but it is dumb luck.

Therefore I’m raising my son not to hate or discriminate against other people based on social position, race, ethnicity, religion, body size, sex, or sexual orientation. It’s the least I could do. And you should too.

The article Privilege first appeared on alexn.org.

Scala Snippet: Safe Passwords

Alexandru Nedelcu — Thu, 27 Aug 2020 00:00:00 +0000

See: Securely Store Sensitive Data in RAM

import cats.effect.{ Resource, Sync } import java.util.ConcurrentModificationException import java.util.concurrent.atomic.AtomicBoolean final case class PasswordValue(value: String) { override def toString = "PasswordValue(****)" } final class SafePassword private (chars: Array[Char]) { private[this] val lock = new AtomicBoolean(false) private[this] var isAvailable = true def read[F[_]](implicit F: Sync[F]): Resource[F, PasswordValue] = Resource[F, PasswordValue](F.suspend { unsafeAcquire() match { case Right(()) => val cancel = F.delay(lock.set(false)) F.pure((PasswordValue(new String(chars)), cancel)) case Left(error) => F.raiseError(error) } }) def onceThenNullify[F[_]](implicit F: Sync[F]): Resource[F, PasswordValue] = read.flatMap { value => Resource(F.pure((value, F.delay(unsafeNullify())))) } private[this] def unsafeNullify(): Unit = { if (isAvailable) { isAvailable = false for (i <- chars.indices) { chars(i) = 0 } } } private[this] def unsafeAcquire(): Either[Throwable, Unit] = if (!lock.compareAndSet(false, true)) { Left( new ConcurrentModificationException( "Cannot use this password from concurrent tasks" ) ) } else if (!isAvailable) { lock.set(false) Left( new IllegalStateException( "Password was nullified already" ) ) } else { Right(()) } } object SafePassword { def apply[F[_]](value: String)(implicit F: Sync[F]): F[SafePassword] = F.delay(unsafe(value)) def unsafe(value: String): SafePassword = new SafePassword(value.toCharArray) }

And usage:

SafePassword[IO]("74kDc2J%dd2&").flatMap { pass => pass.onceThenNullify[IO].use { p => IO { println(p.value) } } }

The article Scala Snippet: Safe Passwords first appeared on alexn.org.

Sample Error Hierarchy in Scala

Alexandru Nedelcu — Fri, 14 Aug 2020 00:00:00 +0000

sealed abstract class KnownException(message: String, cause: Throwable) extends RuntimeException(message, cause) sealed abstract class InputException(message: String, cause: Throwable) extends KnownException(message, cause) object InputException { final case class Validation(message: String, cause: Throwable) extends InputException(message, cause) with ExceptionCaseClassEquality final case class BadInput(message: String, cause: Throwable) extends InputException(message, cause) with ExceptionCaseClassEquality final case class Forbidden(message: String, cause: Throwable) extends InputException(message, cause) with ExceptionCaseClassEquality final case class NotFound(message: String, cause: Throwable) extends InputException(message, cause) with ExceptionCaseClassEquality final case class Conflict(message: String, cause: Throwable) extends InputException(message, cause) with ExceptionCaseClassEquality } sealed abstract class OutputException(message: String, cause: Throwable) extends KnownException(message, cause) object OutputException { final case class Timeout(message: String, cause: Throwable) extends OutputException(message, cause) with ExceptionCaseClassEquality final case class TooManyRequests(message: String, cause: Throwable) extends OutputException(message, cause) with ExceptionCaseClassEquality final case class ResourceUnavailable(message: String, cause: Throwable) extends OutputException(message, cause) with ExceptionCaseClassEquality final case class Unknown(message: String, cause: Throwable) extends OutputException(message, cause) with ExceptionCaseClassEquality } trait ExceptionCaseClassEquality { self: Throwable with Product => override def equals(other: Any): Boolean = { other match { case refTh: Throwable => refTh match { case refProd: Product => productIterator.toSeq.equals(refProd.productIterator.toSeq) && getStackTrace.toSeq.equals(refTh.getStackTrace.toSeq) case _ => false } case _ => false } } }

The article Sample Error Hierarchy in Scala first appeared on alexn.org.

Snippet: turn on JVM debugging in sbt

Alexandru Nedelcu — Thu, 13 Aug 2020 00:00:00 +0000

Remote debugging can be used to debug externally executed programs, useful to activate in sbt in order to keep using it while debugging with your favorite IDE. See for example IntelliJ IDEA’s documentation:

UPDATE: there is now a --jvm-debug parameter to the sbt executable …

sbt --jvm-debug 5005

OLD WAY: — if the above is not suitable, you can do a manual config like this:

fork := true javaOptions ++= { val Digits = "^(\\d+)$".r sys.env.get("JVM_DEBUG") match { case Some("true") => Seq("-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005") case Some(Digits(port)) => Seq(s"-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=$port") case _ => Seq.empty } }

And then set a JVM_DEBUG environment variable, before executing sbt:

JVM_DEBUG=5005 sbt

The article Snippet: turn on JVM debugging in sbt first appeared on alexn.org.

TypeScript Sample: Flow Sensitive Typing

Alexandru Nedelcu — Tue, 11 Aug 2020 00:00:00 +0000

Demonstrating Typescript’s untagged union types:

type Left = { either: "left" value: L } type Right = { either: "right", value: R } type Either = Left | Right function left(value: L): Either { return { either: "left", value } } function right(value: R): Either { return { either: "right", value } } // ---- const value = left("Hello!") // Flow-sensitive typing in action if (value.either == "left") { const l: string = value.value console.log("string: ", l) } else { const r: number = value.value console.log("number", r) }

The article TypeScript Sample: Flow Sensitive Typing first appeared on alexn.org.

Retry Failing Tasks with Cats and Scala

Alexandru Nedelcu — Mon, 03 Aug 2020 00:00:00 +0000

In the face of errors, we could interrupt what we are doing and log the incident for debugging purposes. Some errors are temporary, for example, network connection errors, the web service becoming unavailable for whatever reason, etc. It might be appropriate to do one or multiple retries, as it might not be acceptable to drop a valuable transaction on the floor.

Libraries with DSLs for specifying complex retry logic exist, see cats-retry. In this article, I am not talking about such libraries because implementing your functions is fun, educational, and because you might not need a library where a simple function could do just fine.

Here’s how …

Task Example

We are going to use cats.effect.IO for exemplification, but this can work just as well with the Monix Task, Monix Coeval or any data type that implements the necessary Typelevel Cats and Cats Effect type classes.

import cats.effect.IO import java.io._ // Not very motivating example, but let's go with it def readTextFromFile(file: File, charset: String): IO[String] = IO { val in = new BufferedReader( new InputStreamReader( new FileInputStream(file), charset )) val builder = new StringBuilder() var line: String = null do { line = in.readLine() if (line != null) builder.append(line).append("\n") } while (line != null) builder.toString }

This operation is doing I/O, the file we are looking for could be missing, but only temporarily, or we might have an IOPS capacity problem. In some cases, we might want to keep retrying the task.

Naive Implementation

The ApplicativeError type class from Cats defines these functions:

trait ApplicativeError[F[_], E] extends Applicative[F] { // ... def handleErrorWith[A](fa: F[A])(f: E => F[A]): F[A] def raiseError[A](e: E): F[A] }

The handleErrorWith function works like a flatMap operation, but for errors (the equivalent of Java/Scala’s catch statement). And the raiseError function lifts an E error into the F[A] context (the equivalent of Java’s and Scala’s throw for exceptions).

import cats.implicits._ import cats.{ApplicativeError, Defer} object OnErrorRetry { // WARN: not OK, because we don't have an end condition! def adInfinitum[F[_], A](fa: F[A]) (implicit F: ApplicativeError[F, Throwable], D: Defer[F]): F[A] = { fa.handleErrorWith { _ => // Recursive call describing infinite loop D.defer(loop(fa)) } } } //... OnErrorRetry.adInfinitum(readTextFromFile(file))

Note the usage of ApplicativeError and Defer type classes, added as restrictions for our F[_].

NOTE 1: There’s a caveat with the way we’re using handleErrorWith in such recursive loops. The type we use might not have a memory-safe implementation, which is always a concern in Scala, due to the JVM lacking TCO. The data types that can throw errors, errors based on runtime conditions that can be retried, usually implement a memory-safe handleErrorWith. Still, it’s better if we can ensure this via type restrictions.

We use the Defer type class to force usage of memory-safe (trampolined) implementations, although its laws are probably not strong enough. Still, this restriction will do just fine in practice. The alternative would have been to not put a restriction for memory safety, or to use Cats Effect’s Sync, but this type class is too restricted, to the point that the signature becomes opaque, as it might as well launch missiles.

NOTE 2: We have specialized our E error type, as seen in ApplicativeError[F, E], to Throwable.

There’s no reason to specialize E, except that the Scala compiler ends up having issues inferring the type involved. There are ways to describe a nice API with a generic E and keep the Scala compiler happy, but that’s not the tutorial’s purpose.

This sample has several problems that we’ll have to address:

no end condition

no filtering of errors, since not all errors are recoverable

no protections, like “exponential backoff”

Filtering and End Condition

We must only retry the task in situations in which it can be retried. For example if the task throws a CharacterCodingException, that’s not a task that can be retried, that’s a bug. It’s not always clear when the task can be retried or not, but we can try our best.

And we want to retry, but not forever. So there has to be an end condition in that loop.

/** * Signaling desired outcomes via Boolean is very confusing, * having our own ADT for this is better. */ sealed trait RetryOutcome object RetryOutcome { case object Next extends RetryOutcome case object Raise extends RetryOutcome } /** Module grouping our retry helpers. */ object OnErrorRetry { def withAtMost[F[_], A](fa: F[A], maxRetries: Int) (p: E => RetryOutcome) (implicit F: ApplicativeError[F, Throwable], D: Defer[F]): F[A] = { fa.handleErrorWith { error => if (maxRetries > 0) p(error) match { case RetryOutcome.Next => // Recursive call D.defer(withAtMost(fa, maxRetries - 1)(p)) case RetryOutcome.Raise => // Cannot recover from error F.raiseError(error) } else // Maximum retries reached, triggering error F.raiseError(error) } } }

And usage:

OnErrorRetry.withAtMost(readTextFromFile(file), maxRetries = 10) { case _: CharacterCodingException => RetryOutcome.Raise case _ => RetryOutcome.Next }

Building a Generic Retry Loop

Inspired by Monix’s onErrorRestartLoop, we can describe this function in a generic fashion:

object OnErrorRetry { /** * Saves us from describing recursive functions that accumulate state. */ def loop[F[_], A, S]( fa: F[A], initial: S )( f: (Throwable, S, S => F[A]) => F[A] )(implicit F: ApplicativeError[F, Throwable], D: Defer[F]): F[A] = { fa.handleErrorWith { err => f(err, initial, state => D.defer(loop(fa, state)(f))) } } def withAtMost[F[_], A](fa: F[A], maxRetries: Int)( p: Throwable => RetryOutcome )(implicit F: ApplicativeError[F, Throwable], D: Defer[F] ): F[A] = { loop(fa, maxRetries) { (error, retriesLeft, retry) => if (retriesLeft > 0) p(error) match { case RetryOutcome.Next => retry(retriesLeft - 1) case RetryOutcome.Raise => // Cannot recover from error F.raiseError(error) } else // Maximum retries reached, triggering error F.raiseError(error) } } } // Retrying 10 times at most OnErrorRetry.withAtMost(readTextFromFile(file), maxRetries = 10) { case _: CharacterCodingException => RetryOutcome.Raise case _ => RetryOutcome.Next }

Exponential Backoff

We might also want to introduce exponential backoff because if the resource is busy, the last thing we want to do is to overwhelm it with retry requests. And we are are going to use Timer for introducing delays.

At this point, the state and the configuration are more complicated, so let’s introduce a reusable data structure too, that should be self-explanatory:

import scala.concurrent.duration._ /** * Configuration for retry logic, could be read from a config file, via * something like [[https://github.com/pureconfig/pureconfig PureConfig]]. */ final case class RetryConfig( maxRetries: Int, initialDelay: FiniteDuration, maxDelay: FiniteDuration, backoffFactor: Double, private val evolvedDelay: Option[FiniteDuration] = None, ) { def canRetry: Boolean = maxRetries > 0 def delay: FiniteDuration = evolvedDelay.getOrElse(initialDelay) def evolve: RetryConfig = copy( maxRetries = math.max(maxRetries - 1, 0), evolvedDelay = Some { val nextDelay = evolvedDelay.getOrElse(initialDelay) * backoffFactor maxDelay.min(nextDelay) match { case ref: FiniteDuration => ref case _: Duration.Infinite => maxDelay } } ) }

Finally, we can do this:

object OnErrorRetry { // ... def withBackoff[F[_], A](fa: F[A], config: RetryConfig)( p: Throwable => F[RetryOutcome] )(implicit F: MonadError[F, Throwable], D: Defer[F], timer: Timer[F] ): F[A] = { OnErrorRetry.loop(fa, config) { (error, state, retry) => if (state.canRetry) p(error).flatMap { case RetryOutcome.Next => timer.sleep(state.delay) *> retry(state.evolve) case RetryOutcome.Raise => // Cannot recover from error F.raiseError(error) } else // No retries left F.raiseError(error) } } }

In our predicate we take an F[RetryOutcome] instead of a RetryOutcome. That’s because we might want to trigger additional side effects, like logging.

So to build our final sample, let’s introduce a dependency on typesafe-config in build.sbt, which you probably have anyway:

libraryDependencies += "com.typesafe" % "config" % "1.4.0"

And usage:

object Playground extends LazyLogging with IOApp { // Motivating example def readTextFromFile(file: File, charset: String): IO[String] = ??? override def run(args: List[String]): IO[ExitCode] = { val config = RetryConfig( maxRetries = 10, initialDelay = 10.millis, maxDelay = 2.seconds, backoffFactor = 1.5 ) val task = IO.suspend { val path = args.headOption.getOrElse( throw new IllegalArgumentException("File path expected in main's args") ) readTextFromFile(new File(path), "UTF-8") } val taskWithRetries = OnErrorRetry.withBackoff(task, config) { case _: CharacterCodingException | _: IllegalArgumentException => IO.pure(RetryOutcome.Raise) case e => IO(logger.warn("Unexpected error, retrying", e)) .as(RetryOutcome.Next) } for { t <- taskWithRetries _ <- IO(println(t)) } yield ExitCode.Success } }

Enjoy~

The article Retry Failing Tasks with Cats and Scala first appeared on alexn.org.

Scala Snippet: Cats-Effect Resource to Reactive Streams

Alexandru Nedelcu — Thu, 30 Jul 2020 00:00:00 +0000

Cats-Effect’s Resource can’t be converted directly into a Reactive Streams Publisher. Beware!

import cats.effect.Resource import org.reactivestreams.Publisher import scala.concurrent.Future type Ack[F] = () => F[Unit] // This leaks def fromResource[F[_], A](res: Resource[F, A]): F[Publisher[A]] // Explicit acknowledgement logic is required def fromResource[F[_], A](res: Resource[F, A]): F[Publisher[(A, Ack[F])]] //------------- // Similarly for concrete resources, this leaks! def fromFile(file: File): Publisher[InputStream] // Explicit acknowledgement logic is required def fromFile(file: File): Publisher[(InputStream, Ack[Future])]

The article Scala Snippet: Cats-Effect Resource to Reactive Streams first appeared on alexn.org.

ExecutionContext Must Be Stack-safe

Alexandru Nedelcu — Wed, 27 May 2020 00:00:00 +0000

// Demonstrating that directly executing runnables in your // ExecutionContext is a really bad idea def trigger(cycles: Int): Future[Int] = { implicit val directEC = new ExecutionContext { def execute(r: Runnable) = r.run() def reportFailure(e: Throwable) = throw e } val p = Promise[Int]() val f = (0 until cycles).foldLeft(p.future)((f, _) => f.map(_ + 1)) p.success(0) f } // Throws StackOverflowError trigger(5000)

The article ExecutionContext Must Be Stack-safe first appeared on alexn.org.

Fatal Warnings and Linting in Scala

Alexandru Nedelcu — Tue, 26 May 2020 00:00:00 +0000

The best practices are those enforced by the build tools, as part of the build process. Don’t annoy your colleagues in code reviews; let the build tools do that for you.

The Scala compiler has multiple linting options available and emits some warnings out of the box that would be more useful as errors. Let’s see how we can rely on the Scala compiler to strengthen our code and piss off your colleagues with clean code requirements.

Best Practice: Stop Ignoring Warnings!

Some of it might be noise; however, in that noise, some real gems might be missed, warnings that signal bugs. For example, the compiler can do exhaustiveness checks when pattern matching:

def size(list: List[_]): Int = list match { case _ :: rest => 1 + size(rest) } // On line 2: warning: match may not be exhaustive. // It would fail on the following input: Nil

Here’s what happens next:

size(List(1,2,3)) // scala.MatchError: List() (of class scala.collection.immutable.Nil$)

It shouldn’t be just a warning. Even if you’re diligent, this warning can get lost in a sea of other warnings, like it often does. And here we were lucky, because the runtime exception was triggered on the happy path, but our luck would eventually run out and end up with such exceptions in production.

1. Activate -Xfatal-warnings

To turn all compiler warnings into errors, you can activate -Xfatal-warnings. In build.sbt:

scalacOptions ++= Seq( "-Xfatal-warnings", //... )

NOTE: it’s probably a good idea to exclude this from the console configuration, see section 2.3.

Now if we try out the code above, we get an error like this:

[error] .../Example.scala:10:5: match may not be exhaustive. [error] It would fail on the following input: Nil [error] list match { [error] ^ [error] one error found

Thus we can no longer ignore it.

1.1. Make only some warnings fatal (Scala 2.13)

There’s a new -Wconf option in Scala 2.13.2 (see PR). With it, we can still keep some warnings as warnings.

Use-case: say you’re upgrading an Akka project. It’s going to have a ton of @deprecated warnings that you may not want to fix right now, but you still want to keep the exhaustiveness checks as errors.

scalacOptions ++= Seq( // "-Xfatal-warnings", "-Wconf:cat=deprecation:ws,any:e", )

What this does is to turn all warnings into errors, except for deprecation messages, which are still left as warnings. To break it down:

cat=deprecation refers to deprecation messages (classes/methods marked with @deprecated being called) and :ws says that for these warnings a “warning summary” should be shown

any:e says that for any other kind of warning, signal it via an error

Run this in a terminal for more help:

scalac -Wconf:help

2. Activate All Linting Options

There are many useful compiler options that you could activate. You can find a (possibly non-complete) list on docs.scala-lang.org.

WARNING: the compiler evolves and it's good to keep this list up to date (see next tip)!

For Scala 2.13 at the time of writing, here’s my list of scalac options:

scalacOptions := Seq( // Feature options "-encoding", "utf-8", "-explaintypes", "-feature", "-language:existentials", "-language:experimental.macros", "-language:higherKinds", "-language:implicitConversions", "-Ymacro-annotations", // Warnings as errors! "-Xfatal-warnings", // Linting options "-unchecked", "-Xcheckinit", "-Xlint:adapted-args", "-Xlint:constant", "-Xlint:delayedinit-select", "-Xlint:deprecation", "-Xlint:doc-detached", "-Xlint:inaccessible", "-Xlint:infer-any", "-Xlint:missing-interpolator", "-Xlint:nullary-override", "-Xlint:nullary-unit", "-Xlint:option-implicit", "-Xlint:package-object-classes", "-Xlint:poly-implicit-overload", "-Xlint:private-shadow", "-Xlint:stars-align", "-Xlint:type-parameter-shadow", "-Wdead-code", "-Wextra-implicit", "-Wnumeric-widen", "-Wunused:implicits", "-Wunused:imports", "-Wunused:locals", "-Wunused:params", "-Wunused:patvars", "-Wunused:privates", "-Wvalue-discard", )

There are many useful options in there, from disallowing “adapted args”, detecting inaccessible code, inferred Any, shadowing of values, to unused imports and params and others.

2.1. Use the sbt-tpolecat plugin

Keeping that list of compiler options up to date is exhausting, new useful options get added all the time, others are deprecated, and especially for libraries, you have to deal with multiple Scala versions in the same project.

A better option is to include sbt-tpolecat in your project.

2.2. Exclude annoying linting options

Some linting options can trigger false positives that are too annoying. It’s fine to remove them from your project or specific configurations (e.g. Test, Console).

For example, I do not like the -Wunused:privates option because it triggers an annoying false positive when defining (unused) default values in case class definitions.

If using sbt-tpolecat, as mentioned above, you can include something like this in your build definition:

scalacOptions in Compile ~= { options: Seq[String] => options.filterNot( Set( "-Wunused:privates" ) ) }

2.3. Relax the console configuration

When playing around in the console, it’s a good idea to deactivate Xfatal-warnings, along with some of the more annoying warnings.

You could use sbt-tpolecat, which already does this automatically, or you could add this to build.sbt:

val filterConsoleScalacOptions = { options: Seq[String] => options.filterNot(Set( "-Xfatal-warnings", "-Werror", "-Wdead-code", "-Wunused:imports", "-Ywarn-unused:imports", "-Ywarn-unused-import", "-Ywarn-dead-code", )) } // ... scalacOptions.in(Compile, console) ~= filterConsoleScalacOptions, scalacOptions.in(Test, console) ~= filterConsoleScalacOptions

Credit for this snippet: DavidGregory084/sbt-tpolecat.

3. Silence some warnings

Sometimes you want to ignore a certain warning:

maybe it’s a false positive

maybe you want something “unused”, as a placeholder

3.1. Silencer plugin (Scala < 2.13)

You can use the ghik/silencer compiler plugin.

For our sample above, if we want to silence that exhaustiveness check:

import com.github.ghik.silencer.silent @silent("not.*?exhaustive") def size(list: List[_]): Int = list match { case _ :: rest => 1 + size(rest) }

The annotation accepts a regular expression that matches the warning being silenced.

You can also use silence warnings for an entire folder, useful if you have auto-generated source files:

// Alternative, if using the silencer plugin scalacOptions ++= Seq( "-P:silencer:pathFilters=.*[/]src_managed[/].*", )

3.2. Using @nowarn and -Wconf in Scala 2.13.2

Scala 2.13 has added the -Wconf flag for configurable warnings, @nowarn annotation for local suppression.

@nowarn can be more fine grained. We could do just like the above and silence with a pattern matcher:

import scala.annotation.nowarn @nowarn("msg=not.*?exhaustive") def size(list: List[_]): Int = list match { case _ :: rest => 1 + size(rest) }

But we can do better, because the actual error messages are brittle. We can silence based on the “category” of the warning. To find the category of a warning, we can (temporarily) enable extra verbosity in these messages via this compiler option in build.sbt:

scalacOptions += "-Wconf:any:warning-verbose"

That error will then look like this:

[error] ... [other-match-analysis @ size] match may not be exhaustive.

The category is “other-match-analysis”, so we can silence it like this:

@nowarn("cat=other-match-analysis") def size(list: List[_]): Int = list match { case _ :: rest => 1 + size(rest) }

You can also silence warnings for entire folders, via -Wconf, useful if you have auto-generated files (e.g. from templates):

// Turns off warnings for generated files and for templates scalacOptions ++= Seq( "-Wconf:src=src_managed/.*:silent", )

NOTE: for forward compatibility in older Scala versions, with the Silencer plugin, coupled with scala-library-compat, you can use the new @nowarn annotation with older Scala versions, however only the @nowarn("msg=") filtering is supported.

4. Other linters

You shouldn’t stop at Scala’s linting options. There are other sbt plugins available that can enforce certain best practices. Off the top of my head:

Scalafix (this one can even rewrite the code for you 😉)

Wartremover

Scapegoat

Scalastyle

With these, you could ban null or Any from your project, among other beneficial options.

Any useful plugins that I’m missing?

4.1. Example: disable .toString conversions

I recently disabled unsafe .toString and string + any conversions in our work project, with the purpose to not leak sensitive customer data in logs. To do this I enabled “Wartremover” with this configuration in build.sbt, and it is doing wonders:

wartremoverErrors in (Compile, compile) := Seq(Wart.ToString, Wart.StringPlusAny), wartremoverErrors in (Test, test) := Seq.empty,

Coupled with a custom logger interface making use of its own type class, we can ensure that data structures containing protected data cannot leak by mistake via logging (either Logback or println statements).

Final words

Scala is a static language, but sometimes it isn’t static or opinionated enough. The more you can prove about your code at compile-time, the less defects you can have at runtime.

Now go forth and annoy your colleagues with actually useful compiler errors!

The article Fatal Warnings and Linting in Scala first appeared on alexn.org.

Parallelizing Work via a JavaScript Semaphore

Alexandru Nedelcu — Tue, 21 Apr 2020 16:00:00 +0000

This is a simple, but very effective implementation of a Semaphore, used for easily parallelizing work in JavaScript/TypeScript. And while libraries for doing this might be available, quality varies and implementing your own stuff is fun.

Use-case: many times we iterate over a list of items and then do expensive network requests, like reading and writing to a database.

Scala already has good implementations for doing this, no need to reinvent the wheel. See: cats.effect.concurrent.Semaphore and monix.catnap.Semaphore.

Simple example:

// Dummy function function saveToDatabase(item: number): Promise { return new Promise(f => { if (item % 1000 === 0) { console.log("Progress: ", item) } setTimeout(f, 100 /* millis */) }) } // ... for (let i = 0; i < 100000; i++) { await saveToDatabase(i) }

But this will be very slow, taking ~2.8 hours to finish.

On the other hand, if we try a naive solution via Promise.race, this can easily overwhelm your database and your script can crash with “out of memory” or “out of available file handles” errors. This is also not adequate in case producing those items in the first place is an expensive process (e.g. going over an async iterator). We can certainly hold 100,000 items in memory, but talk about 100 million or more and things start to fall apart.

const promises: Promise[] = [] for (let i = 0; i < 100000; i++) { promises.push(saveToDatabase(i)) } await Promise.race(promises)

Let’s implement a simple data structure that can help us parallelize the workload, but only execute at most 100 tasks in parallel:

const semaphore = new AsyncSemaphore(100) for (let i = 0; i < 100000; i++) { await semaphore.withLockRunAndForget(() => saveToDatabase(i)) } await semaphore.awaitTerminate() console.log("Done!")

Execution time will be ~1.7 minutes (compared with 2.8 hours for the first sample).

The method withLockRunAndForget only waits in case the semaphore doesn’t have handles available. And when the loop is finished, we need awaitTerminate to wait for all active tasks to finish.

Implementation, which can be copy/pasted in your project:

export class AsyncSemaphore { private _available: number private _upcoming: Function[] private _heads: Function[] private _completeFn!: () => void private _completePr!: Promise constructor(public readonly workersCount: number) { if (workersCount <= 0) throw new Error("workersCount must be positive") this._available = workersCount this._upcoming = [] this._heads = [] this._refreshComplete() } async withLock(f: () => Promise): Promise { await this._acquire() return this._execWithRelease(f) } async withLockRunAndForget(f: () => Promise): Promise { await this._acquire() // Ignoring returned promise on purpose! this._execWithRelease(f) } async awaitTerminate(): Promise { if (this._available < this.workersCount) { return this._completePr } } private async _execWithRelease(f: () => Promise): Promise { try { return await f() } finally { this._release() } } private _queue(): Function[] { if (!this._heads.length) { this._heads = this._upcoming.reverse() this._upcoming = [] } return this._heads } private _acquire(): void | Promise { if (this._available > 0) { this._available -= 1 return undefined } else { let fn: Function = () => {/***/} const p = new Promise(ref => { fn = ref }) this._upcoming.push(fn) return p } } private _release(): void { const queue = this._queue() if (queue.length) { const fn = queue.pop() if (fn) fn() } else { this._available += 1 if (this._available >= this.workersCount) { const fn = this._completeFn this._refreshComplete() fn() } } } private _refreshComplete(): void { let fn: () => void = () => {/***/} this._completePr = new Promise(r => { fn = r }) this._completeFn = fn } }

Enjoy~

The article Parallelizing Work via a JavaScript Semaphore first appeared on alexn.org.

Blog Updates

Alexandru Nedelcu — Tue, 21 Apr 2020 10:00:00 +0000

Video link (open on YouTube.com)

I love writing. I write long emails, long chat texts, long code comments. Some of my rants are epic. Sometimes I'm amazed that my colleagues have the patience to read it all 🙂

Yet I’ve neglected my personal blog. This happens because Twitter fulfilled my needs for shitposting and for code I ended up creating snippets as Gists, many of which are private, meant for colleagues.

No more. I dusted off my blog and will commence a regular posting schedule.

This blog is about programming, but take note that I don’t do just functional programming, or just Scala. Today’s article is about imperative, Promise-driven JavaScript.

Website Yak Shaving

Like all professional software developers that want to write more, I started with yak shaving by updating my blog:

Migrated from Middleman to Jekyll for generating my static website

Changed the design from something homemade to a modified brianmaierjr/long-haul; design should be responsive on mobile phones

Disabled comments, replaced widget with an email address — if you want to talk with me about my articles, we can do so in private

Automatic thumbnails for Vimeo/YouTube videos, enabled for Twitter/Facebook cards; expect more videos soon

Integration of MathJax for math formulas

Video link (open on YouTube.com)

If you find anything broken, please let me know.

The article Blog Updates first appeared on alexn.org.

Python Snippet: Send Email via SMTP

Alexandru Nedelcu — Wed, 18 Mar 2020 00:00:00 +0000

#!/usr/bin/env python import smtplib import argparse from datetime import datetime, timedelta from email.message import EmailMessage parser = argparse.ArgumentParser(description='Send email') parser.add_argument( '--smtp-host', dest='smtp_host', type=str, help="SMTP host and port", required=True) parser.add_argument( '--smtp-user', dest='smtp_user', type=str, help="SMTP user for login", required=True) parser.add_argument( '--smtp-pass', dest='smtp_pass', type=str, help="SMTP password for login", required=True) parser.add_argument( '--to', dest='to_address', type=str, help="TO address", required=True) parser.add_argument( '--from', dest='from_address', type=str, help="FROM address", required=True) parser.add_argument( '--title', dest='title', type=str, help="Mail title", required=True) parser.add_argument( '--body', dest='body', type=str, help="Mail body", required=True) args = parser.parse_args() msg = EmailMessage() msg.set_content(args.body) msg['Subject'] = args.title msg['From'] = args.from_address msg['To'] = args.to_address service = smtplib.SMTP_SSL(args.smtp_host) # service.starttls() service.login(args.smtp_user, args.smtp_pass) service.send_message(msg) service.close()

The article Python Snippet: Send Email via SMTP first appeared on alexn.org.

Shell Snippet: Encrypt Files with AES256

Alexandru Nedelcu — Tue, 18 Feb 2020 00:00:00 +0000

#!/usr/bin/env bash # # Quickly encrypt a message or a file for sending to # your colleagues over unsecure messengers (Slack, etc)... # # echo "secret message" | encrypt # # Or to encrypt a file: # # encrypt ./path/to/file # # ----------------------------------------------------------------------------- FILEPATH="$1" if [ -z "$FILEPATH" ]; then BASE64=$(openssl enc -A -aes256 -base64 -md md5) echo echo \# Decrypt this message by running: echo \# ================================ echo echo echo \""$BASE64"\" \| openssl enc -d -aes256 -base64 -md md5 -A echo echo \# ================================ echo elif [ -f "$FILEPATH" ]; then RPATH=$(realpath --relative-base="$HOME" "$FILEPATH") BASE64=$(openssl enc -A -aes256 -base64 -md md5 < "$FILEPATH") echo echo \# Decrypt this file by running: echo \# ================================ echo echo echo \""$BASE64"\" \| openssl enc -d -aes256 -base64 -md md5 -A \> \""~/$RPATH"\" echo echo \# ================================ echo else echo >&2 echo "ERROR - file not found: $FILEPATH" >&2 echo >&2 exit 1 fi

The article Shell Snippet: Encrypt Files with AES256 first appeared on alexn.org.

Haskell Snippet: Sierpinski Triangle

Alexandru Nedelcu — Fri, 29 Nov 2019 00:00:00 +0000

#!/usr/bin/env stack -- stack --resolver lts-14.14 script import Data.Bits main :: IO () main = mapM_ putStrLn lines where n = 32 :: Int line i = [ if i .&. j /= 0 then ' ' else '*' | j <- [0 .. n - 1] ] lines = [line i | i <- [0 .. n - 1]]

NOTE: the code isn’t mine, but I don’t remember from where I got it from.

The article Haskell Snippet: Sierpinski Triangle first appeared on alexn.org.

Regexp Replacement via Function in Java/Scala

Alexandru Nedelcu — Fri, 22 Nov 2019 00:00:00 +0000

Article explains how to have fine grained control over replacements when using String.replaceAll or similar in Java or Scala. The samples given are in Scala, but if you’re a Java user, the snippets can be easily translated without any issues.

The following task should be obvious to accomplish in Scala / Java, yet I’ve lost 2 hours on it, because Java’s standard library is aged and arcane.

Writing this article mostly for myself, to dump this somewhere 🙂

In Ruby if you want to replace all occurrences of a string, via a regular expression, you can use gsub:

"HelloWorld!".gsub(/(?<=[a-z0-9_])[A-Z]/, ' \0') #=> Hello World!

This API is available in Scala/Java as well:

"HelloWorld!".replaceAll("(?<=[a-z0-9_])[A-Z]", " $0") // res: String = Hello World!

However Ruby goes one step further and accepts as the replacement a function block:

"HelloWorld!".gsub(/(?<=[a-z0-9_])[A-Z]/) {|ch| ch.downcase } #=> Hello world! "Apollo 12".gsub(/\d+/) {|num| num.to_i + 1} #=> "Apollo 13"

The Java API exposed by Pattern is a little awkward, so lets see how to do this in Scala / Java:

type Index = Int type MatchGroup = String def replaceAll(regex: Pattern, input: String) (f: (Index, MatchGroup, List[MatchGroup]) => String): String = { val m = regex.matcher(input) val sb = new StringBuffer while (m.find()) { val groups = { val buffer = ListBuffer.empty[String] var i = 0 while (i < m.groupCount()) { buffer += m.group(i + 1) i += 1 } buffer.toList } val replacement = f(m.start(), m.group(), groups) m.appendReplacement(sb, Matcher.quoteReplacement(replacement)) } m.appendTail(sb) sb.toString }

What happens here is that Matcher allows you to replace all occurrences of a string with an iterator-like protocol by using:

.find: for finding the next occurrence in a while loop

.appendReplacement: which appends the remaining text after the discovery of the last match and the current one, plus the replacement that you’ve calculated for the current match

Note this requires the usage of quoteReplacement, because otherwise the logic in appendReplacement will treat certain special chars like \ and $, so this specifies that you want the replacement to be verbatim

.appendTail: to append to the final string whatever is left

We can now describe something like this:

def camelCaseToSnakeCase(input: String): String = replaceAll("[A-Z](?=[a-z0-9_])".r.pattern, input) { (i, ch, _) => (if (i > 0) "_" else "") + ch.toLowerCase } // And usage: camelCaseToSnakeCase("RebuildSubscribersCounts") //=> res: String = rebuild_subscribers_counts

Or like this:

def incrementNumbersIn(input: String): String = replaceAll("\\d+".r.pattern, input) { (_, num, _) => (num.toInt + 1).toString } // And usage: incrementNumbersIn("Apollo 12") //=> res: String = Apollo 13

Enjoy ~

The article Regexp Replacement via Function in Java/Scala first appeared on alexn.org.

Async Queue in TypeScript

Alexandru Nedelcu — Mon, 07 Oct 2019 00:00:00 +0000

type Callback = (a: A) => void; /** * Delays stuff for ensuring fairness. */ export function yieldRunLoop(): Promise { const fn: (cb: (() => void)) => void = typeof setImmediate !== 'undefined' ? setImmediate : cb => setTimeout(cb, 0) return new Promise(fn) } /** * Async queue implementation */ export class Queue { private readonly elements: A[] = [] private readonly callbacks: ([Callback, Callback])[] = [] enqueue = async (a: A) => { const cbs = this.callbacks.shift() if (cbs) { // fairness + guards against stack overflows await yieldRunLoop() const [resolve, _] = cbs resolve(a) } else { this.elements.push(a) } } dequeue = async () => { if (this.elements.length > 0) { return this.elements.shift() as A } else { let cb: [Callback, Callback] | undefined const p = new Promise((resolve, reject) => { cb = [resolve, reject] }) if (!cb) throw new Error("Promise constructor") this.callbacks.push(cb) return await p } } rejectAllActive = (e: Error) => { while (this.callbacks.length > 0) { const cbs = this.callbacks.pop() if (!cbs) continue const [_, reject] = cbs reject(e) } } } /** * Consumer implementation. * * @param workers specifies the number of workers to start in parallel * @param blockProcessFromExiting if `true` then blocks the Nodejs process from exiting * * @returns a `[promise, cancel]` tuple, where `cancel` is a function that can be used * to stop all processing and `promise` can be awaited for the completion of * all workers, workers that complete on cancellation */ export function consumeQueue(queue: Queue, workers: number, blockProcessFromExiting: boolean = false) { const Cancel = new Error("queue-cancel-all") const startWorker = async (isActive: boolean[], process: (a: A) => Promise) => { await yieldRunLoop() try { while (isActive.length > 0 && isActive[0]) { const a = await queue.dequeue() try { await process(a) } catch (e) { console.error("Error while processing queue message:", a, e) } // Fairness + protection against stack-overflow await yieldRunLoop() } } catch (e) { if (e != Cancel) throw e } } // For keeping the process alive, for as long as there is a run-loop active function startTick() { let tickID: Object function tick() { tickID = setTimeout(tick, 1000) } tick() return () => clearTimeout(tickID as any) } return (process: (a: A) => Promise) => { const isActive = [true] const cancelTick = blockProcessFromExiting ? startTick() : () => {} const cancel = () => { isActive[0] = false queue.rejectAllActive(Cancel) cancelTick() } const tasks: Promise[] = [] for (let i=0; i undefined as void) return [all, cancel] as [Promise, () => void] } }

And usage:

async function main() { console.log("Starting...") const queue = new Queue() const [promise, cancel] = consumeQueue(queue, 3, true)( async msg => { await new Promise(r => setTimeout(r, 1000)) console.log(msg) }) process.on('SIGINT', async () => { console.log("\nCancelling...\n") cancel() }) await queue.enqueue("Hello") await queue.enqueue("World!") // Requires `blockProcessFromExiting` to be `true` await promise console.log("Done!") } main().catch(console.error)

The article Async Queue in TypeScript first appeared on alexn.org.

Scala Snippet: Blocking Task

Alexandru Nedelcu — Mon, 02 Sep 2019 00:00:00 +0000

Monix Task implementation for wrapping (suspending) blocking I/O such that it can be canceled. s Imported from gist.github.com.

WARN: not sure if this code is correct.

import monix.eval._ import monix.execution.atomic.Atomic import scala.util.control.NonFatal def blocking[A](f: => A): Task[A] = Task.cancelable0 { (scheduler, cb) => // For capturing the executing thread val thread = Atomic(None : Option[Thread]) // For synchronizing cancellation, ensuring the // interrupted flag is reset, in case it is our fault val wasInterrupted = Atomic(false) // Executing on top of thread-pool scheduler.execute(new Runnable { def run() = { val th = Thread.currentThread() val update = Some(th) var started = false try { if (thread.compareAndSet(None, update)) { started = true scala.concurrent.blocking { cb.onSuccess(f) } } } catch { case e: InterruptedException => () case NonFatal(e) => cb.onError(e) } finally { // If true, then cancellation logic is guaranteed to // interrupt or to have interrupted current thread if (started && !thread.compareAndSet(update, null)) { // Waits for cancellation logic to finish while (!wasInterrupted.get) { // Thread.onSpinWait() on Java 9 Thread.`yield`() } // Clear interruption flag Thread.interrupted() } } } }) // Cancellation logic Task { thread.getAndSet(null) match { case None | null => () case Some(th) => th.interrupt() wasInterrupted.set(true) } } }

The article Scala Snippet: Blocking Task first appeared on alexn.org.

Scala's isInstanceOf is an Anti-Pattern

Alexandru Nedelcu — Sun, 11 Aug 2019 00:00:00 +0000

When you use isInstanceOf[Class] checks, that’s an anti-pattern, as Scala has a much better way of discriminating between types. Scala has implicit parameters, with which you can describe type classes.

When C# developers try Java, one primary complaint is about the lack of reification for Java’s generics and by that they mean the ability to discriminate between different type parameters, so to differentiate between List[Int] and List[String] via isInstanceOf checks. Java and Scala do type erasure so a List[String] at runtime becomes a List[Any].

Interestingly this complaint is not valid for Scala. Because of Scala’s expressiveness, you shouldn’t need to do isInstanceOf checks, unless it’s for interoperability or for awkward micro optimizations that have no place in high level code. Reification is a runtime construct and Scala solves the associated use cases by moving all of that at compile time, via implicit parameters.

Let’s do an exercise …

The Anti-Pattern

Given some piece of heavy logic, let’s say you want to describe a function that can execute a block of code, along with some finalizer, something like this:

def guarantee[R](f: => R)(finalizer: => Unit): R = try f finally finalizer

And usage:

guarantee { println("Executing!") 1 + 1 } { println("Done!") }

Then one of your colleagues comes along and tries it with Future:

import scala.concurrent.Future import scala.concurrent.ExecutionContext.Implicits.global guarantee { Future { println("Executing!") 1 + 1 } } { println("Done!") } //=> Done! //=> Executing!

Oops! This doesn’t work, so one of you might get the bright idea to do this instead:

import scala.util.control.NonFatal def guarantee[R](f: => R)(finalizer: => Unit): R = try { f match { // Anti-pattern case ref: Future[_] => ref.transform { r => finalizer; r } .asInstanceOf[R] case result => finalizer result } } catch { case NonFatal(e) => finalizer throw e }

This logic isn’t extensible — What happens when you’ll want to introduce logic for Java’s CompletableFuture, Monix’s Task, Cats-Effect’s IO and so on? Each type gets its own branch?

We’ve run into the expression problem and yet our R data type is not a tagged union, meaning that the set of possible values in that pattern match is endless.

The second problem is the default branch. We are assuming that, in case R is not a Future, then we are dealing with a side effectful function that executes synchronously. So one of your colleagues comes along and does this:

import monix.eval.Task guarantee { // Oh noes! Task { println("Executing!") 1 + 1 } } { println("Done!") } //=> Done!

Oops again! Now we’ve got a bug.

Type Classes to the rescue

Let us define a type class for discriminating between types at compile-time:

trait CanGuarantee[R] { def guarantee(f: => R)(finalizer: => Unit): R }

Now our function can look like this:

def guarantee[R: CanGuarantee](f: => R)(finalizer: => Unit): R = implicitly[CanGuarantee[R]].guarantee(f)(finalizer)

To get the behavior of the original sample, we can define the default instances in the companion object like this:

object CanGuarantee { // Future instance implicit def futureInstance[A]: CanGuarantee[Future[A]] = new CanGuarantee[Future[A]] { def guarantee(f: => Future[A])(finalizer: => Unit): Future[A] = Future(f).flatten.transform { r => finalizer r } } // Default instance implicit def syncInstance[R]: CanGuarantee[R] = new CanGuarantee[R] { def guarantee(f: => R)(finalizer: => Unit): R = try f finally finalizer } }

The upside is that now the mechanism is extensible, without modifying the original function:

final class Thunk[A](val run: () => A) object Thunk { // Extending our logic with a new data type implicit def canGuarantee[A]: CanGuarantee[Thunk[A]] = new CanGuarantee[Thunk[A]] { def guarantee(f: => Thunk[A])(finalizer: => Unit): Thunk[A] = new Thunk(() => { try f.run() finally finalizer }) } }

The problem with defaults

What happens in case you don’t define CanGuarantee[Thunk[A]]?

guarantee { new Thunk { () => println("Calculating!") 1 + 1 } } { println("Done!") }

The syncInstance that we defined above is incorrect for Thunk. This means that we can introduce silent bugs. What should happen here?

import java.util.concurrent.CompletableFuture guarantee { CompletableFuture.runAsync { () => println("Running!") } } { println("Done!") } //=> Done! //=> Running!

This is a bug waiting to happen.

Therefore one can make the case that the default instance, if it doesn’t have the intended behavior for all data types, should not exist. But you can still provide a helper for creating one:

import scala.annotation.implicitNotFound @implicitNotFound("""Cannot find implicit value for CanGuarantee[${R}]. If this value is synchronously calculated via an effectful function, then use CanGuarantee.synchronous to create one.""") trait CanGuarantee[R] { def guarantee(f: => R)(finalizer: => Unit): R } object CanGuarantee { // No longer implicit def synchronous[R]: CanGuarantee[R] = new CanGuarantee[R] { def guarantee(f: => R)(finalizer: => Unit): R = try f finally finalizer } // Future instance implicit def futureInstance[A]: CanGuarantee[Future[A]] = new CanGuarantee[Future[A]] { def guarantee(f: => Future[A])(finalizer: => Unit): Future[A] = Future(f).flatten.transform { r => finalizer r } } }

Notice the usage of Scala’s @implicitNotFound annotation for providing a nice error message.

Let’s see what happens now when we try running the CompletableFuture code again:

error: Cannot find implicit value for CanGuarantee[CompletableFuture[Void]]. If this value is synchronously calculated via an effectful function, then use CanGuarantee.synchronous to create one.

That’s better. This now goes for Unit as well:

scala> guarantee { println("Executing!") } { println("Done!") } error: Cannot find implicit value for CanGuarantee[Unit]. If this value is synchronously calculated via an effectful function, then use CanGuarantee.synchronous[Unit] to create one. guarantee { println("Executing!") } { println("Done!") } ^

If you want to work with Unit, you have to create an instance for it, but notice how the error message tells you exactly what to do:

implicit val ev = CanGuarantee.synchronous[Unit] guarantee { println("Executing!") } { println("Done!") }

In conclusion

Friends don’t let friends use isInstanceOf checks, because Scala has better ways of handling the related use cases.

Enjoy~

The article Scala's isInstanceOf is an Anti-Pattern first appeared on alexn.org.

On Bifunctor IO and Java's Checked Exceptions

Alexandru Nedelcu — Sun, 06 May 2018 00:00:00 +0000

The Bifunctor IO data type is a hot topic in the Scala community. In this article however I’m expressing my dislike for it because it shares the same problems as Java’s Checked Exceptions.

What is IO?

Normally the IO data type is expressed as:

sealed trait IO[+A] { ??? }

What this means is that IO is like a thunk, like a function with zero parameters, that upon execution will finally produce an A value, if successful. The type also signals possible side effects that might happen upon execution, but since it behaves like a function (that hasn’t been executed yet), when you are given an IO value you can consider it as being pure (and the function producing it has referential transparency). This means that IO can be used to describe pure computations.

Modern IO implementations for JVM are also capable of describing asynchronous processes, therefore you can also think of IO as being:

opaque type IO[+A] = () => Future[A]

If we had opaque types this would work well ;-)

Available implementations are:

cats.effect.IO

monix.eval.Task

scalaz.concurrent.Task (in the 7.2.x and 7.3.x series)

Some cool presentations on this subject:

The Making of an IO (ScalaIO FR, 2017)

What Referential Transparency can do for you (ScalaIO FR, 2017)

Monix Task: Lazy, Async and Awesome (flatMap(Oslo), 2016)

IO[+A] implements MonadError[IO, Throwable]. And if we’re talking of Cats-Effect or Monix, it also implements Sync among others.

This means that IO[+A] can terminate with error, it can terminate in a Throwable, actually reflecting the capabilities of the Java Runtime. This means that this code is legit:

import cats.effect.IO import scala.util.Random def genRandomPosInt(max: Int): IO[Int] = IO(Math.floorMod(Random.nextInt(), max))

The astute reader might notice that this isn’t a total function, as it could throw an ArithmeticException. An easy mistake to make.

What’s the Bifunctor IO?

I’m going to call this data type BIO, to differentiate it from IO above:

sealed trait BIO[E, A] { ??? }

Such a type parameterizes the error type in E. This is more or less like usage of Either to express the error, but as you shall see below, they aren’t exactly equivalent:

opaque type BIO[E, A] = IO[Either[E, A]]

Or in case you’re throwing EitherT in the mix to make that less awkward:

import cats.data.EitherT type BIO[E, A] = EitherT[IO, E, A]

Exposing the error type would allow one to be very explicit at compile time about the error:

def openFile(file: File): BIO[FileNotFoundException, BufferedReader] = // Made up API BIO.delayE { try Right(new BufferedReader(new FileReader(file))) catch { case e: FileNotFoundException => Left(e) } } def genRandomInt: BIO[Nothing, Int] = BIO.delayE(Right(Random.nextInt()))

You can see in the first function that we are very explicit about FileNotFoundException being an error that could happen, instructing readers that they should probably do error recovery.

And in the second function we could use Nothing as the error type to signal that this operation can in fact produce no error (not really, but let’s go with it 😉).

Available implementations:

scalaz/ioeffect, the Scalaz 8 IO, available as a backport for Scalaz 7, by John A. De Goes and other Scalaz contributors

cats-bio by Luka Jacobowitz, inspired by Scalaz 8’s IO, he took Cats-Effect’s IO and changed Throwable to E as a proof of concept

Worthy to mention is also Unexceptional IO, Luka’s precursor to his BIO implementation, inspired by Haskell’s UIO I think

Some articles on this subject:

No More Transformers: High-Performance Effects in Scalaz 8, by John A. De Goes

Rethinking MonadError, by Luka Jacobowitz

The premise of these articles is that:

our type system should stop us from being able to write nonsensical error handling code and give us a way to show anyone reading the code that we’ve already handled errors

the performance of EitherT is bad and usage more awkward

Naturally, I disagree with the first assertion and I don’t think the second assertion is a problem 😀

The Problems of Java’s Checked Exceptions

While I think that the Bifunctor IO is a cool implementation, that’s pretty useful for certain people, or certain use cases, I believe that ultimately it’s not a good default implementation, as it shares the same problems as Java’s Checked Exceptions. Or in other words, it’s ignoring decades of experience with exceptions, since their introduction in LISP and then in C++, Java, C# and other mainstream languages.

The web is littered with articles on why checked exceptions were a bad idea and many of those reasons are also very relevant for an IO[E, A]. Here’s just two such interesting articles:

Checked exceptions I love you, but you have to go

The Trouble with Checked Exceptions, an interview with Anders Hejlsberg

But let me explain in more detail …

1. Composition Destroys Specific Error Types

Let’s go with a more serious example:

import java.io._ def openFile(file: File): BIO[FileNotFoundException, BufferedReader] = // Made up API BIO.delayE { try Right(new BufferedReader(new FileReader(file))) catch { case e: FileNotFoundException => Left(e) } } def readLine(in: BufferedReader): BIO[IOException, String] = BIO.delayE { try Right(in.readLine()) catch { case e: IOException => Left(e) } finally in.close() } def convertToNumber(nr: String): BIO[NumberFormatException, Long] = BIO.delayE { try Right(nr.toLong) catch { case e: NumberFormatException => Left(e) } }

What would be the type of a composition of multiple IO values like this?

for { buffer <- openFile(file) line <- readLine(buffer) num <- convertToNumber(line) } yield num

That’s right, you’ll have a Throwable on your hands. And this is assuming that we’ve got a flatMap that widens the result to the most specific super-type, otherwise you’ll have to take care of conversions manually, at each step. Also note that our usage of Throwable is irrelevant for the problem at hand. You could come up with your own error type, but Throwable is actually more practical, because we can simply cast it.

So assuming a flatMap that doesn’t automatically widen the error type of the result, what you’ll have to deal with is actually worse:

for { buffer <- openFile(file).leftWiden[Throwable] line <- readLine(buffer).leftWiden[Throwable] num <- convertToNumber(line).leftWiden[Throwable] } yield num

Not sure how people feel about this, but to me this isn’t an improvement over the status quo, far from it, this is just noise polluting the code. And before you say anything in its defence, make sure the argument doesn’t also apply to Java and everything you dislike about it 😉

2. You Don’t Recover From Errors Often

Imagine a piece of code like this:

for { r1 <- op1 r2 <- op2 r3 <- op3 } yield r1 + r2 + r3

So we are executing 3 operations in sequence and each of them can fail, we don’t know which or how.

Does it matter? Most of the time, you don’t care. Most of the time it is irrelevant. Most of the time you can’t even recover until later.

Due to this uncertainty about which operations trigger errors and which don’t, the premise of a Bifunctor IO is that we’re forced to do attempt (error recovery) everywhere, but that is not a correct premise. The way exceptions work and why they were introduced in LISP and later in C++, is that you only catch exceptions at the point were you can actually do something about it, otherwise it’s fine to live in blissful ignorance.

Empirical evidence suggests that most checked exceptions in Java are either ignored or re-thrown, forcing people to write catch blocks that are meaningless and even error prone.

You can even find some studies on handling of checked exceptions in Java projects, although I’m unsure about how good they are. For example there’s Analysis of Exception Handling Patterns in Java Projects, which states that:

“Results of this study indicate that most programmers ignore checked exceptions and leave them unnoticed. Additionally, it is observed that classes higher in the exception class hierarchy are more frequently used as compared to specific exception subclasses.”

Consider that in case of a web server the recovery might be something as simple as showing the user an HTTP 500 status. HTTP 500 statuses are a problem, but only if they happen and when they start to show up, you can then go back and fix what needs to be fixed.

Also remember the FileNotFoundException we mentioned above? Well, in most cases there’s not much you can do about it. It’s not like you’ve got much choice in the knowledge that the file is missing, most of the time the important bit being that an error, any error, happened.

To quote Anders Hejlsberg, the original designer of C#:

“It is funny how people think that the important thing about exceptions is handling them. That is not the important thing about exceptions. In a well-written application there’s a ratio of ten to one, in my opinion, of try/finally to try/catch. Or in C#, using statements, which are like try/finally.”

In other words the most important part of exceptions are the finalizers, recovery being less frequent.

3. The Error Type is an Encapsulation Leak

Lets say that we have this function:

def foo(param: A): BIO[FileNotFoundException, B]

By saying that it can end with a FileNotFoundException, we are instructing all callers, at all call sites, to handle this error as part of the exposed API.

It’s pretty obvious that FileNotFoundException can happen due to trying to open a file on disk that is missing. It’s a very specific error, isn’t it, the kind of error we’re supposed to like if we’re fans of EitherT or of the Bifunctor IO.

Well, what happens if we change foo to make an HTTP request instead, or maybe we turn it into something that reads a memory location. Now all of a sudden FileNotFoundException is no longer a possibility.

def foo(param: A): BIO[Unit, B]

This then bubbles down to all call sites, effectively breaking backwards compatibility, so all that depend on your foo will have to upgrade and recompile. And as the author of foo you’ll be faced with two choices:

break compatibility

keep lying to your users that foo can end with a FileNotFoundException and thus leave them with unreachable code - which is something that some Java libraries are known to have done

NOTE: there are cases in which you want to break binary compatibility in case the error type changes. That is precisely the use case for which the Bifunctor IO or EitherT are recommended.

4. It Pushes Complexity to the User

On utility I deeply understand the need to parameterize all things. But the question is, what else could we parameterize and why aren’t we doing it?

we could have a type parameter that says whether the operation is blocking-IO bound, or CPU bound and in this way we could avoid running an IO that’s CPU-bound on a thread-pool meant for blocking I/O or vice-versa

we could add a type parameter for the execution model — is it synchronous or asynchronous?

we could describe the side effect with a type parameter — i.e. is it doing PostgreSQL queries, or ElasticSearch inserts and in this way the type becomes more transparent and you could come up with rules for what’s safe to execute in parallel or what not

add your own pet peeve …

I’m fairly sure that people have attempted these. I’m fairly sure that there might even be libraries around that are useful in certain specific instances. But they are not mainstream.

We aren’t doing it because adding type parameters to the types we are using leads to the death of the compiler, not to mention our own understanding of the types involved, plus usage becomes that much harder, because by introducing type parameters, values with different type arguments no longer compose without explicit conversion / widening, pushing a lot of complexity to the user.

This is why EitherT is cool, even with all of its problems. It’s cool because it can be bolted on, when you need it, adding that complexity only when necessary.

The Bifunctor IO[E, A] looks cool, but what happens downstream to the types using it? Monix’s Iterant for example is Iterant[F[_], A]. Should it be Iterant[F[_], E, A]? Or maybe Iterant[F[Throwable, _], A]? Or Iterant[F[_, _], E, A]?

If I parameterize the error in Iterant, how could it keep on working with the current IO that doesn’t have a E parameter? And if Iterant works with IO[Throwable, _], then what’s the point of IO[E, A] anyway?

Note that having multiple type parameters is a problem in Haskell too. Martin Odersky already expressed his dislike for type classes of multiple type parameters, such as MonadError and it’s pretty telling that type classes with multiple type parameters are not part of standard Haskell.

5. The Bifunctor IO Doesn’t Reflect the Runtime

I gave this piece of code above and I’m fairly sure that you missed the bug in it:

def readLine(in: BufferedReader): BIO[IOException, String] = BIO.delayE { try Right(in.readLine()) catch { case e: IOException => Left(e) } finally in.close() }

The bug is that in.close() can throw exceptions as well. Actually on top of the JVM even pure, total functions can throw InterruptedException for example.

So what happens next?

Well the Bifunctor IO cannot represent just any Throwable. By making E generic, it means that handling of Throwable is out. So at this point there are about 3 possibilities:

crash the process, which would be the default, naive implementation

your thread crashes without making a sound, logging to a stderr that gets redirected to /dev/null

use something like a custom Java Thread.UncaughtExceptionHandler, or Scalaz’s specific “fiber” error reporter to report such errors somewhere

Also the astute reader should notice that by replacing the MonadError handling and recovery by a simple reporter there’s no way to do back-pressured retries. The nature of bugs is that many bugs are non-deterministic. Maybe you’re doing an HTTP request and you’re expecting a number in return, but it gives you an unexpected response - maybe it has a maximum limit of concurrent connections or something.

When making requests to web services, wouldn’t it be better to give them some slack? Wouldn’t it be better to do retries with exponential backoff a couple of times before crashing? Or maybe use utilities such as TaskCircuitBreaker? Of course it is. And in the environments I worked on, such instances are very frequent and the processes have to be really resilient to failure and resiliency is built-in only when having the assumption that everything can fail for unknown reasons.

In the grand scheme of things, the reason for why this is a huge problem is because IO should reflect the runtime, because IO effectively replaces Java’s call-stack. But the Bifunctor IO no longer does.

In the words of Daniel Spiewak, who initiated the Cats-Effect project:

“ The JVM runtime is typed to a first order. Which happens to be exactly what the type parameter of IO reflects. I'm not talking about code in general, just IO. IO is the runtime, the runtime is IO. ”

Source

“ The whole purpose of IO as an abstraction is to control the runtime. If you pretend that the runtime has a property which it does not, then that control is weakened and can be corrupted (in this case, by uncontrolled crashes). ”

Source

“ IO needs to reflect and describe the capabilities of the runtime, for good or for bad. All it takes is an "innocent" throw to turn it all into a lie, and you can't prevent that. ”

Source

I agree with that and it shows which developers worked a lot in dynamic environments, this great divide being between those that think types can prove correctness in all cases and those that don’t.

If you’re in the former camp, I think Hillel Wayne is eager to prove you wrong 😉

IO Cannot Be an Alias of the Bifunctor IO

You might be temped to say that:

type IO[A] = BIO[Throwable, A]

This is not true and it gave birth to, what I like to call, the great “No True Functor” debate and fallacy 😜

But details about it would take another article to explain.

So it’s enough to say that cats.effect.IO and monix.eval.Task has got you covered in all cases, whereas a Bifunctor IO needs to pretend that developers on top of the JVM can work only with total functions, on top of an environment that actively proves you wrong, thus applying the “let it crash” philosophy on top of a runtime that makes this really expensive to do so (i.e. the JVM is not Erlang).

This is another great divide in mentality, although I can see the merits of the arguments on the other side. In such cases it’s relevant by what kind of problems you got burned or not in the past I guess.

Final Words

I am not saying that the Bifunctor IO[E, A] is not useful.

I’m pretty sure it will prove useful for some use-cases, the same kind of use-cases for which EitherT is useful, except with a less orthogonal design. Well you gain some performance in that process, although when you’re using EitherT it’s debatable whether it matters for those particular use cases.

What I am saying is that:

let’s not ignore the two decades of experience we had with Java’s checked exceptions, preceded by another two decades of experience with exceptions in other languages

EitherT is useful because it can be bolted on when the need arises, or otherwise it can be totally ignored by people like myself, so let’s not throw the baby with the bath water

I do think that IO[E, A] will be a great addition to the ecosystem, as an option over the current status quo. Scala is a great environment.

That’s all.

The article On Bifunctor IO and Java's Checked Exceptions first appeared on alexn.org.

In Defense of OOFP

Alexandru Nedelcu — Mon, 12 Feb 2018 00:00:00 +0000

This article is about OOP hierarchies, in comparisson with type classes, using as example Scala’s standard collections, which are involved in a recent debate around their redesign.

1. Background

Scala is a hybrid language, being at the same time its greatest strength and weakness.

The collection-strawman represents a redesign of Scala’s standard collections, planned to be shipped in Scala 2.13, as part of a Scala Center proposal.

There are at the moment of writing two blog posts explaining the upcoming changes on Scala’s blog:

Tribulations of CanBuildFrom

Let Them Be Lazy!

Scala 2.13 Collections Rework

On Performance of the New Collections

It’s now also receiving some criticism. This article represents my 2¢ on why I like this redesign and on why criticism of an OOP hierarchy are not warranted, even if you love FP.

1.1. My Road to Scala

Like many others in the Scala community, I was attracted to Scala because it seemed to be a better, more expressive language, with stricter static type checks than Java or the other languages I worked with. Coming from Python at that time, Scala seemed like a breath of fresh air.

Yes, I came to Scala in 2012 looking for a better OOP language and what I got was a great combination of Smalltalk-esque OOP features, mixed with an actual FP culturre. I have an article about what I liked about it written back then and to be honest, not much has changed.

But what really sold me were the standard collections. In spite of their internal complexity and problems being leaked at the call sites, it was the first time I felt my needs for standard data structures were met and even exceeded.

2. Criticism

There’s plenty to complain about in the current implementation and here I’m doing my best to summarize it:

2.1. CanBuildFrom

Checkout this classic StackOverflow issue:

Is the Scala 2.8 collections library a case of “the longest suicide note in history”?

In it the author rightfully complains that this method signature for List is scary and he doesn’t understand it:

def map[B, That](f: A => B)(implicit bf: CanBuildFrom[Repr, B, That]): That

The Scala core developers first tried to hide this signature, so if you’ll take a look at the current ScalaDoc for List.map, you won’t see it. But that’s only a documentation trick, because the actual source-code (and your IDE) tells a different story, see List.scala#280.

In essence CanBuildFrom is a clever abstraction, as Martin Odersky himself explains, however at this point we can all agree that its complexity is not justified. Not only because it is difficult to understand, but because it has historically created issues for people at the call-sites.

Here’s what Coda Hale was writing back in 2011:

“ Replacing a `scala.collection.mutable.HashMap` with a `java.util.HashMap` in a wrapper produced an order-of-magnitude performance benefit for one of these loops. Again, this led to some heinous code as any of its methods which took a `Builder` or `CanBuildFrom` would immediately land us with a mutable.HashMap. (We ended up using explicit external iterators and a while-loop, too.)
...
The number of concepts I had to explain to new members of our team for even the simplest usage of a collection was surprising: implicit parameters, builder typeclasses, "operator overloading", return type inference, etc. etc. ”

Source

But if you’ll take a look at List.scala in collection-strawman, that signature now looks like this:

def map[B](f: A => B): List[B]

There, that brings justice to everybody that complained for all these years.

TL;DR: CanBuildFrom was a clever solution to an exaggerated problem, but you’ll no longer have to worry about it. Beware of clever solutions to problems you don’t have!

2.2. Complex Hierarchy

This is an awesome infographic, showing how complex the implementation is for List, by Rob Norris:

This is indeed a disaster, there’s no nicer way of saying it.

And this is List, after the collection-strawman refactoring:

(Again credit goes to Rob Norris)

This is indeed vastly simplified and actually understandable.

Some questions are left to be answered. Just by looking at this info-graphic I wonder in what capacity is there still code sharing between the immutable collections and the mutable versions, or between the lazy and the strict - ideally there should be no code sharing between mutable or immutable, between lazy and strict and I hope they’ve been extra careful this time around.

TL;DR: a complex OOP hierarchy, as clever as it may be, brings with it complexity beyond reason and is counter-productive for subtype polymorphism!

2.3. Seq or Other Super-types

Daniel Spiewak made a good attempt at a proposal in 2015.

Here are some recent impressions, as a short introduction to this criticism:

“ You always care about the asymptotic properties of your data structures. So there is never a case where Seq makes sense, even as a concept! ”

Source

The example that Daniel gives in his proposal is this:

def foo(is: Seq[Int]): MyThingy = { val t = new MyThingy for (idx <- 0 until is.length) { t.accumulate(is(idx)) } t }

“If I showed you this code and asked you what the big-O complexity of it was, could you tell me the answer? No, you cannot, because it depends on the runtime type of is!”

This is a very good example showing a very frequent booby trap for beginners that aren’t paying attention to their data structures.

He goes on saying:

“Choosing a collection is a very deliberate and intentional thing. We want to select our collection to match the complexity bounds of our problem. Encouraging (and even enabling!) users to work in terms of generic types that throw away this information is wrong, and very very much an anti-pattern.”

I very much respect this opinion and I know from where he’s coming from, however I disagree with it.

Choosing a collection type means committing to an implementation, it means specializing, which is a bad idea in many cases. And arguing against this is also like arguing against Haskell’s Traversable (cats.Traverse).

N.B. we are not arguing against the merits of Haskell’s Traversable type class and how that compares with Iterable. More on that below.

TL;DR: OOP interfaces might expose leaky operations that are booby traps (e.g. Seq indexing), but this is not specific to OOP interfaces, vigilance in design is always needed!

2.4. Not Using Type-classes

This is actually a separate argument that does have merit:

“ I'll stick to scalaz's separation of functionality and implementation through typeclasses... it's much easier to understand, faster, and easier to extend. ”

Sam Halliday

“ You do not need a top-down hierarchy of collections. They're ready to break back compatibility, so why not do it right like Spiewak's suggestion 3 years ago? `Builder` and `IterableOnce` are symptoms of a deeper problem: poor design choices. ”

Emily Pillmore

The argument for those amongst you not familiar with it is that type-classes can yield a better design for the standard collections, instead of an OOP hierarchy of interfaces.

To re-use the example above:

import cats.Foldable def foo[F[_] : Foldable](list: F[Int]): MyThingy = implicitly[Foldable[F]].foldLeft(list, new MyThingy)(_ accumulate _)

Ah, now we are getting somewhere. But the astute reader should notice at this point that this means exposure of an extra F[_] type parameter that you don’t actually need with OOP — well, OK, this syntax heaviness is an artifact of the Scala language, as in Haskell this wouldn’t be an issue.

Also, not clearly visible here is that type-classes such as Foldable or Traversable, while more generic are also strictly less capable than Iterable. Yes, that’s because of Iterator’s side effecting, but highly efficient and flexible protocol. More on that below.

TL;DR: type-classes are nice, playing well with parametric polymorphism, but in Scala the syntax is heavier than when using OOP, although this isn’t an issue with Haskell

3. OOP vs Constrained Parametric Polymorphism (Type-classes)

Lets imagine a function that sums up numbers:

import scala.math.Numeric def sum[A](list: Seq[A])(implicit A: Numeric[A]): A = list.foldLeft(A.zero)(A.plus)

Note we are already using Numeric, which is a type class exposed by Scala’s standard library. And it’s actually not the best type class we could use, as what we’d need here is a Monoid, but ignore that.

Meet Haskell’s Data.Foldable, also described in cats.Foldable.

With it we can fold arbitrary data-structures, e.g. we could sum up numbers:

import cats.Foldable import scala.math.Numeric def sum[F[_], A](list: F[A]) (implicit F: Foldable[F], A: Numeric[A]): A = { F.foldLeft(list, A.zero)(A.plus) }

(N.B. using F and A as the type names and as the name of the implicit parameters is nothing special, just a convention)

Is this better?

PRO: describing foldLeft as a method on F[_] is no longer required, which makes F[_] data types more generic, more reusable; not very clear in this case, but if you’re unfamiliar with type-class based design, trust me on this one

CON: this method makes F[_] clearly visible, exposing it at the type system level - this is no longer subtyping, this is no longer the Liskov substitution principle, this is parametric polymorphism and it moves the dispatch cost at compile time, with both the good and the bad

TL;DR: With Scala you can actually pick and choose the best approach - this can lead to choice paralysis however and lots of drama!

4. Case Study: Monix Iterant

At Scala World I had a talk titled A Tale of Two Monix Streams in which I described the design of the upcoming monix.tail.Iterant.

It’s a wonderful data structure for modelling streams that makes use of type-classes defined in cats-effect for being able to describe asynchronous computations. In the presentation I describe how I made use of type-classes, with actual restrictions placed on the operations themselves.

If you’re a beginner in FP design via type-classes, I highly recommend the second part of the presentation.

Here’s the gist - suppose we want a pure Iterator that is also capable of deferring the evaluation to a given F[_] (e.g. monix.eval.Task, cats.effect.IO), such that it’s also capable of streaming events from asynchronous sources, we could describe it like this:

sealed trait Iterant[F[_], A] case class Next[F[_], A](first: A, rest: F[Iterant[F, A]], stop: F[Unit]) extends Iterant[F, A] case class Suspend[F[_], A](rest: F[Iterant[F, A]], stop: F[Unit]) extends Iterant[F, A] case class Halt[F[_], A](error: Option[Throwable]) extends ITerant[F, A]

This is a data structure that resembles List, but that:

defers evaluation to an F[_] data type that’s capable of suspending side effects, but note that the actual type class restrictions are not defined on the data structure itself

is capable of asynchronous processing, in case F[_] is capable of asynchronous processing (thus also being equivalent to the async iterators in JavaScript or Ix.NET)

is lazy, if F[_] is lazy, thus being equivalent to Scala’s Stream, or Java’s Stream

is able to do resource handling, to close any open connections in case of interruption (by following stop), thus making it safer than plain iterators

4.1. Type-classes For Super Powers

So how can this simple data structure possibly have such super powers?

The answer is that it defers the hard work to any F[_] data type whose capabilities are defined via type classes.

import cats.effect.Sync def map[F[_], A](fa: Iterant[F, A])(f: A => B) (implicit F: Sync[F]): Iterant[F, B] = { def loop(current: Iterant[F, A]): Iterant[F, A] = current match { case Next(a, rest, stop) => Next(f(b), rest.map(loop), stop) case Suspend(rest, stop) => Suspend(rest.map(loop), stop) case Halt(e) => Halt(e) } loop(fa) }

That part about “implicit F: Sync[F]”, that’s the restriction we have for F[_], defining its capabilities.

What we need here is for F[_] to implement a map operation and thus it needs to have a Functor instance, however our map is used in fact recursively and due to Scala being a strict language, if this map were to be strictly evaluated, then we’d end up with a stack overflow. This is why we require cats.effect.Sync, which implies a Functor, because Sync now suspends evaluation in map by law.

You can see how this process goes: you only add the restrictions you want on the operations themselves, depending on your needs, not on the data structure, thus making the data structure more reusable.

IMPORTANT: F gets exposed via the type system at compile time. It’s should be obvious that given F and G, then an Iterant[F, A] cannot be composed with an Iterant[G, A]. So an Iterant[Task, A] (see Task) cannot be composed with an Iterant[Coeval, A] (see Coeval), or with an Iterant[IO, A] (see IO), unless you convert between these data types explicitly.

For example this should trigger a compile time error:

val tasks: Iterant[Task, Int] = ??? val ios: Iterant[IO, Int] = ??? // Nope, can't do this ;-) tasks ++ ios

TL;DR: constrained parametric polymorphism via type-classes can give you super powers by effectively outsourcing the processing to pluggable data types with various capabilities, the restrictions being on the operations themselves!

4.2. Liskov Substitution Principle: OOP Strikes Back

That last example should make you think - parametric polymorphism implies:

exposing F type parameters at compile time

homogeneity

Because on top of Scala we are looking for opportunities to optimize performance, due to the runtime not being optimized for laziness and IO data types (like Haskell), we want to process items in batches, where possible. For example we’d like to stream arrays, because arrays are contiguous memory blocks and if you don’t find ways to work with arrays, then you’re screwed in terms of throughput:

case class NextBatch[F[_], A]( batch: Array[A], rest: F[Iterant[F, A]], stop: F[Unit]) extends Iterant[F, A]

But why only arrays? What if we allowed Scala’s List or Vector as well? There’s no harm in that and it would still have better throughput, so might as well use Scala’s Seq:

case class NextBatch[F[_], A]( batch: Seq[A], rest: F[Iterant[F, A]], stop: F[Unit]) extends Iterant[F, A]

But wait, because somebody told us that OOP sucks or that the standard collections should not have a hierarchy, lets use type parameters, like we did with F:

case class NextBatch[F[_], Seq[_], A]( batch: Seq[A], rest: F[Iterant[F, A]], stop: F[Unit]) extends Iterant[F, A]

Oh wait, this doesn’t work, unless we’d expose Seq[_] in Iterant as well:

sealed trait Iterant[F[_], Seq[_], A] case class NextBatch[F[_], Seq[_], A]( batch: Seq[A], rest: F[Iterant[F, A]], stop: F[Unit]) extends Iterant[F, Seq, A]

But this sucks, not only due to the types getting more complicated (your Scala compiler is giving you ugly looks right about now), but also because you can’t have heterogeneity:

val node1 = NextBatch(Array(1,2,3), IO(Halt(None)), IO.unit) // Ooops, not compatible, List is not Array, will trigger error ;-) val node2 = NextBatch(List(1,2,3), IO(node1), IO.unit)

No, screw that, let’s backtrack - at this point we need the type class restriction to be on the data structure itself:

case class NextBatch[F[_], Seq[_], A]( batch: Seq[A], rest: F[Iterant[F, A]], stop: F[Unit]) (implicit Seq: Foldable[Seq]) extends Iterant[F, A]

That’s better, right?

Wrong - if you’re passing that type-class instance around, that’s effectively a vtable so congratulations, you have an OOP encoding with extra garbage.

TL;DR: with constrained parametric polymorphism you either expose the data type as a type parameter, making types more complicated and saying goodbye to heterogeneity, or you end up with a shitty OOP substitute!

4.3. Iterator vs Foldable and Traverse

Many people are under the impression that you can substitute the classic Iterator with the Foldable and Traverse type-classes. This is understandable, because these type classes are really potent, really generic, great actually.

Here’s Phil Freeman’s opinion, the creator of the PureScript language:

“ ES2018 will include a whole new language feature in order to implement one instantiation of “traverse”. I’ll say it again - JS won’t be fixed by adding more things. 2ality.com/2017/02/ecmascript-2018.html ”

Source

But that is simply false.

Here’s the big difference: Iterator allows you to pause the processing, until the current iteration cycle has finished, before processing the next item, being a pull-based protocol with the user being completely in charge. The user being responsible for advancing the cursor to the next item, whenever he wants, in a destructive manner is also what makes it error prone, but it’s flexible nonetheless.

So for example, how do you think this works?

val ios: Iterable[IO[Int]] = ??? ios.foldLeft(IO(0)) { (acc, elem) => for (a <- acc; e <- elem) yield a + e }

The short answer - it works, but if that stream is too big, it will blow up your process because there’s no back-pressure in that fold. But I digress.

Iterating over arrays in our NextBatch might prove tricky but is doable, for example in a mapEval implementation that needs to pop an item out of our Array, process that in the F[_] monadic context, then continue from where it left off:

def mapEval[F[_], A](fa: Iterant[F, A])(f: A => F[B]) (implicit F: Sync[F]): Iterant[F, B] = { def loop(index: Int)(fa: Iterant[F, A]): Iterant[F, B] = fa match { case NextBatch(batch, rest, stop) => val eval = if (index >= batch.length) rest.map(loop(0)) else f(batch(index)).map { a => Next(a, F.pure(loop(index + 1)(fa)), stop) } Suspend(eval, stop) case _ => ??? } loop(0)(fa) }

Well, Array can be indexed, but that indexing would be death for List, isn’t it?

Turns out we have a perfectly capable abstraction for iterating over arrays, lists, vectors or what have you, the classic Iterator:

case class NextCursor[F[_], A]( cursor: Iterator[A], rest: F[Iterant[F, A]], stop: F[Unit]) extends Iterant[F, A]

And now we can express this:

def mapEval[F[_], A](fa: Iterant[F, A])(f: A => F[B]) (implicit F: Sync[F]): Iterant[F, B] = { def loop(fa: Iterant[F, A]): Iterant[F, B] = fa match { case NextBatch(batch, rest, stop) => loop(NextCursor(batch.iterator(), rest, stop)) case NextCursor(cursor, rest, stop) => val eval = if (cursor.hasNext) rest.map(loop) else f(batch.next()).map { a => Next(a, F.pure(loop(fa)), stop) } Suspend(eval, stop) case _ => ??? } // Suspends execution, because NextCursor is side-effectful ;-) Suspend(F.delay(loop(fa)), F.unit) }

Can you do that with Foldable / Traverse? No, you can’t!

If a tree falls in a forest and no one is around to hear it, is that a side effect?

The Iterator interface relies on side effects and is thus incompatible with functional programming. However if its mutation is properly suspended and encapsulated, such that referential transparency is preserved, then it really doesn’t matter 😉

To work with type-classes and pure functions, our first intuition should be something like this:

trait IteratorTypeClass[F[_]] { def next[A](fa: F[A]): Option[(A, F[A])] }

This is a variation on the state monad, because yes, we’ve got state to evolve.

Here’s the performance characteristics of such a decomposition for the head and tail operations, needed for this type class:

Vector, has “effectively constant time”, which is worse than List

Queue has “amortized constant time”, which is worse than List

Array has linear O(N) time for tail, aka you can change your job now

Yes, Iterator is efficient for all of them, so it wins by a wide margin. The above is actually horrible and why a type class like that isn’t popular, because it relies on List’s encoding to be efficient 😉

So that’s not it. The pure type-class equivalent is actually this:

trait IteratorTypeClass[F[_]] { type Pointer[T] def start[A](fa: F[A]): Pointer[A] def next[A](fa: F[A], cursor: Pointer[A]): Option[(A, Pointer[A])] } // Sample instance for Array object ArrayIterator extends IteratorTypeClass[Array] { type Pointer[T] = Int def start[A](fa: Array[A]) = 0 def next[A](fa: Array[A], cursor: Int) = if (cursor >= fa.length) None else Some((fa(cursor), cursor + 1)) }

But this will leak implementation details - not bad for our Array instance, but what if we had some sort of self-balancing tree which you then changed to a HAMT. In that case our Pointer would be some sort of node with links to its neighbors, so is it wise exposing it like that? How is Haskell’s binary compatibility anyway? Does it even support dynamic linking?

TL;DR: Iterator actually beats pure, type-class based solution in performance, flexibility or encapsulation! It’s also lower-level, impure and error prone, but sorely needed sometimes.

5. Seq My Love

I actually like Seq and have never regretted using it.

Not everybody agrees and as mentioned in section 2.3., there are people advising against its usage:

“ You always care about the asymptotic properties of your data structures. So there is never a case where Seq makes sense, even as a concept! ”

Daniel Spiewak

Here’s why I disagree …

5.1. Seq on Return

Returning Seq from a function gives you the freedom to change the implementation and this works well because:

you preserve backwards binary compatibility

users might only care to traverse it / fold it, which should be an O(n) operation for any sequence

Real use-case from Monix, the Observable.bufferSliding operator:

/** Returns an observable that emits buffers of items it collects from * the source observable. The resulting observable emits buffers * every `skip` items, each containing `count` items. * * If the source observable completes, then the current buffer gets * signaled downstream. If the source triggers an error then the * current buffer is being dropped and the error gets propagated * immediately. * * For `count` and `skip` there are 3 possibilities: * * 1. in case `skip == count`, then there are no items dropped and * no overlap, the call being equivalent to `buffer(count)` * 1. in case `skip < count`, then overlap between buffers * happens, with the number of elements being repeated being * `count - skip` * 1. in case `skip > count`, then `skip - count` elements start * getting dropped between windows * * @param count the maximum size of each buffer before it should * be emitted * @param skip how many items emitted by the source observable should * be skipped before starting a new buffer. Note that when * skip and count are equal, this is the same operation as * `buffer(count)` */ final def bufferSliding(count: Int, skip: Int): Observable[Seq[A]] = liftByOperator(new BufferSlidingOperator(count, skip))

Facts:

current version (3.0.0-M3) uses an Array as the internal buffer, emitting arrays wrapped in Seq

version 2.0.0 used a ListBuffer, thus emitting List chunks

The change from List to Array was done:

without breaking backwards compatibility

with no consequence to the user, since these chunks are meant to be small and manageable, so all the user cares about is being able to traverse or fold them or to convert them into something else

This is the infamous “program to an interface, not to an implementation” mantra that OOP developers like to repeat so much. You won’t actually need this capability that often, but when you do, it’s golden.

Also note that when using a super-type such as Seq in an interface, due to the covariance of return types, you’re always allowed to override with a more specific type in implementing classes (e.g. to go from Seq to List), and the Liskov substitution principle still applies.

TL;DR: there are perfectly valid use-cases for Seq as the return type!

5.2. Seq on Input

The necessity for Seq is the same as for Foldable, the need to traverse a collection, aggregating its items into some final result.

Seq also implies that items come in a sequence in which order matters and repetition is allowed. For example a Set shouldn’t be a Seq, because Set does not allow repetition and has no ordering.

So here’s another real-world use-case, also from Monix, the Observable.startWith operator:

/** Creates a new Observable that emits the given elements and then * it also emits the events of the source (prepend operation). */ final def startWith[B >: A](elems: Seq[B]): Observable[B] = Observable.fromIterable(elems) ++ self

Here we might have used an Iterator, however the result wouldn’t have been pure and we might have used an Iterable, but as the author of this operator I felt that the input being a sequence is important for getting back a predictable result - after all, that ordering is important is implied by what this operator does.

And note that all the implementation does is to care about traversal, via an Iterator loop.

TL;DR: yes, there are perfectly valid use-cases for Seq as the input!

6. Conclusion

Here’s a summary:

2.1. — CanBuildFrom was a clever solution to an exagerated problem, but you’ll no longer have to worry about it

2.2. — a complex OOP hierarchy, as clever as it may be, brings with it complexity beyond reason and is counter-productive for subtype polymorphism

2.3. — OOP interfaces might expose leaky operations that are booby traps (e.g. Seq indexing), but this is not specific to OOP interfaces, vigilance in design is always needed

2.4. — type-classes are nice, playing well with parametric polymorphism, but in Scala the syntax is heavier than when using OOP, although this isn’t an issue with Haskell

3. with Scala you can actually pick and choose the best approach - this can lead to choice paralysis however and lots of drama

4.1. — constrained parametric polymorphism via type-classes can give you super powers by effectively outsourcing the processing to pluggable data types with various capabilities, the restrictions being on the operations themselves

4.2. — with constrained parametric polymorphism you either expose the data type as a type parameter, making types more complicated and saying goodbye to heterogeneity, or you end up with a shitty OOP substitute

4.3. — Iterator actually beats pure, type-class based solution in performance, flexibility or encapsulation. It’s also lower-level, impure and error prone, but sorely needed sometimes

5.1. — there are perfectly valid use-cases for Seq as the return type

5.2. — there are perfectly valid use-cases for Seq as the input

So yes, there will be people out there, such as myself, genuinely enjoying Scala’s OOP and the new collections.

Thanks for reading ️❤️

The article In Defense of OOFP first appeared on alexn.org.

Minitest: Zero Crap Scala Testing Library

Alexandru Nedelcu — Fri, 10 Nov 2017 00:00:00 +0000

Minitest is my minimal testing library that I’ve been using for developing Monix.

Raison d’être

I dislike most testing frameworks, because of bloat and of heavy DSLs trying to mimic the English language. When I created Minitest, I wasn’t satisfied with any of the available alternatives.

Then I found that SBT can do all the heavy lifting (e.g. running the tests, reporting, etc), exposing a nice sbt/test-interface that you can integrate with. All you need to do is to build your own API on top. And so I did.

NOTE: My opinions in this article disagree with the design choices of popular testing libraries. I know that libraries like ScalaTest or Specs2 are the way they are because people want them that way. And those are awesome projects, having awesome authors. They are just not for me.

Portability

The natural tendency of testing frameworks is to grow beyond all imagination in order to accommodate the various testing styles that people want, these testing frameworks also end up hard to port to new platforms - which is especially relevant in Scala due to new major versions and new targets released all the time (e.g. Scala.js, Scala Native, Dotty).

In 2014 I started to work on Monix and during that time Scala.js was also born.

This awesome Scala compiler that targets JavaScript was really fresh back then and I wanted to target it, however none of the testing libraries (e.g. ScalaTest) were supporting it yet, except for µTest. µTest seemed fine, but it had problems when displaying error messages, plus its DSL was and still is weird.

For some reason I don’t like magic in my tests — and by magic I mean expressions or statements that I don’t immediately understand. That, and I wanted an easy transition in Monix from ScalaTest’s FunSuite, which is what I was using.

Say No to DSLs

I do not want to write x shouldBe greaterThan(y) or other such nonsense. I do not have the memory for that, I always forget the API and the IDE doesn’t help much due to the implicit conversions going on.

Unit tests might be business driven, but so is software in general, nothing makes tests a special snowflake to warrant the abuse of the programming language to make it look like English.

Yes, there are advantages to such a DSL. For example if you express the above with a simple assert, then you won’t get a meaningful error message back, depending on whether the implementation does or doesn’t do macros for assert, but if it does, then that’s a whole other can of worms:

assert(x > y)

Fact: most assertions that you need to do in testing are equality tests and for that rare moment in which you need inequality tests, you can simply add a custom error message:

assert(x > y, s"$x > $y")

Yes, it’s repetitive, but I don’t care, because this is such a rare event that I don’t want to optimize it with a special DSL or even with macros.

Another thing that I absolutely hate are tests marked with English words like “it”, forcing you to phrase the test’s description in a certain way. I frequently end up with “sentences” that makes no sense:

it("left identity") { // ... }

This tendency actually stems from Java OOP, with its “kingdom of nouns”, the idea being that the things you’re testing are all nouns that interact with the world, aka objects. Well, I’m not testing just objects, so it is a bad trend.

And the idea that business folks might be writing tests, hell no, that almost never happens and if they are inclined to do that (like once in a million), then they can just learn programming. It’s not like an English-like DSL is any closer to natural language.

Minimal Implementation, Less is More

Because the implementation is minimal, there’s nothing that I can’t fix in it, there’s nothing that I can’t implement should I need anything.

It’s also easy to port to new targets. I intend to port it to Scala Native as soon as it is available for Scala 2.12 (at the moment of writing, it isn’t).

In fact, if you want to build your own testing framework, Minitest can serve as a sample ;-)

Hypothesis

All you need is the ability to express:

synchronous tests, returning Unit (or an equivalent, as I ended up doing in order to avoid Scala’s annoying implicit conversion)

asynchronous tests, returning Future

the ability to setup an environment before every test, then tear it down after each test

All the asserts that you need:

assert(boolean, string?): general purpose assertion for any condition

assertEquals(received, expected): for equality testing with a nice error message

intercept: for testing that exceptions are thrown

fail(reason?): fails the current test

ignore(reason?): ignores the current test

cancel(reason?): cancels the current test

What you don’t need:

nesting in tests

an English-like DSL

a purely functional base API

Tutorial

Test suites MUST BE objects, not classes. To create a simple test suite, it could inherit from SimpleTestSuite. Here’s a simple test:

import minitest.SimpleTestSuite object MySimpleSuite extends SimpleTestSuite { test("should be") { assertEquals(2, 1 + 1) } test("should not be") { assert(1 + 1 != 3) } test("should throw") { class DummyException extends RuntimeException("DUMMY") def test(): String = throw new DummyException intercept[DummyException] { test() } } test("test result of") { assertResult("hello world") { "hello" + " " + "world" } } test("should be ignored") { if (Platform.isJS) ignore("Blocking not supported on top of JS") val r = Await.result(Future(1), Duration.Inf) assertEquals(r, 1) } }

In case you want to setup an environment for each test and need setup and tearDown semantics, you could inherit from TestSuite. Then on each test definition, you’ll receive a fresh value:

import monix.execution.schedulers.TestScheduler import minitest.TestSuite object MyTestSuite extends TestSuite[TestScheduler] { def setup() = TestScheduler() def tearDown(env: TestScheduler): Unit = assert(env.state.tasks.isEmpty, "Scheduler should not have tasks left") test("simulated async") { implicit ec => val f = Future(1).map(_ + 1) ec.tick() assertEquals(f.value, Some(Success(2))) } }

Minitest supports asynchronous results in tests, just use testAsync and return a Future[Unit]:

import scala.concurrent.ExecutionContext.Implicits.global object MySimpleSuite extends SimpleTestSuite { testAsync("asynchronous execution") { val future = Future(100).map(_+1) for (result <- future) yield { assertEquals(result, 101) } } }

Minitest has integration with ScalaCheck. So for property-based testing:

import minitest.laws.Checkers object MyLawsTest extends SimpleTestSuite with Checkers { test("addition of integers is commutative") { check2((x: Int, y: Int) => x + y == y + x) } test("addition of integers is transitive") { check3((x: Int, y: Int, z: Int) => (x + y) + z == x + (y + z)) } }

That’s everything!

Common Complaints

I do not like Future

That’s too bad, because the Future is needed by the runtime and regardless what alternative you use (e.g. cats.effect.IO, monix.eval.Task), you’ll have to convert it into a Future anyway.

Besides, a good testing framework cannot have dependencies, because it would end in conflict with the project’s dependencies. It’s unwise to depend on Cats or Scalaz.

And you can always build your own testTask, testEffect or testIO utilities on top of testAsync.

I want a purely functional API

Specs2 has a nice functional API. You might like that, however I don’t like it for all the reasons stated above.

And if pure FP is what you want, nothing stops you from implementing your own utilities and I recommend piggybacking on ScalaCheck, e.g:

import cats.effect.IO import org.scalacheck.{Prop, Test} import scala.concurrent.ExecutionContext.Implicits.global trait PureTestSuite extends minitest.api.AbstractTestSuite { private[this] val ts = new SimpleTestSuite {} lazy val properties = ts.properties def test(name: String)(f: => Prop): Unit = ts.test(name) { val result = Test.check(config, f) if (!result.passed) fail(Pretty.pretty(result)) } def testIO(name: String)(f: => IO[Prop]): Unit = ts.testAsync(name) { f.unsafeToFuture.map { result => val result = Test.check(config, f) if (!result.passed) fail(Pretty.pretty(result)) } } }

There’s your purely functional API in just a couple of lines of code.

I don’t want that in Monix though - the integration that we have with ScalaCheck is minimal and enough.

It does not support Maven, CBT or others

Sorry about that, but this library is meant to be minimal and stable, and I don’t have the time to expand support beyond SBT right now.

Pull requests open and only accepted if they don’t complicate the codebase much.

Final Words

Forget DSLs.

All you need for testing are Minitest and ScalaCheck.

The article Minitest: Zero Crap Scala Testing Library first appeared on alexn.org.

What is Functional Programming?

Alexandru Nedelcu — Sun, 15 Oct 2017 00:00:00 +0000

Functional Programming (FP) is programming with functions. Mathematical functions.

There’s no other definition that’s correct, unless it’s equivalent to this one. There’s no other definition that matters.

Background

I first read the rigorous definition of a mathematical function in my first year of high-school, the 9-th grade. A function represents a unique association between elements of a domain (the input set) to the elements of a codomain (the output set). This means that applying the function to some input, you always get the same output.

My 9-th grade self actually read this in a high-school math manual:

Given f: A → B, ∀ x,y ∈ A If f(x) ≠ f(y) then x ≠ y

And in my young mind this condition seemed obvious and redundant, but that’s what you get with mathematical rigorosity, which in our profession is sorely needed.

Given that I’m from Romania, being exposed to an education centered on rote learning, influenced by the French and the Russian / Soviet educational systems, I’m now pretty sure that I have an atypical background, compared to my U.S. peers.

For example we learned some category theory in our 12-th grade, of which I’m grateful, being really intriguing to me how some 6-figures Ivy League graduates can complain about never hearing of the word Monoid, or having to learn the math material of normal teenagers. And don’t get me wrong, our educational system isn’t great, being in a continued decline.

If you want to get even more technical, functional programming has at its foundation the Lambda Calculus, a system for expressing computations that is equivalent to Turing machines, a universal model of computation built on function abstraction and application.

Some languages like Haskell are actually compiled / reduced to an intermediate language that’s very close to Lambda Calculus, which is cool to have as a theoretical foundation, because then you can prove things about your language and have the ability to add new features safely, e.g. without risking type unsoundness. As an aside, Scala doesn’t have that luxury because it’s also an OOP language, so they are developing DOT calculus as an alternative. Interesting stuff.

Why Functional Programming?

Many people are enamored with Functional Programming because it gives us:

Referential Transparency

Equational Reasoning

Software has essential complexity in it and it doesn’t help that our tools also contribute a decent amount of accidental complexity. If for example you think about asynchrony and concurrency, which often lead to non-determinism, the challenges involved have had tremendous cost for this industry.

Functional Programming keeps complexity at a manageable level because FP components can be divorced from their surrounding context and analysed independently. FP components can also be freely composed, an insanely useful property in an industry where software projects are seemingly built like houses of cards.

Memory locks for example don’t compose. Two functions yielding asynchronous results might or might not compose, depending on what shared mutable state they access, or what condition they are waiting on for completion.

There are a few alternatives to FP, like Rust’s draconic borrow checker, which essentially bans uncontrolled sharing. There are advantages and disadvantages to both approaches, however if you ever found it weird or frustrating to deal with pure functions, then fighting Rust’s borrow checker should be even more weird or frustrating (mind you, I think Rust is awesome, but that’s beside the point).

If you no longer require purity, if you change the definition of what kind of “functions” we can accept, then we are no longer talking of Functional Programming, but about …

Procedural Programming

We wouldn’t need the “pure” qualification if we, as programmers, wouldn’t overload terms.

Back in the day of assembly language and Turbo Pascal, we had perfectly good terms for impure functions, such as: procedure, routine, subroutine, these being blocks of code on which you jumped with the code pointer, executed some side-effects, pushed some results on the stack, then jumped back to where you were, with the contract being that such subroutines had a single entry point and a single exit point.

We have had a perfectly good term for describing programming made of procedures / subroutines: Procedural Programming 😉

Impure is Uninteresting

Lately the trend is to classify code making use of “lambda expressions” as functional programming and to classify programming languages that have “first-class functions” as being functional programming languages.

“Lambda expressions” are anonymous functions that can capture the variables in the scope they’ve been defined (implemented using closures).

Well, the problem is that:

The venerable C language has had the ability to pass function pointers around since forever, I know of no mainstream language that doesn’t allow you to pass function references, which makes functions “first class”;

You don’t actually need anonymous functions for doing functional programming, if you have an equivalent — for example Java had “anonymous classes” before Java 8, with the newer lambda expressions actually creating anonymous classes; take a look at this Functional Java library, which was built before Java 8;

It’s 2017 and most languages in use have usable lambda expressions, except for Python which has inherent limitations due to it being statement oriented and the developers refusing to introduce multi-line anonymous functions, which has led to a dozen or so non-orthogonal features to replace the need for it, under the mantra “only one way of doing things”, which by now is surely some kind of joke.

If you reduce your “functional programming” qualifier to usage of first class (impure) functions and lambda expressions, I think the top 15 languages and their use on GitHub qualifies.

Anti-intellectualism Phenomenon

For all the learning that we are doing, software developers are a really conservative bunch, unwilling to accept new concepts easily and in this context “new” is relative to what we’ve learned either in university or at our first job. The vigor with which we defend what we already know is proportional with the time we’ve invested in our knowledge and whatever it is that we are currently doing.

Many people advise against mentioning “Monad”, because it will strike fear in the hearts of the unfaithful, the advice being apparently to either sidestep the issue, to rename it into something that can be supposedly easier to understand, or to compare it with burritos.

Such efforts are like renaming “Water” into “Drinkable” — which obviously makes no sense in certain contexts and deprives people of the correct jargon for seeking help. Although I’ll grant that “Monad” is pretty awful if you’ll look at its etymology, but it doesn’t matter, because it has evolved into a proper noun and has been used by book authors and researchers.

Anyway, want to discredit an idea, opinion, fact, or tool? Classify it as “academic”, a term that now has negative connotations, even though most interesting breakthroughs in computer science come from academia.

Can We do FP in Any Language?

Yes, although some programming languages are better than others.

Doing FP in a programming language like Java feels like doing OOP in C with GObject. Doable, but it makes one think of switching professions in the long run.

But actually it’s not the programming language that’s the biggest problem, because technical challenges can usually be worked around, but the surrounding culture created by the community, along with the libraries available, because as a developer you won’t want to reinvent the wheel and swim against the tide.

This is why in addition to Haskell and OCaml, which are the languages that people refer to when speaking of FP, Scala also shines amongst them, because it has managed to attract and retain talent to work on awesome libraries for FP, that don’t have an equal outside of Haskell.

And for example, yes, you can do actual Functional Programming in JavaScript and there have been libraries helping with that, including really popular ones like RxJS, React, Redux, Immutable.js, Underscore.js amongst others, which were partially inspired by the community’s experience with ClojureScript and now ReasonML, PureScript and others. There are also community efforts, such as my own Funfix, plus a growing ecosystem around Fantasy Land, etc.

But bring up a problem like JavaScript’s Promise Leaks Memory in then chains and dozens of developers will jump on you to re-educate you on how promises work and to make you understand that the concerns you have are a niche, functional programming be damned.

Which does highlight that if you want functional programming, the communities of languages being bred for FP, like Haskell, PureScript, OCaml, Scala, etc. are probably richer and bigger than the FP sub-communities of the top mainstream languages, Java and JavaScript included.

Learning Resources

Don’t believe the opinions of people on the Internet, mine included. Learn some Functional Programming instead, the real stuff, not the pop lambda-infused mumbo jumbo, then you can make up your own mind. At the very least, it’s fun, and you’ve got nothing to lose.

Thus far, I found these two books to be good as an introduction to FP:

Haskell Programming from First Principles

Functional Programming in Scala

Learning Haskell first is not a bad idea, because Haskell is currently the lingua franca of FP and most interesting research is happening in Haskell or in Haskell-derived languages (e.g. Idris, PureScript, etc). Even if you move on, you’ll still refer to concepts you learned in Haskell by name, you’ll still be inspired by ideas from Haskell’s ecosystem, etc. But if you’re into Scala already, then the “red book” is pretty awesome.

After that, you might want to read Category Theory for Programmers (PDF version), which should come only after you’ve gone through one of the above books from cover to cover.

Note you don’t need category theory for FP, but it’s better if you eventually learn the basics, since it’s a formal language for talking about composition and we want composition in our lives 😎

If you want motivation, remember that kids in Romania do it — put that on your fridge 😜

More good books might be out there, but for learning FP, I’d advise against going with anything that’s dynamically typed or LISP based. For one because dynamic languages tend to be more pragmatic, plus their limited type system don’t allow many of the useful abstractions that we’ve discovered in the last decade, so you’d be depriving yourself of many useful concepts and libraries. You should feel free to pick a dynamic language once you have the knowledge to make an informed choice.

Also, SICP (see modern PDF compilation) might have been good for its time and is still a good book, but it’s not that good for learning FP in 2017.

Now go forth and spread the true FP love 💘

The article What is Functional Programming? first appeared on alexn.org.

Scala Days 2017 — Monix Task

Alexandru Nedelcu — Fri, 13 Oct 2017 00:00:00 +0000

Video link (open on YouTube.com)

Presentation from Scala Days, held in Chicago and Copenhagen:

Slides (PDF file)

Video (YouTube)

Video link (open on YouTube.com)

Abstract

Scala’s Future from the standard library is great, but sometimes we need more. A Future strives to be a value, one detached from time and for this reason its capabilities are restricted and for some use-cases its behavior ends up being unintuitive. Therefore, while the Future/Promise pattern is great for representing asynchronous results of processes that may or may not be started yet, it cannot be used as a specification for an asynchronous computation.

The Monix Task is in essence about dealing with asynchronous computations and non-determinism, being inspired by the Scalaz Task and designed from the ground up for performance and to be compatible with Scala.js/Javascript runtimes and with the Cats library. It also makes use of Scala’s Future to represent results, the two being complementary.

In this talk I’ll show you its design, when you should use it and why in dealing with asynchronicity it’s better to work with Task instead of blocking threads.

The article Scala Days 2017 — Monix Task first appeared on alexn.org.

Introduction

Alexandru Nedelcu — Wed, 11 Oct 2017 00:00:00 +0000

This piece of code will leak memory and eventually crash your Node.js process or browser:

// Tested on Node.js 7.10, Firefox 57 and Chrome 61 // // Usage of `setImmediate` is to prove that we have // async boundaries, can be removed for browsers // not supporting it. function signal(i) { return new Promise(cb => setImmediate(() => cb(i))) } function loop(n) { return signal(n).then(i => { if (i % 1000 == 0) console.log(i) return loop(n + 1) }) } loop(0).catch(console.error)

→ Load Sample for Browser

It takes a while to fill GBs of heap, plus Node is less conservative, the GC eventually freezing the process trying to recover memory, so give it a few seconds.

This is equivalent with this async function:

async function loop(n) { const i = await signal(n) if (i % 1000 == 0) console.log(i) // Recursive call return loop(n + 1) }

Of course, if this loop would be synchronous, not using Promise or async / await, then the process would blow up with a stack overflow error because at the moment of writing JavaScript does not do tail calls optimizations (until everybody implements ECMAScript 6 fully at least).

But before you jump to conclusions, this has nothing to do with JavaScript’s lack of TCO support. This is because in our recursive function it’s not JavaScript’s call stack that’s managing that loop, but rather the Promise implementation. That recursive call is asynchronous and so it does not abuse the call stack by definition.

Unfortunately, just like a regular function using the call stack for those recursive calls, the Promise implementation is abusing the heap memory, not chaining then calls correctly. And that sample should not leak, the Promise implementation should be able to do the equivalent of TCO and in such a case eliminate frames in the then chain being created.

The Spec is The Problem

We’re talking of the Promise/A+ specification. Relevant links to known issues on GitHub, explaining why:

Node.js issue, now closed: node/#6673

Promise/A+ spec issue, open since 2014: promises-spec/#179

As you’ll see, there are some reasonable arguments for why the Promise implementation is allowed to leak memory. But I can’t agree.

The Non-leaky Solution

The solution, if you insist on using JavaScript’s Promise, is to work with non-recursive functions:

async function loop(n) { let i = 0 while (true) { i = await signal(i + 1) if (i % 1000 == 0) console.log(i) } }

But at this point any semblance of functional programming, if you ever had any, goes out the window, see below.

Common Objections

Gathering feedback from people, here are the common objections:

1. This is Normal

No, if you judge this implementation, relative to other Promise / Future implementations in the industry, which are setting expectations.

Here are the implementations that I know about that DO NOT leak memory:

Bluebird, probably the most popular non-standard Promise implementation for JavaScript

Scala’s standard Future, in the wild since 2013; the fix for the leaky flatMap chains was added by Rich Dougherty in this PR, inspired by Twitter’s Future implementation

Twitter’s Future, which is used in all of Twitter’s backend infrastructure, being integrated in Finagle

Trane.io, a Java Future implementation providing a TailRec builder meant precissely for this use-case

My very own Funfix Future and Monix CancelableFuture

Interestingly, complaints for Scala’s Future happened due to usage of Play’s Iteratees, with which people have been modeling stream processing.

2. But It Does Unlimited Chaining of Promises

Yes, that’s why it’s leaking memory.

No, the implementation should not chain promises like that and an alternative implementation is possible and well known, as evidenced by the implementations mentioned that don’t leak.

The fault lies with the standard Promise implementation, not with the shown sample, being a legitimate use-case.

3. That Sample Does Not Use the Return Value

The sample is kept simple for didactic purposes, however:

you can easily imagine a loop that processes a very long stream of data, aggregating information along the way, returning a result later

the script does do error handling and without that inner return, the .catch(console.error) would have no effect

4. That’s the Equivalent of a Stack Overflow

Yes, I’ve mentioned this above, but just to set your mind at rest on this point, proper Tail-Calls Optimizations are coming for normal call sites, being part of ECMAScript, see:

ECMAScript specification

ECMAScript 6 Proper Tail Calls in WebKit

You can’t rely on it yet, but you can rest assured that in the future if that sample would be synchronous, it would not trigger a stack overflow.

Therefore, given that asynchronous calls are by definition processes / events happening independently of the current call stack / run loop, this behavior is actually surprising, since one of the reasons to go async is to escape the limitations of the call stack.

5. You Don’t Understand How Promises Work

I’ve been told that I don’t understand promises. So I apologize for the appeal to authority that I’m about to make.

Data types for dealing with asynchrony have been a hobby of mine since 2012 and I’ve been authoring several projects in which I implemented Promise-like data types:

Monix, which implements Task, one of the best ports of Haskell’s IO, along with a complementary CancelableFuture and what I think is the best back-pressured Rx Observable implementations in existence

Funfix, a JavaScript library for FP, delivering a Future and an IO implementation, see below

have contributed to cats-effect, a more conservative IO port

Bonus — see my presentation from Scala Days!

My work, good or bad, has followed a certain pattern, which is why I do understand promises, I do understand at least two solutions to this, hence this article.

6. No Use-cases, This is a Niche

Functional programming might be a niche, however more and more projects, including at big companies such as Facebook, are now using an actual FP style.

You cannot describe any functional programming algorithm involving loops without tail recursions. If folds are used, then folds are described with tail recursions as well. That’s because:

any loop can be described with a tail recursion

you can’t have immutability of the state you’re evolving without it

An example of a use-case is the processing of really long / infinite streams of events, for which it’s really natural to describe algorithms using tail recursions and for which you can’t really work with imperative, mutation-based loops.

Imagine reading chunks of data from a file and describing them with a data structure like this:

interface List class Next implements List { constructor( public readonly head: A, public readonly next: () => Promise> ) {} } class Halt implements List { constructor(public readonly error?: any) {} }

Sample is using TypeScript (could be Flow) for making it clear what the types are. You can work with plain JavaScript of course.

This structure is really cheap and effective, being a lazy, asynchronous, referentially transparent stream. And in fact it’s really similar to JavaScript implementations of async iterators, so yes, you are going to work with something like this in the future, even if you don’t like it ;-)

And describing transformation functions like this one is really fun too:

function map(list: List, f: (a: A) => B): List { if (list instanceof Next) { try { const cons = list as Next return new Next(f(cons.head), () => cons.next().then(xs => map(xs, f))) } catch (e) { return new Halt(e) } } else { return list } }

Alas, with the Promise implementation leaking, this doesn’t work ;-)

Alternatives

Fluture

Project Page: github.com/fluture-js/Fluture

This is the most popular Promise alternative that I know of.

It isn’t a direct replacement however because it has lazy behavior, being meant for suspending side effects. This is more like an IO data type. Which is cool, you should use something like it, but it’s also apples vs oranges.

I don’t have enough experience with it, making this recommendation solely based on its popularity and I double checked that it indeed preserves stack safety.

Funfix

I’ve been building a new project, Funfix, a JavaScript library for functional programming (capital FP), supporting TypeScript and Flow types out of the box.

Funfix exposes Future , an eager Promise alternative that’s safe, cancellable and filled with goodies, along with IO , a lazy, lawful, cancellable data type for handling all kinds of side effects, inspired by Haskell, the two being complementary.

This piece of code powered by Future does not leak:

import { Future } from "funfix" function loop(n) { return Future.of(() => n).flatMap(i => { if (i % 1000 == 0) console.log(i) return loop(n + 1) }) } loop(0).recover(console.error)

And neither does this one, powered by IO:

import { IO } from "funfix" function loop(n) { return IO.of(() => n).flatMap(i => { if (i % 1000 == 0) console.log(i) return loop(n + 1) }) } loop(0).run().recover(console.error)

This IO is a port of Monix’s Task, being a better IO than Haskell’s IO due to its cancellable nature ;-)

In this PR I’ve also fixed the memory leak for Future, doing the same tricks that Scala’s Future is doing. Now released in v6.2.0.

And note that this is harder to do for Funfix’s Future due to also having to deal with chains of Cancelable references, which can also leak.

Author’s Rant — in response to this article I’ve been called a scumbag for “shameless self promotion”.

I’m building stuff that I share with the world and I like talking about it on my personal blog. I’m not going to apologize for it.

Final Words

That the current JavaScript Promise implementation has this leak is a big problem, because tail-recursive calls are the cornerstone of functional programming.

Yes, it’s true that Promise is not a useful monadic type for doing FP, since it does not suspend side effects (which is why you should use IO), but that’s beside the point, plus for the FP purists out there, you can always suspend it in a thunk, assuming that it doesn’t leak in then chains.

This is also why I fear standardization by committee in general. Along with the totally awkward then signature that can’t be safely described with TypeScript’s or Flow’s types, this is another example of how standard solutions can be harmful, because by being pushed as a standard, it makes it hard for alternatives to exist, since most people are just going to use the standard implementation, especially in JavaScript’s ecosystem where people are afraid to take on dependencies.

The article Introduction first appeared on alexn.org.

Automatic Releases to Maven Central with Travis and SBT

Alexandru Nedelcu — Wed, 16 Aug 2017 00:00:00 +0000

Builds and deployments of new versions and snapshots is a pain. This article is an explanation to how I automated this process for monix.io, an open source Scala library that’s making use of SBT as the build tool and travis-ci.org as the continuous integration.

What this setup does is to trigger a publish script that automatically deploys packages on Maven Central:

whenever you tag a release by pushing a version tag in Git, like v1.10.0

whenever you push into the snapshot branch, the result being hashed versions, e.g. 1.10.0-36fa3d3, where the hash appended as a suffix is the Git commit hash; these hashed versions are like snapshot releases, but better because people can rely on them to remain in Maven Central and thus less volatile

After you read this article, you can use the setup of these projects for inspiration:

github.com/monix/shade

github.com/monix/monix

WARNING: with this process you’ll have to trust Travis-ci.org with your PGP private key for signing the published binaries. If that’s an acceptable risk or not, that’s up to you. See below.

Generating a PGP Key Pair

For deployments to Sonatype / Maven Central the built packages need to be signed. If you don’t have an existing PGP key, one can be easily generated.

WARNING: do not give out your personal PGP private key that you use to sign emails or for online transactions. Generate a special PGP key pair just for your project.

I’m currently using a MacOS machine, so for managing PGP keys I’m using the open source GPG Suite, coming with a nice GUI interface.

As far as I know GPG comes installed by default on all major Linux operating systems and for Windows checkout this download page on gnupg.org.

With the GPG command line tools installed you can generate a PGP key pair like this:

$ gpg --gen-key

Steps:

accept the default RSA for the kind of key

enter the desired key size, the bigger the better, so enter 4096

for expiration, I preferred a key that doesn’t expire, although this might not be wise

for an email address, enter a valid one

I recommend encrypting your private key with a generous passphrase that you then store in 1Password / LastPass ;-)

To get the ID of the newly generated key you can do:

$ gpg --list-secret-keys --keyid-format LONG

To export this newly generated key, assuming that 2673B174C4071B0E is the key ID, you’ll need both the public key and the private one, but they can be dumped in the same file:

gpg -a --export 2673B174C4071B0E > my-key.asc gpg -a --export-secret-keys 2673B174C4071B0E >> my-key.asc

I also keep these in 1Password btw.

To configure SBT to sign your packages with a key living in the project’s repository you’ll need a PGP key ring. Such a key ring is basically a database of multiple PGP keys. You need to have one to keep in the repository of your project. Normally these keys are kept in:

$HOME/.gnupg/pubring.gpg for the public keys

$HOME/.gnupg/secring.gpg for the private keys

In the $PROJECT root we need a custom key ring containing just the key we need, like this:

$PROJECT/project/.gnupg/pubring.gpg for the public keys

$PROJECT/project/.gnupg/secring.gpg for the private keys

These files are going to be encrypted, to provide minimal protection. To generate this ring in your project, go to you’re project’s root directory and then:

gpg --no-default-keyring \ --primary-keyring `pwd`/project/.gnupg/pubring.gpg \ --secret-keyring `pwd`/project/.gnupg/secring.gpg \ --keyring `pwd`/project/.gnupg/pubring.gpg \ --fingerprint \ --import path/to/my-key.asc

The my-key.asc file is the one that you’ve created in the previous step.

After you create these files, make sure to delete any junk from $PROJECT/project/.gnupg, so verify the newly created files with git status.

NOTE: check the newly created files, because the gpg command line tools might generate junk. We only want those 2 files (pubring.gpg and secring.gpg), so check your project directory with git status and delete anything extra.

Configuring SBT

Curently in monix.io I’m using the following plugins:

sbt-pgp for signing packages with PGP

sbt-git for making use of Git from SBT, relevant here if you want to do Git-enabled version hashes

sbt-sonatype for automatically publishing artifacts to Maven Central

For PGP the configuration is as follows:

useGpg := false usePgpKeyHex("2673B174C4071B0E") pgpPublicRing := baseDirectory.value / "project" / ".gnupg" / "pubring.gpg" pgpSecretRing := baseDirectory.value / "project" / ".gnupg" / "secring.gpg" pgpPassphrase := sys.env.get("PGP_PASS").map(_.toArray)

Explanation:

useGpg := false says that we do not want to use the GPG tools installed on your computer, but rather the implementation that sbt-pgp ships with; in my experience this is a must, otherwise depending on the GPG tools you have, you won’t be able to make it use a different pgp ring

usePgpKeyHex forces a certain key to be used for signing by specifying its key

pgpPublicRing and pgpPublicRing specify the path to a GPG ring that contains the key you want, instead of the default one which is usually $HOME/.gnupg/pubring.gpg and $HOME/.gnupg/secring.gpg

pgpPassphrase is a GPG passphrase for the used key, that’s taken from the env variable named PGP_PASS; Travis has the ability to set such env variables to be available in your build

For publishing to Sonatype, we’ll need these settings:

sonatypeProfileName := organization.value credentials += Credentials( "Sonatype Nexus Repository Manager", "oss.sonatype.org", sys.env.getOrElse("SONATYPE_USER", ""), sys.env.getOrElse("SONATYPE_PASS", "") ) isSnapshot := version.value endsWith "SNAPSHOT" publishTo := Some( if (isSnapshot.value) Opts.resolver.sonatypeSnapshots else Opts.resolver.sonatypeStaging )

In addition to these options, for Sonatype we also need the required artifact info (e.g. license, homepage, authors). Here’s what I have for Shade, adjust accordingly:

licenses := Seq("MIT" -> url("https://opensource.org/licenses/MIT")) homepage := Some(url("https://github.com/monix/shade")) scmInfo := Some( ScmInfo( url("https://github.com/monix/shade"), "scm:git@github.com:monix/shade.git" )) developers := List( Developer( id="alexelcu", name="Alexandru Nedelcu", email="noreply@alexn.org", url=url("https://alexn.org") ))

TIP, to find out the ID of a license type, see this cool list: spdx.org/licenses/.

You’ll need those two environment variables set in Travis’s settings, more details below.

And then to enable Git versioning for snapshots (e.g. 3.0.0-9d94d3d) you can do:

enablePlugins(GitVersioning) /* The BaseVersion setting represents the in-development (upcoming) version, * as an alternative to SNAPSHOTS. */ git.baseVersion := "3.0.0" val ReleaseTag = """^v([\d\.]+)$""".r git.gitTagToVersionNumber := { case ReleaseTag(v) => Some(v) case _ => None } git.formattedShaVersion := { val suffix = git.makeUncommittedSignifierSuffix(git.gitUncommittedChanges.value, git.uncommittedSignifier.value) git.gitHeadCommit.value map { _.substring(0, 7) } map { sha => git.baseVersion.value + "-" + sha + suffix } }

Now test your setup with this command:

$ PGP_PASS="xxxxxx" sbt publishLocalSigned

Replace xxxxxx with your passphrase. If this command works, then we are good thus far.

Configuring Travis

In build.sbt I configured these 2 commands:

addCommandAlias("ci-all", ";+clean ;+compile ;+test ;+package") addCommandAlias("release", ";+publishSigned ;sonatypeReleaseAll")

Then the .travis.yml file has something like this:

language: scala sudo: required dist: trusty group: edge matrix: include: - jdk: oraclejdk8 scala: 2.12.3 env: COMMAND=ci-all PUBLISH=true script: - sbt -J-Xmx6144m ++$TRAVIS_SCALA_VERSION $COMMAND after_success: - ./project/publish

And then the project/publish script, which I’ve built with Ruby (since I don’t know Bash well :)):

#!/usr/bin/env ruby def exec(cmd) abort("Error encountered, aborting") unless system(cmd) end puts "CI=#{ENV['CI']}" puts "TRAVIS_BRANCH=#{ENV['TRAVIS_BRANCH']}" puts "TRAVIS_PULL_REQUEST=#{ENV['TRAVIS_PULL_REQUEST']}" puts "PUBLISH=#{ENV['PUBLISH']}" puts unless ENV['CI'] == 'true' abort("ERROR: Not running on top of Travis, aborting!") end unless ENV['PUBLISH'] == 'true' puts "Publish is disabled" exit end branch = ENV['TRAVIS_BRANCH'] version = nil unless branch =~ /^v(\d+\.\d+\.\d+)$/ || (branch == "snapshot" && ENV['TRAVIS_PULL_REQUEST'] == 'false') puts "Only triggering deployment on the `snapshot` branch, or for version tags " + "and not for pull requests or other branches, exiting!" exit 0 else version = $1 puts "Version branch detected: #{version}" if version end # Forcing a change to the root directory, if not there already Dir.chdir(File.absolute_path(File.join(File.dirname(__FILE__), ".."))) # Go, go, go exec("sbt release")

Give execution permissions to this script:

$ chmod +x ./project/publish

Remember to push your changes:

$ git add . $ git commit -am 'Build changes for automatic releases' $ git push

Setting environment variables

As a final step we need to set the following environment variables in Travis:

PGP_PASS: the passphrase we used to encrypt our private PGP key

SONATYPE_USER: a user to login to Sonatype, used by SBT to publish and deploy releases on Sonatype

SONATYPE_PASS: a password to login to Sonatype, used by SBT to publish and deploy releases on Sonatype

See the article on adding environment variables to Travis.

NOTE: to get a SONATYPE_USER and a SONATYPE_PASS go to the User Profile on Sonatype page and access the “User Token”, or generate a new one.

Here’s a screenshot of how my setup currently looks like:

Alternative Env with Travis Encryption

As an alternative to setting those environment variables in Travis’s UI, you can use Travis’s mechanism for encrypting stuff to set these values in .travis.yml. See the Encryption Keys document.

First install the travis command line tool:

$ gem install travis

And then do the following, replacing xxxxx with your key:

$ travis encrypt 'PGP_PASS=xxxxx' --add $ travis encrypt 'SONATYPE_USER=xxxxx' --add $ travis encrypt 'SONATYPE_PASS=xxxxx' --add

NOTE: if your env values have special chars, they might need to be escaped for Bash to not trigger any errors. See document above.

These commands will modify your .travis.yml file, adding a section that resembles the following:

env: global: - secure: GRdfKNrJn/zqjaDWE+16HCfuCSf/wsDpL... - secure: SPSIblLKFVns7pVY1x3SEs4/16htY5HUz... - secure: YVx2BSSsqF7LdYTwinf6o8nqJiYL9FeFA...

Now this can be committed in your repository and Travis will take care of decrypting those values.

Publishing

For publishing hashed snapshot versions, we need a snapshot branch, as that’s what the script above looks for.

So create this branch by forking master and pushing it, like so:

$ git checkout master $ git checkout -b snapshot $ git push --set-upstream origin snapshot

If everything goes well, we should have a new hashed version published, but watch the output of Travis for any problems.

Extra Resources

I’ve written this document while preparing the Shade project for automatic deployments. So here’s for inspiration:

commits in monix/shade

snapshot release sample (that published 1.10.0-d712897 on Maven Central)

release of v1.10.0 (that published the final 1.10.0 on Maven Central)

In Closing

So that’s about it. Pretty painful if you ask me, but hopefully we don’t have to do this too often.

I’ve written this article for myself actually, because I keep forgetting what I did the first time.

The article Automatic Releases to Maven Central with Travis and SBT first appeared on alexn.org.

Functional Programming Inception (Presentation)

Alexandru Nedelcu — Wed, 15 Mar 2017 00:00:00 +0000

My presentation from the Bucharest FP meetup.

Resources:

Slides (PDF file)

Links from the presentation:

Monix

Typelevel Cats

Discipline

ScalaCheck

Generic Iterant Implementation

Simplified Task-based Implementation

Abstract

Designing functionality that exhibits the properties of functional programming is hard because it requires a mentality change, coping with immutability and consideration for recursion, performance and polymorphism. This talk is a lesson in FP design that makes use of Scala’s hybrid OOP+FP nature.

We are going to start from Scala’s (and Java’s) ubiquitous Iterator/Iterable types which expose the famous iterator pattern, analyzing its strengths and weaknesses. And then we are going to work our way up to a fully featured FP replacement that has referential transparency and that fixes everything that’s wrong with Iterator, while being more generic.

This lesson in design involves talking about immutability, imperative programming, asynchrony and problems encountered when going FP, like performance considerations, recursion and memory leaks. We are also going to talk about ADTs, higher kinded polymorphism and type-classes versus OOP subtyping. Interestingly the example presented will use both OOP subtyping and type-classes and thus we can make a clear comparison about what to use and when - a problem that the Scala developer has in his daily work.

The article Functional Programming Inception (Presentation) first appeared on alexn.org.

Asynchronous Programming and Scala

Alexandru Nedelcu — Mon, 30 Jan 2017 00:00:00 +0000

Asynchrony is everywhere and it subsumes concurrency. This article explains what asynchronous processing is and its challenges.

1. Introduction

As a concept it is more general than multithreading, although some people confuse the two. If you’re looking for a relationship, you could say:

Multithreading <: Asynchrony

We can represent asynchronous computations with a type:

type Async[A] = (Try[A] => Unit) => Unit

If it looks ugly with those Unit return types, that’s because asynchrony is ugly. An asynchronous computation is any task, thread, process, node somewhere on the network that:

executes outside of your program’s main flow or from the point of view of the caller, it doesn’t execute on the current call-stack

receives a callback that will get called once the result is finished processing

it provides no guarantee about when the result is signaled, no guarantee that a result will be signaled at all

It’s important to note asynchrony subsumes concurrency, but not necessarily multithreading. Remember that in Javascript the majority of all I/O actions (input or output) are asynchronous and even heavy business logic is made asynchronous (with setTimeout based scheduling) in order to keep the interface responsive. But no kernel-level multithreading is involved, Javascript being an N:1 multithreaded platform.

Introducing asynchrony into your program means you’ll have concurrency problems because you never know when asynchronous computations will be finished, so composing the results of multiple asynchronous computations running at the same time means you have to do synchronization, as you can no longer rely on ordering. And not relying on an order is a recipe for nondeterminism.

Wikipedia says: a nondeterministic algorithm is an algorithm that, even for the same input, can exhibit different behaviors on different runs, as opposed to a deterministic algorithm … A concurrent algorithm can perform differently on different runs due to a race condition.

The astute reader could notice that the type in question can be seen everywhere, with some modifications depending on use-case and contract:

in the Observer pattern from the Gang of Four

in Scala’s Future, which is defined by its abstract onComplete method

in Java’s ExecutorService.submit(Callable)

in Javascript’s EventTarget.addEventListener

in Akka actors, although there the given callback is replaced by the sender() reference

in the Monix Task.Async definition

in the Monix Observable and Observer pair

in the Reactive Streams specification

What do all of these abstractions have in common? They provide ways to deal with asynchrony, some more successful than others.

2. The Big Illusion

We like to pretend that we can describe functions that can convert asynchronous results to synchronous ones:

def await[A](fa: Async[A]): A

Fact of the matter is that we can’t pretend that asynchronous processes are equivalent with normal functions. If you need a lesson in history for why we can’t pretend that, you only need to take a look at why CORBA failed.

With asynchronous processes we have the following very common fallacies of distributed computing:

The network is reliable

Latency is zero

Bandwidth is infinite

The network is secure

Topology doesn’t change

There is one administrator

Transport cost is zero

The network is homogeneous

None of them are true of course. Which means code gets written with little error handling for network failures, ignorance of network latency or packet loss, ignorance of bandwidth limits and in general ignorance of the ensuing nondeterminism.

People have tried to cope with this by:

callbacks, callbacks everywhere, equivalent to basically ignoring the problem, as it happens in Javascript, which leads to the well known effect of callback hell, paid for with the sweat and blood of programmers that constantly imagine having chosen a different life path

blocking threads, on top of 1:1 (kernel-level) multithreading platforms

first-class continuations, implemented for example by Scheme in call/cc, being the ability to save the execution state at any point and return to that point at a later point in the program

The async / await language extension from C#, also implemented in the scala-async library and in the latest ECMAScript

Green threads managed by the runtime, possibly in combination with M:N multithreading, to simulate blocking for asynchronous actions; examples including Golang but also Haskell

The actor model as implemented in Erlang or Akka, or CSP such as in Clojure’s core.async or in Golang

Monads being used for ordering and composition, such as Haskell’s Async type in combination with the IO type, or F# asynchronous workflows, or Scala’s Futures and Promises, or the Monix Task or the Scalaz Task, etc, etc.

If there are so many solutions, that’s because none of them is suitable as a general purpose mechanism for dealing with asynchrony. The no silver bullet dilemma is relevant here, with memory management and concurrency being the biggest problems that we face as software developers.

WARNING - personal opinion and rant: People like to boast about M:N platforms like Golang, however I prefer 1:1 multithreaded platforms, like the JVM or dotNET.

Because you can build M:N multithreading on top of 1:1 given enough expressiveness in the programming language (e.g. Scala’s Futures and Promises, Task, Clojure’s core.async, etc), but if that M:N runtime starts being unsuitable for your usecase, then you can’t fix it or replace it without replacing the platform. And yes, most M:N platforms are broken in one way or another.

Indeed learning about all the possible solutions and making choices is freaking painful, but it is much less painful than making uninformed choices, with the TOOWTDI and “worse is better” mentalities being in this case actively harmful. People complaining about the difficulty of learning a new and expressive language like Scala or Haskell are missing the point, because if they have to deal with concurrency, then learning a new programming language is going to be the least of their problems. I know people that have quit the software industry because of the shift to concurrency.

3. Callback Hell

Let’s build an artificial example made to illustrate our challenges. Say we need to initiate two asynchronous processes and combine their result.

First let’s define a function that executes stuff asynchronously:

import scala.concurrent.ExecutionContext.global type Async[A] = (A => Unit) => Unit def timesTwo(n: Int): Async[Int] = onFinish => { global.execute(new Runnable { def run(): Unit = { val result = n * 2 onFinish(result) } }) } // Usage timesTwo(20) { result => println(s"Result: $result") } //=> Result: 40

3.1. Sequencing (Purgatory of Side-effects)

Let’s combine two asynchronous results, with the execution happening one after another, in a neat sequence:

def timesFour(n: Int): Async[Int] = onFinish => { timesTwo(n) { a => timesTwo(n) { b => // Combining the two results onFinish(a + b) } } } // Usage timesFour(20) { result => println(s"Result: $result") } //=> Result: 80

Looks simple now, but we are only combining two results, one after another.

The big problem however is that asynchrony infects everything it touches. Let’s assume for the sake of argument that we start with a pure function:

def timesFour(n: Int): Int = n * 4

But then your enterprise architect, after hearing about these Enterprise JavaBeans and a lap dance, decides that you should depend on this asynchronous timesTwo function. And now our timesFour implementation changes from a pure mathematical function to a side-effectful one and we have no choice in the matter. And without a well grown Async type, we are forced to deal with side-effectful callbacks for the whole pipeline. And blocking for the result won’t help, as you’re just hiding the problem, see section 2 for why.

But wait, things are about to get worse 😷

3.2. Parallelism (Limbo of Nondeterminism)

The second call we made above is not dependent on the first call, therefore it can run in parallel. On the JVM we can run CPU-bound tasks in parallel, but this is relevant for Javascript as well, as we could be making Ajax requests or talking with web workers.

Unfortunately here things can get a little complicated. First of all the naive way to do it is terribly wrong:

// REALLY BAD SAMPLE def timesFourInParallel(n: Int): Async[Int] = onFinish => { var cacheA = 0 timesTwo(n) { a => cacheA = a } timesTwo(n) { b => // Combining the two results onFinish(cacheA + b) } } timesFourInParallel(20) { result => println(s"Result: $result") } //=> Result: 80 timesFourInParallel(20) { result => println(s"Result: $result") } //=> Result: 40

This right here is an example showing nondeterminism in action. We get no ordering guarantees about which one finishes first, so if we want parallel processing, we need to model a mini state machine for doing synchronization.

First, we define our ADT describing the state-machine:

// Defines the state machine sealed trait State // Initial state case object Start extends State // We got a B, waiting for an A final case class WaitForA(b: Int) extends State // We got a A, waiting for a B final case class WaitForB(a: Int) extends State

And then we can evolve this state machine asynchronously:

// BAD SAMPLE FOR THE JVM (only works for Javascript) def timesFourInParallel(n: Int): Async[Int] = { onFinish => { var state: State = Start timesTwo(n) { a => state match { case Start => state = WaitForB(a) case WaitForA(b) => onFinish(a + b) case WaitForB(_) => // Can't be caught b/c async, hopefully it gets reported throw new IllegalStateException(state.toString) } } timesTwo(n) { b => state match { case Start => state = WaitForA(b) case WaitForB(a) => onFinish(a + b) case WaitForA(_) => // Can't be caught b/c async, hopefully it gets reported throw new IllegalStateException(state.toString) } } } }

To better visualize what we’re dealing with, here’s the state machine:

But wait, we aren’t over because the JVM has true 1:1 multi-threading, which means we get to enjoy shared memory concurrency and thus access to that state has to be synchronized.

One solution is to use synchronized blocks, also called intrinsic locks:

// We need a common reference to act as our monitor val lock = new AnyRef var state: State = Start timesTwo(n) { a => lock.synchronized { state match { case Start => state = WaitForB(a) case WaitForA(b) => onFinish(a + b) case WaitForB(_) => // Can't be caught b/c async, hopefully it gets reported throw new IllegalStateException(state.toString) } } } //...

Such high-level locks protect resources (such as our state) from being accessed in parallel by multiple threads. But I personally prefer to avoid high-level locks because the kernel’s scheduler can freeze any thread for any reason, including threads that hold locks, freezing a thread holding a lock means that other threads will be unable to make progress and if you want to guarantee constant progress (e.g. soft real-time characteristics), then non-blocking logic is preferred when possible.

So an alternative is to use an AtomicReference, which is perfect for this case:

// CORRECT VERSION FOR JVM import scala.annotation.tailrec import java.util.concurrent.atomic.AtomicReference def timesFourInParallel(n: Int): Async[Int] = { onFinish => { val state = new AtomicReference[State](Start) @tailrec def onValueA(a: Int): Unit = state.get match { case Start => if (!state.compareAndSet(Start, WaitForB(a))) onValueA(a) // retry case WaitForA(b) => onFinish(a + b) case WaitForB(_) => // Can't be caught b/c async, hopefully it gets reported throw new IllegalStateException(state.toString) } timesTwo(n)(onValueA) @tailrec def onValueB(b: Int): Unit = state.get match { case Start => if (!state.compareAndSet(Start, WaitForA(b))) onValueB(b) // retry case WaitForB(a) => onFinish(a + b) case WaitForA(_) => // Can't be caught b/c async, hopefully it gets reported throw new IllegalStateException(state.toString) } timesTwo(n)(onValueB) } }

PRO-TIP: if you want code that cross-compiles to Javascript / Scala.js, along with performance tweaks and cool utilities for manipulating atomic references, try the Atomic type from Monix.

Are you getting pumped? Let’s take it up a notch 😝

3.3. Recursivity (Wrath of StackOverflow)

What if I were to tell you that the above onFinish call is stack-unsafe and if you aren’t going to force an asynchronous boundary when calling it, then your program can blow up with a StackOverflowError?

You shouldn’t take my word for it. Let’s first have some fun and define the above operation in a generic way:

import scala.annotation.tailrec import java.util.concurrent.atomic.AtomicReference type Async[+A] = (A => Unit) => Unit def mapBoth[A,B,R](fa: Async[A], fb: Async[B])(f: (A,B) => R): Async[R] = { // Defines the state machine sealed trait State[+A,+B] // Initial state case object Start extends State[Nothing, Nothing] // We got a B, waiting for an A final case class WaitForA[+B](b: B) extends State[Nothing,B] // We got a A, waiting for a B final case class WaitForB[+A](a: A) extends State[A,Nothing] onFinish => { val state = new AtomicReference[State[A,B]](Start) @tailrec def onValueA(a: A): Unit = state.get match { case Start => if (!state.compareAndSet(Start, WaitForB(a))) onValueA(a) // retry case WaitForA(b) => onFinish(f(a,b)) case WaitForB(_) => // Can't be caught b/c async, hopefully it gets reported throw new IllegalStateException(state.toString) } @tailrec def onValueB(b: B): Unit = state.get match { case Start => if (!state.compareAndSet(Start, WaitForA(b))) onValueB(b) // retry case WaitForB(a) => onFinish(f(a,b)) case WaitForA(_) => // Can't be caught b/c async, hopefully it gets reported throw new IllegalStateException(state.toString) } fa(onValueA) fb(onValueB) } }

And now we can define an operation similar to Scala’s Future.sequence, because our will is strong and our courage immensurable 😇

def sequence[A](list: List[Async[A]]): Async[List[A]] = { @tailrec def loop(list: List[Async[A]], acc: Async[List[A]]): Async[List[A]] = list match { case Nil => onFinish => acc(r => onFinish(r.reverse)) case x :: xs => val update = mapBoth(x, acc)(_ :: _) loop(xs, update) } val empty: Async[List[A]] = _(Nil) loop(list, empty) } // Invocation sequence(List(timesTwo(10), timesTwo(20), timesTwo(30))) { r => println(s"Result: $r") } //=> Result: List(20, 40, 60)

Oh, you really think we are done?

val list = 0.until(10000).map(timesTwo).toList sequence(list)(r => println(s"Sum: ${r.sum}"))

Behold the glorious memory error that will probably crash your program in production, being considered a fatal error that Scala’s NonFatal does not catch:

java.lang.StackOverflowError at java.util.concurrent.ForkJoinPool.externalPush(ForkJoinPool.java:2414) at java.util.concurrent.ForkJoinPool.execute(ForkJoinPool.java:2630) at scala.concurrent.impl.ExecutionContextImpl$$anon$3.execute(ExecutionContextImpl.scala:131) at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:20) at .$anonfun$timesTwo$1(:27) at .$anonfun$timesTwo$1$adapted(:26) at .$anonfun$mapBoth$1(:66) at .$anonfun$mapBoth$1$adapted(:40) at .$anonfun$mapBoth$1(:67) at .$anonfun$mapBoth$1$adapted(:40) at .$anonfun$mapBoth$1(:67) at .$anonfun$mapBoth$1$adapted(:40) at .$anonfun$mapBoth$1(:67)

As I said, that onFinish call being made without a forced async boundary can lead to a stack-overflow error. On top of Javascript this can be solved by scheduling it with setTimeout and on top of the JVM you need a thread-pool or a Scala ExecutionContext.

Are you feeling the fire yet? 🔥

4. Futures and Promises

The scala.concurrent.Future describes strictly evaluated asynchronous computations, being similar to our Async type used above.

Wikipedia says: Future and Promise are constructs used for synchronizing program execution in some concurrent programming languages. They describe an object that acts as a proxy for a result that is initially unknown, usually because the computation of its value is yet incomplete.

Author’s Rant: The docs.scala-lang.org article on Futures and Promises currently says that “Futures provide a way to reason about performing many operations in parallel– in an efficient and non-blocking way”, but that is misleading, a source of confusion.

The Future type describes asynchrony and not parallelism. Yes, you can do things in parallel with it, but it’s not meant only for parallelism (async != parallelism) and for people looking into ways to use their CPU capacity to its fullest, working with Future can prove to be expensive and unwise, because in certain cases it has performance issues, see section 4.4.

The Future is an interface defined by 2 primary operations, along with many combinators defined based on those primary operations:

import scala.util.Try import scala.concurrent.ExecutionContext trait Future[+T] { // abstract def value: Option[Try[T]] // abstract def onComplete(f: Try[T] => Unit)(implicit ec: ExecutionContext): Unit // Transforms values def map[U](f: T => U)(implicit ec: ExecutionContext): Future[U] = ??? // Sequencing ;-) def flatMap[U](f: T => Future[U])(implicit ec: ExecutionContext): Future[U] = ??? // ... }

The properties of Future:

Eagerly evaluated (strict and not lazy), meaning that when the caller of a function receives a Future reference, whatever asynchronous process that should complete it has probably started already.

Memoized (cached), since being eagerly evaluated means that it behaves like a normal value instead of a function and the final result needs to be available to all listeners. The purpose of the value property is to return that memoized result or None if it isn’t complete yet. Goes without saying that calling its def value yields a non-deterministic result.

Streams a single result and it shows because of the memoization applied. So when listeners are registered for completion, they’ll only get called once at most.

Explanatory notes about the ExecutionContext:

The ExecutionContext manages asynchronous execution and although you can view it as a thread-pool, it’s not necessarily a thread-pool (because async != multithreading or parallelism).

The onComplete is basically our Async type defined above, however it takes an ExecutionContext because all completion callbacks need to be called asynchronously.

All combinators and utilities are built on top of onComplete, therefore all combinators and utilities must also take an ExecutionContext parameter.

If you don’t understand why that ExecutionContext is needed in all those signatures, go back and re-read section 3.3 and don’t come back until you do.

4.1. Sequencing

Let’s redefine our function from section 3 in terms of Future:

import scala.concurrent.{Future, ExecutionContext} def timesTwo(n: Int)(implicit ec: ExecutionContext): Future[Int] = Future(n * 2) // Usage { import scala.concurrent.ExecutionContext.Implicits.global timesTwo(20).onComplete { result => println(s"Result: $result") } //=> Result: Success(40) }

Easy enough, the Future.apply builder executes the given computation on the given ExecutionContext. So on the JVM, assuming the global execution context, it’s going to run on a different thread.

Now to do sequencing like in section 3.1:

def timesFour(n: Int)(implicit ec: ExecutionContext): Future[Int] = for (a <- timesTwo(n); b <- timesTwo(n)) yield a + b // Usage { import scala.concurrent.ExecutionContext.Implicits.global timesFour(20).onComplete { result => println(s"Result: $result") } //=> Result: Success(80) }

Easy enough. That “for comprehension” magic right there is translated to nothing more than calls to flatMap and map, being literally equivalent with:

def timesFour(n: Int)(implicit ec: ExecutionContext): Future[Int] = timesTwo(n).flatMap { a => timesTwo(n).map { b => a + b } }

And if you import scala-async in your project, then you can do it like:

import scala.async.Async.{async, await} def timesFour(n: Int)(implicit ec: ExecutionContext): Future[Int] = async { val a = await(timesTwo(a)) val b = await(timesTwo(b)) a + b }

The scala-async library is powered by macros and will translate your code to something equivalent to flatMap and map calls. So in other words await does not block threads, even though it gives the illusion that it does.

This looks great actually, unfortunately it has many limitations. The library cannot rewrite your code in case the await is inside an anonymous function and unfortunately Scala code is usually full of such expressions. This does not work:

// BAD SAMPLE def sum(list: List[Future[Int]])(implicit ec; ExecutionContext): Future[Int] = async { var sum = 0 // Nope, not going to work because "for" is translated to "foreach" for (f <- list) { sum += await(f) } }

This approach gives the illusion of having first-class continuations, but these continuations are unfortunately not first class, being just a compiler-managed rewrite of the code. And yes, this restriction applies to C# and ECMAScript as well. Which is a pity, because it means async code will not be heavy on FP.

Remember my rant from above about the no silver bullet? 😞

4.2. Parallelism

Just as in section 3.2 those two function calls are independent of each other, which means that we can call them in parallel. With Future this is easier, although its evaluation semantics can be a little confusing for beginners:

def timesFourInParallel(n: Int)(implicit ec: ExecutionContext): Future[Int] = { // Future is eagerly evaluated, so this will trigger the // execution of both before the composition happens val fa = timesTwo(n) val fb = timesTwo(n) for (a <- fa; b <- fb) yield a + b // fa.flatMap(a => fb.map(b => a + b)) }

It can be a little confusing and it catches beginners off-guard. Because of its execution model, in order to execute things in parallel, you simply have to initialize those future references before the composition happens.

An alternative would be to use Future.sequence, which works for arbitrary collections:

def timesFourInParallel(n: Int)(implicit ec: ExecutionContext): Future[Int] = Future.sequence(timesTwo(n) :: timesTwo(n) :: Nil).map(_.sum)

This too can catch beginners by surprise, because those futures are going to be executed in parallel only if the collection given to sequence is strict (not like Scala’s Stream or some Iterator). And the name is sort of a misnomer obviously.

4.3. Recursivity

The Future type is entirely safe for recursive operations (because of the reliance on the ExecutionContext for executing callbacks). So retrying the sample in section 3.3:

def mapBoth[A,B,R](fa: Future[A], fb: Future[B])(f: (A,B) => R) (implicit ec: ExecutionContext): Future[R] = { for (a <- fa; b <- fb) yield f(a,b) } def sequence[A](list: List[Future[A]]) (implicit ec: ExecutionContext): Future[List[A]] = { val seed = Future.successful(List.empty[A]) list.foldLeft(seed)((acc,f) => for (l <- acc; a <- f) yield a :: l) .map(_.reverse) } // Invocation { import scala.concurrent.ExecutionContext.Implicits.global sequence(List(timesTwo(10), timesTwo(20), timesTwo(30))).foreach(println) // => List(20, 40, 60) }

And this time we get no StackOverflowError:

val list = 0.until(10000).map(timesTwo).toList sequence(list).foreach(r => println(s"Sum: ${r.sum}")) //=> Sum: 99990000

4.4. Performance Considerations

The trouble with Future is that each call to onComplete will use an ExecutionContext for execution and in general this means that a Runnable is sent in a thread-pool, thus forking a (logical) thread. If you have CPU-bounded tasks, this implementation detail is actually a disaster for performance because jumping threads means context switches, along with the CPU cache locality being destroyed. Of course, the implementation does have certain optimizations, like the flatMap implementation using an internal execution context that’s trampolined, in order to avoid forks when chaining those internal callbacks, but it’s not enough and benchmarking doesn’t lie.

Also due to it being memoized means that upon completion the implementation is forced to execute at least one AtomicReference.compareAndSet per producer, plus one compareAndSet call per listener registered before the Future is complete. And such calls are quite expensive, all because we need memoization that plays well with multithreading.

In other words if you want to exploit your CPU to its fullest for CPU-bound tasks, then working with futures and promises is not such a good idea.

If you want to see how Scala’s Future implementation compares with Task, see the following recent benchmark:

[info] Benchmark (size) Mode Cnt Score Error Units [info] FlatMap.fs2Apply 10000 thrpt 20 291.459 ± 6.321 ops/s [info] FlatMap.fs2Delay 10000 thrpt 20 2606.864 ± 26.442 ops/s [info] FlatMap.fs2Now 10000 thrpt 20 3867.300 ± 541.241 ops/s [info] FlatMap.futureApply 10000 thrpt 20 212.691 ± 9.508 ops/s [info] FlatMap.futureSuccessful 10000 thrpt 20 418.736 ± 29.121 ops/s [info] FlatMap.futureTrampolineEc 10000 thrpt 20 423.647 ± 8.543 ops/s [info] FlatMap.monixApply 10000 thrpt 20 399.916 ± 15.858 ops/s [info] FlatMap.monixDelay 10000 thrpt 20 4994.156 ± 40.014 ops/s [info] FlatMap.monixNow 10000 thrpt 20 6253.182 ± 53.388 ops/s [info] FlatMap.scalazApply 10000 thrpt 20 188.387 ± 2.989 ops/s [info] FlatMap.scalazDelay 10000 thrpt 20 1794.680 ± 24.173 ops/s [info] FlatMap.scalazNow 10000 thrpt 20 2041.300 ± 128.729 ops/s

As you can see the Monix Task destroys Scala’s Future for CPU-bound tasks.

NOTE: this benchmark is limited, there are still use-cases where usage of Future is faster (e.g. the Monix Observer uses Future for back-pressure for a good reason) and performance is often not relevant, like when doing I/O, in which case throughput will not be CPU-bound.

5. Task, Scala’s IO Monad

Task is a data type for controlling possibly lazy & asynchronous computations, useful for controlling side-effects, avoiding nondeterminism and callback-hell.

The Monix library provides a very sophisticated Task implementation, inspired by the Task in Scalaz. Same concept, different implementation.

The Task type is also inspired by Haskell’s IO monad, being in this author’s opinion the true IO type for Scala.

This is a matter of debate, as Scalaz also exposes a separate IO type that only deals with synchronous execution. The Scalaz IO is not async, which means that it doesn’t tell the whole story, because on top of the JVM you need to represent async computations somehow. In Haskell on the other hand you have the Async type which is converted to IO, possibly managed by the runtime (green-threads and all).

On the JVM, with the Scalaz implementation, we can’t represent async computations with IO and without blocking threads on evaluation, which is something to avoid, because blocking threads is error prone.

In summary the Task type:

models lazy & asynchronous evaluation

models a producer pushing only one value to one or multiple consumers

it is lazily evaluated, so compared with Future it doesn’t trigger the execution, or any effects until runAsync

it is not memoized by default on evaluation, but the Monix Task can be

doesn’t necessarily execute on another logical thread

Specific to the Monix implementation:

allows for cancelling of a running computation

never blocks any threads in its implementation

does not expose any API calls that can block threads

all async operations are stack safe

A visual representation of where Task sits in the design space:

Eager Lazy

Synchronous A () => A

Coeval[A], IO[A]

Asynchronous (A => Unit) => Unit (A => Unit) => Unit

Future[A] Task[A]

5.1. Sequencing

Redefining our function from section 3 in terms of Task:

import monix.eval.Task def timesTwo(n: Int): Task[Int] = Task(n * 2) // Usage { // Our ExecutionContext needed on evaluation import monix.execution.Scheduler.Implicits.global timesTwo(20).foreach { result => println(s"Result: $result") } //=> Result: 40 }

The code seems to be almost the same as the Future version in section 4.1, the only difference is that our timesTwo function no longer takes an ExecutionContext as a parameter. This is because Task references are lazy, being like functions, so nothing gets printed until the call to foreach which forces the evaluation to happen. It is there that we need a Scheduler, which is Monix’s enhanced ExecutionContext.

Now to do sequencing like in section 3.1:

def timesFour(n: Int): Task[Int] = for (a <- timesTwo(n); b <- timesTwo(n)) yield a + b // Usage { // Our ExecutionContext needed on evaluation import monix.execution.Scheduler.Implicits.global timesFour(20).foreach { result => println(s"Result: $result") } //=> Result: 80 }

And just like with the Future type, that “for comprehension” magic is translated by the Scala compiler to nothing more than calls to flatMap and map, literally equivalent with:

def timesFour(n: Int): Task[Int] = timesTwo(n).flatMap { a => timesTwo(n).map { b => a + b } }

5.2. Parallelism

The story for Task and parallelism is better than with Future, because Task allows fine-grained control when forking tasks, while trying to execute transformations (e.g. map, flatMap) on the current thread and call-stack, thus preserving cache locality and avoiding context switches for what is in essence sequential work.

But first, translating the sample using Future does not work:

// BAD SAMPLE (for achieving parallelism, as this will be sequential) def timesFour(n: Int): Task[Int] = { // Will not trigger execution b/c Task is lazy val fa = timesTwo(n) val fb = timesTwo(n) // Evaluation will be sequential b/c of laziness for (a <- fa; b <- fb) yield a + b }

In order to achieve parallelism Task requires you to be explicit about it:

def timesFour(n: Int): Task[Int] = Task.mapBoth(timesTwo(n), timesTwo(n))(_ + _)

Oh, does mapBoth seem familiar? If those two tasks fork threads on execution, then they will get executed in parallel as mapBoth starts them both at the same time.

5.3. Recursivity

Task is recursive and stack-safe (in flatMap) and incredibly efficient, being powered by an internal trampoline. You can checkout this cool paper by Rúnar Bjarnason on Stackless Scala with Free Monads for getting a hint on how Task got implemented so efficiently.

The sequence implementation looks similar with the one for Future in section 4.3, except that you can see the laziness in the signature of sequence:

def sequence[A](list: List[Task[A]]): Task[List[A]] = { val seed = Task.now(List.empty[A]) list.foldLeft(seed)((acc,f) => for (l <- acc; a <- f) yield a :: l) .map(_.reverse) } // Invocation { // Our ExecutionContext needed on evaluation import monix.execution.Scheduler.Implicits.global sequence(List(timesTwo(10), timesTwo(20), timesTwo(30))).foreach(println) // => List(20, 40, 60) }

6. Functional Programming and Type-classes

When working with well grown functions such as map, flatMap and mapBoth, we no longer care that underlying it all is an “(A => Unit) => Unit”, because these functions are, assuming lawfulness, pure and referentially transparent. This means we can reason about them and their result, divorced from their surrounding context.

This is the great achievement of Haskell’s IO. Haskell does not “fake” side-effects, as functions returning IO values are literally pure, the side-effects being pushed at the edges of the program where they belong. And we can say the same thing about Task. Well, for Future it’s more complicated given its eager nature, but working with Future is not bad either.

And can we build interfaces that abstract over such types as Task, Future, Coeval, Eval, IO, Id, Observable and others?

Yes we can, we’ve already seen that flatMap describes sequencing, while mapBoth describes parallelism. But we can’t describe them with classic OOP interfaces, for one because due to the covariance and contravariance rules of Function1 parameters we’d lose type info in flatMap (unless you use F-bounded polymorphic types, which are more suitable for implementation reuse and aren’t available in other OOP languages), but also because we need to describe a data constructor that can’t be a method (i.e. OOP subtyping applies to instances and not whole classes).

Fortunately Scala is one of the very few languages capable of higher kinded types and with the ability to encode type-classes, which means we’ve got everything needed to port concepts from Haskell 😄

Author’s Rant: The dreaded Monad, Applicative and Functor words strike fear in the hearts of the unfaithful, having given rise to the belief that they are “academic” notions disconnected from real-world concerns, with book authors going to great length to avoid using these words, which includes Scala’s API documentation and official tutorials.

But this is a disservice to both the Scala language and its users. In other languages they are only design patterns that are hard to explain primarily because they can’t be expressed as types. You can count the languages having this expressive capability with one hand. And users suffer because in case of trouble they don’t know how to search for existing literature on the subject, having been deprived of learning the correct jargon.

I also feel this is a flavor of anti-intellectualism, as usual born out of fear of the unknown. You can see it coming from people that really know what they are doing, as none of us is immune. For example Java’s Optional type violates the functor laws (e.g. opt.map(f).map(g) != opt.map(f andThen g)), in Swift 5 == Some(5) which is preposterous and good luck explaining to people that Some(null) actually makes sense for as long as null is a valid value of AnyRef and because otherwise you can’t define Applicative[Option].

6.1. Monad (Sequencing and Recursivity)

This article is not about explaining Monads. There are other great articles for that. But if you’re looking to build an intuition, here’s another one: in the context of data types such as Future or Task, Monads describe sequencing of operations and is the only reliable way to ensure ordering.

“Observation: programmers doing concurrency with imperative languages are tripped by the unchallenged belief that “;” defines sequencing.” – Aleksey Shipilëv

A simple encoding of the Monad type in Scala:

// We shouldn't need to do this :-( import scala.language.higherKinds trait Monad[F[_]] { /** Constructor (said to lift a value `A` in the `F[A]` * monadic context). Also part of `Applicative`, see below. */ def pure[A](a: A): F[A] /** FTW */ def flatMap[A,B](fa: F[A])(f: A => F[B]): F[B] }

And providing an implementation for Future:

import scala.concurrent._ // Supplying an instance for Future isn't clean, ExecutionContext needed class FutureMonad(implicit ec: ExecutionContext) extends Monad[Future] { def pure[A](a: A): Future[A] = Future.successful(a) def flatMap[A,B](fa: Future[A])(f: A => Future[B]): Future[B] = fa.flatMap(f) } object FutureMonad { implicit def instance(implicit ec: ExecutionContext): FutureMonad = new FutureMonad }

This is really powerful stuff. We can now describe a generic function that works with Task, Future, IO, whatever, although it would be great if the flatMap operation is stack-safe:

/** Calculates the N-th number in a Fibonacci series. */ def fib[F[_]](n: Int)(implicit F: Monad[F]): F[BigInt] = { def loop(n: Int, a: BigInt, b: BigInt): F[BigInt] = F.flatMap(F.pure(n)) { n => if (n <= 1) F.pure(b) else loop(n - 1, b, a + b) } loop(n, BigInt(0), BigInt(1)) } // Usage: { // Needed in scope import FutureMonad.instance import scala.concurrent.ExecutionContext.Implicits.global // Invocation fib[Future](40).foreach(r => println(s"Result: $r")) //=> Result: 102334155 }

PRO-TIP: this is just a toy example. For getting serious, see Typelevel’s Cats

6.2. Applicative (Parallelism)

Monads define sequencing of operations, but sometimes we want to compose the results of computations that are independent of each other, that can be evaluated at the same time, possibly in parallel. There’s also a case to be made that applicatives are more composable than monads 😏

Let’s expand our mini Typeclassopedia to put on your wall:

trait Functor[F[_]] { /** I hope we are all familiar with this one. */ def map[A,B](fa: F[A])(f: A => B): F[B] } trait Applicative[F[_]] extends Functor[F] { /** Constructor (lifts a value `A` in the `F[A]` applicative context). */ def pure[A](a: A): F[A] /** Maps over two references at the same time. * * In other implementations the applicative operation is `ap`, * but `map2` is easier to understand. */ def map2[A,B,R](fa: F[A], fb: F[B])(f: (A,B) => R): F[R] } trait Monad[F[_]] extends Applicative[F] { def flatMap[A,B](fa: F[A])(f: A => F[B]): F[B] }

And to expand our Future implementation:

// Supplying an instance for Future isn't clean, ExecutionContext needed class FutureMonad(implicit ec: ExecutionContext) extends Monad[Future] { def pure[A](a: A): Future[A] = Future.successful(a) def flatMap[A,B](fa: Future[A])(f: A => Future[B]): Future[B] = fa.flatMap(f) def map2[A,B,R](fa: Future[A], fb: Future[B])(f: (A,B) => R): Future[R] = // For Future there's no point in supplying an implementation that's // not based on flatMap, but that's not the case for Task ;-) for (a <- fa; b <- fb) yield f(a,b) } object FutureMonad { implicit def instance(implicit ec: ExecutionContext): FutureMonad = new FutureMonad }

So we can now define generic functions based on Applicative which is going to work for Future, Task, etc:

def sequence[F[_], A](list: List[F[A]]) (implicit F: Applicative[F]): F[List[A]] = { val seed = F.pure(List.empty[A]) val r = list.foldLeft(seed)((acc,e) => F.map2(acc,e)((l,a) => a :: l)) F.map(r)(_.reverse) }

PRO-TIP: worth repeating, this is just a toy example. For getting serious, see Typelevel’s Cats

6.3. Can We Define a Type-class for Async Evaluation?

Missing from above is a way to actually trigger an evaluation and get a value out. Thinking of Scala’s Future, we want a way to abstract over onComplete. Thinking of Monix’s Task we want to abstract over runAsync. Thinking of Haskell’s and Scalaz’s IO, we want a way to abstract over unsafePerformIO.

The FS2 library has defined a type-class called Effect that goes like this (simplified):

trait Effect[F[_]] extends Monad[F] { def unsafeRunAsync[A](fa: F[A])(cb: Try[A] => Unit): Unit }

This looks like our initial Async type, very much similar with Future.onComplete, with Task.runAsync and could be applied to IO.unsafePerformIO.

However, this is not a real type-class because:

it is lawless and while that’s not enough to disqualify it (after all, useful lawless type-classes like Show exist), the bigger problem is …

as shown in section 3.3, in order to avoid the Wrath of StackOverflowError, we need some sort of execution context or thread-pool that can execute tasks asynchronously without blowing up the stack

And such an execution context is different from implementation to implementation. Java will use Executor, the Scala Future uses ExecutionContext, Monix uses Scheduler which is an enhanced ExecutionContext, FS2 and Scalaz use Strategy which wraps an Executor for forking threads and don’t inject a context when their unsafePerformIO or runAsync gets called (which is why many of the Scalaz combinators are in fact unsafe), etc.

We could apply the same strategy as with Future, to build the type-class instance by taking a implicit whatever: Context from the scope. But that’s a little awkward and inefficient. It’s also telling that we can’t define flatMap only in terms of Effect.unsafePerformIO, not without that execution context. And if we can’t do it, then the type should probably not inherit from Monad because it’s not necessarily a Monad.

So I’m personally not sure - if you have suggestions for what should be introduced in Cats, I’d love to hear them.

I do hope you enjoyed this thought experiment, designing things is fun 😎

7. Picking the Right Tool

Some abstractions are more general purpose than others and personally I think the mantra of “picking the right tool for the job” is overused to defend poor choices.

That said, there’s this wonderful presentation by Rúnar Bjarnason called Constraints Liberate, Liberties Constrain that really drives the point home with concurrency abstractions at least.

As said, there is no silver bullet that can be generally applied for dealing with concurrency. The more high-level the abstraction, the less scope it has in solving issues. But the less scope and power it has, the simpler and more composable the model is. For example many developers in the Scala community are overusing Akka Actors - which is a great library, but not when misapplied. Like don’t use an Akka Actor when a Future or a Task would do. Ditto for other abstractions, like the Observable pattern in Monix and ReactiveX.

Also learn by heart these 2 very simple rules:

avoid dealing with callbacks, threads and locks, because they are very error prone and not composable at all

avoid concurrency like the plague it is

And let me tell you, concurrency experts are first of all experts in avoiding concurrency 💀

The article Asynchronous Programming and Scala first appeared on alexn.org.

Fixing scala.collection.Iterator

Alexandru Nedelcu — Mon, 16 Jan 2017 00:00:00 +0000

The venerable Iterator interface we all love and hate could use some improvements. This is a follow-up to my previous article, in which I talked about getting rid of Traversable because the Iterable and Iterator duo is enough for Scala’s standard library.

As a reminder, the Iterator interface is something like this:

trait Iterator[+A] { def hasNext: Boolean def next(): A }

It’s a destructive interface that is consumed for as long as you call next(), it obviously has “identity” and you’re supposed to use it like this:

val cursor: Iterator[Int] = ??? var sum = 0 while (cursor.hasNext) { sum += cursor.next() }

Problem 1: Both methods (hasNext, next) are side-effecting

You could say that hasNext is not supposed to move the internal cursor / pointer / index and thus it shouldn’t be side-effecting, but that’s not true, because in many cases the only way to know if there is a next element to be served is to trigger a side-effecting read.

And so the problem is that both hasNext and next() are side-effecting and in my opinion the result of the wrong method is getting cached. When you work with Functional Programming for a while, you start noticing when APIs have their side-effects screwed ;-)

We can’t really blame Scala though. This interface has been imported from Java and kept similar probably for remaining familiar.

But let me illustrate by building an iterator for reading an InputStream:

class IteratorFromStream(in: InputStream, chunkSize: Int) extends Iterator[Array[Byte]] { private val buffer = new Array[Byte](chunkSize) private var chunk: Array[Byte] = _ private var hasChunk = false def hasNext: Boolean = { if (!hasChunk) { val len = in.read(buffer) if (len >= 0) { chunk = util.Arrays.copyOf(buffer, len) hasChunk = true } else { in.close() } } hasChunk } def next(): Array[Byte] = { if (hasNext) { val ref = chunk chunk = null // GC purposes hasChunk = false ref } else { throw new NoSuchElementException("InputStream is empty") } } }

Not that particularly exciting and you can see how next() has to duplicate the work of hasNext and that hasNext itself is side-effecting, because we have to read from the InputStream before being able to answer that question.

We can do better and we don’t have to be original about it. Behold the alternative inspired by IEnumerator from C#:

trait Iterator[+A] { // Side-effecting, moves the cursor def moveNext(): Boolean // Not side-effecting, can be called multiple times def current: A }

Usage is straightforward:

val cursor: Iterator[Int] = ??? var sum = 0 while (cursor.moveNext()) { sum += cursor.current }

This interface feels more natural to developers because “moving” the cursor is the side-effect, not reading the current value. And here’s how the above implementation changes:

class IteratorFromStream(in: InputStream, chunkSize: Int) extends Iterator[Array[Byte]] { private val buffer = new Array[Byte](chunkSize) private var chunk: Array[Byte] = _ def moveNext(): Boolean = { val len = in.read(buffer) if (len >= 0) { chunk = util.Arrays.copyOf(buffer, len) true } else { chunk = null in.close() false } } def current: Array[Byte] = { if (chunk == null) throw NoSuchElementException("current") chunk } }

Notice how this simplifies things on the implementation side as well.

UPDATE: M.Odersky points out in the comments that the standard library has a BufferedIterator implementation that caches the current head and can be used for convenience.

Problem 2: Iterator comes with operations attached

At the beginning I gave you a simplified Iterator definition, however I lied. The true scala.collection.Iterator is closer to this:

package scala.collection trait Iterator[+A] extends TraversableOnce[A] { def hasNext: Boolean def next(): A def isTraversableAgain = false def isEmpty: Boolean = !hasNext def hasDefiniteSize = isEmpty def map[B](f: A => B): Iterator[B] = ??? def take(n: Int): Iterator[A] = slice(0, n) def drop(n: Int): Iterator[A] = ??? def slice(from: Int, until: Int): Iterator[A] = ??? def map[B](f: A => B): Iterator[B] = ??? def ++[B >: A](that: => GenTraversableOnce[B]): Iterator[B] = ??? def flatMap[B](f: A => GenTraversableOnce[B]): Iterator[B] = ??? def filter(p: A => Boolean): Iterator[A] = ??? def corresponds[B](that: GenTraversableOnce[B])(p: (A, B) => Boolean): Boolean = ??? def withFilter(p: A => Boolean): Iterator[A] = ??? def filterNot(p: A => Boolean): Iterator[A] = ??? def collect[B](pf: PartialFunction[A, B]): Iterator[B] = ??? def scanLeft[B](z: B)(op: (B, A) => B): Iterator[B] = ??? def scanRight[B](z: B)(op: (A, B) => B): Iterator[B] = ??? def takeWhile(p: A => Boolean): Iterator[A] = ??? def partition(p: A => Boolean): (Iterator[A], Iterator[A]) = ??? def span(p: A => Boolean): (Iterator[A], Iterator[A]) = ??? def dropWhile(p: A => Boolean): Iterator[A] = ??? def zip[B](that: Iterator[B]): Iterator[(A, B)] = ??? def padTo[A1 >: A](len: Int, elem: A1): Iterator[A1] = ??? def zipWithIndex: Iterator[(A, Int)] = ??? def foreach[U](f: A => U) = ??? def forall(p: A => Boolean): Boolean = ??? def exists(p: A => Boolean): Boolean = ??? def contains(elem: Any): Boolean = ??? def find(p: A => Boolean): Option[A] = ??? def indexWhere(p: A => Boolean): Int = ??? def indexOf[B >: A](elem: B): Int = ??? def buffered: BufferedIterator[A] = ??? def grouped[B >: A](size: Int): GroupedIterator[B] = ??? def copyToArray[B >: A](xs: Array[B], start: Int, len: Int): Unit = ??? def sameElements(that: Iterator[_]): Boolean = ??? def toTraversable: Traversable[A] = ??? def toIterator: Iterator[A] = ??? def toStream: Stream[A] = ??? // .... }

Well, at this point you should be thinking that this violates the principles of OOP design. When Iterator comes with operations like map and filter, that are polymorphic and can be overridden, it is no longer just a minimal protocol for “iterating over things”, but a big, fat interface.

You see, there isn’t a single possible implementation for map or for take and by providing such operations with a default implementation the Iterator is imposing to users how those operations should behave. Or more specifically:

these operations have lazy behavior, until overridden in subclasses

assume that the protocol is set in stone

But Iterator is a fine example of an OOP interface because you can add restrictions to it. Lo and behold how OOP inheritance is supposed to work:

trait CloseableIterable[+A] extends Iterable[A] { def iterator: CloseableIterator[A] } trait CloseableIterator[+A] extends Iterator[A] with AutoCloseable { /** Closes this resource, relinquishing any underlying resources. */ def close(): Unit }

BAM, we just invalidated more than 80 operations provided by Scala’s Iterator and Iterable. Are you going to override them all?

If you say yes, then you don’t know what you’re getting yourself into, plus what are you going to do when Scala 2.13 (or whatever the next version is) comes with new operators that need to be overridden? Are you going to remember to do it? It’s a hard problem.

I don’t mind having implementations of map and filter for Iterator, but Scala lacks a minimal interface that provides just the raw protocol. There is value in simplicity. Notice Java’s Iterator, notice C#’s IEnumerator, notice how they don’t come with operators attached. Instead, for C# at least, you can import Ix.NET in your project, which gives you a bunch of extension methods you can work with, no strings attached. Scala could also use type-classes which are much better than extension methods. But instead what currently happens in Scala’s collection library can be seen as inheritance hell.

Not everything needs map and flatMap on it.

UPDATE: there is now an issue for discussing this at collection-strawman/issues/#17.

Non-problem: early termination & resource handling

The Iterator interface alone is not enough to expose streams linked to file handles, network sockets or other resources that need to be disposed when terminated early.

So rookies (also because of problem 2 above) can end up with unconsumed iterators, creating possible connection leaks, because it’s easy:

iterator.take(100)

However I don’t view this as being a problem because:

we don’t need to do resource handling everywhere

as demonstrated in the sample with CloseableIterator above, you can build proper resource handling on top of Iterator

doing I/O by means of an Iterator is often a bad idea, given that Iterator is not capable of asynchronous boundaries, with I/O operations often being asynchronous

Having a CloseableIterator in the standard library wouldn’t be bad though, however given the very complex inheritance hierarchy and the traversable grandparents, I’m afraid to wish for it.

The article Fixing scala.collection.Iterator first appeared on alexn.org.

Why Scala's Traversable Is Bad Design

Alexandru Nedelcu — Fri, 13 Jan 2017 00:00:00 +0000

Given there’s a Scala collection redesign discussion going on, it’s a good time to talk about one of my personal pet peeves: the existence of Traversable in the standard library, along with its variants like TraversableLike and TraversableOnce. Apparently this interface is missing in the new design and that’s awesome.

It’s easy to make API mistakes, we all do it and it’s important to learn from past mistakes, this document serving as a lesson for why Traversable is a bad idea.

Claims:

Traversable has implicit behavior assumptions that are not visible in its exposed signature, the API being error prone

Iterating over a Traversable has worse performance than Iterator

There exists no Traversable data type that doesn’t admit an efficient Iterator implementation, thus Traversable being completely redundant

As a reminder and you can also read the docs, the Traversable is a trait like the following:

trait Traversable[+A] { def foreach(f: A => Unit): Unit }

The standard library also has the venerable Iterable / Iterator:

trait Iterable[+A] { def iterator(): Iterator[A] } trait Iterator[+A] { def hasNext: Boolean def next(): A }

Can you spot the similarities?

You should, because these 2 interfaces are supposed to be duals. So if you think of Traversable as being defined by that foreach function, then Iterable is that function with its arrows reversed:

type Traversable[A] = (A => ()) => () type Iterable[A] = () => (() => A)

Now this is interesting. For one Traversable is a sort of inversion of control technique, so instead of having a cursor that you have to manually advance, you now register a callback to a function and that callback gets called for you on each element. This actually frees us from certain Iterator constraints. For example with a push-based API we should no longer care when those function calls happen.

But you should already spot problems with the above definition. Our Iterable function signature isn’t complete, this one is:

type Iterable[+A] = () => Iterator[A] type Iterator[+A] = () => Try[Option[A]]

Or in other words any Iterator can:

give us the next element,

or signal completion or failure

This means that the actual dual of Iterator is:

type Observer[A] = Try[Option[A]] => Unit

Or for those OOP-oriented among us, I give you the Observer as championed by Rx.NET, as the true dual of Iterator:

trait Observer[-A] { def onNext(a: A): Unit def onComplete(): Unit def onError(ex: Throwable): Unit }

(Hello Monix :-))

This matters because Traversable has no way to signal completion or failure, unless you get a guarantee that all the processing happens synchronously, everything being over after the invocation of its foreach.

As an abstraction, this makes it useless when compared with Iterable and Iterator. If you introduce the synchronous execution constraint, there exists no data type that can implement Traversable and that doesn’t admit an Iterator implementation. None.

Even more problematic in my opinion is that this restriction isn’t visible in its API, unless your eyes are trained for it. With Iterator.next() whether you want it or not, you have to process things synchronously, because the signature says so.

Also problematic is TraversableOnce, which is supposed to be a traversable that can only be traversed once, like its name says. We’ve got this:

trait TraversableOnce[+A] { def foreach(f: A => Unit): Unit } trait Traversable[+A] extends TraversableOnce[A]

Besides the name and the inheritance relationship, there is no difference. This is another problem. Even if the API is effectful/impure, we should still be able to use types serving as documentation. Contrast with the Iterable / Iterator separation. Iterating over an Iterator is known to consume it and you can see this in its API. And the generator/factory part is in Iterable, which is good separation of concerns.

Traversable also has worse performance than Iterator. The ugly truth is that the JVM hasn’t been doing a good job at inlining that function reference you pass to foreach. This is called the inlining problem, which happens for megamorphic function calls in hot inner loops.

So there you have it and I hope that along with the redesign we’ll get rid of Traversable.

My opinions have been highly influenced by the work of Erik Meijer, if you want to learn more checkout this presentation: Contravariance is the Dual of Covariance Implies Iterable is the Dual of Observable.

The article Why Scala's Traversable Is Bad Design first appeared on alexn.org.

Add to NewsBlur Bookmarklet

Alexandru Nedelcu — Fri, 11 Nov 2016 00:00:00 +0000

I’m using NewsBlur.com for consuming RSS feeds. It’s a pretty cool service, however adding a new RSS feed from iOS proves to be a challenge and the “goodies” section provides no way for easily adding a bookmarklet for adding a feed.

The iPad itself makes it difficult to add bookmarklets, so here’s a tutorial:

Go to NewsBlur.com and add it as a bookmark (click on the box with the arrow in it next to the Safari address bar)

We need to edit the new bookmark: tap the address bar in Safari and you should see all bookmarks, press and hold on the new “NewsBlur.com” bookmark that you created and then tap “Edit”

Change the title to something like: “Add to NewsBlur”

Copy/paste the following text for the URL:

javascript:(function%20()%20%7Bvar%20l%3Ddocument.location%2B%27%27%3Bif%20(l.match(%2F%5E(%3F%3Ahttps%3F%3A%5B%2F%5D%7B2%7D(%3F%3Awww.)%3F)%3Fnewsblur.com%2Fi))%20alert(%22Cannot%20add%20NewsBlur.com%20itself!%22)%3B%20else%20window.location%20%3D%20%27https%3A%2F%2Fwww.newsblur.com%2F%3Furl%3D%27%2BencodeURIComponent(l)%3B%7D)()%3Bvoid(0)
For the desktop you can also drag this link:

Add to NewsBlur

Here is the unencoded Javascript for your inspection:

(function () { var l = document.location+''; if (l.match(/^(?:https?:[/]{2}(?:www.)?)?newsblur.com/i)) alert("Cannot add NewsBlur.com itself!"); else window.location = 'https://www.newsblur.com/?url=' + encodeURIComponent(l); })();void(0)

Enjoy!

The article Add to NewsBlur Bookmarklet first appeared on alexn.org.

Monix Task vs Scalaz vs Future, The Benchmark

Alexandru Nedelcu — Thu, 25 Aug 2016 00:00:00 +0000

The Monix Task was heavily inspired by scalaz.concurrent.Task and by scala.concurrent.Future. That’s not a secret and I’ll be forever grateful to their authors. I’ve ran a benchmark and I’m glad to report that the Monix Task beats in performance both.

Such results are actually unexpected, because the Monix Task has to do tricks in order to be “cancelable”, a trait that allows it to close opened resources when race conditions happen, which really means extra footwork. But no, right now, it beats both in performance and by quite the margin.

Details:

Benchmark used is TaskGatherBenchmark in the repository

Monix version: 2.0-RC13

Scalaz version: 7.2.4

Scala version: 2.11.8

Java version: 1.8.0_60

OS: OS X El Captain, version 10.11.6

Sequence

The purpose of this test is the performance of flatMap, or in other words the performance of the run-loop, on both normal/synchronous tasks and tasks that are forked in separate (logical) threads. So in other words:

Task.sequence(tasks)

Which is translated more or less into this:

tasks.foldLeft(init)((acc,et) => acc.flatMap(b => et.map(e => b += e)))

So for synchronous tasks (that evaluate immediately), and note here that Scala’s Future is not applicable since Future is not trampolined:

Source Type Operation Score Error Units

Monix Sync Sequence 6716.906 157.947 ops/s

Scalaz Sync Sequence 3518.888 167.148 ops/s

And for tasks that fork threads on execution:

Source Type Operation Score Error Units

Monix Forked Sequence 2044.624 24.852 ops/s

Scalaz Forked Sequence 1090.355 15.851 ops/s

S.Future Forked Sequence 1753.614 20.871 ops/s

As you can see, the Monix Task has twice the throughput of Scalaz and fares quite better compared with Scala’s standard Future.

Gather

The gather operation would be:

Task.gather(tasks)

This works like sequence, except that the evaluation has non-ordered effects. What this means is that, if the tasks are forking threads, then they get executed in parallel.

For synchronous/immediate tasks the numbers are:

Source Type Operation Score Error Units

Monix Sync Gather 3800.559 341.509 ops/s

Scalaz Sync Gather 2152.441 13.569 ops/s

S.Future Forked Sequence 1753.614 20.871 ops/s

And for forked tasks:

Source Type Operation Score Error Units

Monix Forked Gather 1396.797 17.098 ops/s

Scalaz Forked Gather 1014.452 13.569 ops/s

S.Future Forked Sequence 1753.614 20.871 ops/s

Including the results of Future.sequence as well, because Future has strict evaluation and it can be used to execute futures in parallel. The performance of gather can be worse than Future.sequence, because of the execution model. But if it executes tasks that have immediate execution, or a mixed batch, then it is much better.

Gather Unordered

The gatherUnordered operation would be:

Task.gatherUnordered(tasks)

This behaves like gather, except that it does not care for the order in which the results are served. Can have much better performance if you don’t care about order.

For synchronous/immediate tasks:

Source Type Operation Score Error Units

Monix Sync Unordered 5654.462 150.792 ops/s

Scalaz Sync Unordered 3340.645 244.145 ops/s

S.Future Forked Sequence 1753.614 20.871 ops/s

For forked tasks:

Source Type Operation Score Error Units

Monix Forked Unordered 1658.055 12.114 ops/s

Scalaz Forked Unordered 1657.454 35.218 ops/s

S.Future Forked Sequence 1753.614 20.871 ops/s

Again, performance is really good for synchronous tasks, whereas for forked tasks it evens out with the performance of Future.sequence.

Raw output

[info] # Run complete. Total time: 00:04:28 [info] [info] Benchmark Mode Cnt Score Error Units [info] TaskGatherBenchmark.gatherMonixA thrpt 10 1396.797 ± 17.098 ops/s [info] TaskGatherBenchmark.gatherMonixS thrpt 10 3800.559 ± 341.509 ops/s [info] TaskGatherBenchmark.gatherScalazA thrpt 10 1014.452 ± 13.569 ops/s [info] TaskGatherBenchmark.gatherScalazS thrpt 10 2152.441 ± 24.811 ops/s [info] TaskGatherBenchmark.sequenceFutureA thrpt 10 1753.614 ± 20.871 ops/s [info] TaskGatherBenchmark.sequenceMonixA thrpt 10 2044.624 ± 24.852 ops/s [info] TaskGatherBenchmark.sequenceMonixS thrpt 10 6716.906 ± 157.947 ops/s [info] TaskGatherBenchmark.sequenceScalazA thrpt 10 1090.355 ± 15.851 ops/s [info] TaskGatherBenchmark.sequenceScalazS thrpt 10 3518.888 ± 167.148 ops/s [info] TaskGatherBenchmark.unorderedMonixA thrpt 10 1658.055 ± 12.114 ops/s [info] TaskGatherBenchmark.unorderedMonixS thrpt 10 5654.462 ± 150.792 ops/s [info] TaskGatherBenchmark.unorderedScalazA thrpt 10 1657.454 ± 35.218 ops/s [info] TaskGatherBenchmark.unorderedScalazS thrpt 10 3340.645 ± 244.145 ops/s

Cheers!

The article Monix Task vs Scalaz vs Future, The Benchmark first appeared on alexn.org.

Akka & Monix - Typelevel Summit, Oslo 2016

Alexandru Nedelcu — Sun, 15 May 2016 00:00:00 +0000

Video link (open on YouTube.com)

Video link (open on YouTube.com)

Akka & Monix: Controlling Power Plants - my presentation from the Typelevel Summit, Oslo, 2016:

Slides (PDF file)

Video (YouTube)

Also see:

Monix Task, flatMap(Oslo), 2016

the Monix project

Abstract

This talk is about my experience in dealing with modeling behavior by processing asynchronous soft realtime signals from different sources using Akka, along with Monix, the library for building asynchronous and event-based logic.

It’s an experience report from my work in monitoring and controlling power plants. We do this by gathering signals in real time and modeling state machines that give us the state in which an asset is in. The component I worked on is the one component in the project that definitely adheres to FP principles, the business logic being described with pure functions and data-structures and the communication being handled by actors and by Observable streams. I want to show how I pushed side effects at the edges, in a very pragmatic setup.

This presentation will focus on Akka best practices, the wiring needed for describing purely functional state, along with a presentation of Monix Observable and how that helps.

The article Akka & Monix - Typelevel Summit, Oslo 2016 first appeared on alexn.org.

Monix Task - flatMap(Oslo) 2016

Alexandru Nedelcu — Tue, 10 May 2016 00:00:00 +0000

Video link (open on YouTube.com)

Video link (open on YouTube.com)

Monix Task: Lazy, Async & Awesome - my presentation from flatMap(Oslo) 2016:

Slides (PDF file)

Video (YouTube)

Also see:

Akka & Monix, Typelevel Summit, Oslo, 2016

Monix project

Abstract

Scala’s Future from the standard library is great, but sometimes we need more.

A Future strives to be a value, one detached from time and for this reason its capabilities are restricted and for some use-cases its behavior ends up being unintuitive. Hence, while the Future/Promise pattern is great for representing asynchronous results of processes that may or may not be started yet, it cannot be used as a specification for an asynchronous computation.

The Monix Task is in essence about dealing with asynchronous computations and non-determinism, being inspired by the Scalaz Task and designed from the ground up for performance and to be compatible with Scala.js/Javascript runtimes and with the Cats library. It also makes use of Scala’s Future to represent results, the two being complementary.

In this talk I’ll show you its design, when you should use it and why in dealing with asynchronicity it’s better to work with Task instead of blocking threads.

The article Monix Task - flatMap(Oslo) 2016 first appeared on alexn.org.

Avoid Javaisms: Mocks, Stubs, DI is Code Smell

Alexandru Nedelcu — Tue, 15 Dec 2015 00:00:00 +0000

I’m a man of strong opinions and I truly believe that when we are doing mocking, stubbing, dependency injection and integration testing, such practices represent clear signals for code smell, meaning code that sucks as a symptom of a bigger problem, one of design. The lumping together of these practices is not an accident, as they are related.

Let’s take an example. Often in our components we’ve got dependencies, other components only slightly related and on which we depend for producing the desired effects. Things like database access, for both reads and writes. In true Java spirit, lets build our noun:

trait DBService { def readItemConfig(uuid: UUID): Option[ItemConfig] def saveItemConfig(uuid: UUID, config: ItemConfig): Unit def readDatapoints(item: UUID, offset: Int, count: Int): Seq[Datapoint] def persistDatapoint(item: UUID, dp: Datapoint): Unit }

This interface is reasonably abstract, meaning we aren’t leaking too many underlying storage details. Well, we are assuming synchronous responses and the datapoints are read in batches instead of a nice stream, but those are details that can be corrected and the interface works for a text file, PostgreSQL, MongoDB or what have you. So now we can depend on it:

class ItemActor(db: DBService) extends Actor { def receive = { case Init(uuid) => for (cfg <- db.readItemConfig(uuid)) context.become(active(cfg)) } def active(cfg: ItemConfig): Receive = ??? }

Of course, if you’ve got masochistic tendencies, you might prefer the Cake pattern, being the same thing, only much worse, because now you’ve got garbage enhanced by global state and polymorphic superpowers, sucking as much as Guice, only at compile-time:

trait ItemActorComponentImpl { self: DBServiceComponent => class ItemActor extends Actor { def receive = { case Init(uuid) => for (cfg <- dbService.readItemConfig(uuid)) context.become(active(cfg)) } def active(cfg: ItemConfig): Receive = ??? } }

I personally can’t stand that, being the epitome of good intentions gone wrong. But back to our point, if we want to test this actor, we’d have to mock or stub our DBService, right?

Well, here’s the problem mate: until now this actor only depends on DBService.readItemConfig, yet we have to mock or stub the entire interface of DBService. And having to mock or stub things unrelated to testing this functionality should indicate that this code is too tightly coupled. Right there your nose should reject the air emanated from this code and it’s common sense that often save us, even though we often can’t place our finger on the problem.

OK, OK, lets fix this somewhat using a common Java “best practice”, by splitting this interface into smaller modules. Our DBService interface does too much, or so the popular wisdom would say.

trait ItemConfigsRepository { def read(uuid: UUID): Option[ItemConfig] def save(uuid: UUID, config: ItemConfig): Unit } trait DatapointsRepository { def readList(item: UUID, offset: Int, count: Int): Seq[Datapoint] def persist(item: UUID, dp: Datapoint): Unit }

That feels better, right? By splitting functionality in smaller units of finer-grained stuff, this should ameliorate our dependency woes. Wrong! Now we’ve got two problems:

class ItemActor (icsRepo: ItemConfigsRepository, dpsRepo: DatapointsRepository) extends Actor { def receive = { case Init(uuid) => for (cfg <- icsRepo.read(uuid)) context.become(active(cfg, State.empty)) } def active(cfg: ItemConfig, state: State): Receive = { case Signal(value) => val newState = state.evolve(value) dpsRepo.persist(cfg.uuid, state.powerOutput) context.become(active(cfg, newState)) } }

BAM, more dependencies, more garbage, more mocks and stubs. Does this ring a bell? Cake makes it worse btw. But anyway, now we can see that our solution with ItemConfigsRepository doesn’t work, as readItem is often not used in the same place as writeItem, so our action had an opposite effect of what we wanted.

How can this be, our interfaces are abstract and split in small units according to best practices, yet what are we doing wrong?

Maybe this isn’t so bad, right? I mean surely we can stub the dependencies that aren’t actually used and be done with it, everybody else is doing it. And look, we’ve got dependency injection to deal with all the constructor annoyances. Oh, except when you’ve got more to add, things unrelated to the actual business logic, like persisting more stuff:

def active(cfg: ItemConfig, state: State): Receive = { case Signal(value) => val newState = state.evolve(value) dpsRepo.persist(cfg.uuid, state.powerOutput) dpsRepo.persist(cfg.uuid, state.basepoint) // <-- here context.become(active(cfg, newState)) }

So now with mocks, your tests are broken even though the business logic hasn’t changed, whereas with stubs that ignore those calls, both of those calls might as well not exist. Both outcomes are wrong.

Here we are having a side-effect, which is persisting values in the database in response to that State being evolved when receiving Signal values.

Yet we have a non-obvious implementation leak. From the point of view of this actor, those persistence calls are just signals that a state change happened and that we could do (but not necessarily) something in response, but the actor should not care at all that what we are doing is actual persistence in a database repository, or sending values over an akka remoting connection, or over web-socket, or dumping some logs on disk. These concerns should be totally outside of our component and all we should be testing is if our component is signaling stuff to the outside world. We tried fixing DBService but in fact our Actor is broken.

Meet the famous and underused Observer pattern. And its sibling on steroids ReactiveX. Here’s the sample above using Monifu:

class ItemActor(output: Channel[Signal]) extends Actor { def receive = { case cfg: ItemConfig => context.become(active(cfg, State.empty)) } def active(cfg: ItemConfig, state: State): Receive = { case Signal(value) => val newState = state.evolve(value) output.pushNext(newState) context.become(active(cfg, newState)) case cfg: ItemConfig => context.become(active(cfg, state)) } } // ... // in a galaxy far, far away dbConfigSource.subscribe { config => actor ! itemConfig } output.subscribe { signal => dbService.persist(signal.uuid, signal.powerOutput) dbService.persist(signal.uuid, signal.basepoint) }

OK, I know that Akka actors are cool and all, this is not about you using or not Akka actors. So lets implement the Observer pattern on top of Akka actors to see how that looks like:

class MyActor extends Actor { def receive = active(State.empty, Set.empty) def active(state: State, subscribers: Set[ActorRef]): Receive = { case "register" => val ref = sender() if (!subscribers.contains(ref)) { context.watch(ref) context.become(active(state, subscribers + ref)) } case Terminated(ref) => context.unwatch(ref) context.become(active(state, subscribers - sender)) case Signal(value) => val newState = state.evolve(value) for (subscriber <- subscribers) { for (event <- newState.events) subscriber ! event } context.become(active(newState, subscribers)) } }

It has been my general opinion that actors should mutate their state with context.become and one reason is because that enables us to separate the business logic from the actor and leave the actor to handle just the communication side. Should the above actor be tested? Maybe, if you’ve got time, but it really isn’t a priority, because it doesn’t contain business logic. Let’s go deeper. The business logic, exposed by newState = state.evolve would be something like this:

case class Event(value: Int) case class State (value: Int, lastEvent: Int, events: Seq[Event]) { def evolve(newValue: Int): State = { if (math.abs(newValue - lastEvent) > 100) State(newValue, newValue, Seq(Event(newValue))) else copy(value = newValue) } def popEvents: (Seq[Event], State) = (events, copy(events = Seq.empty)) } object State { val empty = State(0, 0, Seq.empty) }

M’kay, so this does have business logic that would be valuable for testing. AND we are modeling the side-effects in a pure way, by saying on each evolve “here’s a bunch of signals to emit Bob, I don’t care how you do it or who reads them”.

Does this code have any dependencies whatsoever? No, it’s pure and can be tested in total isolation and for things that actually matter, you know, unit testing. This is the essence of functional programming and (I hope) of Scala. Because whenever you’re using Mockito it means that you’re not doing the above.

In other words:

dependency injection, mocking and stubbing is meant for hiding garbage under the rug

for writes, you don’t have to sprinkle your side-effecting calls all over the place, when you can decouple those concerns by implementing signaling by means of the Observer pattern

for reads you can have components that push those configurations into your component and the actual wiring is very often not worth testing, because …

testing has diminishing returns: math formulas, the whiles and the ifs and the decision making are very important, but the interaction with external components or systems? Not so much, especially because you end up testing other people’s libraries and frameworks, essentially duplicating functionality and generally not worth the trouble

integration testing is like meat eating - it’s not that meat eating is bad for you per se, but rather the fact that by eating meat you’re not eating enough vegetables. You see, we have a finite budget and by doing integration testing it means that you’re not doing something else. And people that do integration testing in their code are often the people that gave up on refactoring and unit testing their convoluted and tightly coupled code

mocks and stubs are a definite sign that your components are too tightly coupled and that your business logic is mixed with side-effects of short term value involving third-party components and systems. It’s usually a sign that you need to clean up your mess

testing Akka actors is horrible because of their asynchronous nature and that’s a good thing, because it makes you realize that actors are about communication and that you don’t care about communication in your unit tests, so you’d better not have business logic in them ;-)

your DBService can always fail for reasons outside of your control, so instead of testing DBService, your effort is much better spent in making your own component more resilient to failure and in improving logging, because when it comes to external systems, testing the happy path is worthless, being the edge cases that get you

personally I dislike very much tests that pretend to test things, but have zero value - writing unit tests is just a means to an end, take time and have to be maintained, so don’t burden your team with fragile tests that don’t test anything of value, because that’s not why you’ve been hired

Pain is good. Mocks, stubs, DI, integration tests are about avoiding pain by fixing the symptoms rather than the disease. Don’t treat the symptoms, treat the disease.

Also see the list of best practices I initiated that’s free of Javaisms.

Cheers,

The article Avoid Javaisms: Mocks, Stubs, DI is Code Smell first appeared on alexn.org.

Monifu vs Akka Streams

Alexandru Nedelcu — Sun, 06 Sep 2015 00:00:00 +0000

Back in June I attended Scala Days in Amsterdam and participated at a talk by Mathias Doenitz on The Reactive Streams Implementation Landscape. It was a good talk, yet I felt a little bias towards Akka Streams, which is natural coming from somebody that is contributing to Akka, but let me give you the perspective from the other side.

For a short introduction, we are talking about libraries that are meant for stream processing. Several libraries exist for stream processing, a craze that started with Rx.NET, or with Iteratees on the Haskell side and since then we ended up with ports and enhancements, like RxJava, Rx.js, Bacon.js, Play’s Iteratees, Scalaz Streams and more recently Akka Streams.

The library I’ve been working on is named Monifu. It has just reached 1.0-RC1, it is built on idiomatic Scala principles, is cross-compiled to Scala.js, is freaking awesome and even though it is inspired by Rx, compared to other Rx implementations, it was designed from scratch to deal with back-pressure. It’s also implementing the 1.0 version of the Reactive Streams protocol for interoperability. Checkout this cool sample of client/server communications, where both the server-side and the browser-side is handled by Monifu: code / demo (that’s a free Heroku node, go easy on it :-)).

Code

I am biased, I admit. But personally I find the design of Akka Streams to be conceptually ugly.

Mathias was kind to release his comparison on GitHub, containing the same problem solved with Akka Streams, RxJava and Reactor and I think this provides a good starting point for a comparison. So lets take Mathias’s own sample for Akka Streams and compare it with my own sample for Monifu. Akka Streams is missing some useful operators, like operators for throttling, therefore the code needs to implement it.

To throttle once per second, here’s what the Akka Streams sample is doing:

// Akka Streams sample def onePerSecValve: Flow[State, State, Unit] = Flow() { implicit b ⇒ import FlowGraph.Implicits._ val zip = b.add(ZipWith[State, Tick.type, State](Keep.left) .withAttributes(OperationAttributes.inputBuffer(1, 1))) val dropOne = b.add(Flow[State].drop(1)) Source(Duration.Zero, 1.second, Tick) ~> zip.in1 zip.out ~> dropOne.inlet (zip.in0, dropOne.outlet) }

Monifu has throttling operators built-in, but if that weren’t the case, here’s how you could implement sampling per second:

// Monifu sample def throttleOnePerSec[T](source: Observable[T]): Observable[T] = { val tick = Observable.intervalAtFixedRate(1.second, 1.second) source.whileBusyDropEvents.zip(tick) .map { case (elem, _) => elem } }

Both pieces of code are doing roughly the same thing. A different data-source is being started that generates a tick over every second and that data-source is being zipped with our source. Events emitted by our source are dropped as long as the tick is silent and then when the tick event happens we emit whatever we’ve got. Using zip for sampling is actually very inefficient, as zip is a fundamentally concurrent operator, but this is just for comparing apples with apples ;-)

So why is the code for Akka Streams looking so complicated? Well that’s because of its design, but lets come back to that later. In the Akka Streams sample, here’s how the original data-source is being split in two streams (filterInner and filterOuter) and then merged:

// Akka Streams sample Source(() ⇒ new RandomDoubleValueGenerator()) .grouped(2) .map { case x +: y +: Nil ⇒ Point(x, y) } .via(broadcastFilterMerge) // ... def broadcastFilterMerge: Flow[Point, Sample, Unit] = Flow() { implicit b ⇒ import FlowGraph.Implicits._ val broadcast = b.add(Broadcast[Point](2)) // split one upstream into 2 downstreams val filterInner = b.add(Flow[Point].filter(_.isInner).map(InnerSample)) val filterOuter = b.add(Flow[Point].filter(!_.isInner).map(OuterSample)) val merge = b.add(Merge[Sample](2)) // merge 2 upstreams into one downstream broadcast.out(0) ~> filterInner ~> merge.in(0) broadcast.out(1) ~> filterOuter ~> merge.in(1) (broadcast.in, merge.out) }

Here’s Monifu’s version, doing the same thing.

// Monifu sample val source = Observable .fromIterator(new RandomDoubleValueGenerator()) .buffer(2) .map { case Seq(x, y) ⇒ Point(x, y) } .share() // shares the data-source val innerSamples = source.filter(_.isInner).map(InnerSample) val outerSamples = source.filter(!_.isInner).map(OuterSample) Observable.merge(innerSamples, outerSamples)

At this point I think a pattern emerges.

Design

A stream of information is like a river. Does the river care who observes it or who drinks from it? No, it doesn’t. And yes, sometimes you need to share the source between multiple listeners, sometimes you want to create new sources for each listener. But the listener shouldn’t care what sort of producer it has on its hands or vice-versa. And people are really not good at reasoning about graphs and making those graphs explicit doesn’t make it better, it makes it worse.

In Monifu / Rx you’ve got hot observables (hot data sources shared between an unlimited number of subscribers) and cold observables (each subscriber gets its very own private data source). You can also convert any cold data source into a hot one by using the multicast operator, in combination with Subjects that dictate behavior (e.g. Publish, Behavior, Async or Replay). The ConnectableObservable is meant for hot data sources. In our sample above, we are using share(), an operator that transforms our data source into a hot one and then applies reference counting on its subscribers to know when to stop it. This is what encapsulation is all about.

In Akka Streams the sources have a “single output” port and what you do is you build “flow graphs” and sinks. Akka Streams is thus all about modeling how streams are split. They call it “explicit fan-out” and it’s a design choice. However I consider it an encapsulation leak that makes things way more complicated than they should be and defeats the purpose of using a library for streams manipulation in the first place. In Rx (Rx.NET / RxJava / Monifu) terms, this is like having single-subscriber Observables and then working with Subjects (which is both a listener and a producer) and people that have used Rx know that working with Subjects sucks and when you do, you usually encapsulate it really, really well. This design choice of Akka Streams has also leaked into the “Reactive Streams” specification, as that Processor interface is irrelevant, plus during the talks on what the Producer should be, the original design was for the Producer to be single-subscriber.

Another thing I don’t like is that Akka Streams depends on Akka, the library. You need an Actor System and an ActorFlowMaterializer, whatever that is, with the tasks being executed by actors. I think that’s a design mistake. One reason for why Scala’s Future and ExecutionContext are great is precisely because they model only asynchronous computations, but are completely oblivious to how the required asynchronous execution happens. This is why Future works on top of Scala.js without problems.

And again, I’m biased, but Monifu’s own implementation is conceptually elegant. You’ve got the Scheduler that is used to execute tasks (an evolved ExecutionContext), the Observable interface which is the producer, characterized solely by its onSubscribe function, then you’ve got the Observer which represents the consumer and has the back-pressure protocol baked in its API, the Subject that is both a producer and a consumer, the Channel which represents a way to build Observables in an imperative way without back-pressure concerns and the ConnectableObservable, which represents hot data-sources that are shared between multiple subscribers. Monifu’s internals are self-explanatory and (I hope) a joy to go through.

I mean, this beauty describes Monifu’s design and represents at least half of what you need to know (and skipping over back-pressure concerns, you already know it):

trait Observer[-T] { def onNext(elem: T): Future[Ack] def onError(ex: Throwable): Unit def onComplete(): Unit }

In contrast, I found the source-code and the concepts in Akka Streams to be very hard to read and understand. Give it a try and compare.

Performance

I’ve left the best part of Monifu for last. Here’s running the Monifu sample:

[info] Running swave.rsc.MonifuPi After 1,909,713 samples π is approximated as 3.14102 After 4,132,610 samples π is approximated as 3.14240 After 6,342,356 samples π is approximated as 3.14241 After 8,513,597 samples π is approximated as 3.14239 After 10,696,940 samples π is approximated as 3.14176 After 12,897,762 samples π is approximated as 3.14165 After 15,099,261 samples π is approximated as 3.14160 After 17,326,511 samples π is approximated as 3.14169 After 19,529,832 samples π is approximated as 3.14164 After 21,752,232 samples π is approximated as 3.14170 [success] Total time: 11 s, completed Sep 6, 2015 2:48:06 PM

Here’s Akka Streams:

[info] Running swave.rsc.AkkaPi After 59,812 samples π is approximated as 3.12807 After 258,925 samples π is approximated as 3.13910 After 525,537 samples π is approximated as 3.14173 After 892,973 samples π is approximated as 3.14070 After 1,258,939 samples π is approximated as 3.14127 After 1,545,861 samples π is approximated as 3.14019 After 1,813,374 samples π is approximated as 3.14103 After 2,219,992 samples π is approximated as 3.14137 After 2,515,283 samples π is approximated as 3.14129 After 2,826,297 samples π is approximated as 3.14148 [success] Total time: 13 s, completed Sep 6, 2015 2:49:23 PM

Here’s RxJava (RxScala):

[info] Running swave.rsc.RxScalaPi After 804,127 samples π is approximated as 3.11240 After 1,747,146 samples π is approximated as 3.12848 After 2,761,645 samples π is approximated as 3.13558 After 3,678,029 samples π is approximated as 3.13880 After 4,756,818 samples π is approximated as 3.13960 After 5,787,079 samples π is approximated as 3.14172 After 6,768,352 samples π is approximated as 3.14249 After 7,804,973 samples π is approximated as 3.14320 After 8,812,403 samples π is approximated as 3.14381 After 9,625,207 samples π is approximated as 3.14393 [success] Total time: 12 s, completed Sep 6, 2015 2:54:55 PM

The results:

So my Monifu sample is consistently beating Akka Streams by at least a factor of 7 and RxJava/RxScala by a factor of 2.

The difference is so large that either Akka Streams has some serious improvements to achieve, or I’m doing something terribly wrong, because this is Monifu’s out-of-the-box behavior, as in I haven’t even attempted to fine tune the buffers or the scheduler for this sample.

But yes, Monifu was built for both ease of use and performance.

Final Words

Again, I’m biased, since I’m the author of an alternative, so take this conclusion for what it is.

The precursors of Akka Streams have been Play’s Iteratees and the Akka I/O Pipeline, the former on its way to deprecation, the latter already deprecated. Unfortunately I’m seeing the same mistakes: implementation hard to read, conceptualy very complicated, while exposing custom binary operators in the hope of forming a DSL that will somehow fix this complexity. And I’m all for wheel reinvention when it’s done for the right reasons, but I find this model to be inferior to an evolved Rx (such as Monifu) and this is just one other design heading towards deprecation.

If you’re interested in Monifu, there’s still some work to be done. It’s at release candidate for version 1.0, meaning that we’re done breaking the API and a very functional core is ready. Monifu has been developed in parallel to its usage in production for the past year, should be fairly solid and has very good test coverage as testament to that, but bugs may still happen, since a lot of work went on this past month. Documentation is sadly a work in progress and for now only the API documentation is up to date. But we’re marching towards a 1.0 release you can rely on, which should be available in about two weeks from now (if everything goes well). Checkout the GitHub repo, join our chat channel and give us feedback.

Update (Sep 8, 8:00 AM)

Some interesting questions happened on the reddit thread. To dispel some myths:

currently Akka Streams does no more parallelism than Monifu does

in our sample Monifu is not single-threaded

in our sample Monifu parallelizes the portion that it can, which is the processing of filterInner and filterOuter, plus execution is jumping threads, because Monifu as a matter of policy never keeps a single thread occupied for too long

you can’t parallelize concurrent operations, that being an oxymoron, which is why I’m dumbfounded by claims of parallelism

Cheers,

The article Monifu vs Akka Streams first appeared on alexn.org.

Scala Best Practices

Alexandru Nedelcu — Mon, 20 Oct 2014 00:00:00 +0000

This is a collection of best practices for Scala, something to get you started.

Article has been moved to its own GitHub repo:

alexandru/scala-best-practices

If you find it useful as a starting point for your team, fork it.

The article Scala Best Practices first appeared on alexn.org.

Resources for Learning Scala

Alexandru Nedelcu — Mon, 13 May 2013 00:00:00 +0000

Getting started with a new programming language shouldn't be hard, however navigating the web for resources on getting started with Scala can be a doubting experience, as many such resources are either out of date, or wrong, or both. This post is intended to reduce the noise for my colleagues and other people that are interested in Scala development.

1. Tools of the Trade

All you need for getting started is the Scala interpreter and a good text editor.

First, download the archive from scala-lang.org/downloads, uncompress it and place the bin/ subdirectory on your local PATH. Then start the “scala” interpreter and test if it works:

$ scala Welcome to Scala version 2.10.1 Type in expressions to have them evaluated. Type :help for more information. scala> 1 + 1 res0: Int = 2

You also need a good and simple text editor that can do syntax highlighting for Scala. Your choice should be done in this order:

your existing favorite text editor, if you have one

Sublime Text 2 for a good out-of-the-box experience, although for reasons I won’t go into here, I really keep away from text-editors that aren’t open-source

Vim or Emacs (in combination with scala-mode2 and yasnippet). These text editors are eternal and extremely productive, however if you’re unfamiliar with neither of them, adding the overhead of learning them on top of learning Scala is a bit too much

Other tools you may need for serious development (TM), but not necessarily for learning:

SBT for building projects and managing dependencies, being much like Maven for Java, or Leiningen for Clojure, or Rake+Bundler for Ruby

IntelliJ IDEA, in combination with the sbt-idea plugin, if you need a good IDE, but seriously, when getting your feet wet, stay away from IDEs. The community edition is open-source and fit for Scala development

Typesafe Activator is a pretty recent development if you want to play with the TypeSafe stack (which really means the Play Framework and Akka). I haven’t played with it, but it looks like a neat way to get some sample apps running quickly. It also includes SBT.

But really, for playing around, start with just the Scala compiler + a text editor that does syntax highlighting for Scala. I can’t stress this enough.

2. Books

For getting started, at the moment (May 2013) ignore all books (seriously) other than these 3:

Scala for the Impatient by Cay S. Horstmann, is a good pragmatic book on Scala (not so much on functional programming), but it’s for developers experienced in other languages, so it’s fast-paced while not scaring you away with endless discussions on types. The PDF for the first part (out of 3) is available from the Typesafe website.

Programming in Scala by Martin Odersky is a good book on programming, not just Scala - many of the exercises in Structure and Interpretation of Computer Programs are also present in this book, giving you the Scala-approach for solving those problems, which is good.

Scala in Depth by Joshua Suereth D. - this is an advanced book on Scala, with many insights into how functional idioms work in it or advice on best practices. I’ve yet to finish it, as it’s not really an easy lecture. But it’s a good book. Get the eBook straight from Manning.

NOTE: these are Amazon links (with my affiliate tag) placed here for convenience and for reading other people’s reviews, but if you want the eBook version don’t buy from Amazon, prefer buying directly from the publisher, as you’ll get both a DRM-free Kindle version and a PDF, useful for desktops or iPads.

3. Online Resources

An unspoken rule when searching for online resources about Scala is that you should stay away from the www.scala-lang.org website, because many links are outdated and the website is not properly maintained, as most of the effort these days is going to the documention project (mentioned below), which will probably become the homepage for Scala at some point.

Functional Programming Principles in Scala is an excellent course provided by Coursera / EPFL, taught by Martin Odersky. The course is almost over, so register right now if you want access to the videos and assignments, or you’ll probably have to wait for the next iteration.

Scala Documentation Project - definitely checkout this website, as they aggregate everything good here. If you want to learn more about Scala’s library, especially the collections, this is the place to learn from. Checkout for instance this Scala cheatsheet.

Scala School - a freely available online tutorial by Twitter, which is very friendly to newbies. I’ve read it and it’s pretty good.

Ninety-Nine Scala Problems

a collection of 99 problems to be solved with Scala. If you get stuck, you can view a solution which is often idiomatic. See also this GitHub project that gives you a complete test-suite, to spare you of the effort. Project Euler is also a pretty cool source of problems to solve.

The Scala Overview at StackOverflow.com is a pretty cool aggregate of popular Scala questions. I don’t know if they are compiling this automatically, or by hand, but it almost feels like an online book.

Online videos from nescala.org. These presentations are pure gold and a must see. At the moment, the homepage features the videos from 2013, so start with the 2012 archive.

4. Whom to Follow

The Typesafe Blog usually contains news regarding Scala adoptions in the enterprise, or release announcements about Akka, Play, Scala-IDE, or whatever Typesafe is doing these days and it’s useful to follow their RSS feed.

This Week in #Scala is a weekly-ish article series written by Chris Cundill of Cake Solutions, aggregating the most interesting news happening in the Scala community. I subscribed to their newsletter.

My Twitter and my RSS feed does have subscriptions to interesting people from the Scala community, however following people tends to add noise to your news stream. If you want to learn Scala, then following people’s blogs and tweats is a waist of time.

5. Seeking Help

For seeking help for language usage:

The Scala-User mailing-list

StackOverflow.com, where I got some pretty cool answers on Scala-tagged questions

For seeking help related to usage of various Scala frameworks or libraries, you may want to subscribe to their specific mailing-list. For instance play-framework for problems related to the Play framework.

The article Resources for Learning Scala first appeared on alexn.org.

Towards a Better AtomicReference

Alexandru Nedelcu — Tue, 07 May 2013 00:00:00 +0000

The AtomicReference is like a container for a volatile reference. Usage of volatile references is useful for the issue of visibility in concurrent code, however AtomicReference also supports the atomic Compare-and-Swap operation (CAS for short), which is the pillar of all non-blocking data-structures and algorithms built on top of the JVM, including complex ones like the ConcurrentLinkedQueue, an implementation based on the Michael-Scott non-blocking queues.

However, the interface provided leaves something to be desired:

the compareAndSet operation is too low level and for 99% of everything we do in our day to day code it can be replaced with something much better, as we’ll see

the classes from the java.util.concurrent.atomic package do not implement a common interface, so you can’t use an AtomicInteger in place of an AtomicReference

AtomicInteger and AtomicLong provide incrementAndGet, which is really useful in practice for keeping track of things in non-blocking counters, but why should that be available only for Ints and Longs? Floats, Doubles, BigInt, BigDecimal and all kinds of numbers can be incremented too

IMPORTANT UPDATE (March 31, 2014): The content of this article is slightly obsolete, though still has pedagogical value. For an up to date article on my Atomic references, checkout the wiki page maintained for project Monifu: Atomic References

This is a simple and working example of how Scala can improve your code tremendously. And I’m basically describing the implementation of my own shifter.concurrency.atomic.Ref. You can:

see the code in my GitHub repo

checkout the API docs

The Common Interface

Lets start by mirroring the basics of AtomicReference:

trait Ref[T] { def get: T def set(update: T) def compareAndSet(expect: T, update: T): Boolean } object Ref { def apply[T](initial: T) = new RefAny(initial) }

With a generic implementation that for now just delegates to our inner AtomicReference:

class RefAny[T](initial: T) extends Ref[T] { def get = instance.get() def set(update: T) = instance.set(update) def compareAndSet(expect: T, update: T) = instance.compareAndSet(expect, update) private[this] val instance = new AtomicReference(initial) }

As I was saying, the compareAndSet is too low level. Much better is to work with transformations. How about defining a function with the following signature:

def transformAndGet(cb: T => T): T

And that can be used like this:

val ref = Ref(2) ref.transformAndGet(x => x + 2) // => 4

Well, incrementing numbers is not the only thing that you can do. For instance you could store and transform immutable data-structures, like a queue:

import collection.immutable.Queue val ref = Ref(Queue.empty[String]) ref.transformAndGet(q => q.enqueue("Alex"))

How about that? We got ourselves a non-blocking queue, without having to implement the dreadful Michael-Scott algorithm.

Implementing Ref.transformAndGet is easy:

@tailrec final def transformAndGet(cb: T => T): T = { val oldValue = get val newValue = cb(oldValue) if (!compareAndSet(oldValue, newValue)) // tail-recursive call transformAndGet(cb) else newValue }

I’m using a tail-recursive function, guarded by the tailrec annotation, because that’s how I roll. I also find it much more readable and less error-prone. It creates a loop … as long as the compareAndSet operation is unsuccessful, then it keeps trying.

We can also implement Ref.getAndTransform, which returns the old value before the transformation occurred, instead of the update:

@tailrec final def getAndTransform(cb: T => T): T = { val oldValue = get val update = cb(oldValue) if (!compareAndSet(oldValue, update)) // tail-recursive call getAndTransform(cb) else oldValue }

We can have other utilities too, like transformAndExtract, but lets move on to our next issue … incrementing numbers.

Of course, with our transformation functions, the presence of a shortcut for incrementing numbers is not that required anymore, however it’s still a nice shortcut that can also provide readability and performance advantages. The problem with a generic AtomicReference is that not all reference types are numbers that can be incremented. Fortunately for us, Scala gives us Type Classes and there is already a type class defined in Scala’s standard library for numbers: Numeric[T].

What Numeric[T] does is to define, amongst others, the sum operation for type T and of course the value for one. And we don’t need more than that.

So we can define our Ref.incrementAndGet function, in a generic way, like this:

def incrementAndGet(implicit num : Numeric[T]) = transformAndGet(x => num.plus(x, num.one))

And lo and behold, this stuff works for any kind of number, not just Ints and Longs:

scala> val ref = Ref(BigInt(1)) ref: Ref[scala.math.BigInt] = Ref(1) scala> ref.incrementAndGet res0: scala.math.BigInt = 2

Best of all, if we try doing this on things that aren’t numbers, then it fails with a compile-time error:

scala> val ref = Ref("hello") ref: Ref[String] = Ref(hello) scala> ref.incrementAndGet :9: error: could not find implicit value for parameter num: Numeric[String] ref.incrementAndGet ^

AtomicInteger from Java’s standard library already has an incrementAndGet and who knows, it might be more efficient than our implementation. Maybe at some point it will get translated into a single processor instruction. So we can take advantage of that by specializing our Ref for Int:

final class RefInt(initialValue: Int) extends Ref[Int] { // .... override def incrementAndGet(implicit num: Numeric[Int]): Int = instance.incrementAndGet() private[this] val instance = new AtomicInteger(initialValue) }

And then we can make our primary constructor return a RefInt, in case the initial value is an Int. Well, here’s how the code in shifter.concurrency.atomic.Ref looks like:

object Ref { def apply(initialValue: Int): RefInt = new RefInt(initialValue) def apply(initialValue: Long): RefLong = new RefLong(initialValue) def apply(initialValue: Boolean): RefBoolean = new RefBoolean(initialValue) def apply[T](initialValue: T): Ref[T] = new RefAny[T](initialValue) }

There is still one difference between a Ref[Int] and AtomicInteger. Our interface will be guilty of boxing and unboxing the integers passed to those functions. And in the wild, using AtomicInteger for cheap and non-blocking counters is really common, so it’s a pitty if we’ll have performance degradation here.

The fix is easy though. The Scala compiler can specialize our type T for primitive types, if we annotate our type like this:

trait Ref[@specialized(scala.Int, scala.Long, scala.Boolean) T] { // ... }

In the example, the compiler will specialize the Ref[T] interface for Ints, Longs and Booleans, to avoid the boxing and unboxing overhead.

What this will do is to generate specialized methods for these primitive types. You can inspect the generated interface easily with the javap utility. For instance, let’s see what it generates for compareAndSet:

$ javap shifter.concurrency.atomic.Ref | grep compareAndSet public abstract boolean compareAndSet(T, T); public abstract boolean compareAndSet$mcZ$sp(boolean, boolean); public abstract boolean compareAndSet$mcI$sp(int, int); public abstract boolean compareAndSet$mcJ$sp(long, long);

Or for incrementAndGet:

$ javap shifter.concurrency.atomic.Ref | grep incrementAndGet public abstract T incrementAndGet(scala.math.Numeric); public abstract boolean incrementAndGet$mcZ$sp(scala.math.Numeric); public abstract int incrementAndGet$mcI$sp(scala.math.Numeric); public abstract long incrementAndGet$mcJ$sp(scala.math.Numeric);

So it’s basically method overloading done by the Scala compiler. Clearly this adds some overhead in the generated .class files and it might not do what you think it does, so use it only if you really need it.

As I was saying in the beginning, make sure to also (links update March 31, 2014):

see the code in project Monifu

checkout the API docs

Also, you may be interested in using scala-stm, a library for shared transactional memory, that basically gives you the ability to orchestrate multiple atomic references at once.

The article Towards a Better AtomicReference first appeared on alexn.org.

JVM Multithreading: Monitor Locks and Visibility

Alexandru Nedelcu — Thu, 14 Mar 2013 00:00:00 +0000

Multithreading is a pain to deal with. While interviewing developers, I noticed that surprisingly many don't have knowledge about this topic and I can't blame them really. However, in this day and age, for some problem domains building highly-concurrent architectures may be paramount to the success of demanding projects. As you'll see, there are many high level solutions, but I personally prefer to learn with a bottom up approach, starting from the basic and unsafe primitives, as understanding the problem is always the first step to real solutions.

This is (hopefully) the start of a series of articles giving an overview of the primitives and tools available on top of the JVM for solving concurrency-related problems, with code given in Scala and Java, starting from standard synchronization techniques, going through low-level primitives and non-blocking algorithms based on compare-and-set, up to high-level tools, such as Futures/Promises, actors and optimistic locking with shared transactional memory.

The Problem of Atomicity

Over 100,000 people can watch the same soccer game from the same stadium, at the same time. Those same 100,000 people cannot all take a dump in the same bathroom at the same time. Writing data to a central location requires an agreed-upon protocol for establishing who’s allowed to write and when.

Most of our concurrency-related problems come from our usage of mutable data and data-structures, as both reading and writing are problematic. When updating a mutable data-structure, the data can get into an inconsistent state, so threads that are doing the reading can end-up with garbage. When multiple threads are updating the same data-structure, the result can be far worse as it can lead to irrecoverable data corruption.

To solve the problem, you want updates to seem instantaneous from other threads, with no in-between intermediary and inconsistent state. A piece of code is considered atomic if it seems instantaneous to other threads.

Consider implementing a basic stack. Below is one example in which many things can go wrong, pointing out a few gotchas off the top of my head.

(See here for the Java version)

/** * Class representing nodes in a simple linked list. * * NOTES: * * 1. considering this class is used in the context of a stack, we * never need to add or remove from the middle, so there's no * reason for why this shouldn't be immutable * * 2. leaving this class public exposes the internal implementation * of our stack * */ class Node { var value: Any = null var next: Node = null } /** * Totally unsafe, totally screwed implementation of a stack. */ class Stack { /** * Gotcha: leaving our head public, means other threads can mess * with the internal state of our stack, even more so because our * Node class is mutable. */ var head: Node = null def isEmpty = head == null def push(value: Any) { val node = new Node node.value = value node.next = this.head // Gotcha: by the time we assign the new head, another thread may // have changed it already this.head = node } def pop() = { if (!isEmpty) { // Gotchas: // // 1. by the time the following is executed the `head` can // already be null // // 2. two or more threads may read the same `head` and thus // receive the same value on pop() val node = this.head // 1. again, the `head` can already be different, so the // following assignment may lose data // 2. the new value may not be visible from other threads this.head = node.next node.value } else { null } } }

The standard way to fix this, as preferred by Java developers, is to use the synchronize keyword on all the Stack’s methods. You’ve seen this before, right?

public synchronized boolean isEmpty() { return this.head == null; }

The synchronized keyword creates a monitor lock (also called an intrinsic lock) on the implicit this. So in case of an instance, it creates a monitor on that instance. In the case of Java’s static methods, it creates a monitor on the class object. It’s important to keep this in mind, because synchronized is not some magical tool that solves every problem you may have and I like that Scala doesn’t have such a keyword ;-)

A monitor lock is guaranteed to be acquired by only a single thread at the same time. Other threads that try acquiring it in the process are blocked until the lock is free again.

Lets improve the above using the following:

the Monitor pattern (monitor locks on this)

encapsulation of internal mutable state

Option[T] instead of nulls (Guava’s Optional for the Java version), because that’s how I roll

immutable nodes for our internal linked-list

(See here for the Java version)

/** * Better (mutable) stack implementation. * * For type-safety, changed the interface to take a type parameter. */ class Stack[T] { /** * * Changes: * 1. to prevent implementation leaks, nodes in our linked-list * have to be private, including the Node class * 2. Node instances are now immutable (always prefer immutable * data structures) * 3. never use nulls, prefer proper initialization and Option[T] */ private[this] case class Node( value: Option[T], next: Option[Node] ) /** * The head of our stack. * * Because the Node class is private, if you make this field * public, then the compiler will trigger a compilation error */ private[this] var head: Option[Node] = None def isEmpty = this.synchronized { head.isEmpty } def push(value: T): Stack[T] = { // entering monitor lock this.synchronized { head = Some(Node( value = Option(value), next = head )) this } } def pop(): Option[T] = { // Entering monitor lock // // Note that `isEmpty` is also synchronized, but monitor locks // are reentrant so a lock can be acquired multiple times by the // same thread. this.synchronized { if (!isEmpty) { val node = head.get head = node.next node.value } else None } } }

That’s better. Not perfect though, as Stacks are the easiest immutable data-structures to implement … so how about not using any locks at all? (that’s for another article)

The Big Problem with Locks

Take these 2 stacks:

val stack1 = new Stack.push("World").push("Hello") val stack2 = new Stack

Question: is the following thread safe?

while (!stack1.isEmpty) stack2.push( stack1.pop )

Answer: No. Given that:

A is thread safe

B is thread safe

Then using A + B is NOT thread safe, unless you make it so by using an external lock that protects both at all times.

Code that’s thread-safe through synchronization based on locks is not composable.

The Problem of Visibility

When speaking of multithreading, the most obvious problem is the inconsistency of shared mutable state when being changed and read by multiple threads at the same time. However the problem is actually twofold and the atomicity of code that changes mutable state is not your only problem.

Take this piece of code Scala code ( see here for the Java version):

var result = "Not Initialized" var isDone = false val producer = new Thread(new Runnable { def run() { result = "Hello, World!" isDone = true } }) val consumer = new Thread(new Runnable { def run() { // loops until isDone is true while (!isDone) {} println(result) } }) consumer.start() producer.start() consumer.join()

Question: What does the above print?

Hello, World!

Not initialized

Nothing, goes into an infinite loop

All of the above

The answer may surprise some of you. It’s actually number 4, all of the above. To make matters worse, you can’t predict what happens, as it depends on the CPU architecture you have, on the number of cores, on who made the VM, on what other apps you have running, on whether the laptop is plugged in or not, on planetary alignments and so on.

So what can happen?

On most desktops today, most of the time (as in >50%) it will behave as expected, which kind of sucks really, because it’s far better to have a fast and loud failure than one with subtle effects that may or may not manifest when you’re testing the app.

The JVM doesn’t guarantee that the instructions are executed in the given order. Amongst others, the VM may decide that those instructions are independent of each other and may reverse their order for things like better cache locality, or because processors are pretty smart about executing multi-cycles instructions, being able for example to start subsequent instructions before the previous ones are finished. So it’s pretty common for the compiler to reorder instructions such that longer instructions are executed before shorter ones. The processor itself may decide to execute instructions out of order, even if the VM/compiler is issuing the instructions in the right order. From the point of view of the producer thread, the result is the same as if the instructions are executed in the given order, but you can’t rely on it when viewing the results from outside threads.

The new value for isDone could be cached somewhere (like in a processor register) and the consumer thread may never see this new value. On my desktop in more than 1 out of 10 cases this little example goes into an infinite loop.

Happens-Before Relationships and Memory Barriers

As I was saying at point 2 above, in addition to cached values, you can also have reordered instructions. Say that you’ve got the following calls:

statementA; statementB;

From the point of view of the thread executing these 2 statements, the result of statementA is available to statementB. We call this a happens-before relationship between the two statements. So from the point of view of the executing thread, the result is always the same as if these 2 statements are executing in order, even if these statements are executed in fact out of order.

Outside of the executing thread, this happens-before relationship is not guaranteed. The result of statementB could be visible to other threads, while the result of statementA could be made visible later or never.

In our example, to ensure that isDone is written after result and to ensure the visibility for outside threads for both, you need to create what is called a memory barrier.

The standard way of doing this is (again) through a monitor lock acquired on a certain object.

A synchronization block guarantees two things:

all the writes that happened on other threads on variables, by using the monitor X, are visible to our current thread if it acquired the same monitor X

at the end of the synchronization block, a memory barrier is created and changes made to variables inside that block will be visible to other threads that use the same monitor

To fix our problem with monitor locks, here’s the Scala version ( see here for the Java version):

var result = "Not Initialized" var isDone = false val lock = new AnyRef val consumer = new Thread(new Runnable { def run() { var continueLooping = true while (continueLooping) lock.synchronized { continueLooping = !isDone } println(result) } }) val producer = new Thread(new Runnable { def run() { lock.synchronized { result = "Hello, World!" isDone = true } // <-- memory barrier } }) consumer.start() producer.start() consumer.join()

There is one big gotcha here. The JVM only guarantees visibility and happens-before relationships only if the threads involved in reading/writing to our variables are synchronized with the same monitor lock. This gotcha could happen for a bunch of reasons, for instance the JVM does escape-analysis and it can get rid of locks completely if it decides a lock isn’t used concurrently by multiple threads.

Volatiles

For this particular example, you actually don’t need a lock at all. All you need is a volatile:

@volatile var isDone = false

Or in Java:

volatile boolean isDone = false;

A write to a volatile also creates a memory-barrier. If you need memory barriers, then a write to a volatile on the JVM creates a full-fence. This guarantees not only the visibility of isDone, but it also guarantees the visibility of all other variables written prior to it by the same thread, like result in our example.

Volatiles are useful sometimes in non-blocking algorithms. But even with the strong guarantee of the created memory-barrier, for most purposes where you need volatiles, you’ll end up using atomic instances from the java.util.concurrent.atomic package, like AtomicReference or AtomicInteger.

That’s because you need compare and set for non-blocking algorithms, an atomic operation that’s optimized on most CPUs today that you can’t do with plain volatiles.

But that’s a topic for another article.

Further Reading

Checkout the following:

JSR 133 (Java Memory Model) FAQ

Java Concurrency in Practice (affiliate Amazon link)

The article JVM Multithreading: Monitor Locks and Visibility first appeared on alexn.org.

Notes On JavaScript Client-side Development

Alexandru Nedelcu — Wed, 07 Nov 2012 00:00:00 +0000

Client-side JavaScript development can be quite overwhelming, even for senior developers. I'm describing here what I did in a recent piece of client-side functionality to keep my sanity. This interface I'm talking about is served on mobile-phones, so it must be pretty bloat-free, adding salt over injury.

Dealing with modules, packaging and minifying

This is probably my biggest gripe with JavaScript, that you can’t simply require("some.namespace") without extra glue and tricks. This is a limitation of the browser, as such operations would have to be executed asynchronously, requiring tedious callbacks and because of bandwidth constraints you may want to serve everything in a single optimized and gzipped JS.

Brunch.io is an awesome tool that helps you do that. You split your project in multiple files, then Brunch can do the assembling for you. To have modules in your project (splitting it in several files), Brunch works with the CommonJS modules interface. All that means is that your JavaScript files will end-up looking like this:

// for importing another module utils = require("path/to/utils") function generateRandomString() { var userAgent = navigator.userAgent var rnd = Math.random() * 100000000 return utils.md5(userAgent + rnd); } // exporting function for consumption from other modules modules.export = { generateRandomString: generateRandomString }

Building changes and seeing them in your browser works really fast, as Brunch can watch for changes in real-time and also exposes a built-in server. So Brunch can take care of packaging for all your modules in a single JS file, also minimizing for production if you want. The workflow is pretty sweat, the configuration simple.

Dealing with JavaScript’s Syntax Quirks

JavaScript is a capable language exposed by an awful syntax and plagued by incompatibilities - did you know that placing extra commas or other punctuation will cause syntax errors in older browsers? So it’s not just about API incompatibilities, the syntax itself can lead to surprises.

Brunch can be configured to check your code with JSHint, which is a Linter that can check for syntax errors and you can use it to force upon yourself and your team certain best practices. It has a lot of configuration options and for instance it can trigger errors on uninitialized variables and other tremendously useful stuff.

I’ve used CoffeeScript as Brunch can work with any language, with the proper plugin. I preferred CoffeeScript because I develop in a more functional style and CoffeeScript has a lighter syntax for anonymous functions, plus some other fixes to JavaScript, like “class” or the fat-arrow that binds the defined function to the current this (tremendously useful). I coupled CoffeeScript with CoffeeLint and for instance I disabled the implicit braces when declaring object literals, as I hate that syntax and don’t want to get sucked into it. What I really wanted however was TypeScript, which is like JavaScript, but with nice add-ons. Unfortunately it’s too immature and so it isn’t included by Brunch’s authors yet, but it probably will be at some point.

Dealing with Async Events

Many people use Backbone.js or things based on it. What Backbone gives you, besides a structure for your app with controllers and models, it also gives you a foundation for using the Observer pattern. So your app initiates events that produce data and in response to user actions or new data, you have to update stuff.

However my use-case was simple (the interface can be described by a state-machine, without many things going on as far as the UI is concerned). And I absolutely hate this consumer/producer model with listeners, because I can’t really wrap my head around it even for the simplest of examples, as this model was really designed for usage inside an IDE, where you can right-click on components and see all the registered listeners for certain events.

The Q library - is one library I’m using for dealing with future responses (a promise for a response that may be available at a later time) and for avoiding the Pyramid of Doom.

You see, many people choose something like Backbone because it makes asynchronicity easier to deal with. I chose against it by building small functions that compose, binding them together with Q. Works great for small projects, even if the asynchronous calls are difficult to compose, because working with promises makes that easy - note that jQuery’s implementation is broken, so don’t use it, just wrap jQuery’s ajax calls in a Q promise.

Also, the great thing about preferring small functions with referential transparency that compose is that testing becomes so much easier. Brunch can also run your tests and you can use something like Chai or other helpers for testing nirvana.

This doesn’t mean that Backbone and data flows with something like Q can’t be used together. I can certainly see myself working with Backbone for larger projects.

Going Mobile

Other libraries I’m using are:

Underscore.js, which provides much needed API additions, some of which are available in latest versions of JS, but not on older browsers. You definitely need this if you like to program in a more functional style

Zepto instead of jQuery, because this interface is served on mobile phones and jQuery is pretty bloated. I’ll also probably switch to jqMobi because it’s even lighter.

In regards to jQuery alternatives, well jQuery adds like 33KB of gzipped JavaScript to my download. This may not be an issue in a browser that has jQuery already cached, but in a fresh WebView it’s really big for normal 3G connections. My final JS file that gets downloaded only has 20KB of gzipped JS, in total, including the libraries I mentioned and it won’t grow over 30KB. In combination with a CDN like CloudFront that’s still reasonable.

The ideal would be to use something like the Google Closure compiler, which can do tree-shaking, getting rid of pieces of code you don’t need. However, the code has to follow the Closure conventions, otherwise it will behave no better than a normal minifier and libraries like jQuery don’t follow them. That’s why I have high hopes for TypeScript, because it has a built-in dependency system and you can annotate with types even outside libraries, so a tree-shaker could work for popular libraries if there’s enough interest, even if the library in question does not follow any conventions. I also thought about using ClojureScript, which emits Closure-compatible JavaScript, but I’m not familiar enough with the language, so maybe some other day :-)

As a conclusion

Keep things simple and work with tools provided by the Node.js community, because that’s where the real action is.

The article Notes On JavaScript Client-side Development first appeared on alexn.org.

On Scala, Functional Programming and Type-Classes

Alexandru Nedelcu — Fri, 02 Nov 2012 00:00:00 +0000

I’ve been following the excellent Coursera course on Functional Programming Principles in Scala led by Martin Odersky. This was not my first encounter with Scala as I’ve been using it including for my day job. In parallel, because I felt the need for a Javascript replacement, I’ve been learning Clojure too, because of the excellent ClojureScript.

I’ve fallen in love with both and I can’t really pick a favorite. For what is worth this document represents my (rookie) experience with Scala, being complete yack shaving on my part, or you could call it the intellectual masturbation of a fool.

UPDATE: as if the article wasn’t long enough, I’ve added to it some more stuff (like a couple of times :-)

1. Functional Programming for the Win

It’s not a silver bullet, but on the whole it’s awesome. You really have to experience it, while leaving aside the preconceptions and biases you’ve been building up by honing those imperative skills for years. Students learn functional programing more easily, fresh as they are, otherwise the learning experience can be painful.

But we haven’t evolved much in the last 200,000 years and so our brain finds pleasure mostly in the things that appeal to our inner-animal, being interested in the means to get laid, eat food, sleep and escape wild beasts. Learning can be a pleasure, but not when you’re venturing to unfamiliar grounds, so if you start, hang in there.

We need some definitions though. Functional programming …

deals with computation by evaluating functions with referential transparency as a property (i.e. functions behave like mathematical functions, for the same input you must always get the same output)

the final output of a computation is composed out of multiple transformations of your input data, instead of building that solution by mutating state

A functional programming language is one that:

treats functions as first-class objects, meaning that dealing with higher-order functions is not only possible, but comfortable

gives you the tools needed for composing functions and types

By that definition languages like Ruby and Javascript can be considered decent functional languages and they are. However I would also add:

has a rich collection of immutable/persistent data-structures (in general if you want to assess the viability of any programming language, disregarding the platform it runs on, it’s perfectly characterized by its basic primitives and data-structures; e.g. think of C++, Java, or Javascript)

exposes a type-system that deals efficiently with the expression problem; Rich Hickey calls this “polymorphism a la carte”

You can also go to the extreme of specifying that all side-effects must be modeled with monadic types, but that’s a little too much IMHO, as only one mostly-mainstream language fits that bill (Haskell).

2. Is Scala a Functional Programming Language?

Yes it is. You only need to follow the excellent (I mentioned above) Coursera course and solve the assignments to realize that Scala is indeed a very FP language. The course was a little short, but a follow-up is planned. Now move along …

3. Polymorphism À la Carte

This is a term that I’ve been hearing from Rich Hickey, when he talks about open type-systems, referring primarily to Clojure’s Protocols and Haskell’s Type-Classes.

These mechanisms for polymorphisms are good solutions for dealing with the expression problem being in stark contrast with Object-Oriented Programming as we’ve come to know it from Java and C++.

OOP is often a closed type-system, especially as used in static languages. Adding new classes into an existing hierarchy, adding new functions that operate on the whole hierarchy, adding new abstract members to interfaces, making built-in types to behave in a certain way - all of these cases are cumbersome.

Haskell deals with it through Type Classes. Clojure deals with this through Multi-Methods and protocols, protocols being the dynamic equivalent for type-classes in a dynamic type-system.

4. Yes Virginia, Scala has Type-Classes

So what’s a type class? It’s like an interface in Java, except that you can make any existing types conform to it without modifying the implementation of that type.

As an example, what if we wanted a generic function that can add things up … you know, like a foldLeft() or a sum(), but rather than specifying how to fold, you want the environment to know how to do that for each particular type.

There are several problems with doing this in Java or C#:

there is no interface defined for “+” on types that support addition (like Integers, BigInteger, BigDecimal, floating-point numbers, strings, etc…)

we need to start from some zero (the list of elements you want to fold could be empty)

Well, you can define a type-class, like so:

trait CanFold[-T, R] { def sum(acc: R, elem: T): R def zero: R }

But wait, isn’t this just a simple Java-like interface? Well yes, yes it is. That’s the awesome thing about Scala - in Scala every instance is an object and every type is a class.

So what makes this interface a type-class? Objects in combination with implicit parameters of course. Let’s look at how we’ll implement our sum() function that uses this:

def sum[A, B](list: Traversable[A])(implicit adder: CanFold[A, B]): B = list.foldLeft(adder.zero)((acc,e) => adder.sum(acc, e))

So if the Scala compiler can find an implicit CanFold in scope that’s defined for type A, then it uses it to return a type B. This is awesomeness on multiple levels:

the implicit defined in scope for type A are establishing the return type B

you can define a CanFold for any type you want, integers, strings, lists, whatever

Implicits are also scoped so you have to import them. If you want default implicits for certain types (globally available) you have to define them in the companion object of the trait CanFold, like this:

object CanFold { // default implementation for integers implicit object CanFoldInts extends CanFold[Int, Long] { def sum(acc: Long, e: Int) = acc + e def zero = 0 } }

And usage is as expected:

// notice how the result of summing Integers is a Long sum(1 :: 2 :: 3 :: Nil) //=> Long = 6

I’m not going to lie to you as this stuff gets hard to learn and while learning how to do this, you’ll end-up pulling your hair out wishing for dynamic typing where all of this is not a concern. However you should distinguish between hard and complex (the former is relative and subjective, the later is absolute and objective).

One issue with our implementation is when you want to provide a default implementation for base types. That’s why we’ve made the type parameter T contravariant in the CanFold[-T,R] definition. What contravariance means is precisely this:

if B inherits from A (B <: A), then CanFold[A, _] inherits from CanFold[B, _] (CanFold[A,_] <: CanFold[B,_])

This allows us to define a CanFold for any Traversable and it will work for any Seq / Vector / List and so on.

implicit object CanFoldSeqs extends CanFold[Traversable[_], Traversable[_]] { def sum(x: Traversable[_], y: Traversable[_]) = x ++ y def zero = Traversable() }

So this can sum up any kind of Traversable. The problem is that it loses the type parameter in the process:

sum(List(1,2,3) :: List(4, 5) :: Nil) //=> Traversable[Any] = List(1, 2, 3, 4, 5)

And the reason for why I mentioned this is hard is because after pulling my hair out, I had to ask on StackOverflow on how to get a Traversable[Int] back. So instead of a concrete implicit object, you can provide an implicit def that can do the right thing, helping the compiler to see the type embedded in that container:

implicit def CanFoldSeqs[A] = new CanFold[Traversable[A], Traversable[A]] { def sum(x: Traversable[A], y: Traversable[A]) = x ++ y def zero = Traversable() } sum(List(1, 2, 3) :: List(4, 5) :: Nil) //=> Traversable[Int] = List(1, 2, 3, 4, 5)

Implicits are even more flexible than meets the eye. Apparently the compiler can also work with functions that return the instance you want, instead of concrete instances. As a side-note, what I did above is difficult to do, even in Haskell, because sub-typing is involved, although doing it in Clojure is easy because you simply do not care about the returned types.

NOTE: the above code is not bullet-proof, as conflicts can happen

Say in addition to a CanFold[Traversable,_] you also define something for Sets (which are also traversable) …

implicit def CanFoldSets[A] = new CanFold[Set[A], Set[A]] { def sum(x: Set[A], y: Set[A]) = x ++ y def zero = Set.empty[A] } sum(Set(1,2) :: Set(3,4) :: Nil)

This will generate a conflict error and I’m still looking for a solution that makes the compiler use the most specific type it can find, while still keeping that nice contra-variance we’ve got going (hey, I’m just getting started). The error message looks like this:

both method CanFoldSeqs in object ... and method CanFoldSets in object ... match expected type CanFold[Set[Int], B]

That’s not bad at all as far as error messages go. You could just avoid being too general and in case you want to override the default behavior in the current scope, you can shadow the conflicting definitions:

{ // shadowing the more general definition // (notice the block, representing its own scope, // so shadowing is local) def CanFoldSeqs = null // this now works sum(Set(1,2) :: Set(3,4) :: Nil) //=> Set[Int] = Set(1, 2, 3, 4) }

Another solution that CanBuildFrom uses is to define implicits on multiple levels, such that some implicits take priority over others, likes so:

trait LowLevelImplicits { implicit def CanFoldSeqs[A] = new CanFold[Traversable[A], Traversable[A]] { def sum(x: Traversable[A], y: Traversable[A]) = x ++ y def zero = Traversable() } } object CanFold extends LowLevelImplicits { // higher precedence over the above implicit def CanFoldSets[A] = new CanFold[Set[A], Set[A]] { def sum(x: Set[A], y: Set[A]) = x ++ y def zero = Set.empty[A] } }

And yeah, it will do the right thing. A little ugly though, as it means you have to have specific knowledge about how these implicits are prioritized. In essence, this is heavy stuff already and a little complex too. Good design can make for kick-ass libraries though.

5. Scala’s Collections Library is Awesome

So what does the above buy you anyway? The following are some examples from Scala’s own collections library.

You can sum things up in sequences, as long as you have an implementation of type-class Numeric[T] in scope:

List(1,2,3,4).sum //=> Int = 10

You can sort things, as long as you have an implementation of type-class Ordering[T] in scope:

List("d", "c", "e", "a", "b").sorted //=> List[java.lang.String] = List(a, b, c, d, e)

A collection will always do the right thing, returning the same kind of collection when doing a map() or a flatMap() or a filter() over it. For instance to revert the keys and values of a Map:

Map(1 -> 2, 3 -> 4).map{ case (k,v) => (v,k) } //=> scala.collection.immutable.Map[Int,Int] = Map(2 -> 1, 4 -> 3)

However, if the function you give to map() above does not return a pair, then the result is converted to an iterable:

Map(1 -> 2, 3 -> 4).map{ case (k,v) => v * 2 } //=> scala.collection.immutable.Iterable[Int] = List(4, 8)

Even more awesome than this, take for example the BitSet which is a compressed Set of integers (so it’s optimized for storing integers):

import collection.immutable.BitSet BitSet(1,2,3,4).map(_ + 2) //=> BitSet = BitSet(3, 4, 5, 6)

Mapping over it still returns a BitSet, as expected. However, look at what happens when the mapping function returns Strings:

BitSet(1,2,3,4).map(x => "number " + x.toString) //=> Set[java.lang.String] = Set(number 1, number 2, number 3, number 4)

Again, it did the right thing, because you can’t store Strings in a BitSet, as BitSets are for integers. So it returned a plain Set of strings. How is this possible, you may ask?

The answer is in the CanBuildFrom pattern. The signature of map() used above is a bit of a mouthful:

def map[B, That](f: (Int) => B)(implicit bf: CanBuildFrom[BitSet, B, That]): That

So, similar to my example with CanFold:

the compiler takes type B from the mapping function f: (Int) => B that’s provided as an argument

searches for an implicit in scope of type CanBuildFrom[BitSet, B, _]

the return type is established as the third type parameter of the implicit that is used

the actual building of the result is externalized; the BitSet does not need to know how to build Sets of Strings

So basically, if you define your own types like so:

class People extends Traversable[Person] { /* yada yada... */ } case class Person(id: Int)

Then if you want the mapping (or flatMapping) of a BitSet to return a People collection in case the function returns Person, then you have to implement an implicit object of this type:

CanBuildFrom[BitSet, Person, People]

And then this will work:

BitSet(1,2,3,4).map(x => Person(x)) //=> People = People(Person(1), Person(2), Person(3), Person(4))

So what’s great is that the provided implicits for CanBuildFrom can be overridden by your own implementations and you can provide CanBuildFrom implementations for your own types, etc…

(as a side note, Clojure cannot do conversions based on the given mapping function, even if the Seq protocol is awesome nonetheless and doing something akin to CanBuildFrom in Haskell is difficult from what I’ve been told)

If you want a lazy Iterator (like if you want to wrap JDBC result-sets), you only need to wrap the JDBC result-set in an Iterator by implementing next() and hasNext. You then get filter()/map()/flatMap() for free, but with a twist - Iterators are lazy and can only be traversed once. Applying filter/map/flatMap will not traverse the Iterator, being lazy operations. To convert this into a lazy sequence that also memoizes (stores) the results for multiple traversals, you only need to do iterator.toStream, or to get all the results at once iterator.toList.

Streams in Scala are lazy sequences. You can easily implement infinite lists of things, like Fibonacci numbers or the digits of PI or something. But Streams are not the only lazy collections, Scala also has Views and you can transform any collection into a corresponding view, including Maps.

But that’s not all. Scala also has implementations of collections that do things in parallel. Here’s how to calculate if a number is prime, sequentially:

import math._ def isPrime(n: Int) = { val range = 2 to sqrt(abs(n)).toInt ! range.exists(x => n % x == 0) }

If you have multiple cores around doing nothing, here’s how to calculate it by putting those extra cores at work:

def isPrime(n: Int) = { val range = 2 to sqrt(abs(n)).toInt ! range.par.exists(x => n % x == 0) }

Notice the difference?

6. Is this complex?

I mentioned above that this stuff is not complex, it’s just hard. Scala does have complexities when it comes to really advanced use-cases, as can be seen in this article: True Scala Complexity

It’s worth mentioning however that, as Martin Odersky noted in the Hacker News thread of that article, the author tries to accomplish something that’s not possible in most languages out there, while a solution is still possible in Scala (albeit with small limitations).

7. Are OOP Features Getting in the Way?

I happen to disagree and I actually love the blend of OOP with functional features. Martin Odersky claims that OOP is orthogonal to functional programming. But if you pay attention, you’ll notice it’s not only orthogonal, but complementary in an elegant way.

I’m indicating below instances where I think OOP helps, but as a clear example of what the combination can do, consider Scala’s Set. A Set[T] can be viewed as a function that takes a parameter of type T and returns either True if the value is in the Set, or False otherwise. This means you can do this:

val primaryColors = Set("red", "green", "blue") val colors = List("red", "purple", "yellow", "vanilla", "white", "black", "blue") colors.filter(primaryColors)

This is possible because our set is in fact a subtype of Function1[String, Boolean], so you can pass it to any higher-order function that expects that signature.

But the similarity goes deeper than simple resemblance and syntactic sugar. If you remember from school, a mathematical Set can be perfectly described by what is called a characteristic function, so Sets are interchangeable with functions in mathematics.

This means operations on Sets like unions, intersections, complements, Cartesian products and so on can be replaced with operations on functions and that’s exactly what boolean algebra is about. In mathematical terms, these mathematical structures (sets and functions that take an argument and return 0/1) are equivalent (indistinguishable) because there exists an isomorphism between them, savvy? :-)

And I don’t know how Haskell handles this for Data.Set, or if it handles it at all, but OOP subtyping seems like the easiest way to model something like this in a static language …

for one, the hierarchy is simple to understand, simple to model, as subtyping is something that OOP simply does - you just inherit from Function1[-T, +R] - and you’re done

downcasting to a function is something OOP simply does - you just pass your object to something that expects a function and you can forget the original type of that value

This is just a small and insignificant example of course, like most examples I’m giving here, but to me properly done OOP (where every type is modeled with classes and every value is some kind of object) just feels right … I like this principle of “turtles all the way down”, even if you could probably point to things that aren’t “turtles”, but this also happens in languages that are the epitome of kick-ass turtles-recursion, like Scheme or Smalltalk.

8. Scala versus Haskell

Scala’s static type-system is sometimes less expressive than that of Haskell. In particular Haskell supports rank-2 polymorphism, while Scala only rank-1. One point that Scala wins over Haskell is definitely this one:

List(1,2,3,4,5).flatMap(x => Option(x)) //=> List[Int] = List(1, 2, 3, 4, 5)

Doing the above in Haskell (using the bind operator) triggers a compile-time error, because the return type of the mapping function is expected to be of type List and the Maybe type (the equivalent of Option) is not a List.

Option in Scala is not a collection, but it is viewable as a collection of either 0 or 1 elements. As a consequence, because of good design decisions, the monadic types defined in Scala’s collection library are more composable.

EDIT: this example is simple and shallow. As pointed out in the comments, it’s easy to make the conversion by yourself, however I’m talking about the design choices of Scala’s library and the awesomeness of implicits. As a result, the standard monadic types provided by Scala (all collections, Futures, Promises, everything that has a filter/map/flatMap, etc…) are inherently more composable and friendlier.

It’s also worth pointing out that Scala’s collections library is so awesome precisely because OOP plays a part and there are cases where doing similar things in Haskell require experimental GHC extensions.

For instance, all of the collections in Scala share code in one way or another. If you want to build your own Traversable you only have to implement that trait with the abstract foreach(), but you get all other methods, including filter()/map()/flatMap() for free. As a side-effect your collection will be a monadic type by default.

Haskell is lazy by default. This is good for many problems. In Scala lazyness is a choice. In Haskell this lazyness is awesome, but in my experience while playing with it, it gets very hard to reason about the resulting performance. Sometimes it’s fast without you doing anything, other times - well, profiling and fixing performance issues in Haskell is not for mortals. Scala is more predictable, being strict and lazy when needed. It also has at its disposal the awesome JVM ecosystem for profiling and monitoring.

9. Scala versus F# / Ocaml

F# is good if you want to use C# 2020. But F# has rough edges inherited from Ocaml, while it has not inherited all the benefits. F# has nominative typing, instead of structural typing for OOP (as Ocaml). And you really start wishing for an ad-hoc polymorphism mechanism in which the types are open.

In regards to how one implements CanFold F# takes the crown as the ugly ducklin’ as it follows the (really screwed) C# conventions of defining “+” as static functions on classes (a reminiscence of C++ btw), so even if you know that a T is an Integer, you can’t sum 2 Integers based on the interface definition alone, because the compiler cannot make the connection to T + T, as in OOP interfaces/subtyping only applies to instances, not classes and “static members”. This is why they had to extend the language. Take a look at the signature for List.sum in F#:

List.sum : ^T list -> ^T (requires ^T with static member (+) and ^T with static member Zero)

First of all, this is bad from all perspectives, as it uses the (really fucked up) notion of “static members” that should have never happened in OOP. It’s also not a type-class as it is not open - you cannot modify a built-in type to have the required static members, being the same problem you get with classic OOP inheritance of interfaces. You also cannot override the implementation, as you’d wish in certain contexts.

In Scala there is no such thing as “static members”, “+” operations being plain polymorphic instance methods.

The one thing I really like about F# are quotations, which give you .NET LINQ, with the difference that quotations in F# are more potent than what C# can do. In simple words, quotations in F# give you the possibility of repurposing/recompiling pieces of code at runtime (e.g. macros).

But macros support is an upcoming feature of Scala 2.10, which is already at RC1 and you can play around with the up-coming Scala version of LINQ right now.

Ocaml goes a long way with its structural typing for OOP. Ocaml has the most advanced type-inferencer out of the popular functional languages, being more advanced than the one in Haskell. It’s a potent language, but sadly it has no equivalent for type-classes.

The right way to implement CanFold in Ocaml/SML would be to explicitly pass a dictionary of pointers around, as described here: Typeclass overloading and bounded polymorphism in ML.

Scala, unlike Ocaml and F#, does not have 2 type-systems in the same language, as Scala follows the “uniform access principle”. Type-classes and algebraic data-types are still modeled by means of OOP classes and objects.

Why does it matter? If you ever worked with C++ you can understand this - if OOP is pervasive in your language and not just something completely optional, then every type in the system should be (or considered) polymorphic and extending from some Object, otherwise you’ll end up with lots and lots of pain. It’s also a matter of having to make choices.

In Scala the code is indeed more verbose, but it reduces complexity a lot because a big part of learning Ocaml is learning when OOP is appropriate, or not, as you have to pick from the get-go and combining approaches is very cumbersome.

Take for instance the definition of an immutable and persistent List. A List can be defined efficiently as an algebraic data-type, being either an Empty List, or a Pair of 2 elements, the head and the tail, right?

In Ocaml:

type 'a my_list = Nil | List of 'a * 'a my_list

Extremely elegant and simple. And in Scala:

sealed abstract class List[+T] case class Pair[+t](head: T, tail: List[T]) extends List[T] case object Nil extends List[Nothing]

What a mouthful.

One difference should immediately be noticeable, our List has covariant behavior, meaning that a List[String] is also a List[Any], or a List[j.u.HashMap] is also a List[j.u.AbstractMap]. Arrays in Java have the same behavior and this leads to lots of gotchas, but if our List is immutable, then this is not a problem, but a bonus. For instance this gives you polymorphic behavior without needing type parameters or higher-kinded types or other mechanisms, just plain OOP subtyping relationships:

def length(list: List[Any]) = list match { case Pair(head, tail) = 1 + length(tail) case Nil => 0 }

However, that’s not efficient. A much better approach is to make length() polymorphic (in the OOP sense), after all length() is a defining property of Lists, so there’s no reason for why it shouldn’t be there:

sealed abstract class List[+T] { // abstract definition def length: Int } case class Pair[+T](head: T, tail: List[T]) extends List[T] { val length = 1 + tail.length } case object Nil extends List[Nothing] { val length = 0 }

Now, isn’t that nice? What would it take to turn this into a lazy list?

case class Pair[+T](head: T, tail: () => List[T]) extends List[T] { lazy val length = 1 + tail().length }

You can see how length hasn’t changed for either List[T] or for Nil, just for Pair, which makes it a good candidate for OOP. So why not model this with OOP in Ocaml? Because for algebraic data-types, the compiler helps you, like this:

def sum(list: List[Int]): Int = list match { case Pair(head, tail) => head + sum(tail) //-> oops, no termination } //-> output from the compiler ... warning: match is not exhaustive! missing combination Nil def sum(list: List[Int]): Int = list match { ^

Did I mention Scala also has structural typing if you want it? Yes it can (albeit, without the awesome type-inferencing that Ocaml is capable of and it’s mostly based on runtime reflection):

type Closeable = { def close():Unit } def using[A, B <: Closeable](closable: B)(f: B => A): A = try { f(closable) } finally { closable.close() }

This comparisson isn’t really fair btw, because I’ve been fixating on issues that Scala does really well. Ocaml is great, however I personally find it limiting and awkward at the edges of the 2 type systems it contains. Or maybe I’m just a spoiled brat.

10. Static-type versus Dynamic-type Systems

Static versus dynamic is what polarizes developers most in separate camps. It’s like a never-ending flamewar, with healthy dosages of religiosity.

At its core, a static type system helps you by providing proof at compile-time that the types you’re using behave as you expect them to behave (note I’m speaking of types, not instances). This is good, because you need all the help you can get and static typing can eliminate a lot of errors.

This is a doubly-edged sword though. By definition a static type system will reject pieces of code that are perfectly correct. Also, it’s not a silver bullet, as Rich Hickey said in his excellent Simple Made Easy talk: “What’s the common thing that all bugs in the wild share? They passed the type-checker, they passed all the tests!”

I’ve seen opinions that “structural typing” or “type-inference” are as good as “duck typing”. That couldn’t be further from the truth - the real power of duck typing comes from the ability to create / modify types and functions on the fly at runtime. In other words you can make shit up and as long as it’s correct, then it works. In contrast, a static type system actively rejects pieces of code if it can’t prove that the types you’re using support the computation you’re trying to do, so no matter how smart the type system is, you’ll always end up in lots of instances where you have to spoon-feed the compiler to allow you to do what you mean (n.b. not all compilers are equal).

This is not to say that static typing is bad. Well, it is bad in languages where the type system is designed to help the IDE and not the developer (e.g. Java, Delphi, Visual Basic). Otherwise, especially in combination with referential transparency, it really eliminates a whole class of errors.

Here we define an error as being an incorrect state of the computation or corrupted output that takes the developers by surprise. An exceptional state that’s being controlled is not an error. This is why Haskell makes such a big fuss out of dealing with side-effects by means of monadic types - because it makes you think about exceptional state and either deal with it, or make it somebody else’s problem.

Thinking of Scala versus Clojure and Haskell, in regards to its static-type system Scala sits somewhere in the middle. This is both good and bad. On one hand Scala does not have the same (static) expressive capabilities of Haskell, being a poor substitute for it when working with higher-kinded types. On the other hand you can drill holes in that static-type system to make it do what you want, which I think is a good trade-off.

I personally lean towards dynamic type systems, however the tradeoffs I end up making in Scala are worth it for the extra type safety it brings. On the other hand Clojure, because of its support for multi-methods and protocols and macros, is a dynamic language that’s more expressive than most other languages, including dynamic ones, especially the mainstream, like Python, Ruby, Perl, Javascript or PHP.

11. Performance

I don’t have any experience or proof on this, just personal feelings :-)

Scala runs on top of the JVM. When using closures or immutable data-structures, it is wasteful. However there are a few things to consider:

Scala can be as low-level and as efficient as Java for the hot codepaths and low-level Scala code is still higher-level than Java (for instance the pimp-my-library pattern will have 0 overhead starting with Scala 2.10, while implicit parameters are compile-time)

the built-in immutable data-structures are optimized to be versioned / to reuse memory of prior states - just as when adding a new element to a List the old reference gets used as the tail, this also happens with Vectors and Maps - they are still less efficient than Java’s collections, but it’s a good tradeoff as these data-structures can be used without read-locks, so bye, bye lock-contention of threads

Scala creates lots of short lived objects. This can stress the garbage collector, but on the other hand the JVM has the most advanced garbage collectors available, so you shouldn’t worry about it unless profiling tools tell you to … for instance on the JVM heap allocation is as cheap as stack allocation, it can also do some escape analysis to get rid of some locks and to allocate some short-lived objects on the stack and deallocation of short-lived objects is cheap, because the GC is generational so it deallocates whole chucks of memory at once instead of individual references … so why worry about it?

the only instance to be concerned about is if you’re building on top of Android, as Android does not have a JVM - but even there, Scala is workable (or so I’ve heard)

12. Tools of the Trade

I have a love/hate relationship with SBT, the defacto builds manager for Scala, the replacement for Maven, the slayer of XML files.

The syntax is really weird and leads to cargo-culting. It broke compatibility and so many examples available online are out of date. When you’re reading the Getting Started tome, it describes something about immutable data-structures, settings options that are either lazy or strict, how to transform values with a ~= operator, something about another operator written as <<= and so on.

Comparing this to how you work with Ruby Gems / Rake and Bundler is simply not fun. Only a mother could love this syntax.

Then I’ve already had problems with its Ivy integration, not being able to solve some dependencies. Thankfully I could find a fix.

On the other hand it’s really pragmatic and I prefer it over Maven, even if the Scala Maven plugin is in really good shape right now. Here are some highlights of SBT:

it can do cross-builds between multiple Scala versions; as is well known, major Scala versions are breaking binary compatibility, so if you want your library to support multiple Scala versions then SBT is a must, as it makes cross-building a breeze (it’s almost too easy)

it’s well integrated with ScalaTest, being able of continous compilation and testing, with output in colors - a really good tool for TDD

it makes it easy to deal with multiple sub-projects in the same root project, sub-projects that can be worked-on, tested or published individually or as a whole

all Scala projects have instructions for SBT first, Maven second and missing instructions for everything else - this is particularly painful if you’re dealing with plugins (like doing precompilation of templates with Scalate or something)

I use Emacs.

IDEs are not on the same level as Java. But I tried out IntelliJ IDEA’s Scala plugin and it’s quite decent, with refactoring, intellisense and everything nice. An Eclipse plugin is also available, developed now by TypeSafe, however last time I tried, it was unstable.

So IDEs for Scala are in a worst shape than for Java, but on the other hand these IDEs are functional and completely awesome when compared to what you get by picking other functional languages, except maybe F#.

With Scala you can use all the profiling and monitoring tools and classpath reloading tricks that you can use with Java. Nothing’s stopping you, as every tool meant for the JVM also works with Scala.

13. Concurrency and Parallelism

It’s enough to say that Scala doesn’t restrict you in any way:

Light-weight actors that can process tons of messages (Erlang-style) and that work either on the same machine, in a single process, or distributed over a network

Futures and Promises, which in contrast to other languages (* cough * javascript / jquery * cough *) are properly implemented as monadic types

Software transactional memory, as in Clojure

Parallel collections

Async/await as in C#, though it requires a compiler plugin

The awesome Java NIO, along with Netty, Mina and the whole ecosystem for async I/O (you don’t know what pleasure feels like until you wrap Async-Http-Client in Akka Promises which you can combine in for-comprehensions)

Basically Scala has it all. This may seem like a curse, but what other languages define as built-in / hard to change / hard to evolve features, Scala defines as libraries. So there are definitely upsides ;-)

14. Learning Resources

I’ve found the following to be good resources for learning Scala (note that Amazon links have my affiliate tag, but if you want the eBook version don’t buy from Amazon, prefer buying directly from the publisher, as you’ll get both a DRM-free Kindle version and a PDF):

Functional Programming Principles in Scala, already mentioned, is an excellent course provided by Coursera / EPFL, taught by Martin Odersky. The course is almost over, but the material will be left online, which means you can follow the lectures and do the assignments and I’m pretty sure many students that attended will remain on that forum for answering questions.

Scala School: a freely available online tutorial by Twitter, which is very friendly to newbies. I’ve read it and it’s pretty good.

Scala Documentation Project: definitely checkout this website, as they aggregate everything good here. If you want to learn more about Scala’s library, especially the collections, this is the place to learn from.

Ninety-Nine Scala Problems: a collection of 99 problems to be solved with Scala. If you get stuck, you can view a solution which is often idiomatic. See also this GitHub project that gives you a complete test-suite, to spare you of the effort.

Programming in Scala by Martin Odersky is a good book on programming, not just Scala - many of the exercises in Structure and Interpretation of Computer Programs are also present in this book, giving you the Scala-approach for solving those problems, which is good.

Scala for the Impatient by Cay S. Horstmann, is a good pragmatic book on Scala (not so much on functional programming), but it’s for developers experienced in other languages, so it’s fast-paced while not scaring you away with endless discussions on types (like I just did). The PDF for the first part (out of 3) is available from the Typesafe website.

Scala in Depth by Joshua Suereth D. - this is an advanced book on Scala, with many insights into how functional idioms work in it. I’ve yet to finish reading, as it’s not really an easy lecture. But it’s a good book. Get the eBook straight from Manning.

The End, Finally

A sequel on what makes Clojure great will follow when I have the time or patience for it (or once I finish reading the Joy of Clojure, great book btw).

The article On Scala, Functional Programming and Type-Classes first appeared on alexn.org.

How To Build a Naive Bayes Classifier

Alexandru Nedelcu — Thu, 09 Feb 2012 00:00:00 +0000

In machine learning a classifier is able to predict, given an input, a probability distribution over a set of categories. Some use-cases for building a classifier:

Spam detection, for example you could build your own Akismet API

Automatic assignment of categories to a set of items

Automatic detection of the primary language (e.g. Google Translate)

Sentiment analysis, which in simple terms refers to discovering if an opinion is about love or hate about a certain topic

In general you can do a lot better with more specialized techniques, however the Naive Bayes classifier is general-purpose, simple to implement and good-enough for most applications. And while other algorithms give better accuracy, in general I discovered that having better data in combination with an algorithm that you can tweak does give better results for less effort.

In this article I’m describing the math behind it. Don’t fear the math, as this is simple enough that a high schooler understands. And even though there are a lot of libraries out there that already do this, you’re far better off for understanding the concept behind it, otherwise you won’t be able to tweak the implementation in response to your needs.

0. The Source Code

I published the source-code associated at github.com/alexandru/stuff-classifier. The implementation itself is at lib/bayes.rb, with the corresponding test/test_003_naive_bayes.rb.

1. Introduction to Probabilities

Let’s start by refreshing forgotten knowledge. Again, this is very basic stuff, but if you can’t follow the theory here, you can always go to the probabilities section on khanacademy.org.

1.1. Events and Event Types

An “event” is a set of outcomes (a subset of all possible outcomes) with a probability attached. So when flipping a coin, we can have one of these 2 events happening: tail or head. Each of them has a probability of 50%. Using a Venn diagram, this would look like this:

And another example which clearly shows the dependence between “rain” and “cloud formation”, as raining can only happen if there are clouds:

The relationship between events is very important, as you’ll see next:

2 events are disjoint (exclusive) if they can’t happen at the same time (a single coin flip cannot yield a tail and a head at the same time). For Bayes classification, we are not concerned with disjoint events.

2 events are independent when they can happen at the same time, but the occurrence of one event does not make the occurrence of another more or less probable. For example the second coin-flip you make is not affected by the outcome of the first coin-flip.

2 events are dependent if the outcome of one affects the other. In the example above, clearly it cannot rain without a cloud formation. Also, in a horse race, some horses have better performance on rainy days.

What we are concerned here is the difference between dependent and independent events, because calculating the intersection (both happening at the same time) depends on it. So for independent events, calculating the intersection is easy:

Some examples:

if you have 2 hard-drives, each of them having a 0.3 (30%) probability of failure within the next year, that means there’s a 0.09 (9%) probability of them failing both within the next year

if you flip a coin 4 times, there’s a 0.0625 probability of getting a tail 4 times in a row (0.5 ^ 4)

Things are not so simple for dependent events, which is where the Bayes Theorem comes into play.

1.2. Conditional Probabilities and The Bayes Theorem

Let’s take one example. So we have the following stats:

30 emails out of a total of 74 are spam messages

51 emails out of those 74 contain the word “penis”

20 emails containing the word “penis” have been marked as spam

So the question is: what is the probability that the latest received email is a spam message, given that it contains the word “penis”?

So these 2 events are clearly dependent, which is why you must use the simple form of the Bayes Theorem:

Or visually:

With the solution being:

This was a simple one, you could definitely see the result without complicating yourself with the Bayes formula.

1.3. The Naive Bayes Approach

Let us complicate the problem above by adding to it:

25 emails out of the total contain the word “viagra”

24 emails out of those have been marked as spam

so what’s the probability that an email is spam, given that it contains both “viagra” and “penis”?

Shit just got more complicated, because now the formula is this one:

And you definitely don’t want to bother with it if we keep adding words. But what if we simplified our assumptions and just say that the occurrence of penis is totally independent from the occurrence of viagra? Then the formula just got much simpler:

To classify an email as spam, you’ll have to calculate the conditional probability by taking hints from the words contained. And the Naive Bayes approach is exactly what I described above: we make the assumption that the occurrence of one word is totally unrelated to the occurrence of another, to simplify the processing and complexity involved.

This does highlight the flaw of this method of classification, because clearly those 2 events we’ve picked (viagra and penis) are correlated and our assumption is wrong. But this just means our results will be less accurate.

2. Implementation

I mention it again, you can take a look at the source-code published at github.com/alexandru/stuff-classifier.

2.1. General Algorithm

You simply get the probability for a text to belong to each of the categories you test against. The category with the highest probability for the given text wins:

Do note that above I also eliminated the denominator from our original formula, because it is a constant that we do not need (called evidence).

2.2. Avoiding Floating Point Underflow (UPDATE Feb 27, 2012)

Because of the underlying limits of floating points, if you’re working with big documents (not the case in this example), you do have to make one important optimization to the above formula:

instead of the probabilities of each word, you store the (natural) logarithms of those probabilities

instead of multiplying the numbers, you add them instead

So instead of the above formula, if you need this optimization, then use this one:

2.3. Training

Your implementation must have a training method. Here’s how mine looks like:

def train(category, text) each_word(text) {|w| increment_word(w, category) } increment_cat(category) end

And its usage:

classifier.train :spam, "Grow your penis to 20 inches in just 1 week" classifier.train :ham, "I'm hungry, no I don't want your penis"

For the full implementation, take a look at base.rb.

2.4. Getting Rid of Stop Words / Stemming

First of all, you must get rid of the junk. Every language has words that are so commonly used that make them meaningless for any kind of classification you may want to do. For instance in English you have words such as “the”, “to”, “you”, “he”, “only”, “if”, “it” that you can safely strip out from the text.

I’ve compiled a list of such words in this file: stop_words.rb. You can compile such a list by yourself if you’re not using English for example. Head over to Project Gutenberg, download some books in the language you want, count the words in them, sort by popularity in descending order and keep the top words as words that you can safely ignore.

Also, our classifier is really dumb in the sense that it does not care about the meaning or context of a word. So there’s a problem: consider the word “running”. What you want is to treat this just as “run”, which is the morphological root of the word. You also want to treat “parenting” and “parents” as “parent”.

This process is called stemming and there are lots of libraries for it. I think currently the most up-to-date and comprehensive library for stemming is Snowball. It’s a C library with lots of bindings available, including for Ruby and Python and it even has support for my native language (Romanian).

Take a look at what I’m doing in tokenizer.rb, where I’m getting rid of stop words and stemming the remaining.

each_word('Hello world! How are you?') # => ["hello", "world"] each_word('Lots of dogs, lots of cats! This is the information highway') # => ["lot", "dog", "lot", "cat", "inform", "highwai"] each_word("I don't really get what you want to accomplish. There is a class TestEval2, you can do test_eval2 = TestEval2.new afterwards. And: class A ... end always yields nil, so your output is ok I guess ;-)") # => ["really", "want", "accomplish", "class", # "testeval", "test", "eval", "testeval", "new", # "class", "end", "yields", "nil", "output", # "ok", "guess"]

NOTE: depending on the size of your training data, this may not be a good idea. Stemming is useful in the beginning when you don’t have a lot of data. Otherwise consider “house” and “housing” … the former is used less frequently in a spammy context then the later.

2.5. Implementation Guidelines

When classifying emails for spam, it is a good idea to be sure that a certain message is a spam message, otherwise users may get pissed by too many false positives.

Therefore it is a good idea to have thresholds. This is how my implementation looks like:

def classify(text, default=nil) # Find the category with the highest probability max_prob = 0.0 best = nil scores = cat_scores(text) scores.each do |score| cat, prob = score if prob > max_prob max_prob = prob best = cat end end # Return the default category in case the threshold condition was # not met. For example, if the threshold for :spam is 1.2 # # :spam => 0.73, :ham => 0.40 (OK) # :spam => 0.80, :ham => 0.70 (Fail, :ham is too close) return default unless best threshold = @thresholds[best] || 1.0 scores.each do |score| cat, prob = score next if cat == best return default if prob * threshold > max_prob end return best end

Final Words

My example involved spam classification, however this is not how modern spam classifiers work btw. Because the independence assumptions are often inaccurate, this type of classifier can be gamed by spammers to trigger a lot of false positives, which will make the user turn the feature off eventually.

But it is general purpose, being good enough not only for spam detection, but also for lots of other use-cases and it’s enough to get you started.

The article How To Build a Naive Bayes Classifier first appeared on alexn.org.

Data Mining: Finding Similar Items and Users

Alexandru Nedelcu — Mon, 16 Jan 2012 00:00:00 +0000

How to find related items? Here are recipes based on really simple formulas. If you pay attention, this technique is used all over the web (like on Amazon) to personalize the user experience and increase conversion rates. Because we want to give kick-ass product recommendations.

To get one question out of the way: there are already many available libraries that do this, but as you’ll see there are multiple ways of skinning the cat and you won’t be able to pick the right one without understanding the process, at least intuitively.

Defining the Problem

To find similar items to a certain item, you’ve got to first define what it means for 2 items to be similar and this depends on the problem you’re trying to solve:

on a blog, you may want to suggest similar articles that share the same tags, or that have been viewed by the same people viewing the item you want to compare with

Amazon has this section called “customers that bought this item also bought”, which is self-explanatory

a service like IMDB, based on your ratings, could find users similar to you, users that liked or hated approximately the same movies you did, thus giving you suggestions on movies you’d like to watch in the future

In each case you need a way to classify these items you’re comparing, whether it is tags, or items purchased, or movies reviewed. We’ll be using tags, as it is simpler, but the formula holds for more complicated instances.

Redefining the Problem in Terms of Geometry

We’ll be using my blog as sample. Let’s take some tags:

["API", "Algorithms", "Amazon", "Android", "Books", "Browser"]

That’s 6 tags. Well, what if we considered these tags as dimensions in a 6-dimensional Euclidean space? Then each item you want to sort or compare becomes a point in this space, in which a coordinate (representing a tag) is either one (tagged) or zero (not tagged).

So let’s say we’ve got one article tagged with API and Browser. Then its associated point will be:

[ 1, 0, 0, 0, 0, 1 ]

Now these coordinates could represent something else. For instance they could represent users. If say you’ve got a total of 6 users in your system, 2 of them rating an item with 3 and 5 stars respectively, you could have for the article in question this associated point (do note the order is very important):

[ 0, 3, 0, 0, 5, 0 ]

So now you can go ahead and calculate distances between these points. For instance you could calculate the angle between the associated vectors, or the actual euclidean distance between the 2 points. For a 2-dimensional Euclidean space, here’s how it would look like:

Euclidean Distance

The mathematical formula for the Euclidean distance is really simple. Considering 2 points, A and B, with their associated coordinates, the distance is defined as:

The lower the distance between 2 points, then the higher the similarity. Here’s some Ruby code:

# Returns the Euclidean distance between 2 points # # Params: # - a, b: list of coordinates (float or integer) # def euclidean_distance(a, b) sq = a.zip(b).map{|a,b| (a - b) ** 2} Math.sqrt(sq.inject(0) {|s,c| s + c}) end # Returns the associated point of our tags_set, relative to our # tags_space. # # Params: # - tags_set: list of tags # - tags_space: _ordered_ list of tags def tags_to_point(tags_set, tags_space) tags_space.map{|c| tags_set.member?(c) ? 1 : 0} end # Returns other_items sorted by similarity to this_item # (most relevant are first in the returned list) # # Params: # - items: list of hashes that have [:tags] # - by_these_tags: list of tags to compare with def sort_by_similarity(items, by_these_tags) tags_space = by_these_tags + items.map{|x| x[:tags]} tags_space.flatten!.sort!.uniq! this_point = tags_to_point(by_these_tags, tags_space) other_points = items.map{|i| [i, tags_to_point(i[:tags], tags_space)] } similarities = other_points.map{|item, that_point| [item, euclidean_distance(this_point, that_point)] } sorted = similarities.sort {|a,b| a[1] <=> b[1]} return sorted.map{|point,s| point} end

And here is the test you could do, and btw you can copy the above and the bellow script and run it directly:

# SAMPLE DATA all_articles = [ { :article => "Data Mining: Finding Similar Items", :tags => ["Algorithms", "Programming", "Mining", "Python", "Ruby"] }, { :article => "Blogging Platform for Hackers", :tags => ["Publishing", "Server", "Cloud", "Heroku", "Jekyll", "GAE"] }, { :article => "UX Tip: Don't Hurt Me On Sign-Up", :tags => ["Web", "Design", "UX"] }, { :article => "Crawling the Android Marketplace", :tags => ["Python", "Android", "Mining", "Web", "API"] } ] # SORTING these articles by similarity with an article # tagged with Publishing + Web + API # # # The list is returned in this order: # # 1. article: Crawling the Android Marketplace # similarity: 2.0 # # 2. article: "UX Tip: Don't Hurt Me On Sign-Up" # similarity: 2.0 # # 3. article: Blogging Platform for Hackers # similarity: 2.645751 # # 4. article: "Data Mining: Finding Similar Items" # similarity: 2.828427 # sorted = sort_by_similarity( all_articles, ['Publishing', 'Web', 'API']) require 'yaml' puts YAML.dump(sorted)

The Problem (or Strength) of Euclidean Distance

Can you see one flaw with it for our chosen data-set and intention? I think you can - the first 2 articles have the same Euclidean distance to [“Publishing”, “Web”, “API”], even though the first article shares 2 tags with our chosen item, instead of just 1 tag as the rest.

To visualize why, look at the points used in calculating the distance for the first article:

[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1] [1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1]

So 4 coordinates are different. Now look at the points used for the second article:

[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1] [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1]

Again, 4 coordinates are different. So here’s the deal with Euclidean distance: it measures dissimilarity. The coordinates that are the same are less important than the coordinates that are different. For my purpose here, this is not good - because articles with more tags (or less) tags than the average are going to be disadvantaged.

Cosine Similarity

This method is very similar to the one above, but does tend to give slightly different results, because this one actually measures similarity instead of dissimilarity. Here’s the formula:

If you look at the visual with the 2 axis and 2 points, we need the cosine of the angle theta that’s between the vectors associated with our 2 points. And for our sample it does give better results.

The values will range between -1 and 1. -1 means that 2 items are total opposites, 0 means that the 2 items are independent of each other and 1 means that the 2 items are very similar (btw, because we are only doing zeros and ones for coordinates here, this score will never get negative for our sample).

Here’s the Ruby code (leaving out the wiring to our sample data, do that as an exercise):

def dot_product(a, b) products = a.zip(b).map{|a, b| a * b} products.inject(0) {|s,p| s + p} end def magnitude(point) squares = point.map{|x| x ** 2} Math.sqrt(squares.inject(0) {|s, c| s + c}) end # Returns the cosine of the angle between the vectors #associated with 2 points # # Params: # - a, b: list of coordinates (float or integer) # def cosine_similarity(a, b) dot_product(a, b) / (magnitude(a) * magnitude(b)) end

Also, sorting the articles in the above sample gives me the following:

- article: Crawling the Android Marketplace similarity: 0.5163977794943222 - article: "UX Tip: Don't Hurt Me On Sign-Up" similarity: 0.33333333333333337 - article: Blogging Platform for Hackers similarity: 0.23570226039551587 - article: "Data Mining: Finding Similar Items" similarity: 0.0

Right, so much better for this chosen sample and usage. Ain’t this fun? BUT, you guessed it, there’s a problem with this too …

The Problem with Our Sample; The Tf-Idf Weight

Our data sample is so simple that we could have simply counted the number of common tags and use that as a metric. The result would be the same without getting fancy with Cosine Similarity :-)

Clearly a tag such as “Heroku” is more specific than a general purpose tag such as “Web”. Also, just because Jekyll was mentioned in an article, that doesn’t make the article about Jekyll. Also an article tagged with “Android” may be twice as Android-related as another article also tagged with “Android”.

So here’s a solution to this: the Tf-Idf weight, a statistical measure used to evaluate how important a word is to a document in a collection or corpus. With it you can give values to your coordinates that are much more specific than simple ones and zeros. But I’ll leave that for another day.

Also, related to our simple data-set here, perhaps an even simpler metric, like the Jaccard index would be better.

Pearson Correlation Coefficient

The Pearson Correlation Coefficient for finding the similarity of 2 items is slightly more sophisticated and doesn’t really apply to my chosen data-set. This coefficient measures how well two samples are linearly related.

For example, on IMDB we may have 2 users. One of them, lets call him John, has given the following ratings to 5 movies: [1, 2, 3, 4, 5]. The other one, Mary, has given the following ratings to the same 5 movies: [4, 5, 6, 7, 8]. The 2 users are very similar, as there is a perfect linear correlation between them, since Mary just gives the same rankings as John plus 3. The formula itself or the theory is not very intuitive though. But it is simple to calculate:

Here’s the code:

def pearson_score(a, b) n = a.length return 0 unless n > 0 # summing the preferences sum1 = a.inject(0) {|sum, c| sum + c} sum2 = b.inject(0) {|sum, c| sum + c} # summing up the squares sum1_sq = a.inject(0) {|sum, c| sum + c ** 2} sum2_sq = b.inject(0) {|sum, c| sum + c ** 2} # summing up the product prod_sum = a.zip(b).inject(0) {|sum, ab| sum + ab[0] * ab[1]} # calculating the Pearson score num = prod_sum - (sum1 *sum2 / n) den = Math.sqrt((sum1_sq - (sum1 ** 2) / n) * (sum2_sq - (sum2 ** 2) / n)) return 0 if den == 0 return num / den end puts pearson_score([1,2,3,4,5], [4,5,6,7,8]) # => 1.0 puts pearson_score([1,2,3,4,5], [4,5,0,7,8]) # => 0.5063696835418333 puts pearson_score([1,2,3,4,5], [4,5,0,7,7]) # => 0.4338609156373132 puts pearson_score([1,2,3,4,5], [8,7,6,5,4]) # => -1

Manhattan Distance

There is no one size fits all and the formula you’re going to use depends on your data and what you want out of it.

For instance the Manhattan Distance computes the distance that would be traveled to get from one data point to the other if a grid-like path is followed. I like this graphic from Wikipedia that perfectly illustrates the difference with Euclidean distance:

Red, yellow and blue lines all have the same length and the distance is bigger than the corresponding green diagonal, which is the normal Euclidean distance.

Personally I haven’t found a usage for it, as it is more related to path-finding algorithms, but it’s a good thing to keep in mind that it exists and may prove useful. Since it measures how many changes you have to do to your origin location to get to your destination while being limited to taking small steps in a grid-like system, it is very similar in spirit to the Levenshtein distance, which measures the minimum number of changes required to transform some text into another.

The article Data Mining: Finding Similar Items and Users first appeared on alexn.org.

Blogging Platform for Hackers

Alexandru Nedelcu — Thu, 05 Jan 2012 00:00:00 +0000

Host your own static website using free services. This article is a recipe for:

building your website with Jekyll

hosting your own static website on Heroku’s free plan;

using Google’s App Engine as a CDN, for better responsiveness;

keeping Heroku’s free dyno alive, by using a GAE cron job;

having a very responsive, scalable and secure blog, with ultimate; control and simplicity, for zero bucks per month;

UPDATE (2019-12-19): these days there are better options available for hosting your static website for free, including GitHub Pages (also see GitHub Actions) and Netlify.

You could just skip this article and browse the source code of my blog:

github.com/alexandru/alexn.org

Forget about Wordpress or Blogger. Hacking your own stuff is much more fun. Also, make sure to read Blogging Like a Hacker, by Tom Preston-Werner, GitHub’s cofounder and the author of Jekyll.

Jekyll and Heroku, Sitting in a Tree

I love Jekyll, the static website generator. It is pure awesomeness for me:

all content is hosted in a Git repository, the best CMS ever invented

my articles are written in Markdown, with Emacs, the most potent text editor ever created - think Textmate-snippets, macros, syntax highlighting, keyboard-driven navigation and spelling corrections

static content scales like crazy, without any special gimmicks. A small VPS can serve thousands of requests per second without a sweat

static content is also secure by default, no constant upgrades required, no SQL injections

I always make little tweaks to my design, I’m never satisfied, which is why it makes sense to make my own, but checkout Octopress in case you want a reasonable default

I’ve lost an entire blog when my hosting account got blocked in the past. Never again, as my content is right now saved in 2 Git repositories and on my local machine

by working with my own domain, making my own shit, Google will never make me cry ;-)

Jekyll’s first hosting option you should consider is GitHub Pages, however you will need some dynamic behavior, like having configurable redirects. If you don’t then ignore this post and just read Jekyll’s tutorial, but you can come back to this post when its limits start bothering you.

Heroku’s free plan is awesome, in spite of what I said previously. It’s great for prototyping and for quickly seeing your website online. Instant gratification is awesome. Well, it does have some problems and to tell you the truth, for hosting my blog I would have rather used Google’s App Engine, if only they allowed me to have naked domains. I like my domains to be naked.

One note in regards to the scalability of static content I mentioned above. In Heroku the Bamboo stack features a Varnish frontend. If you set proper expiry headers on your content, subsequent requests will not hit the Ruby server.

Hosting Static Content on Heroku

So this tutorial is about hosting a Jekyll website, which is why I’m going to make some assumptions about your directory structure. However you can modify these instructions for any static website, not just Jekyll-generated stuff.

First, the setup:

# install the heroku command-line utility gem install heroku # change to your website directory cd website/ # initialize a git repo, if you haven't done so git init # ... and commit everything to it git add . git commit -m 'initial commit' # create the heroku app heroku create

OK, now we need a Rake-powered application to serve our content. We’ll need a ./Gemfile …

source 'http://rubygems.org' gem 'rack' gem 'mime-types' group :development do gem 'jekyll' gem 'rdiscount' gem 'hpricot' end

Then install these gems with:

bundle install

You also need a Rake configuration file, ./config.ru. What follows is the configuration that I am using. You can go simpler, a lot simpler than this actually, but I like flexibility and Heroku also does something funny with files served through Rack::File, so I refrained from using it …

# Rack configuration file for serving a Jekyll-generated static # website from Heroku, with some nice additions: # # * knows how to do redirects, with settings taken from ./_config.yaml # * sets the cache expiry for HTML files differently from other static # assets, with settings taken from ./_config.yaml require 'yaml' require 'mime/types' # main configuration file, also used by Jekyll CONFIG = YAML::load_file(File.join(File.dirname(__FILE__), '_config.yml')) # points to our generated website directory PUBLIC = File.expand_path(File.join(File.dirname(__FILE__), CONFIG['destination'] || '_site')) # For cutting down on the boilerplate class BaseMiddleware def initialize(app) @app = app end def each(&block) end end # Rack middleware for correcting paths: # # 1. redirects from the www. version to the naked domain version # # 2. converts directory/paths/ to directory/paths/index.html (most # importantly / to /index.html) class PathCorrections < BaseMiddleware def call(env) env['PATH_INFO'] += 'index.html' if env['PATH_INFO'].end_with? '/' request = Rack::Request.new(env) if request.host.start_with?("www.") [301, {"Location" => request.url.sub("//www.", "//")}, self] else @app.call(env) end end end # Middleware that enables configurable redirects. The configuration is # done in the standard Jekyll _config.yml file. # # Sample configuration in _config.yml: # # redirects: # - from: /docs/some-document.html # to: /archive/some-document.html # type: 301 # # The sample above will do a permanent redirect from ((*/docs/dialer.html*)) # to ((*/archive/some-document.html*)) class Redirects < BaseMiddleware def call(env) request = Rack::Request.new(env) path = request.path_info ext = File.extname(path) path += '/' if ext == '' and ! path.end_with?('/') if redirect = CONFIG['redirects'].find{|x| path == x['from']} new_location = redirect['to'] new_location = request.base_url + new_location \ unless new_location.start_with?("http") [redirect['type'] || 302, {'Location' => new_location}, self] else @app.call(env) end end end # The 404 Not Found message should be a simple one in case the # mimetype of a file is not HTML (like the message returned by # Rack::File). However, in case of HTML files, then we should display # a custom 404 message class Fancy404NotFound < BaseMiddleware def call(env) status, headers, response = @app.call(env) if status == 404 ext = File.extname(env['PATH_INFO']) if ext =~ /html?$/ or ext == '' or !ext headers = {'Content-Type' => 'text/html'} response = File.open(File.join PUBLIC, 'pages', '404.html') end end [status, headers, response] end end # Mimicking Rack::File # # I couldn't work with Rack::File directly, because for some reason # Heroku prevents me from overriding the Cache-Control header, setting # it to 12 hours. But 12 hours is not suitable for HTML content that # may receive fixes and other assets should have an expiry in the far # future, with 12 hours not being enough. class Application < BaseMiddleware class Http404 < Exception; end def guess_mimetype(path) type = MIME::Types.of(path)[0] || nil type ? type.to_s : nil end def call(env) request = Rack::Request.new(env) path_info = request.path_info # a /ping request always hits the Ruby Rake server - useful in # case you want to setup a cron to check if the server is still # online or bring it back to life in case it sleeps if path_info == "/ping" return [200, { 'Content-Type' => 'text/plain', 'Cache-Control' => 'no-cache' }, [DateTime.now.to_s]] end headers = {} if mimetype = guess_mimetype(path_info) headers['Content-Type'] = mimetype if mimetype == 'text/html' headers['Content-Language'] = 'en' headers['Content-Type'] += "; charset=utf-8" end end begin # basic validation of the path provided raise Http404 if path_info.include? '..' abs_path = File.join(PUBLIC, path_info[1..-1]) raise Http404 unless File.exists? abs_path # setting Cache-Control expiry headers type = path_info =~ /\.html?$/ ? 'html' : 'assets' headers['Cache-Control'] = "public, max-age=" headers['Cache-Control'] += CONFIG['expires'][type].to_s status, response = 200, File.open(abs_path, 'r') rescue Http404 status, response = 404, ["404 Not Found: #{path_info}"] end [status, headers, response] end end # # the actual Rack configuration, using # the middleware defined above # use Redirects use PathCorrections use Fancy404NotFound run Application.new(PUBLIC)

This Rack configuration uses settings defined in the standard Jekyll _config.yaml file. Here are some settings needed for it to work as intended:

destination: ./_site expires: html: 3600 # one hour assets: 1314000 # one year redirects: - from: /rss/ to: http://feeds.feedburner.com/bionicspirit type: 302

OK, so once done, test this configuration:

# generating the website jekyll # starting the server rackup

Deployment is as easy as pie:

git push heroku master

One note: Heroku could be configured to automatically generate the website for you. However you either have to use the Cedar stack, or generate the pages on the fly. In case of the Cedar stack, you lose Varnish. Just keep your generated files in Git, it’s easier.

Commenting with Disqus, Facebook or Roll Your Own

For commenting Disqus is a really good service. In case you have a very popular website amongst normal people, it may be even better to integrate Facebook’s commenting widget.

Well, I had some fun a while ago and created my own: TheBuzzEngine.

Unfortunately it doesn’t have all the features I want, but it does get the job done and it isn’t bloated. These days I’ll probably get around to adding some stuff to it, like threaded comments and email subscriptions. This is what happens when working for fun on stuff - once you’re over a certain threshold, the return of investment is too low to bother with extra development.

I recommend Disqus, although rolling your own is fun and keeps you in control (which is the reason I’m using Jekyll in the first place).

Using Google App Engine as Your CDN or Cron Manager

So when using Heroku’s free plan, I feel a little uncomfortable because relying on one dyno can get you in trouble. Having Varnish in front is great, but Varnish is a cache manager. For instance, if you happen to push a new version of your latest article to Heroku, then the Varnish cache gets cleared and the Ruby server can potentially get exposed to a lot of requests and one dyno on Heroku can only serve one request at a time.

So why not push all our static assets, except HTML files, to a CDN? It’s best practice anyway as your website should be more responsive. If you have an Amazon AWS account, then CloudFront + S3 are great.

However, I started with the goal of hosting this for zero bucks (it’s fun, so why not?). Therefore I’m going to teach you how to push your files to Google’s App Engine. I don’t really know how GAE works as a CDN for static files, but it seems that it does have the properties of a CDN (i.e. serving content to users from servers closer to their location).

Another problem with Heroku’s free plan is that the free dyno goes to sleep, to save resources. While I advise you to just pay up for an extra dyno, you can get around this restriction by just configuring GAE to send a periodic ping to your website.

Here’s my GAE configuration file, app.yaml which should sit in your root (assuming ./assets is the directory you want to serve from GAE):

application: assets-bionicspirit version: 1 runtime: python27 api_version: 1 threadsafe: true handlers: - url: /assets static_dir: assets expiration: "365d" # next item is for our cron job, described below, but you can ignore # it if you don't want a cron job ... - url: /tasks/ping script: ping.app

As you can see, I’m setting the expiry of my static assets to a whooping 1 year.

I also have a real handler, at /tasks/ping configured. This will be our cron job that sends a ping to our Heroku app, every X minutes. Here’s the code for ping.py …

import webapp2 from google.appengine.api import urlfetch class PingService(webapp2.RequestHandler): def get(self): self.response.headers['Content-Type'] = 'text/plain' url = "https://alexn.org/ping" try: result = urlfetch.fetch(url, deadline=30) self.response.out.write('HTTP %d - %s' % (result.status_code, (result.content or '').strip())) except: self.response.out.write('ERROR: no response') app = webapp2.WSGIApplication([('/tasks/ping', PingService)], debug=False)

But we are not done. To configure /tasks/ping to run every X minutes, you also need a cron.yaml file …

cron: - description: ping alexn.org to wake it up url: /tasks/ping schedule: every 4 minutes

Assuming you already have the GAE SDK installed, then run this command:

appcfg.py update .

To see it working on this blog, here are the requests:

Heroku URL getting requested: https://alexn.org/ping

GAE Cron Job getting executed: http://assets-bionicspirit.appspot.com/tasks/ping

Extra Tip - CloudFlare

Luigi Montanez kindly pointed out in below’s comments the availability of CloudFlare.

CloudFlare is a proxy that sits between your website and your users. It allegedly prevents DDOS attacks on your website, but it also caches static content, which helps because apparently it also has the properties of a CDN.

I activated it to see how it works. The main reason is that GAE has a 1 GB bandwidth-out daily limit - and this article generated ~ 10,000 visits in only one day, which consumed ~ 700 MB of bandwidth on GAE (for a couple of small images, I don’t want to imagine what would happen for an image-rich post). So that’s not good and I placed CloudFlare in front of GAE and my Heroku instance, which should save some bandwidth for me.

I don’t have a conclusion on CloudFlare. If it works as advertised, then it is awesome. Although be careful about it as I’ve seen reports on the Internet that it may in fact add latency to your website, instead of decreasing it.

For my website however, everything seems to be fine. I am monitoring my website with Pingdom.com, a service which also reports the average responsiveness of the website, calculated by doing requests from multiple locations. The homepage, which is not cached by CloudFlare or served by GAE, has an average load time of 300ms, while cached static resources from GAE and proxied through CloudFlare are doing much better.

So we’ll see.

Conclusion

The result is a really responsive, scalable and kick-ass blog, for zero bucks spent on hosting.

This very blog is hosted using the method described above. Well, I’ll probably return to my trustworthy VPS instance as I’m paying for it anyway, but this was fun.

Enjoy ~

The article Blogging Platform for Hackers first appeared on alexn.org.

Crawling the Android Marketplace

Alexandru Nedelcu — Thu, 15 Dec 2011 00:00:00 +0000

I had a very specific need for fetching the details for some apps in the Android marketplace, in an automated manner. Here I'm describing a script that I used to crawl the marketplace back in 2011.

UPDATE (2019-12-18): this is an old article, the script I used no longer works, but it can still be a good starting point for building your own web crawler.

First I found the supermarket gem, a wrapper for the Android Market API Java implementation. However, it gives unpredictable results (it wouldn’t return the details of our in-house apps or of many other examples I tried) and Google is placing hard-limits on the number of requests you can make per minute. This is an internal API, probably used by the marketplace client and the implementation mentioned above was created through reverse-engineering.

This really pissed me off, this is Google, they should grok APIs. But this info is already available from their website and so I went ahead and crawled it.

The script and the data collected are is available. Read below.

How To Do it By Yourself

The actual script that I created can be found in the AndroidMarketCrawler GitHub Repository, with the relevant files being:

crawler.py - source code with lots of comments, it’s really not complicated, you should go read it

marketplace_database.json_lines.bz2 - compressed file containing the details of the crawled apps, one per each line; this is not a proper JSON file, you use it by reading it line by line, where each line represents a JSON object (personal preference, as otherwise the file is pretty big and you can run out of memory)

UPDATE: The Android Marketplace explicitly bans crawling apparently. This crawler and associated data only serves educational purposes. Don’t abuse it.

for app in AndroidMarketCrawler(concurrency=10): # app is at this point a dictionary with the details needed, like # id, name, developer name, number of installs, etc... fh.write(json.dumps(app) + "\n")

I used the Python programming language, along with Eventlet for fetching URLs in parallel (async I/O with epoll/libevent, providing you with coroutines support and green threads) and PyQuery for selecting DOM elements using CSS3 selectors (instead of XPath or BeautifulSoup). If you fancy Ruby instead, you could use slight equivalents such as em-http-request and Nokogiri.

So you start fetching content from a root and add application links as you encounter them in a queue. We are then using a (green) threadpool to start fetching jobs for each of the links in the queue. So it’s recursive. The results are also pushed in another queue, ready to be consumed by the client.

Be careful though, don’t abuse this, as it will generate a ton of traffic and your IP may end up being banned by Google. It also takes a lot of time; with good bandwidth and a VPS located in California, it still took me 5 hours for the script to finish. Don’t abuse the concurrency settings either, 10 is enough.

155,200 Apps Available From the US

You have to realize that this number is only approximate. Apps are going strong in other countries, such as South Korea and Google does Geo-IP filtering, which means some of the apps were unavailable to me, depending on restrictions set by their developers.

The numbers published by Research2Guidance in October tell the story of 500,000 apps published on the Marketplace. But this gets weird, as I took the number of downloads from those 155,200 apps and it matches the number of downloads published by Google this month. See below.

An Average of 13.63 Billion Downloads

So there have been between 5,514,202,281 and 21,545,335,515 downloads for free apps, making the average 13,529,768,898 downloads.

More interesting however is that according to my data for paid apps, the number of downloads is between 42,576,311 and 164,116,615. This number seems rather low to me, making it clear that Android distribution is freemium based.

The article Crawling the Android Marketplace first appeared on alexn.org.

Android Learning Resources

Alexandru Nedelcu — Mon, 12 Dec 2011 00:00:00 +0000

Starting out learning Android development may be intimidating at first, as with any new platform of reasonable complexity you'll have a lot to learn. However the learning process is fun. So here are some learning resources that I'm currently following.

UPDATE (2019-12-18): this is an old article, listing resources that may be obsolete and content that may not reflect my current views.

DISCLAIMER: the Amazon links in this article contain my affiliate code and I get a commission should you choose to buy from Amazon in the next 24 hours. However I’m including these links primarily because of the awesome reviews included, but you should buy straight from the publisher (publisher links also included, without an affiliate ID).

Books for Learning Android

My problem with books is that technical books get obsolete really fast. Books from 2010, while still useful, are already insufficient now with the release of Ice Cream Sandwich. The upside is that there are a lot of books out there.

For ~ $20 you can buy a 1-year subscription for CommonsWare.com. The author, Mark Murphy, is a very proficient Android developer, trainer and consultant, with a huge StackOverflow reputation :) More seriously - for $20 you get 3 books that are continuously updated, which is great.

Here are the books (but don’t buy them from Amazon, as you won’t get the 1-year subscription, which is the main reason I’m recommending these):

The Busy Coder’s Guide to Android Development

The Busy Coder’s Guide to Advanced Android Development

Android Programming Tutorials

Again, don’t buy these items from Amazon. Buy them with the 1-year subscription from the Author’s Website (I did so myself and btw, I have no affiliation with the author). The subscription is useful because you’ll get upgrades for new versions of Android and bug-fixes, for a whole year.

Another book I’ve been reading is Hello Android by Ed Burnette, published by the Pragmatic Programmers. It’s pretty good, but it is more of an introduction (truly a Hello World).

Other books I have not tried, so my list stops here, but updates will follow.

Free Stuff Available Online

I’m the kind of developer that prefers to rely on freely available stuff, because I learn by doing and technical books on APIs are boring. Plus, I like free stuff, however if you plan on getting serious about it, then a small investment is worth it and will keep you focused (nothing will keep you more focused than spending some money).

Android Developer Guides

Seriously, you won’t find a faster learning path than reading and understanding the source-code of real apps. However to not attempt doing this without going through some of the tutorials in the link above.

Videos from Google I/O

The Official Dev Guide

The videos from Google I/O are a gold mine, providing insight you won’t find anywhere else. Highly recommended. All in all, the official documentation is good, although lacking beginner friendliness and a structured clear path. Plus I noticed it has holes in it, but you’ll be fine.

For asking questions, or browse around for insightful gems, there’s nothing better than:

The Android tag on StackOverflow.com.

At the time of this writing, I don’t have many Android-related blogs to share with you and the ones I know about are of poor quality - apparently not many people blog about their Android experience. Hopefully this will change for the better.

Please help me out in identifying other resources! Thanks!

The article Android Learning Resources first appeared on alexn.org.

Earning Money as an Amazon Affiliate

Alexandru Nedelcu — Tue, 29 Nov 2011 00:00:00 +0000

I published an article that I've meant to publish for a long time. I'm usually lazy to not bother writing many articles, however this time I also thought about doing an experiment - you see I'm (1) on a tight budget and (2) a cheap bastard - so what if I could get enough money to pay for my monthly hosting on Linode, while satisfying my urge to write from time to time?

As such I included Amazon Affiliate links in that post to see what happens.

UPDATE (2019-12-18): this is an old article, listing resources that may be obsolete and content that may not reflect my current views.

Moral Considerations Against Amazon's Associates

The discussion on Hacker News took an interesting turn. A couple of comments are a little unbalanced, however there's a comment that I like and with which I am in agreement:

Having affiliate links creates incentives that may not align with faithfully serving your readers. It does not automatically bias your writing, but it can certainly create the appearance of bias.

A concrete example would be writing an especially glowing review of the new Kindle because you have a vested stake in people buying them. Or, perhaps, NOT writing a glowing review because you fear it will be perceived as shilling for affiliate cash.

In fact On The Media recently did a story about the Washington Post struggling with whether or not to include Amazon affiliate links in its book reviews. I think it presents both sides of the argument: www.onthemedia.org/2011/nov/11/web-links...

This comment is striking, as when I started fantasizing about where should I go from here, the one thing that crossed my mind was that I could write a review for the Kindle (I own one) - and oops, unfortunately in that instance my actual opinion would have been unfaithful - @esd nailed it.

Also, a reply puts the above in balance:

Newspapers likely disallow such practices in order to maintain journalistic integrity, but a blog author who is writing posts on purpose to sell things is probably not interested in maintaining journalistic integrity. The blog author is just interested in selling stuff. Maybe the blog posts are well-written and interesting, or maybe they are not. If they are not, then readers who care principally about content will likely avoid the blog on the lack of merit of the content itself.

(the quality of Hacker News still amazes me)

In my oppinion, putting affiliate links is not bad or evil per se - for instance you could say that customers of Apple or Google have an inherent bias because they feel the need to protect their monetary and/or emotional investment. And we aren't professional journalists, trained to watch out for such things - but a mistake is a mistake. So from now on, no more Amazon-related reviews coming from me (maybe I'll try clearly delimited boxes or something).

Some Facts and Stats

The article I'm talking about:

4 Books for Learning to Design, the Hard Way

I published this article on the 25th, The Black Friday - so it had perfect timing. Then I pushed this link to Hacker News and Reddit. I hope you will forgive me, since this was a little self-promotion and I don't deny it, however I hope you found the content therein to be worth it, as it was published with my best intentions.

The resulting traffic and the fee that Amazon gives for the orders generated took me by surprise. Here's how my traffic looks like (unique visitors):

Nov 24: 45

Nov 25: 10,263

Nov 26: 6,939

Nov 27: 1,713

Nov 28: 1,299

Wow. I never generated this much traffic with my blog. Also, here's the Amazon stats (updated for Nov 28th):

Ordered items: 333

Clicks: 6,259

Conversion: 5.32%

Total items shipped: 294

Total earnings: $367.10

So considering that not all items have been shipped and this ain't over yet (I'm still getting traffic), we're looking at: a half-an-hour effort for a single article, earning $400, in 4 days, on a very low-traffic personal blog.

My goal was achieved too - I now have enough money for ~20 months of hosting on Linode. Thank you dear readers, I am in your debt.

Amazon Associates versus Google AdSense

Google AdSense rewards are per-click and is the first option of many webmasters, because it does generate more money. However I feel that the overall user experience suffers a lot - the links served may be context-dependent, but the quality is poor.

Amazon Associates rewards are instead per-action. When the user buys something, you get a referral fee. This can work very well because the items displayed are hand-picked by you and the links add value to your content - whenever I search for reviews of individuals (which I trust more than those of experts), my first stop is on Amazon.

Why A Single Flower Doesn't Bring Spring

So whenever anybody does this successfully, the appetite only grows bigger. After all, this kind of revenue has the potential of being passive. However don't get your hopes up:

in this instance, the timing was perfect and so the circumstances can't be reproduced every month

to have constant conversions you need a big and loyal following that trusts your recommendations

to have a big and loyal following, you also need lots of traffic driven by search engines

to score well in search engines, you need lots of articles with lots of keywords and good ranking

to have loyal readers and a good ranking on Google, those articles must be high-quality

it takes lots and lots of work for the above, probably 1 or 2 years, considering you have the talent of writing content already, or the money to hire people to do it

if you can't recommend anything else other than books, then you're screwed, as you don't have the time to read so many

if the products getting recommended are not relevant to your audience, then they won't convert

So there you have it. My experiment with Amazon's Associates, while probably a short one, has brought joy in my heart :)

Thank you,

The article Earning Money as an Amazon Affiliate first appeared on alexn.org.

4 Books For Learning to Design, The Hard Way

Alexandru Nedelcu — Fri, 25 Nov 2011 00:00:00 +0000

This is the path I'm taking to not suck at design anymore, as frankly, I'm getting tired of sucking. I've read the first 3 books here and I'm having progress on the last one. I highly recommend all 4.

The Design of Everyday Things (link)

It explains the design of common household objects that you may use daily and it is one of the best books you can read on user experience.

To me this was an eye opener that completely changed the way I think about interfaces. It explains how people interact with the objects around them and how they learn. It explains the importance of a user's mental model of how your product works. It gives you a good feeling on what it means and why it matters to design for errors. It helps you to prevent a lot of design errors that a lot of products have.

The book itself was written in 1990, so it does have here and there some references to products that are outdated, but the analysis itself will never be outdated or obsolete. Quite the contrary - it's fascinating how user centric design guidlines stay the same, even though there are a lot of people out there that repeat the same mistakes over and over again.

When it comes to software, a complement article on the subject is User Interface Design For Programmers, by Joel Spolsky.

Non-Designer's Design Book (link)

Robin Williams does an excelent job introducing you to the basic concepts of designing visuals, with clearly explained principles and techniques.

It's pretty hard for us developers to design anything pleasing to the eye. And that's not the only problem you're facing - the visuals of a site have to give hints to the user about their next actions, so you've got many constraints to worry about. Sometimes you get lucky by just copying and combining other designs you like. Sometimes you have a good idea about what you want, but one day you like the result, then the next it looks like an abomination.

The book goes into some detail about how designers think. It has plenty of visual examples, it gives you many examples of what not to do, it explains how to work around those problems, it is concise and doesn't bore you to tears - while not that useful for someone with design experience, for a developer that sucks at this game, this is a really, really good design manual.

Color: A Course in Mastering the Art of Mixing Colors (link)

One of the most surprisingly difficult problems when creating designs is picking the colors palette. This is so goddamn difficult sometimes. We do know that certain color combinations work better than others, but how do you pick them? How can you achieve harmony as to not make your users' eyes bleed? And not only that, but you also want to emphasise certain portions of your pages in a way to attract attention - did you know that according to statistics, more car accidents involve red cars than any other color?

There are web services out there, like Adobe Kuler or COLOURlovers.com which allow you to choose color palettes created by other people - the PROBLEM being that you'll never know what makes a palette work and so you'll make good changes to it only by luck or with expensive A/B testing.

This book by Betty Edwards is not your only choice for learning Color Theory. It isn't even related to web design in any way, having a whole section on mixing oil paint.

However, I believe that color theory can only be learned from people that have real experience in mixing colors. For this reason, articles or books about color theory that aren't written by painters are quite shallow - and the theory in this book transcends the tools used.

It explains notions on color harmony, on the importance of contrast and gives you valuable insight and advices on how to mix and match colors. It's a fun read too, because of the quotes from faimous people contained - however, it's not a light read because you do have to execute the exercises within for best results. But I think it is worth it.

(I'm now my wife's advisor on colors, although our bedroom ended up looking awful, but there's no substitute for experience earned by making mistakes and at least I know where I went wrong ;))

Drawing on the Right Side of the Brain (link)

I've picked up drawing as my hobby. I'm only a beginner, but it's a fun hobby to have - it's impressive, silent and inexpensive. It also boosts your creativity like nothing else, as the only real limit you have is your imagination (much like software development ;))

Ever tried drawing anything lately? You should. The results will be awful. But do you know why? It's not your hand, it's not from a lack of talent, it's your eyes that are deceiving you - in order to learn how to draw, you have to relearn how to see. That's because everything you see is right now filtered and transformed by your brain - as a cheap/fast exercise, look in a mirror at an arm's length and take a guess if your mirrored head is of the same size as your actual head (then use your hands to measure). The picture you're getting through your mind's eye is deceiving and for drawing skill and creativity to emerge you have to silence your brain.

This book of Betty Edwards has an awesome technique that works for everybody. Or so she says, but as far as I'm concerned I'm progressing in leaps and bounds and can already do drawings that I couldn't hope of doing before starting to read this book ... I also promise to publish some drawings, but only after I'll get decent (only started to do this 2 months ago, a frog doesn't transform into a prince overnight you know).

How does drawing help you in web design? Well, do you really have to ask?

Update: Suggestions Received from Readers

The Design of Design, by Frederick P. Brooks

Visual Language for Designers, by Connie Malamed

Interaction of Color, by Josef Albers (review)

Keys to Drawing, by Bert Dodson

Design for Hackers, by David Kadavy

Design and Form, by Johannes Itten

Elements of Color, by Johannes Itten

Principles of Form and Design, by Wucius Wong

Thinking with a Pencil, by Henning Nelms

The Sciences of the Artificial, by Herbert A. Simon

Designerly Ways of Knowing, by Nigel Cross

Designing Visual Interfaces, by Kevin Mullet and Darrell Sano

Grid Systems: Principles of Organizing Type, by Kimberly Elam (presentation)

101 Things I Learned in Architecture School, by Matthew Frederick

Thanks people, wow, I know have my hands full :) ... keep `em coming!

The article 4 Books For Learning to Design, The Hard Way first appeared on alexn.org.

How I Use Flickr: For Backup

Alexandru Nedelcu — Sat, 29 Oct 2011 00:00:00 +0000

I've got a growing number of personal pictures and the collection is growing since 2003, when I got my first digital camera, a shitty Sanyo that still works and that I still use whenever I forget about my Nikon.

But here’s the thing with digital pictures - they are cheap to make, but also easy to lose. Digital storage is not as reliable as glossy paper. Pictures printed on paper can easily last for a 100 years. That’s not the case with any digital storage medium and we will suffer for it.

Storing My Pictures In The Cloud

Pro accounts on Flickr have unlimited storage and can upload and access full-resolution pictures. This is great, although be careful about believing in “unlimited plans”, as nothing is really unlimited and by abusing Flickr you may find yourself locked out of your account.

UPDATE (2019-12-15): since Flickr has been acquired by SmugMug, the free accounts no longer have unlimited, or the 1 TB of space we had in the Yahoo days. Pro accounts continue to have unlimited storage, but you have to keep paying for Pro, otherwise they’ll eventually delete your archive.

Unfortunately the tools for uploading really suck and I haven’t encountered yet a graphical interface that did what I needed. So for synchronizing, I’ve built my own script in Ruby using the excelent Flickraw gem and exifr, another Ruby gem that reads Exif headers from Jpeg files.

One common problem is that you ALWAYS have duplicates. And you don’t want to upload duplicates. What you really want is an “rsync” command for Flickr. But how do you know if a picture was already uploaded?

The approach I’m using is to add a machine tag to my pictures, which is set like a tag, but has the format “namespace:key=value”. This machine tag represents the MD5 hash of the picture and if you want to see if a certain photo was already uploaded to flickr, you can always search for it. Here’s how it looks on one of my pictures:

checksum:md5=5b2fa91c38a7f878088e1420b924e6d9

Besides this, I have this problem with some of the older photos taken by my Sanyo, where the taken-date is totally fucked and for personal photos the taken date is maybe more important than the actual photo quality. I use the excelent ExifTool to correct those photos. It’s nice building on the hard work of other people ;)

So, currently I have 3545 pictures uploaded on Flickr in full resolution and the number will more than triple as soon as I make an inventory of my pictures stored on old hardware I’ve got lying around.

It is fun being a developer. I can make shit happen.

Flickr is Not A Reliable Backup

Flickr is an online service that isn’t meant for being a backup. I share only a fraction of what I upload, everything else is family only. They may terminate your account at any time for whatever reason. They may also go out of business. Yahoo may sell it, etc, etc… I do think Flickr is awesome btw and one reason that I store my photos on Flickr is to be able to always have the whole archive with me. But for backup alone, that’s not enough.

What you really need to do is:

your main repository should be stored locally and properly maintained - I do that on my main computer currently, but multi-TB external hard-drives are cheap

in case of cloud backup, you always need a secondary service for redundancy

Google’s Picasa is a good option because you can explicitly buy storage. This means that if you pay for 80 GB of storage nobody is going to get upset that you uploaded 80 GB of private photos … also, if your photo collection matters to you, I wouldn’t put my trust in their Google+ offering (photos of up to 2048x2048 pixels do not count towards your free quota). That’s because that offer is not meant for you. Just pay up.

So on Google’s Picasa, I’m currently working on integrating with their API too. The desktop app is nice, but too limited for me.

The article How I Use Flickr: For Backup first appeared on alexn.org.

Why I Find Heroku Suboptimal

Alexandru Nedelcu — Sun, 23 Oct 2011 00:00:00 +0000

I love freebies. I often find myself compelled to search for the best price / convenience ratio, and from this perspective you cannot really argue against something offered for free. And yet, here I am bitching and moaning about Heroku.

Heroku provides a free-quota that’s a LOT more reasonable than all the shitty PHP hosting offerings out there. And when time comes to scale, it lets you scale nicely for a price.

UPDATE (Oct 21, 2013): This article is outdated, sort of. Heroku’s Cedar stack is awesome and cost-effective if you can design your app to be efficient in terms of throughput (which you can only achieve if you use a platform capable of high performance, like the JVM). I still think that when starting out, the next step after the free Heroku dyno, is to get your own VPS.

UPDATE (Dec 18, 2019): this is an old article, listing resources that may be obsolete and content that may not reflect my current views.

Normally you develop your app on your localhost (which is like this warm and cozy place for all developers, no place like 127.0.0.1 and all that), but then you want to deploy. You have to get out of your comfort zone and face the jungle and it’s a true jungle out there, filled with shitty / underpowered and expensive hosting offerings. If going for a normal VPS, you’ll have to configure your application server, your database server, your webserver that sits on top, maybe a reverse proxy cache, a memcached instance or two, a load balancer, a firewall, an email server and it goes on and on. And if going for a classic shared-hosting environment, then God help you.

There’s a reason children with happy childhoods don’t want to grow up - the world is an ugly and scary place.

git push heroku master

Heroku is great. It basically allows you to avoid growing-up. The deployment itself couldn’t be simpler, and when browsing their web interface for available add-ons, I feel like a child in a candy-store.

Basically you start with a free worker, to which you can add other “free” services, like a 5MB PostgreSql database and a 5MB Memcached instance, allowing you to prototype stuff. They even have plugins from third-parties that give you freebies, like a 250MB CouchDB, or a 240MB MongoDB. Then as you grow, you start adding more and more resources as needed. This has been labeled as platform as a service and it’s what the cool kids are talking about these days. Heck, there are people that are living within that free-quota without problems. One such example that I know of is http://tzigla.com … or it was last time I talked to the authors, both acquaintances of mine, and Cristi described how he ended-up doing lots of workarounds to get around limitations and he was really excited about how everything fell into place.

But as I was sitting there admiring their determination and skill, I started wondering why the hell haven’t they rented a normal VPS?

I mean really, if you end up pulling all kinds of crap to get around limitations, wouldn’t it be better to just pay up? And if you’re short on cash or you’re the kind of entrepreneur that likes to spend frugally, then wouldn’t you be better just renting a normal VPS? I asked him just that of course, and his reply was basically:

I hate to do sys-admin stuff, installing and upgrading packages and all that

But it doesn’t have to be that way. It’s really not that hard. The reason for these feelings is the Ubuntu I have had installed on my primary laptop for 5 years already. Once you work with Ubuntu or your favorite Linux distribution, every day, configuring a web-server for starters is something like a half-an-hour chore. Or let’s say 1 hour, and then it’s done. And you don’t have to worry about it again.

And there are disadvantages to Heroku, lots of them: that’s because you lose control and end up on top of a platform that’s designed as a common denominator to appeal to all needs in an equally substandard manner.

Example 1: Nginx

Nginx is a freakishly fast web server that consumes really few resources. Its main appeal is in serving static files and you do have static files to serve. When you grow you may want to move those static files to a CDN, like CloudFront, which serves content from locations closer to the actual users, but for serving css/javascript and small images - a properly configured Nginx is all you need. And you can’t really move any files served from your main domain to a CDN (like HTML content).

You can also be smart about semi-static pages in Rails - you can cache the output inside the public/ directory to be served by Nginx. And if you still want to hit your controller on every request, like when doing A/B Testing on a page, you can send an X-Accel-Redirect header in your response to Nginx and let Nginx to the actual content streaming for you. You can also instruct Nginx to serve files from different locations, based on certain variables like the domain name, thus avoiding hitting the Rails application server on every request.

There’s a lot you can do with Nginx if you’re on a budget, and yet this is not possible within Heroku … which even though it may use Nginx as an http reverse proxy, it certainly doesn’t use it for serving static files. All files are thus served by hitting the Rails server, unless Varnish is involved.

Example 2: Varnish

Varnish is described as being a web application accelerator and the things it can do are truly mind-blowing.

Varnish sits in front of your application servers. It can do load-balancing for you with extreme efficiency, although that’s not its main purpose. Its main purpose is to cache content.

When caching content you have an extreme freedom to specify the Key for fetching cached items. You can use anything when instructing Varnish on what and how to cache, like cookies or the user’s IP or any HTTP header. Do you want to also cache content for logged-in users, even though that content is slightly different from user to user? Not a problem. The configuration language is also extremely flexible, allowing you to tap in the request pipeline with any custom behavior you want. The performance of Varnish coupled with this extreme flexibility is what makes it great. It also has this uncanny ability to reload its configuration without restarting or dropping active connections.

Heroku has Varnish in its stable stack, called Bamboo. But you cannot configure it. The configuration is the same for everybody … you basically set expiry headers on your response, Varnish caches it for you and the cache gets invalidated on every new deployment.

This is actually good and has given rise to the famous Heroku use-case: hosting mostly static websites on it. But Varnish can be much more than that, otherwise it kind of gets in your way, and surprise - Heroku is pulling Varnish out of the configuration, starting with the new Celadon Cedar stack. This is because Varnish gets in the way of their ambitious plans: to make heroku platform-agnostic, thus adding support for Node.js and long-pooling.

The now recommended alternative for serving cached static content is to use Rack::Cache in combination with their Memcached add-on. But this sucks because (1) it hits the Rails server on every request and in the free plan you only have a single process to serve those requests + (2) the free plan for Memcached is only 5MB.

Example 3: asynchronous jobs

One common-sense approach to not having a sluggish web interface is to get slow code out of your HTTP process. Lots of libraries and plugins are available for all web frameworks, like delayed_job for Rails or Celery for Django. And you can just write your own half-baked jobs queue and shove it in your cron.

You cannot have asynchronous jobs using Heroku’s free plan. You must get an extra dyno for that.

Price comparison with Linode

The cheapest Linode instance is $20 per month, and for starters you can have …

1 Nginx server

2 Passenger/Rails processes

1 worker for processing asynchronous jobs, it can even be a plain cron-job ; you do have complete flexibility in configuring cron-jobs

1 PostgreSQL database, configured for 256MB RAM usage, with 18 GB of storage. It’s not much, but it isn’t shared either and does just fine, trust me … btw, the PostgreSQL magazine (first issue) has an article about configuring/optimizing PostgreSQL’s memory usage

1 Postfix email server, for bug reports + sending all the spam you want (Linode lets you configure reverse DNS lookup, so you can have a cheap email server that doesn’t trigger spam alerts)

ability to serve for any domain you want, including wildcard subdomains

your own SSL certificate, for free depending on provider

The equivalent Heroku configuration would cost a minimum of $114 per month.

So lets say that you’re growing and you want to add Ronin, Heroku’s plan for a database of 1.7 GB hot data set (whatever the fuck that means). That will cost you a whooping $200 per month extra, versus $80 for a 2GB of RAM instance on Linode, or even better, $160 for a 4GB of RAM instance.

Linode sucks too, but that’s besides the point

You lose the ability to increase your dynos in response to traffic surges. On the other hand you’ll be amazed at how much you can squeeze out of your rented hardware and if a properly configured setup fails to serve, then the problems you have probably can’t be solved by just adding extra web servers.

Really, do some reading on why Reddit is down so often. Do some reading on why Amazon’s EBS is completely unreliable for databases (btw, Heroku does use EBS and they’ve also had their share of downtime due to AWS experiencing problems).

Stop fearing the penguin and start configuring your own damn servers. As with everything that’s actually worth it in life (like having children of your own), it’s hard at first but the return of investment will be tenfold.

The article Why I Find Heroku Suboptimal first appeared on alexn.org.

Cross-Domain, Cross-Browser AJAX Requests

Alexandru Nedelcu — Thu, 24 Mar 2011 00:00:00 +0000

This article describes how to make cross-browser requests, in all browsers (including IExplorer 6), using web standards along with fallbacks and without using a proxy or JSONP (which is limited and awkward) -- as long as you control the destination server, or if the destination server allows.

I’m explaining this file: crossdomain-ajax.js

Updates

Oct 27, 2011

Added restrictions of usage and removed functionality that doesn’t work on IExplorer. So in case this doesn’t work for you, please see this page: Troubleshooting

In Modern Browsers - Meet Cross-Origin Resource Sharing

Or CORS for short, or HTTP Access Control, available in recent browsers, allows you to make cross-domain HTTP requests; the only requirement being that you must have control over the server-side implementation of the domain targeted in your XMLHttpRequest calls.

This little piece of technology is available since Firefox 3.5 / IExplorer 8 and yet when searching for answers on websites like StackOverflow, it rarely comes up.

For the purposes of this tutorial, we’ll assume we want to make a request from website http://source.com to http://destination.org, and that you control the implementation to both.

var xhr = new XMLHttpRequest(); // NOPE, it doesn't work, yet xhr.open("POST", "http://destination.org", true);

Response of destination.org

It’s pretty simple really, all you need to do is to return these headers in your response:

Access-Control-Allow-Methods: GET, POST, OPTIONS Access-Control-Allow-Credentials: true Access-Control-Allow-Origin: http://source.com Access-Control-Allow-Headers: Content-Type, *

You can find a description of them on Mozilla Doc Center, but the most important one is Access-Control-Allow-Origin, which indicates the Origin(s) allowed to make such a request.

Note: these options allow for wildcards (like you can say that you allow for any Origin by putting a “*” in that header), but it is better to be explicit about what’s allowed, otherwise your request won’t work very well cross-browser.

(New) Note: In regards to Access-Control-Allow-Origin, IExplorer DOES NOT support wildcards. See Troubleshooting for details.

Client-side Implementation of Ajax Request for CORS

On browsers where XMLHttpRequest is valid, support for CORS can be validated by checking for the availability of the withCredentials property.

So we’ve got a tiny issue: IExplorer's implementation is different than that of Firefox’s or the rest of the browsers (naturally). Instead of using the same XMLHttpRequest object, IExplorer 8 adds an XDomainRequest object.

So to initialize an async request, that will work on IExplorer 8, Firefox, Chrome and the other browsers supporting it:

try { var xhr = new XMLHttpRequest(); } catch(e) {} if (xhr && "withCredentials" in xhr){ xhr.open(type, url, true); } else if (typeof XDomainRequest != "undefined"){ xhr = new XDomainRequest(); xhr.open(type, url); } else { xhr = null; }

But we aren’t done yet, the callbacks used by these request objects have different behavior on IExplorer. So let’s say we’ve got 2 callbacks that we want to register, one for success, one for errors, having the following signatures (same as jQuery):

function success(responseText, XHRobj) { ... } function error(XHRobj) { ... }

To have correct behavior cross-browser:

// // combines the success/error handlers into one // higher-order function (getting a little fancy for code-reuse) // var handle_load = function (event_type) { return function (XHRobj) { // // stupid IExplorer won't receive any param on callbacks!!! // thus the object used is the initial `xhr` object // (bound to this function because it's a closure) // var XHRobj = is_iexplorer() ? xhr : XHRobj; // // IExplorer also skips on readyState // Also, it's success/error based on the `event_type` used at the call-site // if (event_type == 'load' && (is_iexplorer() || XHRobj.readyState == 4) && success) success(XHRobj.responseText, XHRobj); else if (error) error(XHRobj); } }; try { // IExplorer throws an exception on this one // // Setting this to `true` is specifying to make the request with Cookies attached. // BUT -- it's pretty useless, as IExplorer doesn't support sending Cookies. // // Also, trying to set cookies from the response is not really possible directly // (workarounds are available though -- you can return anything in the response's // body and use local javascript for persistence/propagation on next request) // xhr.withCredentials = false; } catch(e) {}; // // `onload` + `onerror` are actually new additions to these browsers. // // IExplorer doesn't actually push params on calling these callbacks. // For every other browser, the XHRobj we want is in `e.target`, // where `e` is an event object. // xhr.onload = function (e) { handle_load('load')(is_iexplorer() ? e : e.target) }; xhr.onerror = function (e) { handle_load('error')(is_iexplorer() ? e : e.target) }; xhr.send(data);

Also of notice, here’s how to check if the browser is IExplorer:

function is_iexplorer() { return navigator.userAgent.indexOf('MSIE') !=-1 }

Well, that’s it, unless you want to support the rest of desktop browsers in use.

Fallback for Older Browsers

Opera 10 doesn’t have this feature, neither do IExplorer < 8, Firefox < 3.5 – and I don’t really know when Chrome/Safari added it. Fortunately there’s a workaround – Flash can do whatever you want and runs the same on ~90% of desktop browsers out there, AND it can interact with Javascript.

Not to reinvent the wheel, here’s a cool plugin: flensed.flXHR.

Why bother with CORS?

Flash is not available on the iPhone

Flash loads slower than Javascript

Flash SWF files come with a lot of junk that your browser has to download

The whole experience using flXHR will be visibly slower than with CORS

flXHR Usage

// // Does a request using flXHR (the JS-Flash alternative // implementation for XMLHttpRequest) // function _ajax_with_flxhr(options) { var url = options['url']; var type = options['type'] || 'GET'; var success = options['success']; var error = options['error']; var data = options['data']; // // handles callbacks, just as above // function handle_load(XHRobj) { if (XHRobj.readyState == 4) { if (XHRobj.status == 200 && success) success(XHRobj.responseText, XHRobj); else error(XHRobj); } } var flproxy = new flensed.flXHR({ autoUpdatePlayer: false, instanceId: "myproxy1", xmlResponseText: false, onreadystatechange: handle_load }); flproxy.open(type, url, true); flproxy.send(data); }

We are NOT done. The destination server also needs a file called crossdomain.xml, which represents a Crossdomain Policy File Spec. As a requirement, this file has to be placed in the domain’s root, i.e. http://destination.org/crossdomain.xml …

Not Loading the Junk when Not Needed

Javascript is asynchronous and we should take advantage of that by not loading flensed.flXHR, unless needed and at the last moment too (no need to load it until we want to make a request).

We need a method for asynchronously loading a Javascript file and executing a callback onload. And since we may be executing this function multiple times at once, we need to take care of race-conditions. First things first:

// keeps count of files already included var FILES_INCLUDED = {}; // keeps count of files in the processes of getting loaded // for avoiding race conditions var FILES_LOADING = {}; // stacks of registered callbacks, that will get executed once // a file loads -- this to deal with multiple file inclusions at once, // and not ignoring anything var REGISTERED_CALLBACKS = {}; function register_callback(file, callback) { if (!REGISTERED_CALLBACKS[file]) REGISTERED_CALLBACKS[file] = new Array(); REGISTERED_CALLBACKS[file].push(callback); } function execute_callbacks(file) { while (REGISTERED_CALLBACKS[file].length > 0) { var callback = REGISTERED_CALLBACKS[file].pop(); if (callback) callback(); } }

To asynchronously load a Javascript file, with onload callback, behold:

// // Loads a Javascript file asynchronously, executing a `callback` // if/when file gets loaded. // // Returns `true` if callback got executed immediately, `false` otherwise. // function async_load_javascript(file, callback) { // stores callback in the stack register_callback(file, callback); // dealing with race conditions if (FILES_INCLUDED[file]) { execute_callbacks(file); return true; } if (FILES_LOADING[file]) return false; FILES_LOADING[file] = true; // dynamically adds a

	Eager	Lazy
Synchronous	A	() => A
		Coeval[A], IO[A]
Asynchronous	(A => Unit) => Unit	(A => Unit) => Unit
	Future[A]	Task[A]

Source	Type	Operation	Score	Error	Units
Monix	Sync	Sequence	6716.906	157.947	ops/s
Scalaz	Sync	Sequence	3518.888	167.148	ops/s

Source	Type	Operation	Score	Error	Units
Monix	Forked	Sequence	2044.624	24.852	ops/s
Scalaz	Forked	Sequence	1090.355	15.851	ops/s
S.Future	Forked	Sequence	1753.614	20.871	ops/s

Source	Type	Operation	Score	Error	Units
Monix	Sync	Gather	3800.559	341.509	ops/s
Scalaz	Sync	Gather	2152.441	13.569	ops/s
S.Future	Forked	Sequence	1753.614	20.871	ops/s

Source	Type	Operation	Score	Error	Units
Monix	Sync	Unordered	5654.462	150.792	ops/s
Scalaz	Sync	Unordered	3340.645	244.145	ops/s
S.Future	Forked	Sequence	1753.614	20.871	ops/s