Livebook: Elixir's Swiss Army Knife

The Problem of Adoption Friction

Recently I gave a talk about my first year programming with Elixir and my experience contributing to Elixir's open-source ecosystem. In it, I made some points about how a big part of encouraging language adoption is to reduce friction.

One common point of friction that many languages suffer from is fractured tooling ecosystems. These are language ecosystems where the language is unopinionated about things such as its formatting, linting, testing, build system, etc. These can be good for advanced users who have the benefit of experience when deciding which tools to choose, giving them more options depending on their needs. This can, however, just as easily leave beginners wary to even start. Many people credit Go with popularizing the practice of providing developers with standardized libraries and tooling for all parts of the development lifecycle. Providing code formatters, package managers, package repositories, LSPs, unit testing libraries, etc. straight from the language's core team can drastically reduce adoption friction by preventing the phenomenon of analysis paralysis, where people can get too overwhelmed by the bevy of tooling options that they never get around to actually writing code.

Elixir excels in this respect, providing tools such as ExDoc (for documentation), ExUnit (for testing), Mix (build tool), Hex (package manager), Mix format (for formatting), and more. When writing code, the only decisions left to make (for the most part) are design and implementation details for your code, rather than needing to concern yourself with these details that can (and have been) standardized.

Understanding Your Audience

In that same talk, I also spoke about the idea of "entry-points" to the language and how targeting them can help reduce adoption friction, especially in a world where most people are first introduced to other programming paradigms such as object-oriented as opposed to functional programming like Elixir. Here are some of the most common points of entry I came up with for Elixir.

Understanding the perspective of your potential adopters is crucial in focusing where developer efforts should go, and in general a good first impression can do a lot to convince people to stay.

First impressions can be made in all sorts of ways, from good landing pages, to detailed documentation, to convincing testimonials and showcases, but one common way is the place where many people first interact with the language: its REPL (or interactive session). In Elixir, this is IEx.

Many people prefer, above reading documentation or reading source code, getting their hands dirty by immediately interacting with the language. They prefer the hands-on approach to learning the language, which is a large reason why more and more languages are moving to code playgrounds as their getting started guide. They want to get immediate feedback and iterate on quick ideas to gauge whether the language is worth investing more time into. There is only so much you can do from a REPL, so when they've reached the limits of exploration through that means, they might start working on a small script of sorts, but depending on which entry-point they came from, the next natural step might be quite different.

Wouldn't it be great if there was a place that served as a nexus, where someone coming from the embedded world could be introduced to the language in much the same way as someone coming from the Machine Learning World, while still leaning into those very domains that brought them there?

Enter Livebook, the One-Stop-Shop

Livebook is an interactive Notebook-style Elixir application that lets you write reproducible, shareable, deployable, extensible, interactive, and integrated Elixir workflows (and even has support for Erlang!). With these characteristics in mind, Livebook is fast becoming the premier gateway application to introduce newcomers to all that the language has to offer. Dockyard has even gone as far as writing all of the curriculum for Dockyard Academy as Elixir Livebooks.

For the last couple of months, I've been spending much of my time in Livebook, and have found it to really enhance my workflow when working on my libraries. I prepared much of my ElixirConfUS 2023 talk as a Livebook (also published as a blog post here), as well as wrote the documentation for the EXGBoost Plotting Module as a Livebook, in addition to working on several interactive libraries for Livebook (which I will touch on later).

It's very quick to get started with Livebook as it has standalone Mac and Windows installers that bundle in Erlang/OTP and Elixir, so you can get started without any prerequisites. You also have the options to use Livebook Teams to gain collaboration features or run instances of Livebook in the cloud, so you don't have to worry about managing installs!

Livebook is available in three different runtime options: standalone if you want to use Livebook as its own instance of Elixir, attached if you want Livebook to connect to an existing instance of Elixir, or embedded if you want to install Livebook on an embedded device (see the Nerves Project).

Livebook can be viewed as a next-generation, "supercharged IEx," and is still getting better. Let's go into some more detail regarding some of the earlier points I mentioned to show the power and versatility of Livebook.

Reproducible

Livebooks fundamentally consists of a series of code cells (or other cells such as markdown) which are executed in the order they appear (top-down). Working within the immutable nature of Elixir, each cell receives the output state of the previous cell as its input state, and passes its output state to the next cell, meaning that whenever a cell is run, all previous cells will run first since it depends on their collective state. This ensures reproducibility of workflows. Livebook also has the concept of "Sections" and "Forks", where you can Fork from a section which means that the initial state of the current section relies on the final state of the forked section. This means that Livebook cell execution is represented as a tree, which enables very flexible workflows.

This is fundamentally different from other notebook execution environments such as Jupyter Notebooks, which does not enforce that cells must be executed in order and allows mutable changes to cells such that executing the same cell multiple times can have compounding effects on the state of the notebook.

Shareable

One of my favorite decisions that the Livebook team was making a Livebook's .livemd filetype perfectly valid Markdown, such that you can share Livebooks anywhere that Markdown can be shared. I've written several blog posts entirely within Livebook then simply copied the .livemd Markdown into a blog post as Markdown (see here for an example).

This is in stark contrast to the format used by Jupyter Notebooks, which uses a custom JSON schema to represent its notebooks, so if you want to share the notebook you either have to export to another format (such as HTML, or PDF) or rely on the hosting service to provide rendering (e.g. GitHub supports rendering .ipynb files).

Meta information is stored as Markdown comments, so the Livebook application itself can render additional details. When exporting a Livebook to Markdown, you also have the option of whether to include the output of the cells, which again, is also just valid Markdown!

Lastly, Livebook offers a "Run in Livebook" badge generator that you can place in the exported Markdown, and the badge will allow a one-click import of the Livebook. You can import Livebooks this way, or many other ways such as from local or cloud storage, from a URL, or from source.

Extensible

Out of the box, Livebook comes outfitted with several great integrations such as Smart Cells for Slack, Databases (PostgreSQL / MySQL Microsoft SQL Server, and more), Plotting via VegaLite, and more. You can check out the full list of built-in integrations here.

What's great about these integrations is that most of them are built on top of the Kino library, and are open-source! Anyone has the ability to contribute their own extensions using Kino and its various abstractions, such as Kino.JS, Kino.JS.Live, and Kino.SmartCell. These abstractions are built on top of well-established Elixir concepts such as GenServers, meaning that you will either already be familiar with the abstraction, or by learning the abstraction you will gain useful experience for use in other Elixir projects. Writing custom Kinos is a great way to dip your toes into writing Client/Server behaviours in Elixir!

I won't go into too much detail about the behaviours defined in Kino in this article, but if you're interested in my preliminary thoughts you can refer to this post where I spoke about it a bit:

One example of this extensibility comes from Thomas Millar, who talks about how he wanted the ability to have Livebook hot-reloading where as soon as he changed a file that the Livebook depends on it re-executes the cell, so he went ahead and built it and shared it with the community!

I've also built out a few Kinos for different use cases that I came across and figured I would share.

Here's a simple Kino I built to embed YouTube videos using the YouTube iframe API:

GitHub - acalejos/kino_youtube: A simple Kino that wraps the YouTube Embedded iFrame API to render a YouTube player in a Livebook.

A simple Kino that wraps the YouTube Embedded iFrame API to render a YouTube player in a Livebook. - acalejos/kino_youtube

GitHubacalejos

Kino has a set of built-in components such as Kino.Control.Button, Kino.Control.Form, and even Kino.Input.Audio. I was wanting to use the audio input to emit event live during a recording for a project I was working on, but found that the Kino.Input.Audio component only allowed listening to the upload event for the audio recording as a whole audio file, so you couldn't receive the event with the audio data until the audio finished recording. I decided to write my own Kino for live audio, and published it along with an app showcasing it.

GitHub - acalejos/kino_live_audio: A Kino designed to record a raw audio stream (no client-side encoding) and emit events.

A Kino designed to record a raw audio stream (no client-side encoding) and emit events. - acalejos/kino_live_audio

GitHubacalejos

Interactive

As I showed above, you can add interactive experiences to Livebook using the Kino library. These interactions can be powered by the three major behaviours provided by the Kino library: Kino.JS, Kino.JS.Live, and Kino.SmartCell.

Here's the example I mentioned showcasing the kino_live_audio library I wrote and how to use it to make a live Voice Activity Detection app in Livebook!

There are built-in interactive Kinos such as the kino_bumblebee library, which provides a GUI to the Elixir Bumblebee library.

There's also support for the VegaLite JavaScript library (via Elixir bindings) to build interactive plots and graphics. You can read more about the Vega-Lite capabilities within Livebook here and you can see some plotting with Vega in action in my post about Plotting in XGBoost, which itself is just a Livebook with Vega plots of decision trees from XGBoost. You can see the .livemd file here.

Integrated

As I mentioned before, Livebook has three runtime options, including an option to attach an instance of Livebook to a running Elixir node. This options provides a ton of flexibility and opportunity to not only use Livebook as the first step in prototyping, but to use it seamlessly during development and even production.

For example, you could use Livebook to create dashboard of live-running production systems or otherwise introspect into the state of the system.

Here's an example of Thomas Millar using Livebook to integrate with the Wallaby library to ease the process of creating automated browser test code.

Here's an example of using my Merquery SmartCell to automatically create Postman-like cells to test a Phoenix application's routes with the ability to pre-populate parameter information as well.

Since Livebook is either attached to a running Elixir node, or is a standalone node itself, you can interact with it like you would with any other node on an cluster in Elixir!

Deployable

Livebook isn't even necessarily just for prototyping either! Livebook apps can be deployed either locally or into the cloud (first-class support for HuggingFace Spaces). There are many options when deploying Livebook apps, including support for multi-session apps (apps where each new connection spawns a new instance of the app) and password-protected apps. You can choose whether or not to expose the underlying source code to the users, meaning that you can hide proprietary implementation details if you want, or use let others learn from your implementation.

You also have the option to only include rich output (e.g. the interactive cells I spoke about earlier) in the deployed app, meaning you can create very rich and complex apps and control what gets shown in the deployed app.

Here are some examples of deployed Livebook applications, and you can see examples of all I just discussed.

Livebook

Livebook - a Hugging Face Space by acalejos

Discover amazing ML apps made by the community

a Hugging Face Space by acalejos

Bringing It All Together

Let's look back at the common Elixir entrypoints I mentioned at the top of this article:

I just showed how Livebook has a ton to offer regardless of which of these points of entry you're coming from. You can get exposed to backend development since you can write any Elixir code. Kinos are generally implemented as GenServers, which is a core pattern to become familiar with when writing backend Elixir code. You can also use Livebook as one node in a cluster and practice distributed Elixir.

Livebook itself is a Pheonix LiveView app, so if nothing else, you can refer to the source code for good Phoenix code practices or write Kinos for practice with frontend (albeit, not necessarily with Phoenix).

Livebook is great for machine learning due to the ability to perform exploratory data analysis with libraries such as Explorer / Kino Explorer, create new workflows like I showed before with Serving Spam Detection with XGBoost and Elixir, work with SmartCells (like kino_bumblebee) to learn and generate code snippets, and much more!

Lastly, Livebook even works great for folks working in embedded projects since it supports an embedded runtime. In fact, the Nerves Project supports a firmware image that comes pre-loaded with Livebook.

Conclusion

I hope I convinced you to not only give Livebook a try, but that it should have a permanent spot in your developer toolkit. Whenever I suggest how to get started with Elixir at this point I always point to Livebook. It has so much to offer and can seamlessly grow with you as a developer. When you're just learning the language it can simply serve as a learning tool, but it has the ability to scale with you and eventually you can use it for production use-cases.