Using distrobox containers for development makes it super easy to separate one or more development setups from the host system. I got used to setting up a container with a different image while still having my user and tools setup as I’m used to. It’s one thing to spin up a container to test out a difference between SUSE Linux Enterprise and Tumbleweed, or see how Ubuntu behaves in a certain case. Having your shell setup as you’re used to reduces disruption in a different environment, and you already have access to the repos you work with, too.

Virtualized container machines

Now let’s take this even further. What if the container host could also be swapped out? Of course virtualization is an option. However you need to install the image by hand and sharing files is at best tolerably slow. I actually used to use VM’s a lot like I use containers these days but setting them up was typically more time-consuming. When it became easier to setup a container with performant files access and easy port forwarding it also became my default. Suppose creating a VM that shares data with the host was as easy as setting up a distrobox container, though… wouldn’t that be tempting?

Building virtual machines on the fly

Lima is a tool that makes it easy to create a VM that feels like you’re still on the VM host, much like distrobox does except rather than podman the backend relies on qemu. Conceptionally it’s similar to WSL2 and also interestingly the backend for Rancher Desktop - just without the Electron GUI. The VM’s are setup based on a pretty straightforward YAML document describing all of the parameters. The easiest option is probably to install via Homebrew:

brew install lima

If you’ve never used Homebrew before be sure to check that the installation prefix is in your $PATH. The installer will also tell you what needs to be configured depending on your environment.

You can also install from a release tarball on GitHub. In this case you need to check that you have qemu dependencies installed.

How about Tumbleweed containers?

There’s three ways to create a new VM depending on one’s needs. Let’s start with the easiest:

limactl start

This will prompt you to create a new instance with the default template. You can confirm without modifying anything and it will get you an instance named Default with a user mirroring your host user.

Your home is mounted into the container as well. Except it’s not set as you user’s $HOME within the VM but $USER.linux. Why is that? Because forwarding all container state keeping may lead to problems that are best avoided. For our purposes that’s just fine since we will be using containers for actual work which hides this detail neatly.

limactl start --log-level debug --name=Quellheim template://experimental/opensuse-tumbleweed

You can also specify a name and one of the existing templates which feature different distros and in some cases tools like k3s or podman. Like before you can go ahead as-is, or maybe at this point try customizing the setup a little bit.

If you ended up making a mistake that rendered your YAML invalid that’s okay. You can still pick up where you left off without editing the template from scratch:

limactl start --name=Quellheim ~/lima.REJECTED.yaml

You will learn about the third way, which is to employ your own fully customized YAML in the next section.

And of course if you’re done with everything you may want to stop the instance:

limactl stop Quellheim

How about that distrobox in a VM?

Now for the use case I’m most interested in. Let’s setup a VM which we can use to run distrobox containers as well as podman. The following example is going to work on MacOS on Apple Silicon, MacOS on Intel and on Linux with few tweaks. It’s also easy to add tools to the VM although I mostly want to keep it minimal and focus on being productive in the containers. Let’s go through the main sections:

vmType: vz
rosetta:
  enabled: true
  binfmt: true
mountType: virtiofs

This is what you want on MacOS on Apple Silicon to enable the hypervisor, Rosetta and virtiofs. It’s still considered experimental upstream but in my experience the performance is quite good. If you’re on Linux or Intel you will want to omit this part.

images:
- location: https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img
  arch: x86_64
- location: https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64.img
  arch: aarch64

mounts:
  - location: "~"
    writable: true

ssh:
  localPort: 0
  forwardAgent: true

provision:
  - mode: system
    script: |
      #!/bin/sh
      export DEBIAN_FRONTEND=noninteractive
      apt-get update
      apt-get install -y flatpak podman python3-pip
      pip3 install podman-compose
      grep -q quay.io /etc/containers/registries.conf && exit 0
      echo '[registries.search]' >> /etc/containers/registries.conf
      echo 'registries = ["registry.opensuse.org", "quay.io", "docker.io"]' >> /etc/containers/registries.conf      
  - mode: system
    script: |
      #!/bin/sh
      [ -f /usr/local/bin/distrobox ] || curl -s https://raw.githubusercontent.com/89luca89/distrobox/main/install | sh      
  - mode: user
    script: |
      #!/bin/sh
      sudo chown $USER:$USER $HOME # Fix "read-only file system" due to root-owned files
      echo 'container_image_default="registry.opensuse.org/opensuse/toolbox:latest"' > $HOME/.distroboxrc
      echo 'container_user_custom_home="/Users/$USER"' >> $HOME/.distroboxrc
      echo 'container_name_default="Gralsund"' >> $HOME/.distroboxrc
      echo 'non_interactive="1"' >> $HOME/.distroboxrc
      echo 'container_generate_entry="1"' >> $HOME/.distroboxrc
      echo 'init="1"' >> $HOME/.distroboxrc      
probes:
  - script: |
      #!/bin/sh
      timeout 30s sh -c "until [ -f /usr/bin/podman ]; do sleep 3; done" || exit 1      
    description: podman installed
  - script: |
      #!/bin/sh
      timeout 30s sh -c "until [ -f /usr/local/bin/distrobox ]; do sleep 3; done" || exit 1      
    description: distrobox installed

containerd:
  system: false
  user: false
portForwards:
  - guestSocket: "/run/user/{{.UID}}/podman/podman.sock"
    hostSocket: "{{.Dir}}/sock/podman.sock"

message: |
  limactl shell {{.Name}} distrobox enter
  ------
  podman system connection add lima-{{.Name}} "unix://{{.Dir}}/sock/podman.sock"
  podman system connection default lima-{{.Name}}
  podman{{if eq .HostOS "linux"}} --remote{{end}} run --rm -it archlinux

What this setup does is the following:

  1. Use an Ubuntu cloud image with SSH forwarding and ~ mounted into your VM
  2. Setup podman and podman-compose
  3. Setup distrobox with convenient defaults

Note that I prefer to install distrobox from the upstream release because it’s newer. Of course you can also choose to use a different version. And although my usual host OS would be openSUSE MicroOS on bare metal, I’ve found it to be less reliable for this use case.

One thing I also like to do for my own convenience is something like

export LIMA_INSTANCE=Quellheim

This way you can use the lima command with the default instance, analoguous to the default distrobox container and lima distrobox enter or lima podman run -rm -it fedora will just work with the default instance.

Can I haz more services in the VM?

If you’d like to use k3s you may want to add two snipets to provision and probes respectively:

  - mode: system
    script: |
      #!/bin/sh
      [ -f /usr/local/bin/k3s ] || curl -sfL https://get.k3s.io | sh
...
  - script: |
      #!/bin/sh
      timeout 30s sh -c "until [ -f /etc/rancher/k3s/k3s.yaml ]; do sleep 3; done" || exit 1
    description: k3s setup

Note that this will increase idle battery usage so keep that in mind. And as you can see it’s pretty straightforward to add other tools to the host. The second part is to check that the setup was successful.

Virtual machines in virtual machines in virtual machines

You can in fact install and run lima within lima[ˆ2]. I’ll leave it as an exercise for you and conclude with a quote:

 - Why is it so important to dream?
 - Because, in my dreams we are together.

Incidentally the names Quellheim and Gralsund are two of several towns in the world of Zamonia. It just so happens to be one of my favorite series of novels and is usually how I name machines, VM’s and containers.

Now go and customize your new VM setup and create many lovely containers!

[ˆ2]: You will need to ensure your home mount is not $USER.linux.