Stéphane Graber: Incus in 2024 and beyond!
<p></p>
<figure class="wp-block-image size-full"><a href="https://stgraber.org/wp-content/uploads/2025/01/image.png"><img alt="" class="wp-image-1657" height="412" src="https://stgraber.org/wp-content/uploads/2025/01/image.png" width="874" /></a></figure>
<p>A lot has happened in 2024 for the <a href="https://linuxcontainers.org/incus/">Incus</a> project, so I thought it’d be interesting to see where we started, what we did and where we ended up after that very busy year, then look forward to what’s next in 2025!</p>
<h1 class="wp-block-heading">Where we started</h1>
<p>We began 2024 right on the heels of the Incus 0.4 release at the end of December 2023.</p>
<p>This is notable as Incus 0.4 was the last Incus release that could directly import changes from the LXD project due to <a href="https://stgraber.org/2023/12/12/lxd-now-re-licensed-and-under-a-cla/">Canonical’s decision to re-license LXD as AGPLv3</a>.</p>
<p>This means that effectively everything that made it into Incus in 2024 originated directly from the Incus community. There is one small exception to that as LXD 5.0 LTS still saw some activity and as that’s still under Apache 2.0, we were able to import a few commits (83 to be exact) from that branch.</p>
<h1 class="wp-block-heading">What we did</h1>
<h2 class="wp-block-heading">Some numbers</h2>
<ul class="wp-block-list">
<li>Releases
<ul class="wp-block-list">
<li>12 feature releases (monthly cadence)</li>
<li>1 LTS release (6.0.0)</li>
<li>3 LTS bugfix releases (6.0.1, 6.0.2, 6.0.3)</li>
</ul>
</li>
<li>Changes
<ul class="wp-block-list">
<li>2317 commits</li>
<li>751 pull requests</li>
<li>124 individual contributors</li>
</ul>
</li>
<li>110394 people tried <a href="https://stgraber.org/category/planet-ubuntu/feed/">Incus online</a></li>
<li>7375 posts were published on <a href="https://discuss.linuxcontainers.org">our forum</a></li>
<li>396 <a href="https://github.com/lxc/incus/issues">Github issues</a> were closed</li>
<li>4700 hours of <a href="https://www.youtube.com/@TheZabbly">Incus videos</a> were watched on Youtube</li>
<li>670194 container and VM <a href="https://images.linuxcontainers.org">images</a> downloaded</li>
</ul>
<h2 class="wp-block-heading">Our first LTS release</h2>
<p>Incus 6.0 LTS was released at the beginning of April, alongside LXC and LXCFS 6.0 LTS.<br />All of which get 5 years of security support.</p>
<p>That was a huge milestone for Incus as it now allowed production users who don’t feel like going through an update cycle every month to switch over to Incus 6.0 LTS and have a stable production release for the years to come.</p>
<p>It also provides a much easier packaging target for Linux distributions as the monthly releases can be tricky to follow, especially when they introduce new dependencies.</p>
<p>Today, Incus 6.0 LTS represents around 50% of the Incus user base.</p>
<h2 class="wp-block-heading">Notable feature additions</h2>
<p>It’s difficult to come up with a list of the most notable new features because so much happened all over the place and deciding what’s notable ends up being very personal and subjective, depending on one’s usage of Incus, but here are a few!</p>
<ul class="wp-block-list">
<li>Application container support (OCI), gives us the ability to natively run Docker containers on Incus</li>
<li>Clustered LVM storage backend, adds support for iSCSI/NVMEoTCP/FC storage in clusters</li>
<li>Network integrations (OVN inter-connect), allows for cross-cluster overlay networking</li>
<li>Automatic cluster re-balancing, simplifies operation of large clusters</li>
</ul>
<h2 class="wp-block-heading">Performance improvements</h2>
<p>As more and more users run very large Incus systems, a number of performance issues were noticed and have been fixed.</p>
<p>An early one was related to how Incus handled OVN. The old implementation relied on the OVN command line tools to drive OVN database changes. This is incredibly inefficient as each call to those tools would require new TLS handshakes with all database servers, tracking down the leader, fetching a new copy of the database, performing a trivial operation and exiting. The new implementation uses a native database client directly in Incus which maintains a constant connection with the database, gets notified of changes and can instantly perform any needed configuration changes.</p>
<p>Then there were 2-3 different cases of database performance issues.<br />Two of them were caused by our auto-generated database helpers which weren’t very smart about handling of profiles, effectively causing a situation where performance would get exponentially worse as more profiles would be present in the database. Addressing this issue resulted in dramatic performance improvement for users operating with hundreds or even thousands of profiles.<br /><br />Another was related to loading of instances on Incus startup, specifically loading the device definitions to check whether anything needed to be done on startup. This logic was always hitting configuration validation which can be costly, in this case, so costly that Incus would fail to startup during the allotted time by the init system (10 minutes). After some fixes to that logic, the affected system, running over 2000 virtual machines (on a single server) at the time, is now able to process all running VMs in just 10-15s.</p>
<p>On top of those issues, special attention was also put in optimizing resource usage on large systems, especially systems with multiple NUMA nodes, supporting basic NUMA balancing of virtual machines as well as selecting the best GPU devices based on NUMA cost.</p>
<h2 class="wp-block-heading">Distribution integration</h2>
<p>Back at the beginning of 2024, Incus was only available through my own packages for Debian or Ubuntu, or through native packages on Gentoo and NixOS.</p>
<p>This has changed considerably through 2024 with Incus now being <a href="https://linuxcontainers.org/incus/docs/main/installing/">readily available</a> on:</p>
<ul class="wp-block-list">
<li>Alpine Linux</li>
<li>Arch Linux</li>
<li>Chimera Linux</li>
<li>Debian</li>
<li>Fedora</li>
<li>Gentoo</li>
<li>NixOS</li>
<li>openSUSE</li>
<li>Rocky Linux</li>
<li>Ubuntu</li>
<li>Void Linux</li>
</ul>
<p>Additionally, it’s also available as a Docker container to run on most any other platforms as well as available on MacOS through Colima. The client tool itself is available everywhere that Go supports.</p>
<h2 class="wp-block-heading">Deployment tooling</h2>
<h3 class="wp-block-heading">Terraform/OpenTofu provider</h3>
<p>The Incus Terraform/OpenTofu <a href="https://github.com/lxc/terraform-provider-incus">provider</a> has seen quite a lot of activity this year.</p>
<p>We’re slowly headed towards a 1.0 release for it, basically ensuring that it can drive every single Incus feature and that its resources are defined in a clear and consistent way.</p>
<p>There is only one issue left in the 1.0 release milestone and there is an open pull request for it, so we are very close to where we want as far as feature coverage and with a few more bugfixes here and there, we should have that 1.0 release out in the coming weeks/month!</p>
<h3 class="wp-block-heading">incus-deploy</h3>
<p><a href="https://github.com/lxc/incus-deploy">incus-deploy</a> was introduced in February and is basically a collection of Ansible and Terraform that allows for easy deployment of Incus, whether standalone or clustered and whether for testing/development or production.</p>
<p>This is commonly used by the Incus team to quickly deploy test clusters, complete with Ceph, OVN, clustered LVM, … all in a very reproducible way.</p>
<h3 class="wp-block-heading">Incus OS</h3>
<p>While incus-deploy provides an automated way to deploy Incus on top of traditional Linux servers, <a href="https://github.com/lxc/incus-os">Incus OS</a> is working on providing a solution for those who don’t want to have to deal with maintaining traditional Linux servers.</p>
<p>This is a fully immutable OS image, kept as minimal as possible and solely focused on running Incus.</p>
<p>It heavily relies on systemd tooling to provide a secure environment, starting from SecureBoot signing, to having every step of the boot be TPM measured, to having storage encrypted using that TPM state and the entire read-only disk image being verified through dm-verity.</p>
<p>The end result is an extremely secure and locked down environment which is designed for just one thing, running Incus!</p>
<p>We’re getting close to having something ready for early adopters with automated builds and update logic now working, but it will be a few more weeks before it’s safe/useful to install on a server.</p>
<h1 class="wp-block-heading">Where we ended up</h1>
<p>Over that year, Incus really turned into a full fledged Open Source project and community.</p>
<p>We have kept on with our release cadence, pushing out a new feature release every month while very actively backporting bugfixes and smaller improvements to our LTS release.</p>
<p>Distributions have done a great job at getting Incus packaged, making it natively available just about everywhere (we’re still waiting on solid EPEL packaging).</p>
<p>Our supporting projects like terraform-provider-incus, incus-deploy and incus-os are making it easier than ever to deploy and operate large scale Incus clusters as well as providing a simpler, more repeatable way of running Incus.</p>
<p>2024 was a very very good year for Incus!</p>
<h1 class="wp-block-heading">What’s coming in 2025</h1>
<p>Looking ahead, 2025 has the potential to be and even better year for us!</p>
<p>On the Incus front, there are no single huge feature to be looking forward to, but just the continual improvement, whether it be for containers, VMs, networking or clustering. We have a lot of small new features and polishing in mind which will help fill in some of the current gaps and provide a nice and consistent experience.</p>
<p>But it’s on the supporting projects that a lot of the potential now rests.</p>
<p>This will hopefully be the year of Incus OS, making installing Incus as easy as writing a file to a USB stick, booting a machine from it and accessing it over the network. Want to make a cluster, no problem, just boot a few more machines onto Incus OS and join them together as a cluster!</p>
<p>But we’re also going to be expanding incus-deploy. It’s currently doing a good job at deploying Incus on Ubuntu servers with Ansible but we want to expand that to also cover Debian and some of the RHEL derivatives so we can cover the majority of our current production users with it. On top of that, we want to also have incus-deploy handle setting up the common support services used by Incus clusters, typically OpenFGA, Keycloak, Grafana, Prometheus and Loki.</p>
<p>We also want to improve our testing and development lab, add more systems, add the ability to test on more architectures and easily test more complex features, whether it’s 100Gb/s+ networking with full hardware offload or confidential computing features like AMD SEV.</p>
<h2 class="wp-block-heading has-text-align-center"><strong>Sovereign Tech Fund</strong></h2>
<p>Thankfully a lot of that is going to be made a whole lot easier thanks to funding by the <a href="https://www.sovereign.tech/">Sovereign Tech Fund</a> who’s going to be supporting a variety of Incus related projects, especially focusing on the kind of work that’s not particularly exciting but is very much critical to the proper running of a project like ours.</p>
<p>This includes a big refresh of our testing and development lab, work on our LTS releases, new security features through the stack, improved support for other Linux distributions and OSes across our projects and more!</p>
<p>I for one am very excited about 2025!</p>
