<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
    <id>https://www.mirkolenz.com/</id>
    <title>Mirko Lenz</title>
    <updated>2026-06-04T00:00:00.000Z</updated>
    <generator>https://github.com/jpmonette/feed</generator>
    <author>
        <name>Mirko Lenz</name>
        <uri>https://www.mirkolenz.com/</uri>
    </author>
    <link rel="alternate" href="https://www.mirkolenz.com/"/>
    <link rel="self" href="https://www.mirkolenz.com/en/atom.xml"/>
    <subtitle>Posts about AI, Nix, and software engineering</subtitle>
    <logo>https://www.mirkolenz.com/assets/avatar.CHOyV6Mm_Z1q0fHg.png</logo>
    <icon>https://www.mirkolenz.com/assets/avatar.CHOyV6Mm_Z1q0fHg.png</icon>
    <rights>All rights reserved 2026, Mirko Lenz</rights>
    <entry>
        <title type="html"><![CDATA[Serving and routing local LLMs with LLMhop]]></title>
        <id>https://www.mirkolenz.com/en/posts/llmhop/</id>
        <link href="https://www.mirkolenz.com/en/posts/llmhop/"/>
        <updated>2026-06-04T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[A tiny OpenAI-compatible router paired with a NixOS module that runs the inference servers for you.]]></summary>
        <content type="html"><![CDATA[<p>High-performance inference servers like <a href="https://github.com/vllm-project/vllm">vLLM</a> and <a href="https://github.com/sgl-project/sglang">sglang</a> are built around a simple assumption: one model per process.
Each worker loads a single set of weights, binds to its own port, and exposes an OpenAI-compatible API.
This is great for throughput, but it leaves you with a fleet of disconnected endpoints the moment you want to serve more than one model on a machine.
<a href="https://github.com/mirkolenz/llmhop">LLMhop</a> closes that gap from two sides at once:
A tiny router that puts a single endpoint in front of those workers, and a NixOS module that can provision the workers themselves.</p>
<h2 id="the-router">The router</h2>
<p>At its core, LLMhop is a stateless single Go binary with zero third-party dependencies and no CGO.
It reads the <code>model</code> field of an incoming OpenAI-compatible request, looks it up in its config, and reverse-proxies the request verbatim to the matching backend.
Because it keeps no state, there is no database, no cache, and no background worker, so it is safe to place behind any load balancer.
A minimal <code>config.json</code> maps model names to upstream URLs:</p>
<pre class="astro-code astro-code-themes gruvbox-light-hard gruvbox-dark-hard" style="background-color:#f9f5d7;--shiki-dark-bg:#1d2021;color:#3c3836;--shiki-dark:#ebdbb2;overflow-x:auto" tabindex="0" data-language="json"><code><span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">{</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">  &quot;</span><span style="color:#689D6A;--shiki-dark:#689D6A">listen</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span><span style="color:#7C6F64;--shiki-dark:#A89984">:</span><span style="color:#7C6F64;--shiki-dark:#A89984"> &quot;</span><span style="color:#076678;--shiki-dark:#83A598">:8080</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span><span style="color:#7C6F64;--shiki-dark:#A89984">,</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">  &quot;</span><span style="color:#689D6A;--shiki-dark:#689D6A">models</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span><span style="color:#7C6F64;--shiki-dark:#A89984">:</span><span style="color:#7C6F64;--shiki-dark:#A89984"> {</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">    &quot;</span><span style="color:#689D6A;--shiki-dark:#689D6A">qwen3-8b</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span><span style="color:#7C6F64;--shiki-dark:#A89984">:</span><span style="color:#7C6F64;--shiki-dark:#A89984"> {</span><span style="color:#7C6F64;--shiki-dark:#A89984"> &quot;</span><span style="color:#689D6A;--shiki-dark:#689D6A">url</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span><span style="color:#7C6F64;--shiki-dark:#A89984">:</span><span style="color:#7C6F64;--shiki-dark:#A89984"> &quot;</span><span style="color:#076678;--shiki-dark:#83A598">http://localhost:18001</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span><span style="color:#7C6F64;--shiki-dark:#A89984"> },</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">    &quot;</span><span style="color:#689D6A;--shiki-dark:#689D6A">openai-gpt-4o</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span><span style="color:#7C6F64;--shiki-dark:#A89984">:</span><span style="color:#7C6F64;--shiki-dark:#A89984"> {</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">      &quot;</span><span style="color:#689D6A;--shiki-dark:#689D6A">url</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span><span style="color:#7C6F64;--shiki-dark:#A89984">:</span><span style="color:#7C6F64;--shiki-dark:#A89984"> &quot;</span><span style="color:#076678;--shiki-dark:#83A598">https://api.openai.com</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span><span style="color:#7C6F64;--shiki-dark:#A89984">,</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">      &quot;</span><span style="color:#689D6A;--shiki-dark:#689D6A">headers</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span><span style="color:#7C6F64;--shiki-dark:#A89984">:</span><span style="color:#7C6F64;--shiki-dark:#A89984"> {</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">        &quot;</span><span style="color:#689D6A;--shiki-dark:#689D6A">Authorization</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span><span style="color:#7C6F64;--shiki-dark:#A89984">:</span><span style="color:#7C6F64;--shiki-dark:#A89984"> &quot;</span><span style="color:#076678;--shiki-dark:#83A598">Bearer ${env:OPENAI_KEY}</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">      }</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">    }</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">  }</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">}</span></span></code></pre>
<p>Two optional features solve common authentication use cases.
First, incoming requests can be gated with a list of bearer tokens, which are compared in constant time and stripped before forwarding, so the client-facing token never leaks upstream.
Second, per-model headers can inject an <code>Authorization</code> (or any other) header when forwarding, which lets you consolidate self-hosted workers and hosted providers like OpenAI behind the same endpoint.
Secrets never have to live in the config file in plaintext: string values are expanded at startup from <code>${env:NAME}</code> or <code>${file:path}</code> references, with the latter resolving against systemd’s <code>$CREDENTIALS_DIRECTORY</code> when present.</p>
<h2 id="the-nixos-module">The NixOS module</h2>
<p>The router alone already removes a lot of glue code, but wiring up the inference servers by hand is still tedious.
The NixOS module therefore goes one step further and can run the workers themselves.
You declare your models once, and each entry becomes an isolated worker bound to a loopback port, with the matching route registered with the router automatically.</p>
<pre class="astro-code astro-code-themes gruvbox-light-hard gruvbox-dark-hard" style="background-color:#f9f5d7;--shiki-dark-bg:#1d2021;color:#3c3836;--shiki-dark:#ebdbb2;overflow-x:auto" tabindex="0" data-language="nix"><code><span class="line"><span style="color:#076678;--shiki-dark:#83A598">services</span><span style="color:#427B58;--shiki-dark:#8EC07C">.</span><span style="color:#076678;--shiki-dark:#83A598">llmhop</span><span style="color:#CC241D;--shiki-dark:#CC241D"> =</span><span style="color:#7C6F64;--shiki-dark:#A89984"> {</span></span>
<span class="line"><span style="color:#B57614;--shiki-dark:#FABD2F">  enable</span><span style="color:#427B58;--shiki-dark:#8EC07C"> =</span><span style="color:#8F3F71;--shiki-dark:#D3869B"> true</span><span style="color:#7C6F64;--shiki-dark:#A89984">;</span></span>
<span class="line"><span style="color:#B57614;--shiki-dark:#FABD2F">  llama-cpp</span><span style="color:#3C3836;--shiki-dark:#EBDBB2">.</span><span style="color:#B57614;--shiki-dark:#FABD2F">models</span><span style="color:#3C3836;--shiki-dark:#EBDBB2">.</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span><span style="color:#79740E;--shiki-dark:#B8BB26">qwen3-8b</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span><span style="color:#427B58;--shiki-dark:#8EC07C"> =</span><span style="color:#7C6F64;--shiki-dark:#A89984"> {</span></span>
<span class="line"><span style="color:#B57614;--shiki-dark:#FABD2F">    port</span><span style="color:#427B58;--shiki-dark:#8EC07C"> =</span><span style="color:#8F3F71;--shiki-dark:#D3869B"> 18001</span><span style="color:#7C6F64;--shiki-dark:#A89984">;</span></span>
<span class="line"><span style="color:#B57614;--shiki-dark:#FABD2F">    settings</span><span style="color:#3C3836;--shiki-dark:#EBDBB2">.</span><span style="color:#B57614;--shiki-dark:#FABD2F">hf-repo</span><span style="color:#427B58;--shiki-dark:#8EC07C"> =</span><span style="color:#7C6F64;--shiki-dark:#A89984"> &quot;</span><span style="color:#79740E;--shiki-dark:#B8BB26">unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span><span style="color:#7C6F64;--shiki-dark:#A89984">;</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">  };</span></span>
<span class="line"><span style="color:#B57614;--shiki-dark:#FABD2F">  vllm</span><span style="color:#3C3836;--shiki-dark:#EBDBB2">.</span><span style="color:#B57614;--shiki-dark:#FABD2F">models</span><span style="color:#3C3836;--shiki-dark:#EBDBB2">.</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span><span style="color:#79740E;--shiki-dark:#B8BB26">llama-3-8b</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span><span style="color:#427B58;--shiki-dark:#8EC07C"> =</span><span style="color:#7C6F64;--shiki-dark:#A89984"> {</span></span>
<span class="line"><span style="color:#B57614;--shiki-dark:#FABD2F">    port</span><span style="color:#427B58;--shiki-dark:#8EC07C"> =</span><span style="color:#8F3F71;--shiki-dark:#D3869B"> 20001</span><span style="color:#7C6F64;--shiki-dark:#A89984">;</span></span>
<span class="line"><span style="color:#B57614;--shiki-dark:#FABD2F">    model</span><span style="color:#427B58;--shiki-dark:#8EC07C"> =</span><span style="color:#7C6F64;--shiki-dark:#A89984"> &quot;</span><span style="color:#79740E;--shiki-dark:#B8BB26">meta-llama/Meta-Llama-3-8B-Instruct</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span><span style="color:#7C6F64;--shiki-dark:#A89984">;</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">  };</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">}</span><span style="color:#CC241D;--shiki-dark:#CC241D">;</span></span></code></pre>
<p>Three backends are supported and can be enabled side by side: llama.cpp, sglang, and vLLM.
The choice of how to run each one reflects how it is best deployed.
llama.cpp runs as a native, hardened systemd system unit under <code>DynamicUser</code>.
sglang and vLLM are launched as rootless Podman containers through <a href="https://github.com/mirkolenz/quadlet-nix">quadlet-nix</a>, each owned by a dedicated lingering system user that holds its own cache directory and rootless container store.
No worker ever runs as root, and the router itself runs under the same aggressive sandboxing as the llama.cpp units.</p>
<p>Running the GPU backends as systemd user units rather than system units is a deliberate workaround.
When systemd’s system manager launches a rootless container, Podman ends up in a UID-mapped namespace, and <code>nvidia-cdi-hook</code> then fails to read the OCI bundle’s config, so the GPU never gets exposed.
Running each Quadlet unit under a real, lingering user’s systemd instance keeps Podman in a <code>keep-id</code>-style mapping where the CDI hook works and the GPU is passed through correctly.</p>
<p>sglang already ships its own router, the SGL Model Gateway, so it is worth being clear about why LLMhop exists alongside it.
That gateway speaks sglang’s own API rather than a generic OpenAI surface:
It probes each worker through sglang-specific endpoints like <code>/get_model_info</code>, and it only adds value when you actually want sglang’s IGW dispatch features such as custom routing or prefix caching across workers.
It is also a much heavier component, a full Rust service running as its own container, where LLMhop is a single dependency-free Go binary that fronts llama.cpp, vLLM, and hosted providers just as happily as sglang.
For maximum flexibility, the module does not force the choice: the SGL Model Gateway can be enabled with a single option and placed in front of the sglang workers, while LLMhop keeps routing across every backend.</p>
<p>The result is that the whole stack, from pulling weights to the public endpoint, lives in one declarative configuration.
The full list of per-backend options is available in the <a href="https://mirkolenz.github.io/llmhop/">reference</a>, and feedback on the project is always welcome.</p>]]></content>
        <author>
            <name>Mirko Lenz</name>
        </author>
        <category label="nix"/>
        <category label="llm"/>
        <published>2026-06-04T00:00:00.000Z</published>
    </entry>
    <entry>
        <title type="html"><![CDATA[Packaging llmster with Nix]]></title>
        <id>https://www.mirkolenz.com/en/posts/llmster-nix/</id>
        <link href="https://www.mirkolenz.com/en/posts/llmster-nix/"/>
        <updated>2026-02-04T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Insights into packaging the headless LM Studio server using Nix for reliable deployment.]]></summary>
        <content type="html"><![CDATA[<p>As part of <a href="https://lmstudio.ai/blog/0.4.0">LM Studio 0.4</a>, a new headless server called <code>llmster</code> has been released, intended to be run on servers without GUIs.
Just like many tools distributed as binary releases, <code>llmster</code> is intended to be installed through a <a href="https://lmstudio.ai/install.sh">bash script</a>:</p>
<pre class="astro-code astro-code-themes gruvbox-light-hard gruvbox-dark-hard" style="background-color:#f9f5d7;--shiki-dark-bg:#1d2021;color:#3c3836;--shiki-dark:#ebdbb2;overflow-x:auto" tabindex="0" data-language="shellsession"><code><span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">$</span><span style="color:#3C3836;--shiki-dark:#EBDBB2"> curl -fsSL https://lmstudio.ai/install.sh </span><span style="color:#427B58;--shiki-dark:#8EC07C">|</span><span style="color:#B57614;--shiki-dark:#FABD2F"> bash</span></span></code></pre>
<p>People using NixOS or the Nix package manager cannot easily use such installation methods–for instance, due to missing dynamic linking of libraries.
Besides this installation script, little information is provided about the <code>llmster</code> binary; only some instructions on how to create a <a href="https://lmstudio.ai/docs/developer/core/headless_llmster">systemd service</a> are available.
Inspecting the installation script, we observe that it downloads a tarball as well as sha512 checksums from an S3 bucket with the following content:</p>
<pre class="astro-code astro-code-themes gruvbox-light-hard gruvbox-dark-hard" style="background-color:#f9f5d7;--shiki-dark-bg:#1d2021;color:#3c3836;--shiki-dark:#ebdbb2;overflow-x:auto" tabindex="0" data-language="shellsession"><code><span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">$</span><span style="color:#3C3836;--shiki-dark:#EBDBB2"> tree -a -F -L 2 </span><span style="color:#AF3A03;--shiki-dark:#FE8019">.</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">├── .bundle/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">│   ├── bin/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">│   ├── bundled-plugins/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">│   ├── daemon/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">│   ├── daemon-mac-updater.sh*</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">│   ├── lib/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">│   └── lms*</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">└── llmster*</span></span></code></pre>
<p>The <code>llmster</code> binary is compiled via <a href="https://bun.com/docs/bundler/executables">Bun</a>, a JavaScript runtime similar to Node.js, and it depends on the files in <code>.bundle</code> to be available in the same directory.
A symlink can then be created to make the binary available when installed system-wide.
Precompiled binaries like this need to be patched via <code>patchelf</code> on Linux to resolve their dynamic library dependencies correctly.
This usually involves stripping them as well (i.e., removing unnecessary RPATH entries), but Bun-compiled ones become non-functional when stripped, so we need <code>dontStrip</code> here.</p>
<pre class="astro-code astro-code-themes gruvbox-light-hard gruvbox-dark-hard" style="background-color:#f9f5d7;--shiki-dark-bg:#1d2021;color:#3c3836;--shiki-dark:#ebdbb2;overflow-x:auto" tabindex="0" data-language="nix"><code><span class="line"><span style="color:#076678;--shiki-dark:#83A598">nativeBuildInputs</span><span style="color:#CC241D;--shiki-dark:#CC241D"> =</span><span style="color:#7C6F64;--shiki-dark:#A89984"> [</span></span>
<span class="line"><span style="color:#076678;--shiki-dark:#83A598">  makeBinaryWrapper</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">]</span></span>
<span class="line"><span style="color:#427B58;--shiki-dark:#8EC07C">++</span><span style="color:#076678;--shiki-dark:#83A598"> lib</span><span style="color:#427B58;--shiki-dark:#8EC07C">.</span><span style="color:#076678;--shiki-dark:#83A598">optionals</span><span style="color:#076678;--shiki-dark:#83A598"> stdenv</span><span style="color:#427B58;--shiki-dark:#8EC07C">.</span><span style="color:#076678;--shiki-dark:#83A598">hostPlatform</span><span style="color:#427B58;--shiki-dark:#8EC07C">.</span><span style="color:#076678;--shiki-dark:#83A598">isLinux</span><span style="color:#7C6F64;--shiki-dark:#A89984"> [</span></span>
<span class="line"><span style="color:#076678;--shiki-dark:#83A598">  autoPatchelfHook</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">]</span><span style="color:#CC241D;--shiki-dark:#CC241D">;</span></span>
<span class="line"></span>
<span class="line"><span style="color:#076678;--shiki-dark:#83A598">installPhase</span><span style="color:#CC241D;--shiki-dark:#CC241D"> =</span><span style="color:#7C6F64;--shiki-dark:#A89984"> &#39;&#39;</span></span>
<span class="line"><span style="color:#79740E;--shiki-dark:#B8BB26">  runHook preInstall</span></span>
<span class="line"></span>
<span class="line"><span style="color:#79740E;--shiki-dark:#B8BB26">  mkdir -p $out/libexec</span></span>
<span class="line"><span style="color:#79740E;--shiki-dark:#B8BB26">  mv llmster .bundle $out/libexec/</span></span>
<span class="line"><span style="color:#79740E;--shiki-dark:#B8BB26">  makeWrapper $out/libexec/llmster $out/bin/llmster</span></span>
<span class="line"></span>
<span class="line"><span style="color:#79740E;--shiki-dark:#B8BB26">  runHook postInstall</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">&#39;&#39;</span><span style="color:#CC241D;--shiki-dark:#CC241D">;</span></span></code></pre>
<p>Another challenge involves the GPU support provided by <code>llmster</code>.
Nixpkgs provides an <code>autoAddDriverRunpath</code> hook to inject the required drivers during runtime, but this also involves stripping under the hood, meaning we need to do this manually.
We not only search for <code>.so</code> files but also <code>.node</code> files, as these can contain native code as well.</p>
<pre class="astro-code astro-code-themes gruvbox-light-hard gruvbox-dark-hard" style="background-color:#f9f5d7;--shiki-dark-bg:#1d2021;color:#3c3836;--shiki-dark:#ebdbb2;overflow-x:auto" tabindex="0" data-language="nix"><code><span class="line"><span style="color:#076678;--shiki-dark:#83A598">nativeBuildInputs</span><span style="color:#CC241D;--shiki-dark:#CC241D"> =</span><span style="color:#076678;--shiki-dark:#83A598"> lib</span><span style="color:#427B58;--shiki-dark:#8EC07C">.</span><span style="color:#076678;--shiki-dark:#83A598">optionals</span><span style="color:#076678;--shiki-dark:#83A598"> stdenv</span><span style="color:#427B58;--shiki-dark:#8EC07C">.</span><span style="color:#076678;--shiki-dark:#83A598">hostPlatform</span><span style="color:#427B58;--shiki-dark:#8EC07C">.</span><span style="color:#076678;--shiki-dark:#83A598">isLinux</span><span style="color:#7C6F64;--shiki-dark:#A89984"> [</span></span>
<span class="line"><span style="color:#076678;--shiki-dark:#83A598">  addDriverRunpath</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">]</span><span style="color:#CC241D;--shiki-dark:#CC241D">;</span></span>
<span class="line"></span>
<span class="line"><span style="color:#076678;--shiki-dark:#83A598">postFixup</span><span style="color:#CC241D;--shiki-dark:#CC241D"> =</span><span style="color:#076678;--shiki-dark:#83A598"> lib</span><span style="color:#427B58;--shiki-dark:#8EC07C">.</span><span style="color:#076678;--shiki-dark:#83A598">optionalString</span><span style="color:#076678;--shiki-dark:#83A598"> stdenv</span><span style="color:#427B58;--shiki-dark:#8EC07C">.</span><span style="color:#076678;--shiki-dark:#83A598">hostPlatform</span><span style="color:#427B58;--shiki-dark:#8EC07C">.</span><span style="color:#076678;--shiki-dark:#83A598">isLinux</span><span style="color:#7C6F64;--shiki-dark:#A89984"> &#39;&#39;</span></span>
<span class="line"><span style="color:#79740E;--shiki-dark:#B8BB26">  find $out/libexec/.bundle -type f \( -name &#39;*.so&#39; -o -name &#39;*.so.*&#39; -o -name &#39;*.node&#39; \) | while read -r lib; do</span></span>
<span class="line"><span style="color:#79740E;--shiki-dark:#B8BB26">    addDriverRunpath &quot;$lib&quot;</span></span>
<span class="line"><span style="color:#79740E;--shiki-dark:#B8BB26">  done</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">&#39;&#39;</span><span style="color:#CC241D;--shiki-dark:#CC241D">;</span></span></code></pre>
<p>Finally, we need to ignore missing references to the actual GPU driver, which is not known during build time.</p>
<pre class="astro-code astro-code-themes gruvbox-light-hard gruvbox-dark-hard" style="background-color:#f9f5d7;--shiki-dark-bg:#1d2021;color:#3c3836;--shiki-dark:#ebdbb2;overflow-x:auto" tabindex="0" data-language="nix"><code><span class="line"><span style="color:#076678;--shiki-dark:#83A598">autoPatchelfIgnoreMissingDeps</span><span style="color:#CC241D;--shiki-dark:#CC241D"> =</span><span style="color:#7C6F64;--shiki-dark:#A89984"> [</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">  &quot;</span><span style="color:#79740E;--shiki-dark:#B8BB26">libcuda.so.1</span><span style="color:#7C6F64;--shiki-dark:#A89984">&quot;</span></span>
<span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">]</span><span style="color:#CC241D;--shiki-dark:#CC241D">;</span></span></code></pre>
<p>The complete package can be found in the corresponding <a href="https://github.com/NixOS/nixpkgs/pull/485978">pull request</a>.
I have tested it on a CUDA-capable machine and was able to run <code>llmster</code> successfully with GPU support, but this is still a work in progress.
In its current state, running the <code>llmster</code> binary creates the directory <code>.lmstudio</code> as well as the file <code>.lmstudio-home-pointer</code> in the user’s home.</p>
<pre class="astro-code astro-code-themes gruvbox-light-hard gruvbox-dark-hard" style="background-color:#f9f5d7;--shiki-dark-bg:#1d2021;color:#3c3836;--shiki-dark:#ebdbb2;overflow-x:auto" tabindex="0" data-language="shellsession"><code><span class="line"><span style="color:#7C6F64;--shiki-dark:#A89984">$</span><span style="color:#3C3836;--shiki-dark:#EBDBB2"> tree -a -F -L 1 .lmstudio</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">.lmstudio/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">├── bin/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">├── config-presets/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">├── conversations/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">├── credentials/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">├── dev-logs/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">├── extensions/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">├── hub/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">├── .internal/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">├── mcp.json</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">├── models/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">├── projects/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">├── server-logs/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">├── settings.json</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">├── user-files/</span></span>
<span class="line"><span style="color:#3C3836;--shiki-dark:#EBDBB2">└── working-directories/</span></span></code></pre>
<p>While in theory it is possible to create these directly inside the Nix store, it would be impractical due to the store’s immutability–for instance, users would be unable to download models.
As such, we might need to accept this stateful behavior for now.
Please help test the PR and provide feedback for improvements!</p>]]></content>
        <author>
            <name>Mirko Lenz</name>
        </author>
        <category label="nix"/>
        <category label="llm"/>
        <published>2026-02-04T00:00:00.000Z</published>
    </entry>
</feed>