Each task execution (aka run) got a disposable cluster and a plain-language objective. 905 clusters, 2,316 full virtual machines, 59,349 recorded commands. The harness, built on Harbor, kept the whole timeline: what was issued, what came back, and what state was left behind. Of the 905 executions, 886 produced usable verifier scores. The other 19 ended without a score, often because the agent timed out and the cluster was terminated. Tasks cover single-host, multi-node services, stateful clusters, and four network operating systems: VyOS, OpenWrt, SONiC and OPNsense. In another category, VyOS and OPNsense are combined in more complex networking scenarios, such as an IPsec tunnel between a VyOS LAN and an OPNsense LAN. Clusters and network devices are provisioned through Antrieb, a testbed creator I built to facilitate my experiments with LLMs and infrastructure.

The headline counts cover the current published corpus. Several finding sections below use the earlier 804-run analysis set and retain its task-level measurements.

The raw data is on GitHub and Hugging Face. I welcome new tasks, runs, or mining of traces. Here's what I found, including a couple of things I had wrong going in.

The 2,316 VMs were never running at the same time. Clusters are provisioned in small batches, on the order of 50 machines at once, and torn down when the run finishes.

Commands 59,349 · Runs scored 886 · VMs booted 2,316 · Left residue 98.8% · Linux distros 8 · Network OSes 4 · Cluster size 1–4 · VM launch 848 ms

Eight findings

1. LLMs almost always complete the job successfully

Across the current corpus, 886 of 905 runs produced usable verifier scores. Nineteen runs ended without a score, often because the agent timed out and the cluster was terminated. The category table below retains the earlier 804-run analysis set: 759 of those scored runs came back with a perfect score, with every requirement met and checked against captured evidence. A run counts as successful if it met 80% of the task's functional requirements.

Perfect-score rate by category

2. LLM-friendliness varies by release

The fail rate is the ratio of failed commands to all commands run on that release. This is a command-level measure; task success is measured separately. A release can therefore have a high command fail rate alongside a high task success rate.

RHEL

Ubuntu

I observe three times as many commands fail on RHEL 7.9 as on RHEL 9.8, running the same 29 tasks with the same wording. Ubuntu follows the same direction, with a weaker effect.

I suspect that an important factor in this failure rate is how much material about a release exists publicly, and how long that material has stood before a newer version supersedes it. RHEL 9 superseded RHEL 7, so most of what the model has read about RHEL describes 9 rather than 7. RHEL 10 is newer than RHEL 9, but it has not superseded RHEL 9 in the written record yet, which would explain why it fails more often than the release it replaces.

These numbers exclude file-integrity-baseline, where one run distorted a column; the simple-task trap below describes it.

3. The leftovers: 98.8% of runs leave residue

In this experiment, every run carries an operational-hygiene score. It asks whether the run mutated things the task never called for, left residue behind, or broke something unrelated. When the model leaves absolutely no residue behind, the run scores a perfect 1.000.

Operational hygiene across 804 scored runs

The table shows that ten of 804 scored runs left the machine in a clean state. The other 98.8% left something behind: a package pulled in to test a theory, a service stopped and never restarted, scratch files in /tmp, a config edited and not reverted.

Residue is a security problem

Residue is attack surface. The debugging packages the agent installed are now unpatched software on your host. The service it stopped may be auditd or a log shipper.

Many tasks left private key material in /tmp when the run ended, including /tmp/pgca/ca.key, the signing key of the certificate authority the model had just created for the cluster (post-run snapshot, line 44). WireGuard private keys and client keys turn up the same way.

Whether that is a breach depends on where the box sits. On a throwaway lab VM it is untidy. Anywhere else, that key is now outside the store it belongs in, sitting in a scratch directory, and it is the key that signs certificates for every node in the cluster. Anyone who can read it can mint a certificate the whole cluster trusts. Backups and snapshots pick the file up as well, which moves the key somewhere with a different and usually longer retention. Under a compliance boundary, such as payment processing, medical devices or plant control, key material outside its intended store is an audit finding on its own, before anyone has to show it was read.

The model has no concept of your data classification. It writes where the shell put it, and it does not come back for it.

4. Configuration is cheap. Coordination is expensive.

As I suspected, cost climbs wherever two or more nodes have to agree on replication, quorum, state transfer or failover, because the result has to be demonstrated through a real state transition instead of being read from a config file. The table shows the observed completion-time ranges across three scenarios.

Observed completion-time range by task shape

A PostgreSQL failover on Ubuntu 24.04 ran for 36 minutes and hit the wall. Same model, same fleet, same day as three-minute single-host tasks that scored clean.

5. The model reaches for force as a first resort

An engineer who is stuck usually knows it. They slow down as the system gets harder to read, they get careful around the parts they do not understand, and past a certain point they stop and ask someone. The model has none of those habits. It resorts to forceful options, and it does so even on runs that are otherwise going fine.

865 commands in the dataset stop a service, kill a process, delete a directory or flush a network configuration. 254 of the 804 scored runs contain at least one of them, so roughly one run in three used force somewhere.

Forceful commands by kind

The linked PostgreSQL failover run contains examples of destructive state-directory removal: node2 and node3 at lines 83–86 and node1 at lines 127–128. The dataset aggregate reports 177 such commands across 55 runs.

6. Unclear provenance for repository URLs

Installing software outside the distribution repositories often requires the agent to configure a vendor repository or download a repository definition. Across the command logs, agents sometimes use URLs that are obsolete, invalid, or incorrect. The logs frequently show the URL and the resulting failure without showing how the agent derived the address.

The RabbitMQ runs provide several examples. On AlmaLinux, the agent tried el9.rabbitmq.com and received DNS resolution failures while writing RPM repository files (lines 7–14). On Ubuntu, it placed the retired ppa1.rabbitmq.com repository in /etc/apt/sources.list.d/rabbitmq.list (lines 99–104); the subsequent APT update reported that the hostname could not be resolved (lines 117–120). RabbitMQ later moved its Debian repositories to deb1.rabbitmq.com and deb2.rabbitmq.com. The evidence supports obsolete or incorrect repository selection, while the agent's reasoning or source for the URLs remains unknown.

Examples of repository URLs used by agents

The provenance gap creates operational and security risk

Incorrect URLs cause DNS failures, repository errors, retries, long waits, and fallback to distribution packages. Those fallbacks can change the software version and alter the result of the task. An address that resolves later to an unrelated service could direct repository configuration toward an unintended source. Repository and package signature checks provide an important control, while the URL's provenance remains unverified. Since the logs rarely reveal where the agent obtained the address, the same error is difficult to predict, prevent, or reproduce. These runs document installation and reproducibility risk.

7. Failure persisting kernel-facing changes

Some Linux tools allow agents to create transient configurations that disappear after a reboot. This applies to both firewall-cmd and iptables.

I wanted to simulate how an administrator would actually use an LLM, so I left persistence implicit. The tasks asked the agent to reboot the machine after making its changes.

firewall-cmd runs that never used --permanent

Across the 237 firewalld cases in the table, 210 runs—88.6%—left the firewall configuration working after reboot. In the remaining 27 runs—11.4%—the changes disappeared. The agent made the service work during the run, then reported success before the reboot exposed the missing persistence.

The consequence is a delayed operational failure. A service may be reachable when the agent finishes, then lose its required access after a routine restart. Cluster communication, remote administration, or firewall protection can fail later, when the original change is no longer in anyone’s immediate view.

8. The simple-task trap

A task that looks simple can produce a surprisingly complicated execution path when an LLM performs it. In the file-integrity-baseline task, the instruction was straightforward: establish an integrity baseline for /etc so later changes can be identified and reported.

Ubuntu 24.04 AIDE check

On most other releases, the task completed in about four minutes on average. Ubuntu 16.04 took about 12 minutes, while Ubuntu 24.04 took 32 minutes.

The difference follows the size of the AIDE scan. Ubuntu 24.04 covered nearly 90,000 filesystem entries, Ubuntu 16.04 covered about 70,000, and the other releases generally covered 30,000 to 40,000. Ubuntu 24 crossed the synchronous request timeout, causing the agent to relaunch AIDE with nohup and poll it with fixed sleep intervals. The scan often finished before the polling sleep ended, so the agent waited until the end of the interval before it could observe the result. The same pattern repeated across the later AIDE operations, adding substantial delay to the total.

The visible task may be simple while the model's execution strategy introduces retries, detached processes, polling, and timeout effects.

Open questions

1. An LLM analyzed the work of an LLM

Every run here was scored by an LLM verifier, and the findings on this page come from an LLM reading the command logs. That is not a rigorous method, and it is fair to hold the conclusions loosely because of it.

This is precisely why I am publishing the dataset: to invite the community to mine it.

2. One model, one setting

Every run here was executed by Claude Sonnet 5 at medium reasoning effort. Nothing in the dataset says whether a different model, or the same model at a different effort, behaves the same way.

I believe most of these observations apply to current models generally. That is a belief, and it needs validating.

The lab

The Antrieb testbed cluster running in my basement. USB fans keep the mini PCs from melting while spinning up disposable VM chains.

All of it runs in my basement on five machines. The ProLiant hosts the MCP server; the four mini PCs carry the virtual machines. An HPE OfficeConnect 1620 switch ties them together, and a UPS keeps the fleet up through the short outages that would otherwise kill an experiment halfway through.

The fleet

What I take from this

Using the traditional definition of completing a system administration task, the LLM succeeds 99% of the time. I tested on eight releases across five distributions, spanning ten years, on jobs ranging from a one-line sysctl change to a three-node quorum. This level of success was surprising.

More surprises were waiting past that definition, starting with how much gets left behind. Almost every run leaves something on the box, and in many tasks that something was a private key left in /tmp, including the signing key of the cluster's own certificate authority. About a third of runs reach for force somewhere, while 11.4% of kernel-facing changes are not persisted. Several runs used repository URLs that were obsolete, invalid, or incorrect, and the logs often do not show how the agent derived them.

I suspect even more surprises are lurking in the data. Please join the mining, and reach out if you have any questions. Everything is on GitHub and Hugging Face: every task, every command timeline, every score.