iSCSI on Talos: Why the Obvious Path Doesn't Work

May 6, 2026 kubernetes · talos · storage · sre

Talos Linux is, in many ways, exactly what you want for a Kubernetes node OS: immutable, minimal, API-driven, nothing to SSH into. It removes an entire class of configuration drift problems. It also means the moment you want to reach for a binary that isn’t in the default image, you’re in a different kind of trouble.

iSCSI is that binary.

The setup

I was running a single-node Talos cluster backed by a FreeNAS box. A handful of apps — Jellyfin, Sonarr, Radarr, Plex — needed config volumes that would survive pod restarts. iSCSI LUNs were already provisioned. The obvious thing was to point Kubernetes at them.

The obvious thing turned out to be a detour.

The in-tree trap

Kubernetes has had in-tree iSCSI volume support for years. It works fine on a standard Linux node. The volume spec looks like this:

volumes:
- name: config
  iscsi:
    targetPortal: 10.100.10.28
    iqn: iqn.2005-10.org.freenas.ctl:sonarr
    iscsiInterface: default
    lun: 0

Simple. Direct. No moving parts. And on Talos, it produces this:

MountVolume.WaitForAttach failed for volume "config":
executable file not found in $PATH

The in-tree iSCSI plugin works by shelling out to iscsiadm from inside the kubelet process. On a standard Linux node, iscsiadm is on the host, kubelet can find it, done. On Talos, the kubelet runs as an OCI container — and that container’s PATH doesn’t include anything from the host filesystem unless you put it there.

The Talos documentation points at the iscsi-tools extension, which you can bake into a custom factory image. I did that. The iscsid daemon started. The iscsiadm binary showed up at /usr/local/sbin/iscsiadm on the host. The kubelet still couldn’t find it, because the kubelet’s container has its own filesystem, and /usr/local/sbin from the host isn’t in it.

Trying to bridge the gap

The natural next move is machine.kubelet.extraMounts — Talos lets you add OCI bind mounts to the kubelet container. I tried a file-level bind mount first:

kubelet:
  extraMounts:
    - destination: /usr/local/bin/iscsiadm
      type: bind
      source: /usr/local/sbin/iscsiadm
      options: [bind, ro]

After the reboot:

Failed to create runner: mkdir /usr/local/sbin/iscsiadm: not a directory

The Talos OCI runner was treating the source path as a directory to create rather than a file to bind. A known rough edge with file-level bind mounts in Talos’s OCI runner.

I switched to a directory-level bind mount — mount the whole /usr/local/sbin from the host into the kubelet container at the same path:

kubelet:
  extraMounts:
    - destination: /usr/local/sbin
      type: bind
      source: /usr/local/sbin
      options: [bind, ro]

That cleared the kubelet startup error. The mount showed up in the kubelet container’s namespace. Whether the in-tree iSCSI plugin would then find iscsiadm correctly is a question I stopped asking, because at this point the approach had already earned its way off the list.

The actual problem with in-tree iSCSI

Even if the binary access works, you’re still managing inline volume specs in every deployment manifest. Every app has its targetPortal, IQN, and LUN baked directly into the pod spec:

- name: pvc
  iscsi:
    iqn: iqn.2005-10.org.freenas.ctl:radarr
    iscsiInterface: default
    lun: 0
    targetPortal: 10.100.10.28

There’s no abstraction. The app has to know where its storage lives. You can’t move a LUN without editing a deployment. And the in-tree iSCSI plugin has been in maintenance mode for years — the Kubernetes project isn’t interested in making it better.

The right tool is the CSI driver.

csi-driver-iscsi

The Kubernetes CSI driver for iSCSI runs as a DaemonSet. The driver pods carry their own iscsiadm binary. Kubelet talks to the driver over a Unix socket, the driver handles the iSCSI operations from its own container. Kubelet never needs to find iscsiadm itself.

This sidesteps the entire Talos binary problem. The kubelet container doesn’t need any changes. The extraMounts config goes away. The iscsi-tools extension is still useful (for iscsid and kernel module loading), but it’s no longer the thing your kubelet depends on directly.

Install the driver via Helm or the manifests from the repo. Then the storage pattern looks like this.

StorageClass — one per cluster, defines the CSI driver:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: manual
provisioner: iscsi.csi.k8s.io

PersistentVolume — one per LUN, bound to a specific IQN:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: sonarr-iscsi-pv
  labels:
    name: sonarr-iscsi-pv
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 60Gi
  csi:
    driver: iscsi.csi.k8s.io
    volumeHandle: iscsi-sonarr-id
    volumeAttributes:
      targetPortal: "10.100.10.28:3260"
      iqn: "iqn.2005-10.org.freenas.ctl:sonarr"
      lun: "0"
      portals: "[]"
      iscsiInterface: "default"
      discovery: "true"
      discoveryCHAPAuth: "false"
      sessionCHAPAuth: "false"

PersistentVolumeClaim — what the app references, with a label selector to bind to the right PV:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: sonarr-iscsi-pv
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 60Gi
  storageClassName: manual
  selector:
    matchExpressions:
      - key: name
        operator: In
        values: ["sonarr-iscsi-pv"]

Deployment — just references the claim:

volumes:
- name: config
  persistentVolumeClaim:
    claimName: sonarr-iscsi-pv

The IQN and target portal are in the PV, not the deployment. Move a LUN, update the PV. The app doesn’t care.

The Talos-specific pieces

A few things to get right before the CSI driver will work:

Factory image with iscsi-tools. The iscsid daemon needs to run on the node, and the iscsi_tcp kernel module needs to load. The Talos image factory lets you bake in the siderolabs/iscsi-tools extension. You can also reference the schematic directly in your machine config’s install.image and run talosctl upgrade to push it to a running node without going back to maintenance mode.

Kernel module. Add iscsi_tcp to machine.kernel.modules in your machine config:

machine:
  kernel:
    modules:
      - name: iscsi_tcp

iscsid.conf. The ext-iscsid service will start without a config file, but it logs warnings and may not connect to targets reliably. In Talos v1.13, machine.files with op: create is restricted to paths under /var. Writing to /etc/iscsi/iscsid.conf directly isn’t allowed. An ExtensionServiceConfig resource is the right approach for configuring extension services going forward — that’s a separate step, and basic connectivity works without it for most setups.

What didn’t need to change

The rest of the Talos machine config stays clean. The extraMounts block on machine.kubelet is unnecessary with the CSI driver approach — take it out entirely. The only kubelet-level config you need is the default.

The in-tree approach asks you to solve a Talos-specific problem (binary access) just to get back to table stakes. The CSI driver approach skips the problem entirely and gives you better storage abstractions as a byproduct. On an immutable OS, moving the iSCSI work into its own container isn’t just tidier — it’s the only path that doesn’t accumulate sharp edges.