Fixing large uploads on OpenMediaVault's K3S

I recently decided to use OpenMediaVault's Kubernetes install for setting up some services in my home network. Among other things, I wanted to use immich for managing my photos and videos locally. Thanks to OMV's Lubernetes recipes, getting a basic install up and running is very easy.

Unfortunately, syncing larger videos (>= 1 GiB) from my phone to the server always fails. The pod logs contained various ECONNRESET messages, indicating connection resets. Some Googling indicated this might be caused by Traefik's default read timeout of 60 seconds.

To fix this, I read a nice blog post on how to increase Traefik ingress controller timeouts in K3S which describes exactly what I wanted to do. It even includes a manifest code snippet that can be applied via kubectl or by copy'n'pasting to http://my-omv-instance/#/services/k8s/apply. There is only one problem: It doesn't work.

The helm chart config can easily be applied, but as soon as Traefik restarts, all web services on my NAS stop working. And this does not only affect services like Immich -- it also brings down the Kubernetes Dashboard at port 4443 and the entire OMV web interface. Fix can be fixed by restarting k3s systemd service, but then the timeouts are also gone and we are back to square one.

Fortunately, after some digging in /var/lib/rancher/k3s/server/manifests/ directory I was able to identify the cause of this problem. In addition the traefik.yaml Helm chart shipped with K3S, there is also a corresponding Helm chart config in openmediavault-traefik.yaml, which contains the following snippet:

---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
  labels:
    app.kubernetes.io/part-of: openmediavault
spec:
  valuesContent: |-
    ports:
      dashboard:
        expose:
          default: true
        exposedPort: 4443
        port: 4443
        protocol: TCP
        tls:
          enabled: true
      web:
        exposedPort: 8080
      websecure:
        exposedPort: 8443
---
# ....

The new helm chart config with the timeout values apparently overrides the helm chart config shipped with OpenMediaVault. Hence, it's possible to just copy the OMV's config and then extend it. Here's the final helm chart config I used:

---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
  labels:
    app.kubernetes.io/part-of: openmediavault
spec:
  valuesContent: |-
    ports:
      dashboard:
        expose:
          default: true
        exposedPort: 4443
        port: 4443
        protocol: TCP
        tls:
          enabled: true
      web:
        exposedPort: 8080
        transport:
          respondingTimeouts:
            readTimeout: 3600
      websecure:
        exposedPort: 8443
        transport:
          respondingTimeouts:
            readTimeout: 3600

Et voilà, the upload issue is fixed and immich is able to sync large files now.