[BUG] Create Mount Pod Error Because Of Default Toleration Error

Mar 13, 2025 by ADMIN 65 views

**BUG: Create Mount Pod Error Due to Default Tolerance Error**

Introduction

In this article, we will discuss a bug that occurs when creating a mount pod due to a default tolerance error. The bug is related to the JuiceFS CSI Driver and is caused by a special character in the default toleration inherited from the CSI node.

What Happened

The error message is as follows:

E0312 13:14:30.222876       7 driver.go:107] "GRPC error" err="rpc error: code = Internal desc = Could not mount juicefs: Pod \"juicefs-[10.1.1.81](http://10.1.1.81/)-canoe-sto-cubhtiy710-icsrkb\" is invalid: spec.tolerations[5].key: Invalid value: \"[node.kubernetes.io/unreachable\\x00\\x00](http://node.kubernetes.io/unreachable//x00//x00)\": name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName',  or 'my.name',  or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]')" logger="driver"

What You Expected to Happen

The expected behavior is that the mount pod should be created successfully without any errors.

How to Reproduce It

To reproduce this bug, you need to have the JuiceFS CSI Driver installed and running on your Kubernetes cluster. Then, try to create a mount pod using the JuiceFS CSI Driver. The bug will occur when the default toleration inherited from the CSI node contains special characters.

Anything Else We Need to Know?

Yes, there are a few more things that we need to know:

JuiceFS CSI Driver version: v0.26.4
Kubernetes version: v1.22.5-tke.8
Object storage: Tencent Cloud
Metadata engine info: Self-maintained
Network connectivity: JuiceFS to metadata engine: 10.1.1.81, JuiceFS to object storage: 10.1.1.81

Environment

The environment details are as follows:

JuiceFS CSI Driver version: v0.26.4
Kubernetes version: v1.22.5-tke.8
Object storage: Tencent Cloud
Metadata engine info: Self-maintained
Network connectivity: JuiceFS to metadata engine: 10.1.1.81, JuiceFS to object storage: 10.1.1.81

CSI Node YAML

The CSI node YAML is as follows:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    tke.cloud.tencent.com/networks-status: |-
      [{
          "name": "tke-bridge",
          "interface": "eth0",
          "ips": [
              "10.32.65.75"
          ],
          "mac": "62:72:d2:f0:04:35",
          "default": true,
          "dns": {}
      }]
  creationTimestamp: "2025-03-12T13:20:59Z"
  generateName: juicefs-csi-node-
  labels:
    app: juicefs-csi-node
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    controller-revision-hash: d54d495b9
    infra.xaminim.com/task-id: t-f9fl29vvnb
    pod-template-generation: "22"
  name: juicefs-csi-node-khncr
  namespace: kube-system
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: DaemonSet
    name: juicefs-csi-node
    uid: 54bc5785-b398-48ae-be8f-d539fb79a53b
  resourceVersion: "15384045645"
  selfLink: /api/v1/namespaces/kube-system/pods/juicefs-csi-node-khncr
  uid: d0963274-32be-461a-b42a-a292255afaa2
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - 10.1.1.81
  containers:
  - args:
    - --endpoint=$(CSI_ENDPOINT)
    - --logtostderr
    - --nodeid=$(NODE_NAME)
    - --v=5
    - --enable-manager=true
    - --config=/etc/config/config.yaml
    env:
    - name: CSI_ENDPOINT
      value: unix:/csi/csi.sock
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: JUICEFS_MOUNT_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: HOST_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.hostIP
    - name: KUBELET_PORT
      value: "10250"
    - name: JUICEFS_MOUNT_PATH
      value: /var/lib/juicefs/volume
    - name: MMTest
      value: aaa
    - name: JUICEFS_CONFIG_PATH
      value: /var/lib/juicefs/config
    - name: JUICEFS_CE_MOUNT_IMAGE
      value: harbor.xaminim.com/minimax-pub/juicer-csi-mount:0.0.3
    - name: JUICEFS_EE_MOUNT_IMAGE
      value: harbor.xaminim.com/minimax-pub/juicedata/mount:ee-5.1.11-ca2c117
    - name: ENABLE_APISERVER_LIST_CACHE
      value: "true"
    image: harbor.xaminim.com/minimax-pub/juicefs-csi-driver:v0.26.4
    imagePullPolicy: IfNotPresent
    lifecycle:
      preStop:
        exec:
          command:
          - /bin/sh
          - -c
          - rm /csi/csi.sock
    livenessProbe:
      failureThreshold: 5
      httpGet:
        path: /healthz
        port: 9909
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 3
    name: juicefs-plugin
    ports:
    - containerPort: 9909
      protocol: TCP
    - containerPort: 6060
      name: metrics
      protocol: TCP
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 512Mi
    securityContext:
      privileged: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/kubelet
      mountPropagation: Bidirectional
      name: kubelet-dir
    - mountPath: /csi
      name: plugin-dir
    - mountPath: /dev
      name: device-dir
    - mountPath: /jfs
      mountPropagation: Bidirectional
      name: jfs-dir
    - mountPath: /root/.juicefs
      mountPropagation: Bidirectional
      name: jfs-root-dir
    - mountPath: /etc/config
      name: juicefs-config
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-qlhzg
      readOnly: true
  - args:
    - --csi-address=$(ADDRESS)
    - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
    - --v=5
    - --health-port=9809
    env:
    - name: ADDRESS
      value: /csi/csi.sock
    - name: DRIVER_REG_SOCK_PATH
      value: /var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock
    image: harbor.xaminim.com/minimax-pub/csi-node-driver-registrar:v2.9.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthz
        port: healthz
        scheme: HTTP
      initialDelaySeconds: 5
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    name: node-driver-registrar
    ports:
    - containerPort: 9809
      name: healthz
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /csi
      name: plugin-dir
    - mountPath: /registration
      name: registration-dir
    - mountPath:<br/>
**Q&A: BUG Create Mount Pod Error Due to Default Tolerance Error**
===========================================================

**Q: What is the cause of the bug?**
--------------------------------

A: The bug is caused by a special character in the default toleration inherited from the CSI node. The default toleration contains the key `node.kubernetes.io/unreachable\\x00\\x00`, which is not a valid key.

**Q: What is the expected behavior?**
-----------------------------------

A: The expected behavior is that the mount pod should be created successfully without any errors.

**Q: How to reproduce the bug?**
-------------------------------

A: To reproduce the bug, you need to have the JuiceFS CSI Driver installed and running on your Kubernetes cluster. Then, try to create a mount pod using the JuiceFS CSI Driver. The bug will occur when the default toleration inherited from the CSI node contains special characters.

**Q: What are the environment details?**
-----------------------------------------

A: The environment details are as follows:

* JuiceFS CSI Driver version: v0.26.4
* Kubernetes version: v1.22.5-tke.8
* Object storage: Tencent Cloud
* Metadata engine info: Self-maintained
* Network connectivity: JuiceFS to metadata engine: 10.1.1.81, JuiceFS to object storage: 10.1.1.81

**Q: What is the CSI node YAML?**
------------------------------

A: The CSI node YAML is as follows:

```yml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    tke.cloud.tencent.com/networks-status: |-
      [{
          "name": "tke-bridge",
          "interface": "eth0",
          "ips": [
              "10.32.65.75"
          ],
          "mac": "62:72:d2:f0:04:35",
          "default": true,
          "dns": {}
      }]
  creationTimestamp: "2025-03-12T13:20:59Z"
  generateName: juicefs-csi-node-
  labels:
    app: juicefs-csi-node
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    controller-revision-hash: d54d495b9
    infra.xaminim.com/task-id: t-f9fl29vvnb
    pod-template-generation: "22"
  name: juicefs-csi-node-khncr
  namespace: kube-system
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: DaemonSet
    name: juicefs-csi-node
    uid: 54bc5785-b398-48ae-be8f-d539fb79a53b
  resourceVersion: "15384045645"
  selfLink: /api/v1/namespaces/kube-system/pods/juicefs-csi-node-khncr
  uid: d0963274-32be-461a-b42a-a292255afaa2
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - 10.1.1.81
  containers:
  - args:
    - --endpoint=$(CSI_ENDPOINT)
    - --logtostderr
    - --nodeid=$(NODE_NAME)
    - --v=5
    - --enable-manager=true
    - --config=/etc/config/config.yaml
    env:
    - name: CSI_ENDPOINT
      value: unix:/csi/csi.sock
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: JUICEFS_MOUNT_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: HOST_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.hostIP
    - name: KUBELET_PORT
      value: "10250"
    - name: JUICEFS_MOUNT_PATH
      value: /var/lib/juicefs/volume
    - name: MMTest
      value: aaa
    - name: JUICEFS_CONFIG_PATH
      value: /var/lib/juicefs/config
    - name: JUICEFS_CE_MOUNT_IMAGE
      value: harbor.xaminim.com/minimax-pub/juicer-csi-mount:0.0.3
    - name: JUICEFS_EE_MOUNT_IMAGE
      value: harbor.xaminim.com/minimax-pub/juicedata/mount:ee-5.1.11-ca2c117
    - name: ENABLE_APISERVER_LIST_CACHE
      value: "true"
    image: harbor.xaminim.com/minimax-pub/juicefs-csi-driver:v0.26.4
    imagePullPolicy: IfNotPresent
    lifecycle:
      preStop:
        exec:
          command:
          - /bin/sh
          - -c
          - rm /csi/csi.sock
    livenessProbe:
      failureThreshold: 5
      httpGet:
        path: /healthz
        port: 9909
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 3
    name: juicefs-plugin
    ports:
    - containerPort: 9909
      protocol: TCP
    - containerPort: 6060
      name: metrics
      protocol: TCP
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 512Mi
    securityContext:
      privileged: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/kubelet
      mountPropagation: Bidirectional
      name: kubelet-dir
    - mountPath: /csi
      name: plugin-dir
    - mountPath: /dev
      name: device-dir
    - mountPath: /jfs
      mountPropagation: Bidirectional
      name: jfs-dir
    - mountPath: /root/.juicefs
      mountPropagation: Bidirectional
      name: jfs-root-dir
    - mountPath: /etc/config
      name: juicefs-config
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-qlhzg
      readOnly: true
  - args:
    - --csi-address=$(ADDRESS)
    - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
    - --v=5
    - --health-port=9809
    env:
    - name: ADDRESS
      value: /csi/csi.sock
    - name: DRIVER_REG_SOCK_PATH
      value: /var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock
    image: harbor.xaminim.com/minimax-pub/csi-node-driver-registrar:v2.9.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthz
        port: healthz
        scheme: HTTP
      initialDelaySeconds: 5
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    name: node-driver-registrar
    ports:
    - containerPort: 9809
      name: healthz
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /csi
      name: plugin-dir
    - mountPath: /registration
      name: registration-dir
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-qlhzg
      readOnly: true
  - args:
    - --csi-address=$(ADDRESS)
    - --health-port=$(HEALTH_PORT)
    env:
    - name: ADDRESS
      value: /csi/csi.sock
    - name: HEALTH_PORT
      value: "9909"
    image: harbor.xaminim.com/minimax-pub/juicefs-livenessprobe:v2.12.0
    imagePullPolicy: IfNotPresent
    name: liveness-probe
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /csi
      name: plugin-dir
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-qlhzg
      readOnly: true
  dnsPolicy: ClusterFirstWithHostNet
  enableServiceLinks: true
  nodeName: 10.1.1.81
  preemptionPolicy: PreemptLowerPriority
  priority: 2000001000
  priorityClassName: system-node-critical
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: juicefs-csi-node-sa
  serviceAccountName: juicefs-c