Skip to main content
Version: v1.28.x

Troubleshooting Kubernetes environments

Troubleshooting Kubernetes environments

The following topics contain information that can help you troubleshoot problems when you encounter unexpected behavior installing and using Zowe™ containers in a Kubernetes environment.

ISSUE: /tmp directory is not writable

Problem:

readOnlyRootFilesystem SecurityContext is enabled by default in Deployment object definition. As a result, /tmp is read-only and not writable to zowe runtime user.

Recommended solution:

Adjust your component to check the TMPDIR or TMP environment variable to determine the location of the temporary directory. Zowe runtime customizes those variables and points them to /home/zowe/instance/tmp directory, which is writable.

Alternative solution:

Disabling readOnlyRootFilesystem SecurityContext is not recommended. But you can make /tmp writable by replacing it with a newly mounted volume. Here is an example of defining the /tmp volume.

apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
volumes:
- name: tmp
emptyDir: {}
containers:
- name: <my-component-name>
volumeMounts:
- name: tmp
mountPath: "/tmp"

With this added to your Deployment, your component should be able to write to /tmp directory.

ISSUE: Permission denied showing in pod log

Problem:

You see error messages similar to the following one in your pod log.

cp: cannot create regular file '/home/zowe/instance/workspace/manifest.json': Permission denied
mkdir: cannot create directory '/home/zowe/instance/workspace/api-mediation': Permission denied
mkdir: cannot create directory '/home/zowe/instance/workspace/backups': Permission denied
cp: cannot create regular file '/home/zowe/instance/workspace/active_configuration.cfg': Permission denied
/home/zowe/runtime/bin/internal/prepare-instance.sh: line 236: /home/zowe/instance/workspace/active_configuration.cfg: Permission denied
/home/zowe/runtime/bin/internal/prepare-instance.sh: line 240: /home/zowe/instance/workspace/active_configuration.cfg: Permission denied
/home/zowe/runtime/bin/internal/prepare-instance.sh: line 241: /home/zowe/instance/workspace/active_configuration.cfg: Permission denied

It means zowe user (UID 20000) does not have write permission to your persistent volume. It's very likely the persistent volume is mounted as root user.

Solution:

To solve this issue, you can modify workload files with extra initContainers step like this:

apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
initContainers:
- name: update-workspace-permission
image: busybox:1.28
command: ['sh', '-c', 'OWNER=`stat -c "%u:%g" /home/zowe/instance/workspace` && PERMISSION=`stat -c "%a" /home/zowe/instance/workspace` && echo "Zowe workspace owner is ${OWNER} with ${PERMISSION} permission" && if [ "${OWNER}" != "20000:20000" -a "${PERMISSION}" != "777" ]; then chown -R 20000:20000 /home/zowe/instance/workspace; fi']
imagePullPolicy: Always
resources:
requests:
memory: "64Mi"
cpu: "10m"
limits:
memory: "128Mi"
cpu: "100m"
volumeMounts:
- name: zowe-workspace
mountPath: "/home/zowe/instance/workspace"
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop:
- all
add:
- CHOWN
runAsUser: 0
runAsGroup: 0

ISSUE: Deployment and ReplicaSet failed to create pod

Problem:

If you are using OpenShift and see these error messages in ReplicaSet Events:

Generated from replicaset-controller
Error creating: pods "api-catalog-??????????-" is forbidden: unable to validate against any security context constraint: []

That means the Zowe ServiceAccount zowe-sa doesn't have any SecurityContextConstraint attached.

Solution:

You can run this command to grant a certain level of permission, for example, privileged, to zowe-sa ServiceAccount:

oc admin policy add-scc-to-user privileged -z zowe-sa -n zowe

ISSUE: Failed to create services

Problem:

If you are using OpenShift and apply services, you may see this error:

The Service "api-catalog-service" is invalid: spec.ports[0].appProtocol: Forbidden: This field can be enabled with the ServiceAppProtocol feature gate

Solution:

To fix this issue, you can simply find and comment out this line in the Service definition files:

appProtocol: https

With OpenShift, you can define a PassThrough Route to let Zowe handle TLS connections.