Installation Troubleshooting

Suggest Edits

Pods not starting

You may find that one or more KSOC pods are not running after installation. Below you can see that only one pod is running.

kubectl get pods -n ksoc
NAME                          READY   STATUS    RESTARTS   AGE
ksoc-guard-5c499786fd-b6mnr   2/2     Running   0          5m38s
ksoc-sbom-7f5d994f8-lfjmw     0/2     Pending   0          5m38s
ksoc-sync-6fcdfc5b9c-9z5m5    0/1     Pending   0          5m38s
ksoc-watch-58bd475f5-pkjx7    0/1     Pending   0          5m38s

Copy one of the pod names into the commandkubectl describe pod <podname> -n ksoc. Below we can see that the pod ksoc-sync-6fcdfc5b9c-9z5m5 has insufficient CPU in the bottom Events section.

kubectl describe pod ksoc-sync-6fcdfc5b9c-9z5m5 -n ksoc
Name:             ksoc-sync-6fcdfc5b9c-9z5m5
Namespace:        ksoc
Priority:         0
Service Account:  ksoc-sync
Node:             <none>
Labels:           app_name=ksoc-sync
                  app_version=v0.0.28
                  maintained_by=ksoc
                  pod-template-hash=6fcdfc5b9c
Annotations:      linkerd.io/inject: disabled
                  sidecar.istio.io/inject: false
Status:           Pending
IP:
IPs:              <none>
Controlled By:    ReplicaSet/ksoc-sync-6fcdfc5b9c
Init Containers:
  ksoc-bootstrapper:
    Image:      us.gcr.io/ksoc-public/ksoc-bootstrapper:v0.0.9
    Port:       <none>
    Host Port:  <none>
    Environment:
      AGENT_VERSION:       v0.0.9
      CHART_VERSION:       0.0.18
      KSOC_API_URL:        https://api.sbx.ksoc.com
      KSOC_CLUSTER_NAME:   Digital Ocean Sock Store Sandbox
      KSOC_NAMESPACE:      ksoc
      KSOC_ACCESS_KEY_ID:  <set to the key 'access-key-id' in secret 'ksoc-access-key'>  Optional: false
      KSOC_SECRET_KEY:     <set to the key 'secret-key' in secret 'ksoc-access-key'>     Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qc6gr (ro)
Containers:
  ksoc-sync:
    Image:      us.gcr.io/ksoc-public/ksoc-sync:v0.0.28
    Port:       <none>
    Host Port:  <none>
    Limits:
      cpu:     200m
      memory:  256Mi
    Requests:
      cpu:     100m
      memory:  128Mi
    Environment:
      AGENT_VERSION:       v0.0.28
      CHART_VERSION:       0.0.18
      KSOC_API_URL:        https://api.sbx.ksoc.com
      KSOC_NAMESPACE:      ksoc
      KSOC_ACCESS_KEY_ID:  <set to the key 'access-key-id' in secret 'ksoc-access-key'>  Optional: false
      KSOC_SECRET_KEY:     <set to the key 'secret-key' in secret 'ksoc-access-key'>     Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qc6gr (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  ksoc-configuration:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ksoc-plugins-configuration
    Optional:    false
  kube-api-access-qc6gr:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason             Age                From                Message
  ----     ------             ----               ----                -------
  Warning  FailedScheduling   6m1s               default-scheduler   0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
  Warning  FailedScheduling   52s                default-scheduler   0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
  Normal   NotTriggerScaleUp  1s (x6 over 5m2s)  cluster-autoscaler  pod didn't trigger scale-up:

The cluster had a single node assigned. Increasing the node count addressed the problem and the remaining KSOC pods automatically started once the additional nodes were available.

Verify User-Supplied Helm Values

You may find that the cluster name is not correct in the KSOC UI or other options such as policy blocking are not working as expected. One step in troubleshooting is to check the custom values set at the time of installation of the KSOC plugins.

helm -n ksoc get values ksoc

Example output:

helm -n ksoc get values ksoc
USER-SUPPLIED VALUES:
ksoc:
  apiUrl: https://api.prd.ksoc.com
  base64AccessKeyId: xxxxxxxxx=
  base64SecretKey: xxxxxxxxx==
  clusterName: MyClusterName
ksocGuard:
  config:
    BLOCK_ON_POLICY_VIOLATION: true

If you find the values are incorrect, edit values.yaml, delete the KSOC plugins, and reinstall them.

Delete KSOC Plugins

helm delete -n ksoc ksoc

Install KSOC Plugins

helm install  
  ksoc ksoc/ksoc-plugins  
  --namespace ksoc  
  -f values.yaml

Reverify User-Supplied Helm Values

helm -n ksoc get values ksoc
USER-SUPPLIED VALUES:
ksoc:
  apiUrl: https://api.prd.ksoc.com
  base64AccessKeyId: xxxxxxxxx=
  base64SecretKey: xxxxxxxxx==
  clusterName: MyUpdatedClusterName
ksocGuard:
  config:
    BLOCK_ON_POLICY_VIOLATION: true

Updated 30 days ago