Kubernetes "Failed create pod sandbox" error when deploying services on Linux/Windows cluster

When I try to deploy a service to my cluster I get the error: "Failed create pod sandbox." The status of the containers remains stuck on "ContainerCreating." I followed this tutorial: https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows

I have a two node Kubernetes cluster made up of a Windows Server 2016 worker node (named kubernetes) and a Ubuntu Server 16.04 LTS master node (named kuberL). Both instances are EC2's running on AWS.

kuber@kuberL:~$ kubectl get nodes
NAME         STATUS    ROLES     AGE       VERSION
kuberL  Ready     <none>    1h        v1.9.0-beta.1
kubernetes    Ready     <none>    1h        v1.10.0-alpha.0.865+73bb9810e0125a

kuber@kuberL:~$ kubectl get pods
NAME                             READY     STATUS              RESTARTS   AGE
iispod-5d75c55f46-vj2b8          0/1       ContainerCreating   0          18m
win-webserver-55566fdb4d-ncjnk   0/1       ContainerCreating   0          18m

Ouput of the command: kubectl describe pods

Name:           iispod-5d75c55f46-vj2b8
Namespace:      default
Node:           kubernetes/10.0.1.4
Start Time:     Tue, 19 Dec 2017 03:19:03 +0000
Labels:         pod-template-hash=1831711902
                run=iispod
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"iispod-5d75c55f46","uid":"5d27c9a3-e46b-11e7-ac05-000d3a145b63",...
Status:         Pending
IP:             
Controlled By:  ReplicaSet/iispod-5d75c55f46
Containers:
  iispod:
    Container ID:   
    Image:          microsoft/iis
    Image ID:       
    Port:           <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qchfs (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  default-token-qchfs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-qchfs
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     <none>
Events:
  Type     Reason                  Age               From                 Message
  ----     ------                  ----              ----                 -------
  Normal   Scheduled               22s               default-scheduler    Successfully assigned iispod-5d75c55f46-vj2b8 to kubernetes
  Normal   SuccessfulMountVolume   22s               kubelet, kubernetes  MountVolume.SetUp succeeded for volume "default-token-qchfs"
  Normal   SandboxChanged          2s (x7 over 19s)  kubelet, kubernetes  Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  1s (x8 over 21s)  kubelet, kubernetes  Failed create pod sandbox.

Name:           win-webserver-55566fdb4d-ncjnk
Namespace:      default
Node:           kubernetes/10.0.1.4
Start Time:     Tue, 19 Dec 2017 20:35:13 +0000
Labels:         app=win-webserver
                pod-template-hash=1112298608
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"win-webserver-55566fdb4d","uid":"d385ad85-e465-11e7-ac05-000d3a1...
Status:         Pending
IP:             
Controlled By:  ReplicaSet/win-webserver-55566fdb4d
Containers:
  windowswebserver:
    Container ID:  
    Image:         microsoft/windowsservercore
    Image ID:      
    Port:          <none>
    Command:
      powershell.exe
      -command
       {...omitted}
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qchfs (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  default-token-qchfs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-qchfs
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  beta.kubernetes.io/os=windows
Tolerations:     <none>
Events:
  Type     Reason                  Age                From                 Message
  ----     ------                  ----               ----                 -------
  Normal   Scheduled               5m                 default-scheduler    Successfully assigned win-webserver-55566fdb4d-ncjnk to kubernetes
  Normal   SuccessfulMountVolume   5m                 kubelet, kubernetes  MountVolume.SetUp succeeded for volume "default-token-qchfs"
  Warning  FailedCreatePodSandBox  2m (x12 over 5m)   kubelet, kubernetes  Failed create pod sandbox.
  Normal   SandboxChanged          20s (x59 over 5m)  kubelet, kubernetes  Pod sandbox changed, it will be killed and re-created.

On the worker node running kubelet I found the following in the output for both the win-server and iispod pods:

E1219 20:35:39.919989 5676 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to start sandbox container for pod "win-webserver-55566fdb4d-ncjnk": Error response from daemon: container 7286a6000122f336349952b0bab6b330cfeddd72d65b1176d4e21e30529c703c encountered an error during CreateContainer: failure in a Windows system call: The operating system of the container does not match the operating system of the host. (0xc0370101) extra info:

I was able to run the microsoft/iis image on the Windows Server 2016 node with no problem using: docker run -d --name myIIS -p 80:80 microsoft/iis

Any ideas on how to get the pods to deploy?

This is a Windows dockeree specific error, the pause image can't be run by as it was built with the wrong OS base image.

The current version of the url you were following has a section on creating the pause image using powershell. The line:

docker build -t kubeletwin/pause .

Builds the pause image (based on nanoserver or windowsservercore) and tags as kubeletwin/pause.

What the instructions omit is that once you've built the pause image, you can test it by running on the node: docker run kubeletwin/pause.

If you get a container/host OS mismatch:

• type dockerfile to get the base image type the image is using
• For a RTM OS build run (Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion").ReleaseId to get the 4 digit host OS version
• For an insider OS build, find the full build number e.g. 10.0.17677.1000, you will also have to add -insider to the base image type.

Then run the following to tag the correct OS base build to match the base image requested by the pause image:

docker pull microsoft\<base image type>:<version>
docker tag microsoft\<base image type>:<version> microsoft\<base image type>:latest
docker build .
docker run <image id from build>