Minikube apiserver stopped frequently

0

I am using minikube v1.11.0 on Microsoft Windows 10 Pro . my minikube stopped frequently ,Find minikube status

Minikube Status:

type: Control Plane
host: Running
kubelet: Running
apiserver: Stopped
kubeconfig: Configured

minikube logs:

Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

Minikube Details:

Minikube Version: 1.11.0
Assigned Memory : 4096
Processor :4 virtual processors

'minikube ssh dmesg' command Result:

D:\IMP\DevOps Implementation\Python>minikube ssh dmesg
[    0.000000] Linux version 4.19.107 (jenkins@jenkins) (gcc version 7.4.0 (Buildroot 2019.02.10)) #1 SMP Thu May 28 15:07:17 PDT 2020
[    0.000000] Command line: BOOT_IMAGE=/boot/bzImage root=/dev/sr0 loglevel=3 console=ttyS0 noembed nomodeset norestore waitusb=10 random.trust_cpu=on hw_rng_model=virtio systemd.legacy_systemd_cgroup_controller=yes initrd=/boot/initrd
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: xstate_offset[3]:  832, xstate_sizes[3]:   64
[    0.000000] x86/fpu: xstate_offset[4]:  896, xstate_sizes[4]:   64
[    0.000000] x86/fpu: Enabled xstate features 0x1f, context size is 960 bytes, using 'compacted' format.
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000f7feffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000f7ff0000-0x00000000f7ffefff] ACPI data
[    0.000000] BIOS-e820: [mem 0x00000000f7fff000-0x00000000f7ffffff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000101ffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.3 present.
[    0.000000] DMI: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
[    0.000000] Hypervisor detected: Microsoft Hyper-V
[    0.000000] Hyper-V: features 0x2e7f, hints 0x20c2c
[    0.000000] Hyper-V Host Build:18362-10.0-0-0.836
[    0.000000] Hyper-V: LAPIC Timer Frequency: 0x30d40
[    0.000000] tsc: Marking TSC unstable due to running on Hyper-V
[    0.000000] Hyper-V: Using hypercall for remote TLB flush
[    0.000000] tsc: Detected 1800.006 MHz processor
[    0.000686] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000687] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000690] last_pfn = 0x102000 max_arch_pfn = 0x400000000
[    0.000705] MTRR default type: uncachable
[    0.000706] MTRR fixed ranges enabled:
[    0.000707]   00000-9FFFF write-back
[    0.000707]   A0000-DFFFF uncachable
[    0.000707]   E0000-FFFFF write-back
[    0.000708] MTRR variable ranges enabled:
[    0.000709]   0 base 0000000000 mask 7F00000000 write-back
[    0.000709]   1 base 0100000000 mask 7000000000 write-back
[    0.000709]   2 disabled
[    0.000710]   3 disabled
[    0.000710]   4 disabled
[    0.000710]   5 disabled
[    0.000710]   6 disabled
[    0.000711]   7 disabled
[    0.000718] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT
[    0.000727] last_pfn = 0xf7ff0 max_arch_pfn = 0x400000000
[    0.006501] found SMP MP-table at [mem 0x000ff780-0x000ff78f]
[    0.006515] Scanning 1 areas for low memory corruption
[    0.006583] Using GB pages for direct mapping
[    0.006586] BRK [0x8da02000, 0x8da02fff] PGTABLE
[    0.006587] BRK [0x8da03000, 0x8da03fff] PGTABLE
[    0.006588] BRK [0x8da04000, 0x8da04fff] PGTABLE
[    0.006599] BRK [0x8da05000, 0x8da05fff] PGTABLE
[    0.006600] BRK [0x8da06000, 0x8da06fff] PGTABLE
[    0.006625] BRK [0x8da07000, 0x8da07fff] PGTABLE
[    0.006632] BRK [0x8da08000, 0x8da08fff] PGTABLE
[    0.006669] RAMDISK: [mem 0x75db4000-0x7fffffff]
[    0.006709] ACPI: Early table checksum verification disabled

[    3.211586] Freeing unused kernel image memory: 1428K
[    3.217804] Write protecting the kernel read-only data: 20480k
[    3.218711] Freeing unused kernel image memory: 2004K
[    3.218811] Freeing unused kernel image memory: 648K
[    3.218813] Run /init as init process
[    3.420796] psmouse serio1: trackpoint: failed to get extended button data, assuming 3 buttons
[    3.606951] tar (1220) used greatest stack depth: 14064 bytes left
[    3.658466] systemd[1]: systemd 240 running in system mode. (-PAM -AUDIT -SELINUX -IMA -APPARMOR -SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP -GCRYPT -GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 -IDN -PCRE2 default-hierarchy=hybrid)
[    3.658536] systemd[1]: Detected virtualization microsoft.
[    3.658540] systemd[1]: Detected architecture x86-64.
[    3.667515] systemd[1]: Set hostname to <minikube>.
[    3.667543] systemd[1]: Initializing machine ID from random generator.
[    3.667835] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid argument
[    3.678215] systemd-fstab-generator[1225]: Ignoring "noauto" for root device
[    3.679587] systemd[1]: File /usr/lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
[    3.679589] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[    3.684942] systemd[1]: /usr/lib/systemd/system/vmtoolsd.service:7: PIDFile= references path below legacy directory /var/run/, updating /var/run/vmtoolsd.pid \xe2\x86\x92 /run/vmtoolsd.pid; please update the unit file accordingly.
[    3.688167] systemd[1]: /usr/lib/systemd/system/rpc-statd.service:13: PIDFile= references path below legacy directory /var/run/, updating /var/run/rpc.statd.pid \xe2\x86\x92 /run/rpc.statd.pid; please update the unit file accordingly.
[    4.077349] systemd-journald[1482]: Received request to flush runtime journal from PID 1
[    4.082397] journalctl (1875) used greatest stack depth: 14032 bytes left
[    4.328094] hv_vmbus: Vmbus version:5.0
[    4.346743] hv_vmbus: registering driver hid_hyperv
[    4.347880] input: Microsoft Vmbus HID-compliant Mouse as /devices/0006:045E:0621.0001/input/input4
[    4.347953] hid-generic 0006:045E:0621.0001: input: <UNKNOWN> HID v0.01 Mouse [Microsoft Vmbus HID-compliant Mouse] on
[    4.347953] hid-generic 0006:045E:0621.0001: input: <UNKNOWN> HID v0.01 Mouse [Microsoft Vmbus HID-compliant Mouse] on
[    4.348317] hv_vmbus: registering driver hv_storvsc
[    4.349279] hv_vmbus: registering driver hyperv_keyboard
[    4.350678] hv_utils: Registering HyperV Utility Driver
[   52.468905] hv_balloon: Max. dynamic memory size: 4000 MB
[   96.728628] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   96.729009] Bridge firewalling registered
[   96.734466] audit: type=1325 audit(1594295640.425:2): table=nat family=2 entries=0
[   96.734568] audit: type=1300 audit(1594295640.425:2): arch=c000003e syscall=313 success=yes exit=0 a0=5 a1=41a8e6 a2=0 a3=5 items=0 ppid=1078 pid=2441 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmod" subj=kernel key=(null)
[   96.734671] audit: type=1327 audit(1594295640.425:2): proctitle=2F7362696E2F6D6F6470726F6265002D71002D2D0069707461626C655F6E6174
[   96.753305] audit: type=1325 audit(1594295640.444:3): table=nat family=2 entries=5
[   96.753308] audit: type=1300 audit(1594295640.444:3): arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=1a1ea60 items=0 ppid=2350 pid=2470 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-legacy-multi" subj=kernel key=(null)
[   96.753309] audit: type=1327 audit(1594295640.444:3): proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D74006E6174002D4E00444F434B4552
[  355.252138] audit: type=1325 audit(1594295898.980:72): table=filter family=2 entries=30
[  355.252142] audit: type=1300 audit(1594295898.980:72): arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=207d050 items=0 ppid=2350 pid=6087 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-legacy-multi" subj=kernel key=(null)
[  355.252144] audit: type=1327 audit(1594295898.980:72): proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D740066696C746572002D4100444F434B45520000002D6900646F636B657230002D6F00646F636B657230002D7000746370002D64003137322E31372E302E34002D2D64706F727400343433002D6A00414343455054
[  569.369364] docker0: port 13(veth97c2039) entered disabled state
[  569.369455] audit: type=1700 audit(1594296113.095:87): dev=veth97c2039 prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
[  569.377656] audit: type=1300 audit(1594296113.095:87): arch=c000003e syscall=44 success=yes exit=32 a0=e a1=c002858760 a2=20 a3=0 items=0 ppid=1 pid=2350 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="dockerd" exe="/usr/bin/dockerd" subj=kernel key=(null)
[  569.377659] audit: type=1327 audit(1594296113.095:87): proctitle=2F7573722F62696E2F646F636B657264002D48007463703A2F2F302E302E302E303A32333736002D4800756E69783A2F2F2F7661722F72756E2F646F636B65722E736F636B002D2D64656661756C742D756C696D69743D6E6F66696C653D313034383537363A31303438353736002D2D746C73766572696679002D2D746C7363
[  575.768707] audit: type=1325 audit(1594296119.496:88): table=nat family=2 entries=27
[  575.768710] audit: type=1300 audit(1594296119.496:88): arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=10a24b0 items=0 ppid=2350 pid=8742 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-legacy-multi" subj=kernel key=(null)
[  575.768712] audit: type=1327 audit(1594296119.496:88): proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D74006E6174002D4400444F434B4552002D7000746370002D6400302F30002D2D64706F727400343433002D6A00444E4154002D2D746F2D64657374696E6174696F6E003137322E31372E302E343A3434330000002D6900646F636B657230
[  575.770459] audit: type=1325 audit(1594296119.498:89): table=filter family=2 entries=32
[  575.770461] audit: type=1300 audit(1594296119.498:89): arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=17ad4d0 items=0 ppid=2350 pid=8744 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-legacy-multi" subj=kernel key=(null)
[  575.770463] audit: type=1327 audit(1594296119.498:89): proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D740066696C746572002D4400444F434B45520000002D6900646F636B657230002D6F00646F636B657230002D7000746370002D64003137322E31372E302E34002D2D64706F727400343433002D6A00414343455054
[  575.772344] audit: type=1325 audit(1594296119.500:90): table=nat family=2 entries=26
[  575.772356] audit: type=1300 audit(1594296119.500:90): arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=814150 items=0 ppid=2350 pid=8746 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-legacy-multi" subj=kernel key=(null)
[  575.772358] audit: type=1327 audit(1594296119.500:90): proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D74006E6174002D4400504F5354524F5554494E47002D7000746370002D73003137322E31372E302E34002D64003137322E31372E302E34002D2D64706F727400343433002D6A004D415351554552414445
[  575.775023] audit: type=1325 audit(1594296119.503:91): table=nat family=2 entries=25
[  576.333512] veth3126200: renamed from eth0
[  576.342533] docker0: port 1(vethb62d4b0) entered disabled state
[  576.518935] docker0: port 2(vetha808974) entered disabled state
[  576.521396] vethe98d518: renamed from eth0
[  576.708284] docker0: port 3(vethc4912f9) entered disabled state
[  576.709295] vethbd779ad: renamed from eth0
[  579.921427] docker0: port 1(vethb62d4b0) entered disabled state
[  579.926503] device vethb62d4b0 left promiscuous mode
[  579.926508] docker0: port 1(vethb62d4b0) entered disabled state
[  580.745101] docker0: port 2(vetha808974) entered disabled state
[  580.748501] device vetha808974 left promiscuous mode
[  580.748514] docker0: port 2(vetha808974) entered disabled state
[  582.042311] docker0: port 3(vethc4912f9) entered disabled state
[  582.047229] device vethc4912f9 left promiscuous mode
[  582.047236] docker0: port 3(vethc4912f9) entered disabled state
[  582.047260] kauditd_printk_skb: 14 callbacks suppressed
[  582.047261] audit: type=1700 audit(1594296125.769:96): dev=vethc4912f9 prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
[  582.050812] audit: type=1300 audit(1594296125.769:96): arch=c000003e syscall=44 success=yes exit=32 a0=e a1=c001dcf7e0 a2=20 a3=0 items=0 ppid=1 pid=2350 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="dockerd" exe="/usr/bin/dockerd" subj=kernel key=(null)
[  582.050918] audit: type=1327 audit(1594296125.769:96): proctitle=2F7573722F62696E2F646F636B657264002D48007463703A2F2F302E302E302E303A32333736002D4800756E69783A2F2F2F7661722F72756E2F646F636B65722E736F636B002D2D64656661756C742D756C696D69743D6E6F66696C653D313034383537363A31303438353736002D2D746C73766572696679002D2D746C7363
kubernetes
minikube
asked on Stack Overflow Jul 8, 2020 by raviteja • edited Jul 9, 2020 by raviteja

1 Answer

0

Based on the info you provided the most common cause of this issue might be the lack of resources which would lead to the apiserver being evicted. There are few things to look at:

  1. Increase the amount of allocated memory and CPU to make sure it is enough for your use case.

  2. Check if you have enough storage at /dev/sda1. Hitting a threshold might cause some of the processes to be evicted, apiserver included.

  3. If this is still not enough, try to get more logs by executing: minikube ssh 'docker logs $(docker ps -a -f name=k8s_kube-api --format={{.ID}})'

  4. Eventually you can restart your minikube with minikube stop and minikube start (you can add minikube delete in between if you want to start from scrach). If problem will return, try to catch its cause with journalctl.

Please let me know if that helped.

answered on Stack Overflow Jul 16, 2020 by OhHiMark

User contributions licensed under CC BY-SA 3.0