7月31日 k8s实战

霍格沃兹测试学院校内交流
1

PPT地址

https://pdf.ceshiren.com/ck23/k8s/k8s与普罗米修斯.html

远程访问k8s集群

安装kubectl的文档：
https://kubernetes.io/zh-cn/docs/tasks/tools/install-kubectl-macos/

kubeconfig内容如下：

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeU1EY3pNVEEyTXpNeE1Wb1hEVE15TURjeU9EQTJNek14TVZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS3IvCmFrWkZsMnlqMkg5ZnJBdkNncU9UR1ZtTmpWQlo3NWpMQzhIYUJGYzUyemlmWFRkTGp5U3BWTUhZVUwrZFRjVnYKSTFqWDZmcXc2azJPMlkzS0FVYmQ4VlMrVURWNWlhWFRSODFpVmJzWXZXd3ZkZUkrZVlqZitETERpa1puak5wVQpRMWFkWEN6S2wzUjU2YzlxRlJESVpLUFNETkFuVVZFN2w5OXB5UjFUSC9TaWN6Z1E0M29aR3UrRUpCT1dUcHpYCld5RU9HK0o1NFJTaDlwUmg5Vk5EL1k1MnhuN0hzVC9NTFJ0cG1YY2lqTlFEQjFtSU1mNjkvL1JvTnNtRTFlOEgKaUpIYjEydUlodGZHTEt6cDZKaDdVbEQ1cjNqMmZjZFZKdUZnZUQzRVZvRFdtQXBXbWc0eXlyNFBCa083VlpRRgo1UlFjRnFxdWIxWS95eW5DK3VjQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFIcERIT2RJekhscm5peDgzblpXUDRJTHZUWFoKNkU1MjUraHB0bGRzU05TU3c1ZTltM3AzSGFqZ3FhNUptTWpwYXZ3TldnQXNzNEI5SFhFOFNEN3ZsbVVLTnpkMQpCK1pjQ3B1UG1DMUhTeUZwK29ZK1RQSlMvTWpQcEVHYUJrZ0VGckt3aWc2UXNMQ2MzdEdJemhIQmxHaEdaNjJsClpVNUNwRS9GQVlZd2dNN0VBdWY0Nkc3YXFuMGZxWXlscld3eGVBakRnLzFGbjVLN05QQVVGb3JETFcvRnA4cU8KaGg5R3JKdHpjb1ZOcnB5ZmhYeGN6dzIzbEQ2RHpmclNmZ1lSN2daVytjby8yYTlDV3lJaW1uT0dHZGovNjdFRAplaTFIVy9Ddk1uSzIyYlM1d2dJZ0VrQ2JQR241bHRHZVVhbExsaFg2VTU1dTJ1V2lyd01jaTdYdXAydz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://kubernetes:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4akNDQWRxZ0F3SUJBZ0lJTGJaa3VTVW1LZ1l3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TWpBM016RXdOak16TVRGYUZ3MHlNekEzTXpFd05qTXpNVE5hTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXpYS29zQ3NCZmlmQW5kZ2kKMkVsVlk3b0dNanQ3MW8wQ3FIaHAxNy9HS2YrU2krN3RoRUVQUVdoSXBLNkV1U25FRTlmTVRLTlhBVW5KOEs3WApyOTErWnM1ajFXdTlRZVZVOVB4a1RVelVwVTJrRmEwWFhMK0NXVW5WM2RlMWxLV1dGQmtPWHFLS0JoMUlMUGY5CkMzaUVyZ2k3aFJzck4zUlVXd2lFdU1WSkFzU0RzUHAyVEdSdDNXUUtCekRJL2FsekpLZFhiYTd5RjRzbTh2Q1QKVWxYTllPUDFxZS9ZcFVUMnFBS09XRnh5UGlDaUpVc0k3dUozMGU3WHZGTVdFeitLSzJNTDFvcDlaL0ZNc3pCVApabjl1VVEwUXVxWWZ6b3V5N29jN3V0dHJBRVlGN1pRNFVLTzRrZk92ZDdsSDhETW5OY0ZwcEZXM0EzRW94RlVJCjZqK0tYUUlEQVFBQm95Y3dKVEFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFIRDdoYXNPYnhXRyswc2tGOHhOUEdmTlNxQkgrT3dTSWZtMgpEVUdyOXgwdHZsR2JpMGkvc3cvUXZvcklvaDZ2R2I1ZVFtL0k1aEt1aWliMUo3VDdsa3FXRWxYRTZRUGtoOWdhCjZuWXo2c1ViVHBRV2RndTkyeFdOcU1MNFZ5aEU4WTQrakQwa3lacFY5emRIM0g0V1FTdjVEL1BndUpPS2JDZ1IKcU5yc3c2amlMWEQxaXBpSnlZYTRob0xUYjNUTlliUFFCcE93Nk5nTXRLbkZiZUpxVWtzTmNMbCtJRVQxWVJjQwo4Q3dNNzNMZEhpWkRGU3VRaXNXTzhKbCs5R0NLb29JWExkcGpTeVVZYU9GTFpuR01XUEdLTEVpT0lVbnJ5dFNqCmN2TWhrUVZjN0xTWEFRcFNmUFpIcDZaNEVhZGZ1ZURIWWtvZWZKYnFVZnNvNVVSMWNGcz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBelhLb3NDc0JmaWZBbmRnaTJFbFZZN29HTWp0NzFvMENxSGhwMTcvR0tmK1NpKzd0CmhFRVBRV2hJcEs2RXVTbkVFOWZNVEtOWEFVbko4SzdYcjkxK1pzNWoxV3U5UWVWVTlQeGtUVXpVcFUya0ZhMFgKWEwrQ1dVblYzZGUxbEtXV0ZCa09YcUtLQmgxSUxQZjlDM2lFcmdpN2hSc3JOM1JVV3dpRXVNVkpBc1NEc1BwMgpUR1J0M1dRS0J6REkvYWx6SktkWGJhN3lGNHNtOHZDVFVsWE5ZT1AxcWUvWXBVVDJxQUtPV0Z4eVBpQ2lKVXNJCjd1SjMwZTdYdkZNV0V6K0tLMk1MMW9wOVovRk1zekJUWm45dVVRMFF1cVlmem91eTdvYzd1dHRyQUVZRjdaUTQKVUtPNGtmT3ZkN2xIOERNbk5jRnBwRlczQTNFb3hGVUk2aitLWFFJREFRQUJBb0lCQUZrT3YvTUY0WExIVnFiVgp2MTBVaFBCaS9qYWVOSE52MHR3bXZuTG96Y1Q4NVpERk5LNms2VEVJUEowdWpaZ2FjOFhLUU9SajFkNVlGeksvCkIvNVRQR3ppUU1LT0hvOUhFMG5UY3NOVGI4WnNYcG1LMjlxM3BaQ0JBalZWUVlBN08xQmhBQ3FqdU1NbnhGYkIKOVM0WXd5TVI0Lyt5SkV5NG42eUtoOFgwdDA1OEM0c1FjRFA4dk9CSVF0azY2VjhWMS9McUN0MU1hVkppcVlNQQpqMlVMdkU3eXJyb1pyS3VaTmRpRjY3WER0SGd2d3JNYUs0NDlWcTI0djNXOTdaTU5RWEozYUMyclFHYW02dzk1CmQ1WDVVQTFXRkUyZGNWd2xaaGVKYkEzeVg5SDRvbG9zbjAwbXBWYU1yamFpNWlnVFpKLzRTNU5HOTlaUUg2b3QKcGpFYXE4RUNnWUVBKzlhaUJ4SVVNdmlvYUhSZktObGw5Rkl0Qkwydk9NMlY1eU84VXJ2bjdkWkpGUU9PSmZ2NQp6L2wrbGdxNVRBaXVhUWJXNXBYUjBXSmR1U2U1dFZGY1l4aVJqeU51aFcvUkNPejFMYzZvcXFDY0ZobUY2RWZZCnZjMm1QbG8vUEdva05zVkpwZWREdXY4YnM4OEtsSFhwUEFKUDJueFFwckZuaXk2dDZmOXJaOUVDZ1lFQTBOZkgKQkpONGxXU2d5UmVqTkFzQTFId3Z0a2JLVTZCRWgrM2xUU1RRZjNkd0xSOEZOVzRISE1EMFg3Z09ZWG9DczRCdgpGUHNHV2QzcUh5bHFoRWVpTFZ0WjIzTzRhOE92MDBHd2F4NnFkTG9iODhycHg4RXMwc0JOQjNDSnYwSTAxb2ZCCnFQZ0JVcnBibGMxWkhVSy9ZZ0J3MVZsQWtxZ0Y1elFmRjhxbzZNMENnWUVBMm04VUlxWGZ0MFpPYkN0anpuNmkKSEFTYTM1M1hPYUczLzcvVFBMK1hMNGZKNDl3dGsyeTNCVmtXSHZZMk9tamRQbU9jU214d2pQaVE2bVEyZzdxSwpNN20vcHpQSVRSenJacDZCR3JaQlBaNEpSSEhtMzg1UmhHKzNYaVhRNjRuMDF6elA0SUp6RU9MdEdFemdsRnloCmJGZGxBOUVIODAyMHpkZ3pLTjlPUGNFQ2dZQjdMc3MwNThjNFZ5Si9lbkhKUUxROG9RT0RnRDNYRmJGV2UvdkUKaFBKMlBQenRvRjV5SlhRTWdjT2lzZTNKa1M0R3NyclNwVDRadzh0VWxoazk3UUs3dmlaQ1N2OThrUEFrcTVwagpsTEwySnRCVi9Gbm9lUWozUm5ZbzdUaFNsN3FTejRCcktkMmQ3azRYRGdpQ09keW84Wm1jUGdqWWVFTURITHM2ClRVOVVBUUtCZ0Q5RlNxcy9nVm1Ub0ZaNURpSjJ3UXhqYWNvNTk2ZkljdTlMQTdIK2NzeEluNitwUWJVa2NrWTMKWEZvcFUvTGVtN3lPcFZ5WFVmdUE2ODczelJxNzBLZmVYaHFSa0ZYV3NYMEY5Z3JtWVZ1WlNteHZHNGxTTkFFeAp4ZVNWeTBIQjZTZEdScHVuKzVkVms5VGtvckZUd2FGMGI4VmI3Z0JUb0hOdnhkQ0Y2NHpvCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==

放到家目录下的：~/.kube/confg

在/etc/hosts里添加一条记录：
47.93.32.161 kubernetes

centos安装k8s集群

配置源

/etc/yum.repos.d/kubernetes.repo

[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

安装工具

yum install -y kubelet-1.17.17-0 kubectl-1.17.17-0 kubeadm-1.17.17-0

初始化集群

kubeadm init --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.17.17 --pod-network-cidr=10.244.0.0/16

安装网络插件

kubectl create -f https://docs.projectcalico.org/archive/v3.17/manifests/tigera-operator.yaml

删除master节点上的污点

kubectl get nodes 查看节点
kubectl taint nodes 节点名称 node-role.kubernetes.io/master:NoSchedule-

在k8s中部署服务

apiVersion: apps/v1
kind: Deployment
metadata:
  name: selenium-hub
  labels:
    name: selenium-hub
spec:
  replicas: 1
  selector:
    matchLabels:
      name: selenium-hub
  template:
    metadata:
      labels:
        name: selenium-hub
    spec:
      containers:
        - name: selenium-hub
          image: selenium/hub:4.0.0-rc-2-prerelease-20210923
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 4444
            - containerPort: 4442
            - containerPort: 4443
            - containerPort: 5557
          env:
            - name: TZ
              value: "Asia/Shanghai"
          volumeMounts:
            - mountPath: "/etc/localtime"
              name: "host-time"
          livenessProbe:
            httpGet:
              path: /grid/console
              port: 4444
            initialDelaySeconds: 30
            timeoutSeconds: 1
            periodSeconds: 5
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /grid/console
              port: 4444
            initialDelaySeconds: 30
            timeoutSeconds: 1
            periodSeconds: 5
            failureThreshold: 3
      volumes:
        - name: "host-time"
          hostPath:
            path: "/etc/localtime"
---
apiVersion: v1
kind: Service
metadata:
  name: selenium-hub
  labels:
    name: selenium-hub
spec:
  type: NodePort
  ports:
    - name: port1
      protocol: TCP
      port: 4442
      targetPort: 4442
    - name: port2
      protocol: TCP
      port: 4443
      targetPort: 4443
    - name: port3
      protocol: TCP
      port: 5557
      targetPort: 5557
    - port: 4444
      targetPort: 4444
      name: port0
      nodePort: 32757
  selector:
    name: selenium-hub
  sessionAffinity: None

node:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: selenium-node-chrome
  labels:
    name: selenium-node-chrome
spec:
  replicas: 2
  selector:
    matchLabels:
      name: selenium-node-chrome
  template:
    metadata:
      labels:
        name: selenium-node-chrome
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels:
                  name: selenium-node-chrome
      containers:
        - name: selenium-node-chrome
          image: selenium/node-chrome:4.0.0-rc-2-prerelease-20210923
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5900
            - containerPort: 5553
          env:
            - name: SE_EVENT_BUS_HOST
              value: "selenium-hub"
            - name: SE_EVENT_BUS_PUBLISH_PORT
              value: "4442"
            - name: SE_EVENT_BUS_SUBSCRIBE_PORT
              value: "4443"
            - name: SE_NODE_MAX_SESSIONS
              value: "20"
            - name: SE_NODE_OVERRIDE_MAX_SESSIONS
              value: "true"
            - name: TZ
              value: "Asia/Shanghai"
          resources:
            requests:
              memory: "500Mi"
          volumeMounts:
            - mountPath: "/dev/shm"
              name: "dshm"
            - mountPath: "/etc/localtime"
              name: "host-time"
      volumes:
        - name: "dshm"
          hostPath:
            path: "/dev/shm"
        - name: "host-time"
          hostPath:
            path: "/etc/localtime"
---
apiVersion: v1
kind: Service
metadata:
  name: selenium-node-chrome
  labels:
    name: selenium-node-chrome
spec:
  type: NodePort
  ports:
    - port: 5900
      targetPort: 5900
      name: port0
      nodePort: 31002
  selector:
    name: selenium-node-chrome
  sessionAffinity: None

python client

文档：GitHub - kubernetes-client/python: Official Python client library for kubernetes
https://github.com/kubernetes-client/python/blob/master/kubernetes/README.md

k8s client

import os

from kubernetes.client import V1Deployment, V1Service, V1ConfigMap, V1Secret, V1Pod
from kubernetes import client, config
from kubernetes.stream import stream


# ns = client.CoreV1Api().list_namespace()
# print(ns)


class K8SClient(object):
    def __init__(self, kube_config_path=None):
        if os.path.exists(kube_config_path):
            self.kube_config_path = kube_config_path
            config.load_kube_config(kube_config_path)
        else:
            config.load_incluster_config()

        self.corev1 = client.CoreV1Api()
        self.appsv1 = client.AppsV1Api()

    def get_deployment_info(self, deployment_name, namespace) -> V1Deployment:
        deployment_list = self.appsv1.list_namespaced_deployment(namespace)
        for d in deployment_list.items:
            if deployment_name in d.metadata.name:
                return d

    def get_service_into(self, service_name, namespace) -> V1Service:
        return self.corev1.read_namespaced_service(service_name, namespace, pretty=True)

    def get_configmap_info(self, configmap_name, namespace) -> V1ConfigMap:
        return self.corev1.read_namespaced_config_map(name=configmap_name, namespace=namespace, pretty=True)

    def get_secret_info(self, secret_name, namespace) -> V1Secret:
        return self.corev1.read_namespaced_secret(secret_name, namespace, pretty=True)

    def pod_exec_info(self, pod_name, namespace, exec_command, container_name):
        return stream(self.corev1.connect_get_namespaced_pod_exec, pod_name, namespace, container=container_name,
                      command=exec_command, stderr=True, stdin=False, stdout=True, tty=False)


def get_master_k8s_client():
    return K8SClient("/Users/cainsun/.kube/config")
#
#
# def get_edge_k8s_client():
#     return K8SClient(resource.get_file('kube_config/master_config'))

test.py:

from k8s import K8SClient, get_master_k8s_client

import os

if __name__ == '__main__':
    client = get_master_k8s_client()
    # namespaces = client.corev1.list_namespace()
    # for ns in namespaces.items:
    #     pods = client.corev1.list_namespaced_pod(ns.metadata.name)
    #     for p in pods.items:
    #         # print(p.spec)
    #         print(p.metadata.name)

    deploys = client.appsv1.list_deployment_for_all_namespaces()
    for d in deploys.items:
        # print(d)
        # print(d.spec.template.spec.containers)
        # print(d.spec.replicas)
        for c in d.spec.template.spec.containers:
            if c.readiness_probe is None:
                print("%s 没有就绪探针" % d.metadata.name)

k8s 故障注入

安装文档：https://chaos-mesh.org/zh/docs/production-installation-using-helm/

手动注入故障：

  1. kubectl get pod -o wide 找到容器对应的节点
  2. 登陆到该节点中。 用docker ps命令 找到该容器
  3. docker inspect 容器名称 查询该容器的pid
  4. nsenter -t pid -n 进入容器的网络名称空间
  5. iptables/tc 注入故障。