远程执行Shell命令

安装 Fabric

pip3 install fabric

远程执行 Shell 命令脚本（remote_execute_shell_command.py） #!/usr/bin/python import argparse from fabric import Connection, Config # 您要远程操作的计算机，username@ip HOSTS = ['root@192.168.0.1', 'root@192.168.0.2'] PASSWORDS = ['admin', 'admin'] if name == 'main': parser = argparse.ArgumentParser() parser.add_argument('-c', '--command', type=str, help='execute shell command.') args = parser.parse_args() if not args.command: args.command = 'uname -a' print('➜ Execute shell command: ', args.

fabricsshremote-executionpythonshellautomationscripting

安装Kubernetes Dashboard

用户界面 Dashboard

Dashboard 是基于网页的 Kubernetes 用户界面。您可以使用 Dashboard 将容器应用部署到 Kubernetes 集群中，也可以对容器应用排错，还能管理集群资源。您可以使用 Dashboard 获取运行在集群中的应用的概览信息，也可以创建或者修改 Kubernetes 资源（如 Deployment，Job，DaemonSet 等等）。例如，您可以对 Deployment 实现弹性伸缩、发起滚动升级、重启 Pod 或者使用向导创建新的应用。

部署 Dashboard

安装 Dashboard

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.4.0/aio/deploy/recommended.yaml

过几分钟后您将看到 kubernetes-dashboard 名字空间下会有 Pod 运行。

kubernetesdashboardkubectlcluster-setuprbacserviceaccountnodeportproxymacos

安装Kubernetes 1.21.5

删除 Kubernetes 旧版本，安装 Kubernetes 1.21.5。

必备安装

每台服务器都需要

root 登录

su - root

设置时区

timedatectl set-timezone Asia/Shanghai

设置主机名

hostnamectl set-hostname new-hostname

安装 Docker-CE

apt-get -y update
apt-get -y install apt-transport-https ca-certificates curl software-properties-common

# 安装GPG证书
curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | apt-key add -

# 写入软件源信息
add-apt-repository "deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"

# 更新并安装
apt-get -y update
apt-get -y install docker-ce docker-ce-cli containerd.io
kuberneteskubeadmkubectldockerinstallationcluster-setupmaster-nodeworker-nodeweave-netcgroups

使用PaddleOCR进行文字识别

安装

pip install paddleocr

测试

import cv2
import numpy as np

from paddleocr import PaddleOCR


ocr = PaddleOCR(use_angle_cls=True)
image_path = 'test.jpg'
img = cv2.imread(image_path)

img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
img_gray1 = img_gray[:,:, np.newaxis]
img_gray3 = np.concatenate([img_gray1, img_gray1, img_gray1], axis=-1)

texts = ocr.ocr(img_gray3)
for text in texts:
    """
    box   坐标1　　　　　　　　　坐标2
          坐标4　　　　　　　　　坐标3
    """
    box = text[0]
    t = text[1][0]
    score = text[1][1]

可视化（图像上画出文本和得分） import os import shutil import cv2 import numpy as np import uuid from PIL import ImageFo

pythonpaddleocrpaddlepaddleopencvcomputer-visionocrimage-processingdeep-learninginference

在 Python 中解析和修改 XML

XML 数据

xml_data = '''<?xml version="1.0" encoding="UTF-8"?>
<books>
    <book SN="12460901">
        <name>国富论</name>
        <author nation="English">亚当·斯密</author>
        <price>108</price>
        <ISBN>9787511373229</ISBN>
    </book>
    <book SN="12257413">
        <name>原则</name>
        <author nation="America">瑞·达利欧</author>
        <price>98</price>
        <ISBN>9787508684031</ISBN>
    </book>
</books>
'''

xml.etree.ElementTree 导入 import xml.

pythonxmlelementtreeparsingstdlib

OpenCV Python实践

安装

Python

sudo apt install python3
sudo apt install python3-pip
sudo pip3 install --upgrade pip

OpenCV

sudo pip3 install opencv-python
sudo pip3 install opencv-contrib-python

图像

读取图像

import cv2

img_file = 'python-logo@2x.png'
img = cv2.imread(img_file)

获取图像大小

width = img.shape[1]
height = img.shape[0]

显示图像并等待按任意键退出

import cv2

img_file = 'python-logo@2x.png'
img = cv2.imread(img_file)
cv2.imshow('', img)
cv2.waitKey(0)

宽高缩小一倍 比例 import cv2 img_file = 'python-logo@2x.png' img = cv2.imread(img_file) img = cv2.resize(img, None, fx=0.5, fy=0.5) cv2.

pythonopencvcomputer-visionimage-processingnumpypiltkintervideochinesegui

使用 Python 临时文件模块

file.write() 内容超过 4K 才会写入磁盘

验证代码

import os
import tempfile

for count in range(1, 4100):
    content = '0'*count
    with tempfile.NamedTemporaryFile() as file:
        print(file.name)
        file.write(content.encode())

        with open(file.name, 'r') as tf:
            content_len = len(tf.read())
            if content_len > 0:
                print(f'{count} bytes written successfully.')

运行结果 /var/folders/bc/7lz308t90gb1h1xw6k4j65x80000gn/T/tmpj458ozas /var/folders/bc/7lz308t90gb1h1xw6k4j65x80000gn/T/tmpmrxo8sg1 /var/folders/bc/7lz308t90gb1h1xw6k4j65x80000gn/T/tmp0hu_i4hz 4097 bytes written successfully.

pythontempfilefile-iostdlibfile

Kubernetes中的计算资源管理

查看节点资源总量

$ kubectl describe nodes ln1

Name:               ln1
...
Capacity:
  cpu:                40
  ephemeral-storage:  574345384Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             264015508Ki
  pods:               110
Allocatable:
  cpu:                40
  ephemeral-storage:  529316705019
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             263913108Ki
  pods:               110
  • Capacity, 节点的资源总量
  • Allocatable, 可分配给 Pod 的资源量

Pod 中的容器申请资源 定义 Pod，通过容器指定 CPU 和内存的资源请求量 #requests-pod.

kubernetesresource-managementresourceslimitrangeresourcequotaqospodkubectlgpu

Docker 网络

ifconfig & route

$ docker run -d --name centos1 centos
$ docker run -it --name centos2 centos bash
# yum install net-tools -y

# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.3  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:ac:11:00:03  txqueuelen 0  (Ethernet)
        RX packets 3475  bytes 14166022 (13.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2988  bytes 214839 (209.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         _gateway        0.0.0.0         UG    0      0        0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth0
dockercontainer-networknetworkingbridgelinux

iptables

规则

添加规则

iptables -t raw -A OUTPUT -p icmp -j TRACE
iptables -t raw -A PREROUTING -p icmp -j TRACE

查看规则

# iptables -t raw -L -n --line-numbers 
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    TRACE      icmp --  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    TRACE      icmp --  0.0.0.0/0            0.0.0.0/0

删除规则

iptables -t raw -D PREROUTING 1
iptables -t raw -D OUTPUT 1

参考资料

iptablesfirewallnetworkinglinuxsecurity

Docker SDK for Python Examples

安装 Docker SDK for Python

pip install docker

例子

将本地文件或目录添加到容器，生成新的镜像。

  • put_archive
  • commit
import docker
import tarfile
import tempfile
import os

def simple_tar(path):
    f = tempfile.NamedTemporaryFile()
    t = tarfile.open(mode='w', fileobj=f)
    abs_path = os.path.abspath(path)
    t.add(abs_path, arcname=os.path.basename(path))
    t.close()
    f.seek(0)
    return f

client = docker.from_env()
// ...

参考资料

dockerpythonsdkcontainer-managementtempfilescripting

如何使用 Docker 打包已注册的模型

模型打包成镜像

手工打包

project_dir=platen-switch
darknet_model_name=darknet-model-platen-switch

cd $project_dir
docker run -d --name $darknet_model_name alpine
docker cp model/ $darknet_model_name:/
docker commit -a 'wang-junjian@qq.com' -m 'darknet model [platen-switch recognition]' \
  $darknet_model_name gouchicao/$darknet_model_name:latest
docker rm -v $darknet_model_name

docker push gouchicao/$darknet_model_name:latest

Python 脚本 import docker import tarfile import tempfile import os def simple_tar(path): f = tempfile.NamedTemporaryFile() t = tarfile.

dockerkubernetesmodel-deploymentinference-servingdarknetinit-containerspythonsdkcontainer-management

Kubernetes JSONPath实践

Node

标签信息

查看单个节点

$ kubectl get node ln1 -o=jsonpath='{.metadata.labels}{"\n"}'
map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:ln1 kubernetes.io/os:linux node-role.kubernetes.io/master:]

查看集群所有节点 $ kubectl get nodes -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels}{"\n"}{end}' gpu1 map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux k8s.amazonaws.com/accelerator:vgpu kubernetes.io/arch:amd64 kubernetes.io/hostname:gpu1 kubernetes.

kuberneteskubectljsonpathnode-managementlabelstaintpodquery

minikube

安装

安装 kubectl

curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x ./kubectl
mv ./kubectl /usr/local/bin/kubectl

安装 minikube

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
install minikube-linux-amd64 /usr/local/bin/minikube

启动 Kubernetes 集群 # minikube start --force 😄 Centos 8.1.1911 (amd64) 上的 minikube v1.22.

kuberneteskubectldockerminikubecluster-setuplocal-developmentaddonscentos

在Kubernetes上安装Ingress

官方的安装

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx

在中国

  1. 下载 [Ingress Release][ingress-nginx-releases]
wget https://github.com/kubernetes/ingress-nginx/releases/download/ingress-nginx-3.3.0/ingress-nginx-3.3.0.tgz
  1. 解压
tar xzvf ingress-nginx-3.3.0.tgz
cd ingress-nginx

替换镜像 vim values.yaml repository: k8s.gcr.

kubernetesingressnginxhelmnetworkingload-balancingcontainer-networkinstallation

Kubernetes API 服务器的安全防护

认证机制

API 服务器接收的请求经过认证插件列表，当遇到第一个成功返回用户名、用户ID和组信息给 API 服务器后，将停止剩余的认证调用，进入授权阶段。

ServiceAccount

镜像拉取密钥

向 ServiceAccount 中添加 imagePullSecrets，可以不必对每个 Pod 单独添加 imagePullSecrets。

创建 docker-registry Secret

kubectl create secret docker-registry mydockerhubsecret \
  --docker-username=myusername \
  --docker-password=mypassword \
  --docker-email=myemail

编写 ServiceAccount YAML 文件（sa-image-pull-secret.yaml）

apiVersion: v1
kind: ServiceAccount
metadata:
  name: image-pull-secret
imagePullSecrets:
- name: mydockerhubsecret

创建 ServiceAccount 对象

kubectl apply -f sa-image-pull-secret.yaml

编写 Pod YAML 文件（private-serviceacc

kubernetessecurityauthenticationrbacserviceaccountauthorizationprivate-registrysecret

Kubernetes中如何恢复误删的节点

在删除标签的时候不小心删除了节点。

$ kubectl delete nodes ln6 server-type-
node "ln6" deleted
Error from server (NotFound): nodes "server-type-" not found

主节点

在 Master 节点创建加入节点用的 Token

kubeadm token create --print-join-command
W0706 08:09:27.126498  609025 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm join 172.16.33.157:6443 --token xxx.yyyyyy     --discovery-token-ca-cert-hash sha256:zzzzzzzzzzzzzzzzzzzzzz

工作节点（被删除的）

登录 root 用户

su - root

重置 # kubeadm reset [reset] WARNING: Changes made to this host by &#3

kuberneteskubectlkubeadmnode-managementnode-recoverycluster-operationsworker-nodemaster-node

ConfigMap和Secret：配置应用程序

ConfigMap

向容器传递命令行参数

在 Docker 中定义命令与参数

下面是 Dockerfile 中的指令 ENTRYPOINT 和 CMD。

指令 解释
ENTRYPOINT 容器启动时调用的命令
CMD 传递给 ENTRYPOINT 指定命令的参数

构建带参数的程序（date:args）

编写脚本 date.sh

#!/bin/sh
INTERVAL=$1
while :
do
  echo $(date)
  sleep $INTERVAL
done

编写 Dockerfile

FROM busybox
ADD date.sh /date.sh
RUN chmod +x /date.sh
ENTRYPOINT ["/date.sh"]
CMD ["1"]

参数默认值为 1，在运行容器可以设置参数覆盖默认值。

构建镜像

docker build -t wangjunjian/date:args .

在 Docker 中覆盖命令和参数

使用 Docker 运行镜像来设置命令和参数

docker run [--entrypoint=] <image> [arg1, arg2, arg3]

启动镜像 $ docker run wangjunjian/date:args Tue Aug 3 13:08:04 UTC 2021 Tue Aug 3

kubernetesconfigmapsecretdockerdockerfilenginxhttpsenvironment-variablesvolumesport-forward

Kubernetes中的卷：将磁盘挂载到容器

通过卷在容器之间共享数据

Build [Date HTML Generator] Image

编写 HTML 生成器（date-html-generator.sh）

#!/bin/sh
mkdir /var/htdocs
while :
do
  echo $(date +'%Y-%m-%d %H:%M:%S') Writing to /var/htdocs/index.html
  echo $(date +'%Y-%m-%d %H:%M:%S') > /var/htdocs/index.html
  sleep 1
done
  • %Y : 完整年份 (0000-9999)
  • %m : 月份 (01-12)
  • %d : 日 (01-31)
  • %H : 小时(00-23)
  • %M : 分钟(00-59)
  • %S : 秒(00-60)

编写 Dockerfile

FROM busybox
ADD date-html-generator.sh /bin/date-html-generator.sh
RUN chmod +x /bin/date-html-generator.sh
ENTRYPOINT /bin/date-html-generator.sh

生成 date-html-generator 镜像 docker build -t wangjunjian/date-htm

kubernetesvolumesstorageemptydirhostpathnfspersistent-storagedockerport-forward