116 篇文章带有标签 “python”

深度学习准确性验证框架

例子

进入 accuracy_checker 目录

cd open_model_zoo/tools/accuracy_checker

下载数据集

wget https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
tar xvf cifar-10-python.tar.gz -C sample

配置文件结构

models:
  - name: model_name

    launchers:
      - framework: openvino
        adapter: adapter_name

    datasets:
      - name: dataset_name

评估 accuracy_check -c sample/sample_config.yml -m data/test_models/ -s sample/ 2022-05-18 11:18:38.663810: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.

OpenVINO

Open Visual Inference and Neural network Optimization

OpenVINO 安装

Development

pip 安装

• 下载

安装 OpenVINO 开发工具

python -m venv openvino_env

• Linux

source openvino_env/bin/activate

• Windows

openvino_env\Scripts\activate.bat

python -m pip install --upgrade pip
pip install openvino-dev[onnx,pytorch,kaldi,mxnet,caffe,tensorflow2]==2022.1.0 -i https://mirrors.aliyun.com/pypi/simple/

• Install OpenVINO™ Development Tools

源代码编译安装(没有成功)👹

Build OpenVINO™ Inference Engine

创建项目

创建目录

mkdir project
cd project

创建虚拟环境

python -m venv env
source env/bin/activate
# 退出命令 deactivate

创建 requirements.txt 文件

fastapi
python-multipart
aiofiles
uvicorn
gunicorn

安装需要的库

pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/

项目目录结构

project
├── app
│   ├── __init__.py
│   ├── dependencies.py
│   ├── main.py
│   └── routers
│       ├── __init__.py
│       ├── files.py
│       └── users.py
└── requirements.txt

main.py import uvicorn from fastapi import FastAPI from .routers import users from .routers import files app = FastAPI(title='REST API Interface', version='1.

服务器 CPU 40核，内存 256G，操作系统 Ubuntu 20.04，Python3.9

RESTAPI 基于 FastAPI 实现的文件上传和下载 router = APIRouter(prefix='/file_benchmarking', tags=['Files']) @router.post('/upload/binary/chunk/async_func/async_r_sync_w', tags=['Upload', 'binary']) async def upload_binary_chunk_async_func_async_r_sync_w(request: Request): file_path = get_random_filename() with open(file_path, "wb") as file: async for chunk in request.stream(): file.write(chunk) return {'file_path': file_path} @router.

使用 FastAPI 实现了文件的上传和下载，部署服务使用了 uvicorn 和 gunicorn+uvicorn 两种方法。

基准测试工具使用的是 wrk

服务器 CPU 40核，内存 256G，操作系统 Ubuntu 20.04，Python3.9

测试流程

使用的测试图片 health.jpg (256kb)

生成测试数据

生成通过 HTTP POST 发送二进制数据的文件。

python make_http_postdata.py make health.jpg postdata

file: /home/wjj/test/postdata
boundary: gouchicao0123456789

创建用于 wrk 的 lua 脚本：postfile.lua

wrk.method = "POST"
local f = io.open("postdata", "rb")
wrk.body   = f:read("*all")
wrk.headers["Content-Type"] = "multipart/form-data; boundary=gouchicao0123456789"

部署 FastAPI 应用 uvicorn uvicorn app.

问题

基于之前安装 Python3.8 的经验，运行下面的命令就可以成功安装 Python3.9 和 pip，但是这回失败了。

sudo apt install build-essential python3.9 python3.9-dev -y
sudo ln -s /usr/bin/python3.9 /usr/bin/python
curl https://bootstrap.pypa.io/get-pip.py | sudo python -

Traceback (most recent call last):
  File "/home/lnsoft/wjj/get-pip.py", line 33324, in <module>
    main()
  File "/home/lnsoft/wjj/get-pip.py", line 135, in main
    bootstrap(tmpdir=tmpdir)
  File "/home/lnsoft/wjj/get-pip.py", line 111, in bootstrap
    monkeypatch_for_cert(tmpdir)
  File "/home/lnsoft/wjj/get-pip.py", line 92, in monkeypatch_for_cert
    from pip._internal.commands.install import InstallCommand
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpcw3afq6v/pip.zip/pip/_internal/commands/__init__.py", line 9, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpcw3afq6v/pip.zip/pip/_internal/cli/base_command.py", line 15, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpcw3afq6v/pip.zip/pip/_internal/cli/cmdoptions.py", line 23, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpcw3afq6v/pip.zip/pip/_internal/cli/parser.py", line 12, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpcw3afq6v/pip.zip/pip/_internal/configuration.py", line 26, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpcw3afq6v/pip.zip/pip/_internal/utils/logging.py", line 27, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpcw3afq6v/pip.zip/pip/_internal/utils/misc.py", line 39, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpcw3afq6v/pip.zip/pip/_internal/locations/__init__.py", line 14, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpcw3afq6v/pip.zip/pip/_internal/locations/_distutils.py", line 9, in <module>
ModuleNotFoundError: No module named 'distutils.cmd'

健康码识别服务使用了 FastAPI 进行开发的，本周主要工作是为了对健康码识别的服务进行性能调优。接口函数使用了 async 关键字，但是内部的实现并没有使用 await。由于改写成异步代码需要时间，这里并没有改写代码，只是删除了 async 关键字。部署服务使用了 uvicorn 和 gunicorn+uvicorn 两种方法。

基准测试工具使用的是 ab

测试流程

生成测试数据

准备测试图片 health.jpg

echo -n '{"base64": "' > health.json
base64 -w0 health.jpg >> health.json
echo -n '"}' >> health.json

部署服务

uvicorn

docker run --runtime=nvidia --rm -it -e NVIDIA_VISIBLE_DEVICES=2 -p 20001:8000 \
    -v $(pwd):/health_code_service --name=health-uvicorn  health-code-service \
    uvicorn controller:app --host 0.0.0.0 --workers 1

• workers 并发进程数

安装依赖库

pip install typer python-docx

编写生成个人周报的应用 制作个人周报模版 个人周报－{}－{}.xlsx 编写应用 weekly.py import shutil import openpyxl import typer from datetime import datetime, timedelta def get_weekly_info(): now = datetime.now() # 工作表的标题，格式: yymmdd(周一)-yymmdd(周五) monday = now-timedelta(now.weekday()) friday = monday+timedelta(4) sheet_title = '{}-{}'.format(monday.strftime('%y%m%d'), friday.strftime('%y%m%d')) # 报告日期，格式: yyyy/mm/dd(周四) report_date = now.strftime("%Y/%m/%d") # 计划的开始时间与结束时间 base_date = datetime(1899, 12, 30, 0, 0) work_tasks_begin_time = (monday-base_date).

操作 Word 文档

安装依赖库 python-docx

pip install python-docx

示例 from docx import Document from docx.shared import Inches document = Document() document.add_heading('Document Title', 0) p = document.add_paragraph('A plain paragraph having some ') p.add_run('bold').bold = True p.add_run(' and some ') p.add_run('italic.').italic = True document.add_heading('Heading, level 1', level=1) document.add_paragraph('Intense quote', style='Intense Quote') document.

系统

指定本地时区

FROM ubuntu:20.04
LABEL maintainer="wang-junjian@qq.com"

ARG TIME_ZONE=Asia/Shanghai
RUN DEBIAN_FRONTEND=noninteractive apt-get install tzdata -y && \
    ln -fs /usr/share/zoneinfo/$TIME_ZONE /etc/localtime && \
    echo $TIME_ZONE > /etc/timezone

• /etc/localtime 修改时区
• /etc/timezone 描述本机所属时区
• timedatectl 显示修改好的效果 参考资料
• linux时区（/etc/localtime和/etc/timezone）
• 在 Linux 中查看你的时区

Python 开发环境 构建 Python, OpenCV 开发环境的镜像 FROM ubuntu:20.04 LABEL maintainer="wang-junjian@qq.com" # 设置 apt 源（阿里云），设置完必须 update。 RUN sed -i 's/archive.ubuntu.com/mirrors.aliyun.

构建镜像

FROM paddlepaddle/paddle:2.2.2-gpu-cuda10.2-cudnn7
LABEL maintainer="wang-junjian@qq.com"

RUN apt-get update && apt-get install libjpeg-dev zlib1g-dev -y

RUN pip install -i https://mirrors.aliyun.com/pypi/simple/ \
    numpy fastapi paddleocr opencv-python

EXPOSE 20000

WORKDIR /inference-serving
ADD . ./

CMD ["python", "app.py"]

官方推荐：非安培架构的GPU，推荐使用CUDA10.2，性能更优。

自己构建 paddlepaddle 镜像

通过官方的 Docker Hub 没有找到 runtime 版本，想着节省几个G的空间，于是考虑自己来构建。

test.json

{ "stuff": { "that": [1,2,3], "isin": true, "json": "end"}}

jq

在命令行运行

jq . <<< '{ "stuff": { "that": [1,2,3], "isin": true, "json": "end"}}'

jq . test.json

在vim的命令模式下运行

%!jq .

python json.tool

在命令行运行

python -m json.tool <<< '{ "stuff": { "that": [1,2,3], "isin": true, "json": "end"}}'

python -m json.tool test.json

在vim的命令模式下运行

%!python -m json.tool

在线格式化 Format JSON JSON Formatter JSON Formatter, Validator, Vi

构建 ONNXRuntime-GPU 镜像

编写 requirements.txt

$ vim requirements.txt

flask
connexion[swagger-ui]
connexion
gunicorn
numpy
opencv-python
scikit-image
psutil
pynvml
onnxruntime-gpu

编写 Dockerfile 需要带 cudnn 库的 CUDA 作为基镜像 $ vim Dockerfile FROM nvidia/cuda:11.4.0-cudnn8-runtime-ubuntu20.04 LABEL maintainer="wang-junjian@qq.com" RUN rm /etc/apt/sources.list.d/cuda.list /etc/apt/sources.list.d/nvidia-ml.list && \ sed -i 's/archive.ubuntu.com/mirrors.aliyun.com/g' /etc/apt/sources.

安装 Fabric

pip3 install fabric

远程执行 Shell 命令脚本（remote_execute_shell_command.py） #!/usr/bin/python import argparse from fabric import Connection, Config # 您要远程操作的计算机，username@ip HOSTS = ['root@192.168.0.1', 'root@192.168.0.2'] PASSWORDS = ['admin', 'admin'] if name == 'main': parser = argparse.ArgumentParser() parser.add_argument('-c', '--command', type=str, help='execute shell command.') args = parser.parse_args() if not args.command: args.command = 'uname -a' print('➜ Execute shell command: ', args.

安装

pip install paddleocr

测试

import cv2
import numpy as np

from paddleocr import PaddleOCR


ocr = PaddleOCR(use_angle_cls=True)
image_path = 'test.jpg'
img = cv2.imread(image_path)

img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
img_gray1 = img_gray[:,:, np.newaxis]
img_gray3 = np.concatenate([img_gray1, img_gray1, img_gray1], axis=-1)

texts = ocr.ocr(img_gray3)
for text in texts:
    """
    box   坐标1　　　　　　　　　坐标2
          坐标4　　　　　　　　　坐标3
    """
    box = text[0]
    t = text[1][0]
    score = text[1][1]

可视化（图像上画出文本和得分） import os import shutil import cv2 import numpy as np import uuid from PIL import ImageFo

XML 数据

xml_data = '''<?xml version="1.0" encoding="UTF-8"?>
<books>
    <book SN="12460901">
        <name>国富论</name>
        <author nation="English">亚当·斯密</author>
        <price>108</price>
        <ISBN>9787511373229</ISBN>
    </book>
    <book SN="12257413">
        <name>原则</name>
        <author nation="America">瑞·达利欧</author>
        <price>98</price>
        <ISBN>9787508684031</ISBN>
    </book>
</books>
'''

xml.etree.ElementTree 导入 import xml.

安装

Python

sudo apt install python3
sudo apt install python3-pip
sudo pip3 install --upgrade pip

OpenCV

sudo pip3 install opencv-python
sudo pip3 install opencv-contrib-python

图像

读取图像

import cv2

img_file = 'python-logo@2x.png'
img = cv2.imread(img_file)

获取图像大小

width = img.shape[1]
height = img.shape[0]

显示图像并等待按任意键退出

import cv2

img_file = 'python-logo@2x.png'
img = cv2.imread(img_file)
cv2.imshow('', img)
cv2.waitKey(0)

宽高缩小一倍 比例 import cv2 img_file = 'python-logo@2x.png' img = cv2.imread(img_file) img = cv2.resize(img, None, fx=0.5, fy=0.5) cv2.

file.write() 内容超过 4K 才会写入磁盘

验证代码

import os
import tempfile

for count in range(1, 4100):
    content = '0'*count
    with tempfile.NamedTemporaryFile() as file:
        print(file.name)
        file.write(content.encode())

        with open(file.name, 'r') as tf:
            content_len = len(tf.read())
            if content_len > 0:
                print(f'{count} bytes written successfully.')

运行结果 /var/folders/bc/7lz308t90gb1h1xw6k4j65x80000gn/T/tmpj458ozas /var/folders/bc/7lz308t90gb1h1xw6k4j65x80000gn/T/tmpmrxo8sg1 /var/folders/bc/7lz308t90gb1h1xw6k4j65x80000gn/T/tmp0hu_i4hz 4097 bytes written successfully.

安装 Docker SDK for Python

pip install docker

例子

将本地文件或目录添加到容器，生成新的镜像。

• put_archive
• commit

import docker
import tarfile
import tempfile
import os

def simple_tar(path):
    f = tempfile.NamedTemporaryFile()
    t = tarfile.open(mode='w', fileobj=f)
    abs_path = os.path.abspath(path)
    t.add(abs_path, arcname=os.path.basename(path))
    t.close()
    f.seek(0)
    return f

client = docker.from_env()
// ...

参考资料

• Docker SDK for Python
• Docker SDK for Python Document
• Develop with Docker Engine API
• docker-py : Docker Python sdk container.put_archive not working

模型打包成镜像

手工打包

project_dir=platen-switch
darknet_model_name=darknet-model-platen-switch

cd $project_dir
docker run -d --name $darknet_model_name alpine
docker cp model/ $darknet_model_name:/
docker commit -a 'wang-junjian@qq.com' -m 'darknet model [platen-switch recognition]' \
  $darknet_model_name gouchicao/$darknet_model_name:latest
docker rm -v $darknet_model_name

docker push gouchicao/$darknet_model_name:latest

Python 脚本 import docker import tarfile import tempfile import os def simple_tar(path): f = tempfile.NamedTemporaryFile() t = tarfile.