6 篇文章带有标签 “file”

创建项目

创建目录

mkdir project
cd project

创建虚拟环境

python -m venv env
source env/bin/activate
# 退出命令 deactivate

创建 requirements.txt 文件

fastapi
python-multipart
aiofiles
uvicorn
gunicorn

安装需要的库

pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/

项目目录结构

project
├── app
│   ├── __init__.py
│   ├── dependencies.py
│   ├── main.py
│   └── routers
│       ├── __init__.py
│       ├── files.py
│       └── users.py
└── requirements.txt

main.py import uvicorn from fastapi import FastAPI from .routers import users from .routers import files app = FastAPI(title='REST API Interface', version='1.

服务器 CPU 40核，内存 256G，操作系统 Ubuntu 20.04，Python3.9

RESTAPI 基于 FastAPI 实现的文件上传和下载 router = APIRouter(prefix='/file_benchmarking', tags=['Files']) @router.post('/upload/binary/chunk/async_func/async_r_sync_w', tags=['Upload', 'binary']) async def upload_binary_chunk_async_func_async_r_sync_w(request: Request): file_path = get_random_filename() with open(file_path, "wb") as file: async for chunk in request.stream(): file.write(chunk) return {'file_path': file_path} @router.

使用 FastAPI 实现了文件的上传和下载，部署服务使用了 uvicorn 和 gunicorn+uvicorn 两种方法。

基准测试工具使用的是 wrk

服务器 CPU 40核，内存 256G，操作系统 Ubuntu 20.04，Python3.9

测试流程

使用的测试图片 health.jpg (256kb)

生成测试数据

生成通过 HTTP POST 发送二进制数据的文件。

python make_http_postdata.py make health.jpg postdata

file: /home/wjj/test/postdata
boundary: gouchicao0123456789

创建用于 wrk 的 lua 脚本：postfile.lua

wrk.method = "POST"
local f = io.open("postdata", "rb")
wrk.body   = f:read("*all")
wrk.headers["Content-Type"] = "multipart/form-data; boundary=gouchicao0123456789"

部署 FastAPI 应用 uvicorn uvicorn app.

file.write() 内容超过 4K 才会写入磁盘

验证代码

import os
import tempfile

for count in range(1, 4100):
    content = '0'*count
    with tempfile.NamedTemporaryFile() as file:
        print(file.name)
        file.write(content.encode())

        with open(file.name, 'r') as tf:
            content_len = len(tf.read())
            if content_len > 0:
                print(f'{count} bytes written successfully.')

运行结果 /var/folders/bc/7lz308t90gb1h1xw6k4j65x80000gn/T/tmpj458ozas /var/folders/bc/7lz308t90gb1h1xw6k4j65x80000gn/T/tmpmrxo8sg1 /var/folders/bc/7lz308t90gb1h1xw6k4j65x80000gn/T/tmp0hu_i4hz 4097 bytes written successfully.

拷贝一批文本文件(10000)到目录

cp

• time: 0.470s
• 当文件更新了或者缺少时才拷贝。(-u)
• 速度最快

cp -u labels/*/*.txt datasets/yolo/sign/labels/

xargs

• time: 30.003s

ls labels/*/*.txt | xargs -I {} cp {} datasets/yolo/sign/labels/

find -exec

• time: 32.521s

find labels/ -type f -iname '*.txt' -exec cp {} datasets/yolo/sign/labels/ \;

for

• time: 41.259s

for i in `ls labels/*/*.txt`; do cp $i datasets/yolo/sign/labels/; done

参考资料

• cp Command in Linux (Copy Files)
• HowTo: Use bash For Loop In One Line

导入包

>>> import os
>>> import shutil

文件

拷贝文件

目录路径可以是目录名

>>> shutil.copy('/home/python/main.py', '/home/python/main.py.bak')
>>> shutil.copy('/home/python/main.py', '/home/python/bak/')

创建文件

• 方法1

open(filename, 'w').close()

• 方法2

def touch(path):
    with open(path, 'a'):
        os.utime(path, None)

• 方法3

OS X需要root特权

os.mknod(filename)

修改文件名

>>> os.rename('filename', 'new_filename')

删除文件

>>> os.remove('/home/python/main.py')

目录

拷贝目录树 >>> shutil.