Python爬虫实践

在头信息中加入 User-Agent 来解决。在 Safari 浏览器中通过选择菜单[开发]->[显示JavaScript控制台],然后选择[网络]->[文稿]->[标头],在内容里的[请求]节可以找到 User-Agent。

import requests

url = "https://movie.douban.com/cinema/later/beijing/"
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15'}
page = requests.get(url, headers=headers)
print(page.status_code)
200

Install NVIDIA device plugin for Kubernetes

  1. 重启服务
sudo systemctl restart docker
  1. 使用Helm安装
helm install --generate-name nvdp/nvidia-device-plugin
  • 失败(gpu2节点的Docker没有配置好)
$ kubectl logs -n kube-system nvidia-device-plugin-1614240442-wfh6c 
2021/02/26 07:03:48 Loading NVML
2021/02/26 07:03:48 Failed to initialize NVML: could not load NVML library.
2021/02/26 07:03:48 If this is a GPU node, did you set the docker default runtime to `nvidia`?
2021/02/26 07:03:48 You can check the prerequisites at: https://github.com/NVIDIA/k8s-device-plugin#prerequisites
2021/02/26 07:03:48 You can learn how to set the runtime at: https://github.com/NVIDIA/k8s-device-plugin#quick-start
2021/02/26 07:03:48 If this is not a GPU node, you should set up a toleration or nodeSelector to only deploy this plugin on GPU nodes

基于腾讯云物联网开发平台的光照传感器

  1. 导入成功
  1. 创建成功
  1. 新建设备的二维码,可以使用腾讯连连扫描快速增加。
  1. 在线调试
  • light.py
from machine import ADC
from machine import Pin
class Light():
    """
    基于 PT550 环保型光敏二极管的光照传感器元器件,它的灵敏度更高,测量范围是 0Lux~6000Lux。
    Lux(勒克斯)是光照强度的单位,它和另一个概念 Lumens(流明)是不同的。Lumens 是指一个光源(比如电灯、投影仪)发出的光能力的总量,而 Lux 是指空间内一个位置接收到的光照的强度。
    因为 ADC 支持的最大位数是 12bit,所以这个数值范围是 0~4095 之间。这里按照线性关系做一个转换。
    """
    def __init__(self, pin):
        self._light = ADC(Pin(pin))

    def value(self):
        value = self._light.read()
        print("Light ADC value:", value)
        return int(value/4095*6000)
  • tencent_mqtt.py
import time
import ujson
from umqtt.simple import MQTTClient
from light import Light
"""
QCloud Device Info
"""
DEVICE_NAME = ""
PRODUCT_ID = ""
DEVICE_KEY = ""
"""
MQTT topic
"""
MQTT_DEVICE_CONTROL_TOPIC = "$thing/down/property/"+PRODUCT_ID+"/"+DEVICE_NAME
MQTT_DEVICE_STATUS_TOPIC = "$$thing/up/property/"+PRODUCT_ID+"/"+DEVICE_NAME
MQTT_SERVER = PRODUCT_ID + ".iotcloud.tencentdevices.com"
MQTT_PORT = 1883
MQTT_CLIENT_ID = PRODUCT_ID+DEVICE_NAME
MQTT_USERNAME = ""
MQTT_PASSWORD = ""
class LightSensor():
    def __init__(self):
        self.light = Light(36)
class TencentMQTT():
    light_sensor = LightSensor()

    def __init__(self):
        self.mqtt_client = None

    def connect(self):
        self.mqtt_client = MQTTClient(MQTT_CLIENT_ID, MQTT_SERVER, MQTT_PORT, MQTT_USERNAME, MQTT_PASSWORD, 60)
        self.mqtt_client.set_callback(TencentMQTT.control)
        self.mqtt_client.connect()
 
    """
    接收订阅的消息(用于远程控制)
    """
    @staticmethod
    def control(topic, msg):
        pass

    """
    发送设备的状态消息
    """
    def report(self):
        msg = {
            "method": "report",
            "clientToken": "clientToken-0123456789",
            "params": {
                "Illuminance": self.light_sensor.value()
            }   
        }

        self.mqtt_client.publish(MQTT_DEVICE_STATUS_TOPIC.encode(), ujson.dumps(msg).encode())

    """
    消息循环
    """
    def loop(self):
        self.mqtt_client.subscribe(MQTT_DEVICE_CONTROL_TOPIC.encode())

        up_time = 5000
        time_sleep = 50
        time_count = 0
        while True:
            self.mqtt_client.check_msg()

            time_count += time_sleep
            if (time_count >= up_time):
                self.report()
                time_count = 0

            time.sleep_ms(time_sleep)

基于腾讯云物联网开发平台的智能电灯

  1. 保存
  1. 选择智能家居
  1. 新建产品
  1. 保存
  • 配置小程序,快速入口配置
  1. 新建设备
  1. 设备的详细信息,用于开发的三元组(产品ID、设备名称、设备密钥)
  1. 新建设备的二维码,可以使用腾讯连连扫描快速增加。
  • led.py
from machine import PWM
from machine import Pin
class LED():
    """
    LED模块
    """
    def __init__(self, rpin, gpin, bpin, freq=1000):
        """
        构造函数
        :param pin: 接LED的管脚,必须支持PWM
        :param freq: PWM的默认频率是1000
        """
        self.pin_red = Pin(rpin)
        self.pin_green = Pin(gpin)
        self.pin_blue = Pin(bpin)

        self.led_red = PWM(self.pin_red, freq = freq)
        self.led_green = PWM(self.pin_green, freq = freq)
        self.led_blue = PWM(self.pin_blue, freq = freq)

    def rgb_light(self, red, green, blue, brightness):
        if red in range(256) and \
            green in range(256) and \
            blue in range(256) and \
            0.0 <= brightness and \
            brightness <=1.0:
            self.led_red.duty(int(red/255*brightness*1023))
            self.led_green.duty(int(green/255*brightness*1023))
            self.led_blue.duty(int(blue/255*brightness*1023))
        else:
            print("red green blue must between 0 and 255, and brightness from 0.0 to 1.0")
        
    def deinit(self):
        """
        析构函数
        """
        self.led_red.deinit()
        self.led_green.deinit()
        self.led_blue.deinit()

基于NodeMCU(ESP32)搭建Python开发环境

  • 数据线,一头是 USB-A 接口,另一头是 Micro-USB 接口。
  1. 擦除 Flash 芯片
esptool.py --chip esp32 --port /dev/cu.usbserial-1410 erase_flash
  1. 烧录固件
esptool.py --chip esp32 --port /dev/cu.usbserial-1410 --baud 460800 write_flash -z 0x1000 esp32-idf3-20210202-v1.14.bin
  • 如果不先打开WIFI将会出现下面的错误
import ubluetooth
ble = ubluetooth.BLE()
ble.active(True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 110] ETIMEDOUT

基于NodeMCU(ESP8266)搭建Python开发环境

  • 数据线,一头是 USB-A 接口,另一头是 Micro-USB 接口。
  1. 运行 esptool.py read_mac 命令,确认连接成功。
$ esptool.py read_mac
esptool.py v3.0
Found 2 serial ports
Serial port /dev/cu.usbserial-0001
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: f4:cf:a2:ec:0a:86
Uploading stub...
Running stub...
Stub running...
MAC: f4:cf:a2:ec:0a:86
Hard resetting via RTS pin...
  1. 查看设备文件
$ ls /dev/cu*
/dev/cu.Bluetooth-Incoming-Port /dev/cu.usbserial-0001
  1. 擦除 Flash 芯片
esptool.py --port /dev/cu.usbserial-0001 erase_flash

烧录固件 esptool.py --port /dev/cu.

Linux Shell 实践

  • 执行时的所有信息都输出到文件(&>)
echo hello &> log.info
$ cat log.info 
hello
  • 创建一个文件并写入内容(> filename <<EOF)
cat > hello.sh << EOF
#!/bin/bash
echo hello
EOF
  • 整数四则运算(let)
let n=10-3+4/2
echo $n
9
  • 四种执行模式运行
$ vim test.sh
#!/bin/bash
echo $str
$ chmod u+x test.sh
$ bash test.sh 
$ ./test.sh 
$ source test.sh 
hello world
$ . test.sh 
hello world
  • 导出变量(export),父进程定义的变量子进程可见。
$ export str
$ bash test.sh 
hello world
$ source test.sh 
hello world
  • 移除变量(unset)
$ unset str
$ echo $str
  • $$ 当前进程ID
$ echo $$
26102
  • $0 当前进程名
$ echo $0
-bash
  • 显示所有数组元素
$ echo ${ips[@]}
10.0.0.1 10.0.0.2 10.0.0.3

命令tar

    -c, --create Create a new archive.  Arguments supply the names of the files to be archived.
    -x, --extract, --get Extract files from an archive.
    -z, --gzip, --gunzip, --ungzip Filter the archive through gzip(1).
    -j, --bzip2 Filter the archive through bzip2(1).
    -f, --file=ARCHIVE Use archive file or device ARCHIVE.  If this option is not given, tar will first examine the environment variable `TAPE'.  If it is set, its value will be used as the archive  name.   Otherwise, tar will assume the compiled-in default.
    -t, --list List the contents of an archive.
    -v, --verbose Verbosely list files processed.

命令zip

排除所有的 yaml 文件

zip -r yolov5.zip yolov5/ -x "*yolo*.yaml"

只查看文件名 $ zipinfo -1 yolov5.zip yolov5/ yolov5/hubconf.py yolov5/tutorial.ipynb yolov5/Dockerfile yolov5/README.md yolov5/.dockerignore yolov5/models/ yolov5/models/common.py yolov5/models/export.py yolov5/models/yolov5s.yaml yolov5/models/yolo.py yolov5/models/init.py yolov5/models/experimental.py yolov5/models/yolov5m.yaml yolov5/models/yolov5l.yaml yolov5/models/yolov5x.yaml yolov5/LICENSE yolov5/.gitignore yolov5/test.py yolov5/train.py yolov5/weights/ yolov5/weights/download_weights.sh yolov5/requirements.

命令du

du - 估计文件空间使用量

    -s, --summarize display only a total for each argument
    -h, --human-readable print sizes in human readable format (e.g., 1K 234M 2G)

命令top

  • 查看1号进行
top -p 1
top -p1
top - 22:58:02 up 323 days, 12:09,  2 users,  load average: 0.64, 0.61, 0.38
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.2 us,  1.8 sy,  0.0 ni, 90.3 id,  0.0 wa,  0.5 hi,  0.2 si,  0.0 st
MiB Mem :   3780.8 total,    400.2 free,   1749.6 used,   1631.0 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   1939.3 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    1 root      20   0  178540   7680   3792 S   0.0   0.2   7:50.54 systemd

vim实践

Ubuntu

sudo apt install vim-gtk3

运行

vim -g
gvim
  • :!command 运行shell命令
    • :!ls -l 查看当前目录列表
    • :!ifconfig 查看本地IP地址
  • 选择
    • v 进入字符可视
      • w 选择下一个单词
      • 0 选择到行首
      • $ 选择到行尾
    • c 剪切选择的字符
    • d 删除选择的字符
    • x 删除选择的字符
    • ggVG 选择所有的字符
  • 复制
    • y 复制选择的字符
    • :%y 复制所有的字符
    • 当前行
      • dd 剪切当前行
        • 3dd 剪切3行
      • y$ 复制当前字符到行尾
      • :y 复制当前行
      • yy 复制当前行
        • 3yy 复制3行
    • 剪切板(用于应用之间的交换)
      • :y+ 复制光标所在行(也可以先使用块的方式进行选择再复制)
      • :%y+ 复制所有数据
  • p 粘贴
  • 删除
    • x 删除当前字符
    • :1,$d 删除所有行(1第一行开始,$直到文件末尾,d删除)
    • dw 删除光标所在字符到单词的末尾
    • diw 删除光标所在的整个单词
    • d0 删除光标所在字符到行首
    • d$ 删除光标所在字符到行尾
    • dG 删除光标所在行到文件末尾
  • :%s/search_word/replace_word/g 替换(全文)
    • :1,$s/search_word/replace_word/g 替换(全文)
    • :3,4s/search_word/replace_word/g 替换(3-4行)
  • :%s/search_word//g 删除(全文)
    • :1,$s/search_word//g 删除(全文)
    • :3,4s/search_word//g 删除(3-4行)

命令ln

在文件之间建立链接

    ln [OPTION]... [-T] TARGET LINK_NAME
    ln [OPTION]... TARGET
    ln [OPTION]... TARGET... DIRECTORY
    ln [OPTION]... -t DIRECTORY TARGET...
  1. rm
rm /var/www/ubuntu