Linux系统DNS设置

之前在文件/etc/resolv.conf中设置了，过段时间总是自动恢复默认值。（注释中写的很详细，是不可编辑的由系统自动生成的文件）

/etc/resolv.conf # This file is managed by man:systemd-resolved(8). Do not edit. # # This is a dynamic resolv.conf file for connecting local clients to the # internal DNS stub resolver of systemd-resolved. This file lists all # configured search domains. # # Run "resolvectl status" to see details about the uplink DNS servers # currently in use. # # Third party programs must not access this file directly, but only through the # symlink at /etc/resolv.conf. To manage man:resolv.

linuxubuntudnsnetworkingsystemdsysadmin

Linux系统上修改用户名

今天同事安装了一台新的服务器Ubuntu20.04，但用户名和主机名不是我想要的，这里尝试了直接修改Linux文件的方式。

登录root用户

$ su - root

修改用户信息

/etc/passwd

# nano /etc/passwd
username:x:1000:1000:username:/home/username:/bin/bash

/etc/shadow

# nano /etc/shadow
username:D78/D2/DdYW.FVG.GlqDlZsZ4sK21gSxhDooqWlJtCVl3oUbDUTKtGxBWkCE3E/Oha40kjDrk0pBbsvT4TwtzuH61vYmnJ/GY.bAHWbVv1:18545:0:99999:7:::

/etc/group

# nano /etc/group
adm:x:4:syslog,username
cdrom:x:24:username
sudo:x:27:username
dip:x:30:username
plugdev:x:46:username
lxd:x:116:username
username:x:1000:

修改HOME路径

# mv /home/old_username /home/new_username

修改主机名 /etc/hostname # nano /etc/hostname hos

linuxubuntuuser-managementhostnamesysadmincommand-line

在Ubuntu上安装nvidia-docker2

在 Ubuntu20.04 上安装 nvidia-docker2

配置 apt 仓库(repository)

$ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | \
  sudo apt-key add -
$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
$ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-docker.list

更新安装包的列表

$ sudo apt-get update

安装 nvidia-docker2

$ sudo apt-get install -y nvidia-docker2
  • 安装后可以查看 nvidia runtime 配置
$ nano /etc/docker/daemon.json
{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}
dockernvidiagpuubuntucontainersinstallationcudalinux

在Ubuntu上安装Docker

在 Ubuntu20.04 上安装 Docker

安装 Docker

$ curl -fsSL https://get.docker.com | sh -

把用户名(username)加入到 docker 组，这样以后在非 root 用户下不用每次操作都 sudo。

$ sudo usermod -aG docker username

需要退出用户会话，重新登录方可生效。

卸载Docker

  • 删除Docker及其依赖
$ sudo apt-get remove --auto-remove docker
  • 删除所有数据
$ sudo rm -rf /var/lib/docker
dockerubuntuinstallationcontainerslinuxsysadmincommand-line

基于Apt-Cacher NG创建本地Ubuntu存储库

安装 Apt-Cacher-NG

$ sudo apt install apt-cacher-ng
$ sudo systemctl enable apt-cacher-ng
$ sudo service apt-cacher-ng start

配置

$ sudo nano /etc/apt-cacher-ng/acng.conf
  • CacheDir: /var/cache/apt-cacher-ng
  • LogDir: /var/log/apt-cacher-ng

查看缓存目录的数据

$ du -sh /var/cache/apt-cacher-ng/
19M	/var/cache/apt-cacher-ng/

客户端配置

$ sudo nano /etc/apt/apt.conf.d/00aptproxy
Acquire::http::Proxy "http://172.16.33.157:3142";

安装软件

$ sudo apt install nodejs

参考资料 使用apt-cacher-ng的快速Debian/Ubuntu軟件包緩存代理設置 使用“ apt-cacher”设置“ apt-cache”服务器 使用 apt-mir

ubuntuaptcacheproxysysadminrepositorynetworking

Welcome to Jekyll!

You’ll find this post in your _posts directory. Go ahead and edit it and re-build the site to see your changes. You can rebuild the site in many different ways, but the most common way is to run jekyll serve, which launches a web server and auto-regenerates your site when a file is updated.

Jekyll requires blog post files to be named according to the following format:

YEAR-MONTH-DAY-title.MARKUP

Where YEAR is a four-digit number, MONTH and DAY are both two-digit numbers, and MARKUP is the file extension representing the format used in the file. After that, include the necessary front matter.

jekyllstatic-generatorrubyblogsmarkdown

安装Harbor

下载最新的 Harbar Release

wget -c https://github.com/goharbor/harbor/releases/download/v2.1.3/harbor-offline-installer-v2.1.3.tgz
tar -zxvf harbor-offline-installer-v2.1.3.tgz
cd harbor

生成证书（用于支持 HTTPS 访问）

在生产环境中，请始终使用HTTPS，您应该从CA获得证书。 在测试或开发环境中，您可以生成自己的CA。

生成CA证书 生成CA证书私钥 openssl genrsa -out ca.key 4096 Generating RSA private key, 4096 bit long modulus (2 primes) .............................................................................................................................................++++ ..............................................................................................

dockerharborubunturegistryhttpssslcontainersdevopsdeploymentprivate-registry

NFS配置

Ubuntu

服务端

  • 安装服务端
sudo apt install nfs-kernel-server
  • 查看服务状态
sudo systemctl status nfs-server
  • 查看开启的NFS协议
sudo cat /proc/fs/nfsd/versions

-2 +3 +4 +4.1 +4.2
  • 配置访问的路径
sudo nano /etc/exports

/data/nfs        172.16.33.0/24(rw,sync,fsid=0,crossmnt,no_subtree_check)
  • 应用配置
sudo exportfs -ra
  • 查看当前应用
sudo exportfs -v
  • 重启服务
sudo systemctl restart nfs-server

客户端

  • 安装客户端
sudo apt install nfs-common
  • 查看NFS服务器导出列表
showmount -e 172.16.33.157

Export list for 172.16.33.157:
/data/nfs 172.16.33.0/24,172.16.128.164
  • 挂载NFS
sudo mount -t nfs 172.16.33.157:/ $(pwd)/nfs
  • 移除挂载
sudo umount $(pwd)/nfs

参考资料 如何在Ubuntu 18.04上安装和配置NFS服务器 ubuntu18.

linuxubuntunfsnetworkingsysadminfile-sharingstorage

磁盘：分区－格式化－挂载

分区

列出块设备信息

lsblk

NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0    7:0    0 71.3M  1 loop /snap/lxd/18772
loop1    7:1    0 55.4M  1 loop /snap/core18/1944
loop2    7:2    0 31.1M  1 loop /snap/snapd/10492
loop3    7:3    0 31.1M  1 loop /snap/snapd/10707
loop4    7:4    0 71.2M  1 loop /snap/lxd/18674
loop5    7:5    0 55.4M  1 loop /snap/core18/1932
sda      8:0    1  558G  0 disk 
├─sda1   8:1    1    1M  0 part 
└─sda2   8:2    1  558G  0 part /
sdb      8:16   1  2.2T  0 disk

创建分区 sudo fdisk /dev/sdb Welcome to fdisk (util-linux 2.34). Changes will remain in memory only, until you decide to write them. Be careful before using the write command.

linuxdiskpartitionfilesystemmountsysadminubuntucommand-linestorage

使用RetinaNet算法训练自定义数据集

训练自己的数据集

标注数据

LabelImg

#标注后的目录结构
project
└── labelimg
    ├── 20190128155421222575013.jpg
    ├── 20190128155421222575013.xml
    ├── 20190128155703035712899.jpg
    ├── 20190128155703035712899.xml
    ├── 20190129091126392737624.jpg
    └── 20190129091126392737624.xml

构建镜像

  • 拉取
$ sudo docker pull gouchicao/keras-retinanet:latest
  • 手动构建
FROM gouchicao/tensorflow:2.2.0-gpu-jupyter-opencv4-pillow-wget-curl-git-nano
LABEL maintainer="wang-junjian@qq.com"
WORKDIR /
RUN mkdir -p /root/.keras/models/ && \
    wget -O /root/.keras/models/ResNet-50-model.keras.h5 https://github.com/fizyr/keras-models/releases/download/v0.0.1/ResNet-50-model.keras.h5
RUN git clone --depth 1 --recurse-submodules https://github.com/gouchicao/keras-retinanet.git
WORKDIR /keras-retinanet/keras-retinanet
# 提前安装指定版本 keras==2.3.1 解决错误 TypeError: type object got multiple values for keyword argument 'training'
RUN pip install keras==2.3.1 && \
    pip install . && \
    python setup.py build_ext --inplace
WORKDIR /keras-retinanet
目标检测retinanettensorflowkerasdockertensorboarddeep-learningcomputer-visionpythongpu

将注释从PASCAL VOC转换为YOLO Darknet

PASCAL VOC(Visual Object Classes) 格式

格式

  • 每张图片对应一个xml标注文件。
  • object　 标注的对象，一张图片可以有1个或者多个
  • name　 类别名
  • bndbox 标注的边框（使用的坐标信息）
  • difficult 目标是否难以识别（0表示容易识别）

0001.xml <annotation> <folder>car</folder> <filename>0001.jpg</filename> <path>/dataset/car/0001.

目标检测yolodarknetpascal-vocpythondata-formatannotationcomputer-visiondeep-learning

Linux设置时区

查看系统的时区

$ ll /etc/localtime
lrwxrwxrwx. 1 root root 35 2月  18 2020 /etc/localtime -> ../usr/share/zoneinfo/UTC

列出有效的时区

$ timedatectl list-timezones
Africa/Abidjan
......
America/New_York
......
Asia/Shanghai
......

设置时区

# timedatectl set-timezone Asia/Shanghai

查看系统的时区

$ ll /etc/localtime
lrwxrwxrwx. 1 root root 35 2月  18 2020 /etc/localtime -> ../usr/share/zoneinfo/Asia/Shanghai
linuxtimezonesysadminubuntucommand-line

SSH使用密匙登录

生成身份验证密钥

ssh-keygen -t rsa，在~/.ssh/目录下生成私匙id_rsa和公匙id_rsa.pub两个文件。

$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/username/.ssh/id_rsa): 直接回车
Enter passphrase (empty for no passphrase): 直接回车
Enter same passphrase again: 直接回车
Your identification has been saved in /home/username/.ssh/id_rsa.
Your public key has been saved in /home/username/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:foo-YWwIv/a/HGEGt9P6vvmff/QjBGEvzlYM4hBWeR0 username@hostname
The key's randomart image is:
+---[RSA 2048]----+
|         +oo.ooE.|
|        . o..o+. |
|      . .  .. .+ |
|  .    o o  o.o  |
|   o o  S .  +.  |
|    o =+ +  .. ..|
|     + .+ . . o..|
|    o .o = . + .o|
|   . o+o=o+o. o..|
+----[SHA256]-----+
linuxsshssh-keygensecurityauthenticationsysadmincommand-line

构建YOLOv4容器应用在自定义数据集上

构建YOLOv4容器

  • 编写Dockerfile
FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04
LABEL maintainer="wang-junjian@qq.com"

#auto install tzdata(opencv depend)
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y \
    git wget nano \
    libopencv-dev python3-opencv \
    && rm -rf /var/lib/apt/lists/*

#set your localtime
RUN ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

WORKDIR /
// ...
  • 构建容器
docker build -t darknet:latest-gpu-yolov4 .

创建工程 工程目录结构 ├── yolov4.conv.137 预训练模型 ├── darknet └── project 工程目录 ├── backup 存储模型训练时权重值 ├── cfg 配置目录 │ ├── train.

目标检测yoloyolov4darknetdockernvidiagpudeep-learningcomputer-visioncontainers

使用Detectron在自定义数据集上训练MaskRCNN

拉取镜像 Detectron

sudo docker pull gouchicao/detectron:latest

创建工程（COCO格式）

└── helmet　　　　　　　　　　工程目录
    ├── images　　　　　　　 样本图片目录
    ├── helmet_train.json　训练的样本标注信息
    ├── helmet_val.json　　验证的样本标注信息
    ├── test               测试图片目录
    ├── predict            预测图片目录
    ├── model　　　　　　　　模型目录
    └── e2e_mask_rcnn_R-101-FPN_2x.yaml  网络配置文件

运行容器 Detectron，挂载工程目录。

sudo docker run -it --runtime=nvidia --name detectron-helmet \
                -v /helmet-project-realpath:/detectron/project \
                gouchicao/detectron:latest

修复BBOX_XFORM_CLIP错误

nano /detectron/detectron/utils/env.py 

yaml_load = lambda x: yaml.load(x, Loader=yaml.Loader)

工程设置 配置数据集 nano /detectron/detectron/datasets/dataset_catalog.

目标检测detectronmask-rcnndockermachine-learningcomputer-visioninstance-segmentationtutorial

Jupyter Notebook实践

本地安装

云端环境

Binder

Google Colab

多用户版本

部署

jupyterjupyterhubpythondata-sciencenotebooktutorial

基于Darknet框架的YOLOv3算法开发的模型训练和部署的容器化产品

基于 Darknet 的容器化产品

  • 模型训练工程化 [darknet]
  • 模型部署服务器 [darknet-serving]

模型训练工程化 [darknet]

  1. 训练前的数据准备
    • 工程目录结构
    project/            # 工程根目录
├── cfg             # 配置目录
│   └── voc.names   # 标签名
├── images          # 样本
│   ├── ...
│   └── ...
├── labels          # YOLO标注格式
│   ├── ...
│   └── ...
└── test            # 训练完后用于测试（非必须）
├── ...
└── ...
  • 举例：这里以platen-switch为例
    platen-switch/
├── cfg
│   └── voc.names
├── images
│   ├── IMG_9255.JPG
│   ├── IMG_9263.JPG
│   ├── IMG_9266.JPG
│   └── IMG_9280.JPG
├── labels
│   ├── IMG_9255.txt
│   ├── IMG_9263.txt
│   ├── IMG_9266.txt
│   └── IMG_9280.txt
└── test
├── IMG_9256.JPG
└── IMG_9271.JPG

运行darknet容器 将工程目录作为挂载点绑定到容器 # 使用您的工程绝对路径设置变量 project_dir

目标检测darknetyoloyolov3dockercontainersmachine-learningcomputer-visiondeployment

使用Darknet在自定义数据集上训练YOLOv3

编译darknet

git clone https://github.com/pjreddie/darknet
cd darknet/
make

创建工程 工程目录结构 ├── darknet53.conv.74 基于imagenet的预训练模型 └── open-close 工程目录 ├── backup 存储模型训练时权重值 ├── cfg 配置目录 │ ├── train.txt 存储用于训练的图像路径 │ ├── valid.txt 存储用于验证的图像路径 │ ├── voc.data 配置文件 │ ├── voc.names 标签名 │ └── yolov3.cfg YOLOv3神经网络文件 ├── data │ └── labels 预测时用于显示标签名字 │ ├── 100_0.png │ ├── 100_1.png │ ├── ...... │ └── make_labels.py ├── images 图像样本集 │ ├── IMG_9255.JPG │ ├── IMG_9263.JPG │ ├── IMG_9266.JPG │ └── IMG_9280.JPG ├── labels PASCAL VOC格式的标注 │ ├── IMG_9255.xml │ ├── IMG_9263.xml │ ├── IMG_9266.xml │ └── IMG_9280.

目标检测darknetyoloyolov3machine-learningcomputer-visiontutorial

Linux系统网络配置

Ubuntu

修改IP

$ sudo nano /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
  ethernets:
    eno1:
      addresses:
      - 172.16.33.1/24
      gateway4: 172.16.33.254
      nameservers: {}
  version: 2

应用配置

$ sudo netplan apply
# 推荐使用 debug 参数
$ sudo netplan --debug apply

或者

sudo netplan generate

验证

ip a | grep eno1

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 172.16.33.1/24 brd 172.16.33.255 scope global eno1

参考资料 How to setup a static IP on Ubuntu Server 18.04 How to configure Network Settings in Ubuntu 18.

ubuntulinuxnetworkingnetplanstatic-ipsysadmintutorial