搭建GitLab流水线

我们希望可以借助Gitlab CI/CD流水线的能力,实现源码从提交开始,经过平台问题拦截、API文档生成、代码编写规范校验、源码加密、版本编译打包发布,到最终部署到测试环境等端到端的完全自动化。

_static/cicd_pipeline.png

编写流水线配置脚本

我们通过编写pipeline流水线脚本(.gitlab-ci.yml)来定义流水线流程。该文件定义了流水线有4条阶段:安装install、测试test、构建build、发布release、部署deploy。

_static/gitlab_pipeline.png
  • 在Windows主机安装依赖库

  • 在Linux主机安装依赖库

流水线脚本示例

下面是编写的pipeline yaml文件。

  1# You can override the included template(s) by including variable overrides
  2# SAST customization: https://docs.gitlab.com/ee/user/application_security/sast/#customizing-the-sast-settings
  3# Secret Detection customization: https://docs.gitlab.com/ee/user/application_security/secret_detection/pipeline/#customization
  4# Dependency Scanning customization: https://docs.gitlab.com/ee/user/application_security/dependency_scanning/#customizing-the-dependency-scanning-settings
  5# Container Scanning customization: https://docs.gitlab.com/ee/user/application_security/container_scanning/#customizing-the-container-scanning-settings
  6# Note that environment variables can be set in several places
  7# See https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence
  8stages:
  9  - install
 10  - test
 11  - build
 12  - release
 13  - deploy
 14
 15cache:
 16  paths:
 17    - dist/
 18    - myenv/
 19
 20before_script:
 21  - cat /etc/os-release
 22  - cat /proc/version
 23  - apt update
 24  - apt-cache policy python3
 25  - apt install python3 -y
 26  - python3 -V
 27  - apt install python3-venv -y
 28  - python3 -m venv myenv
 29  - source myenv/bin/activate
 30  - which python
 31  - which python3
 32  - which pip
 33  - which pip3
 34
 35install-dependencies:
 36  stage: install
 37  tags:
 38    - gitlab-org
 39  only:
 40    - main
 41    - alpha
 42    - beta
 43    - rc
 44  before_script:
 45    - cat /etc/os-release
 46    - cat /proc/version
 47    - apt update
 48    - apt-cache policy python3
 49    - apt install python3 -y
 50    - python3 -V
 51    - apt install python3-venv -y
 52    - python3 -m venv myenv
 53    - source myenv/bin/activate
 54    - which python
 55    - which python3
 56    - which pip
 57    - which pip3
 58  script:
 59    - echo "Install->Dependencies..."
 60    - python3 -m pip install -r requirements-dev.txt -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn
 61
 62test-unittest:
 63  stage: test
 64  tags:
 65    - gitlab-org
 66  only:
 67    - main
 68    - alpha
 69    - beta
 70    - rc
 71  before_script:
 72    - export PYTHONIOENCODING=utf8
 73    - source myenv/bin/activate
 74  script:
 75    - echo "Test->Run Unittest..."
 76    - python3 setup.py test
 77
 78build-api-docs:
 79  stage: build
 80  tags:
 81    - gitlab-org
 82  only:
 83    - main
 84    - alpha
 85    - beta
 86    - rc
 87  before_script:
 88    - export PYTHONIOENCODING=utf8
 89    - source myenv/bin/activate
 90  script:
 91    - echo "Build->Generate APIs Documentation..."
 92    - rm -rf dist/html/*
 93    - make html
 94
 95build-packages:
 96  stage: build
 97  tags:
 98    - gitlab-org
 99  only:
100    - main
101    - alpha
102    - beta
103    - rc
104  before_script:
105    - export PYTHONIOENCODING=utf8
106    - source myenv/bin/activate
107    - echo "CI_COMMIT_BRANCH=$CI_COMMIT_BRANCH"
108    - echo "CI_COMMIT_REF_NAME=$CI_COMMIT_REF_NAME"
109  script:
110    - echo "Build->Packaging..."
111    - rm -rf dist/*.whl
112    - python3 setup.py bdist_wheel
113
114release-packages:
115  stage: release
116  tags:
117    - gitlab-org
118  only:
119    - main
120    - alpha
121    - beta
122    - rc
123  before_script:
124    - export PYTHONIOENCODING=utf8
125    - source myenv/bin/activate
126  script:
127    - echo "Release->Packages..."
128    - TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*.whl --skip-existing --verbose
129
130release-tag:
131  stage: release
132  tags:
133    - gitlab-org
134  only:
135    - main
136    - alpha
137    - beta
138    - rc
139  before_script:
140    - export PYTHONIOENCODING=utf8
141    - source myenv/bin/activate
142  script:
143    - echo "Release->Create Tag Branch..."
144    - git --version
145    - git remote remove origin
146    - git remote add origin http://${GITLAB_USERNAME}:${GITLAB_PASSWORD}@gitlab.com/nuangua/testbot.git
147    - git config user.name "%GITLAB_USER_NAME%"
148    - git config user.email "%GITLAB_USER_EMAIL%"
149    - version=$(date +%Y.%W.%u.%H%M)
150    - git tag -a "v${version}" -m "Version ${version} created by gitlab-ci Release stage"
151    - echo git push origin "v${version}"
152
153pages:
154  stage: deploy
155  tags:
156    - gitlab-org
157  only:
158    - main
159    - alpha
160    - beta
161    - rc
162  script:
163    - echo "Deploy->Update APIs Documentation"
164    - rm -rf public/*
165    - cp -r dist/html public/
166  artifacts:
167    paths:
168      - public

流水线变量配置

流水线脚本需要从环境变量读取到GitLab和NEXUS服务器的认证信息,请到TATF仓库的 CI/CD Settings 的Variables页面,添加以下参数:

_static/cicd_variables.png
NEXUS_USERNAME = nuangua
NEXUS_PASSWORD = {NEXUS PASSWORD}
GITLAB_USERNAME = {GITLAB USERNAME}
GITLAB_USERNAME = {GITLAB EMAIL}
GITLAB_PASSWORD = {GITLAB PASSWORD}
TWINE_REPOSITORY_URL = {TWINE REPOSITORY URL}
TWINE_USERNAME = nuangua
TWINE_PASSWORD = {TWINE PASSWORD}

安装流水线代理程序

流水线的执行,依赖Gitlab Runner,由于我们的流水线脚本既需要在Windows主机上执行,也需要在Linux主机上执行。因此,我们需要分别在一台Windows主机和一台Linux主机上安装GitLab Runner。

请到TATF仓库的 CI/CD Settings 的Runner页面,点击链接 Install GitLab Runner ,选择操作系统类型进行相应的安装。

_static/cicd_runners.png

Windows主机安装代理

  • 下载最新的gitlab-runner程序

  • 创建文件夹 D:/GitLab-Runner/tatf ,拷贝gitlab-runner程序到该目录,修改程序名称为gitlab-runner.exe

mkdir D:\GitLab-Runner\testbot
cd D:\GitLab-Runner\testbot
  • 以管理员权限打开命令行窗口,注册一个Runner,过程会提示你输入URL和注册token等信息,该信息可以在Runner页面找到;tag输入为windows

D:\GitLab-Runner\testbot\gitlab-runner.exe register
_static/runner_register.png
  • 注册完成后,会在当前目录生成一个config.toml配置文件,修改配置文件的shell参数为cmd

_static/runner_config.png
  • 以管理员权限打开命令行窗口,安装Runner服务

D:\GitLab-Runner\testbot\gitlab-runner.exe install --service tatf-runner --config D:\GitLab-Runner\testbot\config.toml
  • 打开命令行窗口,启动Runner服务

D:\GitLab-Runner\testbot\gitlab-runner.exe start --service testbot-runner
  • 打开PowerShell窗口,查看Runner日志流

Get-WinEvent -ProviderName gitlab-runner
  • 到浏览器Runner页面检查Runner状态

_static/runner_list.png

Raspbian树莓派Linux主机安装代理

  • 安装最新版本的gitlab-runner程序

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
sudo apt install -y gitlab-runner
  • 创建文件夹 /opt/GitLab-Runner/tatf

sudo mkdir /opt/GitLab-Runner/testbot
cd /opt/GitLab-Runner/testbot
  • 打开命令行窗口,注册一个Runner,过程会提示你输入URL和注册token等信息,该信息可以在Runner页面找到;tag输入为raspbian

sudo gitlab-runner register
_static/runner_register_raspbian.png
  • 打开命令行窗口,安装Runner服务,安装过程会提示你输入URL和注册token等信息

sudo gitlab-runner install install --service testbot-runner --user gitlab-runner
  • 打开命令行窗口,启动Runner服务

sudo gitlab-runner start --service testbot-runner
  • 打开命令行窗口,查看Runner日志流

journalctl -xefu testbot-runner.service

触发流水线执行

GitLab CICD流水线支持3种流水线触发方式:代码提交触发、手动触发、定时触发。

代码提交触发

任何一笔代码提交到仓库TATF,都会触发该仓库相应CICD流水线的执行。

手动触发

_static/trigger_pipeline.png

定时触发

_static/schedule_trigger.png

版本发布

当前我们仅提供pip安装的手动方式,后续我们将会提供一键式安装python、tatf库的命令。

版本分类

流水线会自动对源码进行编译、打包,生成一个pypi安装包。安装包版本类型有:alpha版本、beta版本、rc版本和正式版本,分别对应着alpha分支、beta分支、rc分支和master分支的代码。

_static/cicd_pipeline.png

版本存储

安装包会被上传到一个pypi仓库进行持久化储存。

_static/pypi_repo.png

安装更新版本

  • 安装最新版本

会安装最新的testbot版本,以及其依赖库。

python -m pip install -U testbot
  • 安装指定版本

会安装指定的testbot版本,以及其依赖库。

python -m pip install testbot==2024.45.4
  • 检查当前安装的版本

python -m pip show testbot

返回信息如下:其中Version是当前已安装工具的版本号,Summary里提交ID是当前版本是基于代码仓库的哪个commitid编译打包的,方便我们定位问题。

Name: testbot
Version: 2024.45.4
Summary: TESTBOT测试框架,分支名称:master,提交ID:7e33f39f
Home-page: https://gitlab.com/nuangua/testbot
Author: Nuanguang Gu(Sunny)
Author-email: nuanguang.gu@aliyun.com
License: Copyright (c) 2024 Nuanguang Gu(Sunny) Reserved.
Location: c:\python38\lib\site-packages
Requires: adbutils
Required-by:
  • 卸载

python -m pip uninstall -y testbot