深圳幻海软件技术有限公司 欢迎您！
更多

Kubernetes 集群备份工具 velero 使用指南

2023-03-20

root io to
​​想了解更多关于开源的内容，请访问：​​​​51CTO 开源基础软件社区​​​​https://ost.51cto.com​​简介Velero 是一款可以安全的备份、恢复和迁移Kubernetes集群资源和持久卷等资源的备份恢复软件。Velero实现的kubernetes资源备

​​想了解更多关于开源的内容，请访问：​​

​​51CTO 开源基础软件社区​​

​​https://ost.51cto.com​​

简介

Velero 是一款可以安全的备份、恢复和迁移 Kubernetes 集群资源和持久卷等资源的备份恢复软件。

Velero 实现的 kubernetes 资源备份能力，可以轻松实现 Kubernetes 集群的数据备份和恢复、复制 kubernetes 集群资源到其他 kubernetes 集群或者快速复制生产环境到测试环境等功能，这种备份就类似于把资源的 yaml 文件进行整体备份，从而保障资源的完整性。

Velero 对存储的支持较好，可以支持很多种存储资源，比如 AWS S3、Azure Blob、Google Cloud Storage、Alibaba Cloud OSS、Swift、MinIO 等等.

​​Storage-support​​。

流程图

注意：本文档采用 MinIO 作为后端存储。

Docker 部署 MinIO

因为我们要保证存储和集群隔离，这样能保证在集群出问题的时候，不影响备份，所以就将 MinIO 部署在其他服务器上，与 Kubernetes 环境隔离。避免因意外情况宕机的时候影响更多。

mkdir -pv /data/{minio,config}
docker pull minio/minio
docker pull minio/mc:latest

docker run -d -p 9000:9000 -p 9090:9090 --name=minio --restart=always -e "MINIO_ROOT_USER=admin" -e "MINIO_ROOT_PASSWORD=admin123456" -v /data/minio:/data -v /data/config:/root/.minio  minio/minio server /data --console-address ":9090" --address ":9000"

# 对minio 授权
docker run -it --entrypoint=/bin/sh -v /data/config:/root/.minio minio/mc

mc config host add velero http://127.0.0.1:9000 admin admin123456 --api S3v4 
mc mb -p velero/velero
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.

注意，mc 使用的是 api 端口，并不是 console 界面端口，所以 mc 在授权的时候需要使用 API 端口。


然后浏览器通过配置的账号和 password 即可登录：

安装 velero 在 k8s master 上

下载 velero：

https://github.com/vmware-tanzu/velero/releases/download/v1.9.0/velero-v1.9.0-linux-amd64.tar.gz。

1、创建密钥并安装velero

cat > credentials-velero <<EOF
[default]
aws_access_key_id = admin
aws_secret_access_key = admin123456
EOF
# 安装velero，配置自动补全
cp velero /usr/local/bin/
echo 'source <(velero completion bash)' >> /etc/profile && source /etc/profile
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.

2、K8s集群安装velero

# 需要提前下载镜像 velero/velero-plugin-for-aws:v1.2.1 和velero/velero:v1.9.0
velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.2.1 \
    --bucket velero \
    --secret-file ./credentials-velero \
    --use-volume-snapshots=false \
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://10.0.2.196:9000
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.

安装时产生的日志。

CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: attempting to create resource client
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource client
CustomResourceDefinition/resticrepositories.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: attempting to create resource client
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: attempting to create resource client
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero: attempting to create resource
Namespace/velero: attempting to create resource client
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: attempting to create resource client
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: attempting to create resource client
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: attempting to create resource client
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: attempting to create resource client
BackupStorageLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: attempting to create resource client
Deployment/velero: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.
  • 35.
  • 36.
  • 37.
  • 38.
  • 39.
  • 40.
  • 41.
  • 42.
  • 43.
  • 44.
  • 45.
  • 46.
  • 47.
  • 48.
  • 49.
  • 50.
  • 51.
  • 52.
  • 53.

验证是否安装完毕。

[root@master ~]# kubectl get pod -n velero 
NAME                     READY   STATUS    RESTARTS   AGE
velero-cf79784ff-zdfp9   1/1     Running   0          47s
[root@master ~]# kubectl  api-versions  | grep velero
velero.io/v1
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.

3、velero 命令

velero create backup NAME [flags]
# 剔除 namespace
--exclude-namespaces stringArray                  namespaces to exclude from the backup
# 剔除资源类型
--exclude-resources stringArray                   resources to exclude from the backup, formatted as resource.group, such as storageclasses.storage.k8s.io
# 包含集群资源类型 
--include-cluster-resources optionalBool[=true]   include cluster-scoped resources in the backup
# 包含 namespace
--include-namespaces stringArray                  namespaces to include in the backup (use '*' for all namespaces) (default *)
# 包含 namespace 资源类型
--include-resources stringArray                   resources to include in the backup, formatted as resource.group, such as storageclasses.storage.k8s.io (use '*' for all resources)
# 给这个备份加上标签
--labels mapStringString                          labels to apply to the backup
-o, --output string                               Output display format. For create commands, display the object but do not send it to the server. Valid formats are 'table', 'json', and 'yaml'. 'table' is not valid for the install command.
# 对指定标签的资源进行备份
-l, --selector labelSelector                      only back up resources matching this label selector (default <none>)
# 对 PV 创建快照
--snapshot-volumes optionalBool[=true]            take snapshots of PersistentVolumes as part of the backup
# 指定备份的位置
--storage-location string                         location in which to store the backup
# 备份数据多久删掉
--ttl duration                                    how long before the backup can be garbage collected (default 720h0m0s)
# 指定快照的位置，也就是哪一个公有云驱动
--volume-snapshot-locations strings               list of locations (at most one per provider) where volume snapshots should be stored
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.

4、测试

velero 在安装包中有一个测试demo，我们可以利用测试demo来进行测试验证。

[root@master ~]# cd velero-v1.9.0-linux-amd64/examples/nginx-app/
[root@master nginx-app]# ls
base.yaml  README.md  with-pv.yaml
# 注意修改 with-pv 中的 storageclass
storageClassName: longhorn
# 修改 svc 类型为 ClusterIP
type: ClusterIP
[root@master nginx-app]# kubectl apply -f with-pv.yaml 
namespace/nginx-example created
persistentvolumeclaim/nginx-logs created
deployment.apps/nginx-deployment created
service/my-nginx created
[root@master nginx-app]# kubectl get svc
NAME       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
my-nginx   ClusterIP   10.108.189.178   <none>        80/TCP    79s
[root@master nginx-app]# kubectl get all -n nginx-example 
NAME                                    READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-6779884c68-4nv2s   2/2     Running   0          2m16s

NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/my-nginx   ClusterIP   10.108.189.178   <none>        80/TCP    2m16s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   1/1     1            1           2m16s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-deployment-6779884c68   1         1         1       2m16s
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.

5、备份

[root@master nginx-app]# velero backup create nginx-backup --include-namespaces nginx-example
Backup request "nginx-backup" submitted successfully.
Run `velero backup describe nginx-backup` or `velero backup logs nginx-backup` for more details.
  • 1.
  • 2.
  • 3.
[root@master nginx-app]# velero backup describe nginx-backup
Name:         nginx-backup
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.23.5
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=23
Phase:  Failed (run `velero backup logs nginx-backup` for more information)
Errors:    0
Warnings:  0
Namespaces:
  Included:  nginx-example
  Excluded:  <none>
Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto
Label selector:  <none>
Storage Location:  default
Velero-Native Snapshot PVs:  auto
TTL:  720h0m0s
Hooks:  <none>
Backup Format Version:  1.1.0
Started:    2022-06-30 17:44:49 +0800 CST
Completed:  2022-06-30 17:44:49 +0800 CST
Expiration:  2022-07-30 17:44:49 +0800 CST
Velero-Native Snapshots: <none included>
[root@master nginx-app]# velero backup logs nginx-backup
An error occurred: gzip: invalid header
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.

登录 minio 查看，备份记录。

6、恢复测试

删除 nginx-app：

[root@master velero-v1.9.0-linux-amd64]# kubectl delete -f examples/nginx-app/base.yaml
  • 1.

恢复 nginx：

[root@master velero-v1.9.0-linux-amd64]# velero restore create --from-backup nginx-backup --wait
Restore request "nginx-backup-20220630192521" submitted successfully.
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.

Restore completed with status: Completed. You may check for more information using the commands `velero restore describe nginx-backup-20220630192521` and `velero restore logs nginx-backup-20220630192521`.
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
[root@master velero-v1.9.0-linux-amd64]# kubectl  get pods -n nginx-example
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-6779884c68-4nv2s   2/2     Running   0          27s
  • 1.
  • 2.
  • 3.

恢复会新建一个 restores。

目录注意：​​velero restore​​​ 恢复不会覆盖​​已有的资源​​​，只恢复当前集群中​​不存在的资源​​。已有的资源不会回滚到之前的版本，如需要回滚，需在restore之前提前删除现有的资源。

备份所有命名空间：

velero backup logs k8s-all
  • 1.

7、定时备份

比较推荐做定时全量备份，这个能保证我们是一直全量最新备份的状态。

# 每天12点备份一次
velero create schedule k8s-all --schedule="0 0 * * *"
  • 1.
  • 2.

查看备份：

velero  get  backup   #备份查看
velero  get  schedule #查看定时备份
velero  get  restore  #查看已有的恢复
velero  get  plugins  #查看插件
  • 1.
  • 2.
  • 3.
  • 4.

​​想了解更多关于开源的内容，请访问：​​

​​51CTO 开源基础软件社区​​

​​https://ost.51cto.com​​。

决定从云端回迁之前需要考虑什么
探索5G与边缘计算跨行业协同