Migration Cluster from vContainer to VKS

Tools used: Velero + vStorage (s3)

1. Process

Process of migrating from vContainer to vKS (Customer + VNG Cloud)

Step 1: Evaluate the current version of the vContainer cluster and the corresponding vKS cluster to migrate. At this point, there will be 2 cases (step 2 and step 3)

Step 2: If the vContainer version is lower than the versions currently supported by vKS, then the Custom Resources (CRDs) need to be reviewed for compatibility with the new kubernetes versions.

• If compatible with the new vKS version, continue with step 3.

• If not compatible, need to manually update and reconfigure CRD as well as related Applications.

Step 3: If the vContainer version is supported by vKS (1.27, 1.28 and 1.29), then you need to check the resources on vContainer before backing up. You need to pay attention to the following types of resources:

• PV/PVC: Velero does not support backup with hostPath type, only Local type is supported.

• Ingress resources : Ingress resources managed by container-ingress-nginx-controller will not work after migration.

• Label and Taint nodes : Velero does not re-attach Labels and Taints to nodes in the vKS

Step 4 : Backup resources on vContainer cluster

Step 5 : Restore resources on vKS cluster

Step 6 : Perform testing and adjustments

2. Important Note:

1. Mapping StorageClass on vKS cluster

Check the existing StorageClasses on vContainer and create corresponding StorageClasses on vKS. Then perform 1:1 mapping of StorageClasses between vContainer and vKS cluster.

Example ConfigMap file in Step 3.

2. Use PersistentVolume as hostPath

Velero does not support hostPath volume backup, only local.

For clusters using PV type hostPath, it is necessary to convert to local type as follows:

Note: Need to delete and redeploy the Application using PV type hostPath

Case 1: Type Local supports hostPath path

For example, PV hostPath is configured to mount at the /opt/data folder, convert to Local type and mount back to Pods, according to the following form:

apiVersion: storage.k8s.io/v1 

kind: StorageClass 

metadata: 

  name: manual 

provisioner: kubernetes.io/no-provisioner 

volumeBindingMode: WaitForFirstConsumer 

--- 

apiVersion: v1 

kind: PersistentVolume 

metadata: 

  name: pv-volume 

  labels: 

    type: local 

spec: 

  storageClassName: manual 

  capacity: 

    storage: 1Gi 

  accessModes: 

    - ReadWriteOnce 

  local: 

    path: "/opt/data" 

  nodeAffinity: 

    required: 

      nodeSelectorTerms: 

      - matchExpressions: 

        - key: kubernetes.io/hostname 

          operator: Exists 

--- 

apiVersion: v1 

kind: PersistentVolumeClaim 

metadata: 

  name: pv-claim 

spec: 

  storageClassName: manual 

  accessModes: 

    - ReadWriteOnce 

  resources: 

    requests: 

      storage: 1Gi 

Case 2: Type Local does not support the original hostPath path

When creating PV/PVC according to Case 1, Pod cannot mount PVC and an error appears

MountVolume.NewMounter initialization failed for volume "pvc" : path "/mnt/data" does not exist

Copy the data to a new folder (eg /var, /opt, /tmp, ...) and repeat Case 1, then mount the PVC into the Pod as usual:

cp -R /mnt/data /var 

3. Detailed implementation steps

Step 1: Install Velero on both clusters (vContainer and vKS)

Create a vStorage project, Container and corresponding S3 key to store backup data

On both clusters:

• Create a credentials-velero file with the following content:

[default] 
aws_access_key_id=________________________ # <= Adjust here 
aws_secret_access_key=________________________ # <= Adjust here 

• Install Velero CLI

curl -OL  https://github.com/vmware-tanzu/velero/releases/download/v1.14.1/velero-v1.14.1-linux-amd64.tar.gz 

tar -xvf velero-v1.14.1-linux-amd64.tar.gz 
cp velero-v1.14.1 -linux-amd64/velero /usr/local/bin 

• Install Velero on 2 kubernetes clusters

velero install \ 
    --provider aws \ 
    --plugins velero/velero-plugin-for-aws:v1.9.0 \ 
    --use-node-agent \ 
    --use-volume-snapshots=false \ 
    --secret-file ./credentials-velero \ 
    --bucket __________\                    # <= Adjust here 
    --backup-location-config region=hcm03,s3ForcePathStyle="true",s3Url=https://hcm03.vstorage.vngcloud.vn 

Step 2: Perform backup on vContainer Cluster

• Annotate the Persistent Volumes to be backed up

./velero_helper.sh mark_volume --confirm 

• Annotate non-backup resources of kube-system

./velero_helper.sh mark_exclude --confirm 

• Annotate other resources (not marked in the velero_helper.sh file), such as CSI, LoadBalancer Controller, or other resources that you do not want to migrate (note that you need to label all resources of objects that do not need to be backed up).

  • For example, an application includes DaemonSet, Deployment, Pod, ... then it is necessary to mark labels for all those resources.

# Thêm label velero.io/exclude-from-backup=true cho từng resources 

 

## Với Cinder CSI 

kubectl -n kube-system label StatefulSet/csi-cinder-controllerplugin velero.io/exclude-from-backup=true 

kubec kubectl -n kube-system label DaemonSet/csi-cinder-nodeplugin velero.io/exclude-from-backup=true  

 

## Với vcontainer-ingress-nginx-controller 

kubectl -n kube-system label Deployment/ vcontainer-ingress-nginx-controller velero.io/exclude-from-backup=true 

kubectl -n kube-system label Deployment/ vcontainer-ingress-nginx-default-backend velero.io/exclude-from-backup=true 

• Perform check and attach Labels and Taints on vContainer nodes to vKS nodes before restore

# Kiểm tra các Label nodes 
./velero_helper.sh check_node_label 
# Kiểm tra các Taint nodes 
./velero_helper.sh check_node_taint 

• Create 2 backups for Cluster resources and Namespace resources according to the syntax

# Tạo cluster resource backup 
velero backup create vcontainer-cluster --include-namespaces ""  --include-cluster-resources=true --wait 

 

# Tạo cluster namespace backup 

velero backup create vcontainer-namespace --exclude-namespaces velero --wait 

 

# Xóa các bản backup (nếu cần) 

velero backup delete vcontainer-namespace vcontainer-cluster --confirm 

• View created backups and details of backed up resources (note the STATUS of the backup)

# List các bản backup được tạo 
velero get backup 

# Chi tiết của một bản backup 

velero backup describe <bk-name> --details 

# Xem logs quá trình backup 

velero backup logs <bk-name> 

Step 3: Perform Restore on VKS Cluster

• If in the vContainer cluster using CSI is cinder.csi.openstack.org , need to perform StorageClass mapping between 2 clusters vContainer and vKS

  • Mapping csi-sc-cinderplugin-nvme-5000 (vContainer) and vngcloud-nvme-5000-delete (vKS), similarly for other StorageClasses

  • In vKS, it is necessary to create corresponding StorageClasses.

Create sc-mapping.yaml file and apply on VKS cluster

apiVersion: v1 

kind: ConfigMap 

metadata: 

  name: change-storage-class-config 

  namespace: velero 

  labels: 

    velero.io/plugin-config: "" 

    velero.io/change-storage-class: RestoreItemAction 

data: 

  csi-sc-cinderplugin-nvme-5000: vngcloud-nvme-5000-delete 

  #_______old_storage_class_______: _______new_storage_class_______ # <= Add here 

• Add permissions for Velero to restore data PersistentVolume

Create add-permission.yaml file and apply on VKS cluster

apiVersion: v1 

kind: ConfigMap 

metadata: 

  name: fs-restore-action-config 

  namespace: velero 

  labels: 

    velero.io/plugin-config: "" 

    velero.io/pod-volume-restore: RestoreItemAction 

data: 

  secCtx: | 

    capabilities: 

      drop: [] 

      add: [] 

    allowPrivilegeEscalation: false 

    readOnlyRootFilesystem: true 

    runAsUser: 0 

    runAsGroup: 0 

• Perform restore in order

  • Note, for each restore, you need to check whether the restore process was successful or not before continuing to execute other commands.

# Kiểm tra restore 

velero get restore 

velero describe restore <restore-name> --details 

velero restore create --item-operation-timeout 1m --from-backup vcontainer-cluster --exclude-resources="MutatingWebhookConfiguration,ValidatingWebhookConfiguration" 

velero restore create --item-operation-timeout 1m --from-backup vcontainer-namespace 

velero restore create --item-operation-timeout 1m --from-backup vcontainer-cluster 

• In case of migrating to VKS and still using vcontainer-nginx-ingress-controller, it is necessary to change the Service type to LoadBalancer

kubectl patch service -n kube-system vcontainer-ingress-nginx-controller -p '{"spec": {"typ