K8s offline deployment

Environments：

• Online env
  • Ubuntu-22.04.3-live-server-amd64, 4 CPU, 8G Memory
• Offline env(target node)
  • Ubuntu-22.04.3-live-server-amd64, 4 CPU, 8G Memory, 100GB available disk space
• Any VM (refer to source VM, localhost)
  • Python 3.10
  • kubespray 2.22.1, kubespray-offline 2.22.1
  • 100GB available disk space
    Note: You must execute this process on the same OS of k8s target nodes.(Online env & target node)

On localhost

Clone kubespray on localhost

git clone https://github.com/kubernetes-sigs/kubespray.git

Install pip

curl -sS https://bootstrap.pypa.io/get-pip.py | python3

Install Ansible

# if your python3 is python3.8
alias python3='/usr/bin/python3.10'

# install venv
sudo apt update
sudo apt install python3.10-venv

# source python venv
VENVDIR=kubespray-venv
KUBESPRAYDIR=kubespray
python3 -m venv $VENVDIR
source $VENVDIR/bin/activate
cd $KUBESPRAYDIR
python3 -m pip install -U -r requirements.txt

Add authorized_keys & update

ssh-keygen
ssh-copy-id {online-env-user}@{online-env-ip}

Install k8s to online-env

apt-get update
Change K8s version & calico to ovn in inventory/online_env/group_vars/k8s_cluster/k8s-cluster.yml

# Copy ``inventory/sample`` as ``inventory/mycluster``
cp -rfp inventory/sample inventory/online_env

# Update Ansible inventory file with inventory builder
declare -a IPS=(online-env-ip)
CONFIG_FILE=inventory/online_env/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

# Review and change parameters under ``inventory/mycluster/group_vars``
cat inventory/online_env/group_vars/all/all.yml
# Update 
cat inventory/online_env/group_vars/k8s_cluster/k8s-cluster.yml

# Clean up old Kubernetes cluster with Ansible Playbook - run the playbook as root
# The option `--become` is required, as for example cleaning up SSL keys in /etc/,
# uninstalling old packages and interacting with various systemd daemons.
# Without --become the playbook will fail to run!
# And be mind it will remove the current kubernetes cluster (if it's running)!
ansible-playbook -i inventory/online_env/hosts.yaml  --become --user={online_env_username} --become-user=root --ask-become-pass reset.yml

# Deploy Kubespray with Ansible Playbook - run the playbook as root
# The option `--become` is required, as for example writing SSL keys in /etc/,
# installing packages and interacting with various systemd daemons.
# Without --become the playbook will fail to run!
# Add--user={username} & --ask-become-pass
ansible-playbook -i inventory/online_env/hosts.yaml  --become --user={online_env_username} --become-user=root --ask-become-pass cluster.yml

Error

Or add Add–user={username} & –ask-become-pass to reset and cluster

• fatal: [node1]: FAILED! => {“attempts”: 4, “censored”: “the output has been hidden due to the fact that ‘no_log: true’ was specified for this result”, “changed”: false}
  Run: vi inventory/{yourcluster}/group_vars/all/mirror.yml, add mirror

Test

Run:sudo kubectl get pods -A

On online env

Clone Kubespray-offline on online-env

Note: You must execute this process on the same OS of k8s target nodes.

git clone https://github.com/kubespray-offline/kubespray-offline.git

Before download offline files, check and edit configurations in kubespray-offlineconfig.sh.
If you don’t have container runtime (docker or containerd), install it first.

• To use Docker CE
  • run install-docker.sh to install Docker CE.
• To use containerd
  • run./install-containerd.sh to install containerd and nerdctl.
  • set docker environment variable to /usr/local/bin/nerdctl in config.sh.
    Then, download all files

Download offline files

Change k8s version

Change download-kubespray-files.sh line 82:

/bin/bash ${KUBESPRAY_DIR}/contrib/offline/generate_list.sh -e kube_version="v1.26.3" || exit 1

Run: download-all.sh
All artifacts are stored in ./outputs directory.
Package outputs:

tar -czvf outputs.tar.gz outputs/

Use the SCP command to transfer the output archive to the machine where you need to install k8s in an offline environment.

scp outputs.tar.gz xxx@192.168.xxx.xxx:.

Offline installation

Target node support scripts

Unpackage tar on the machine where you need to install k8s to an offline environment.

tar -xzvf outputs.tar.gz

Run: outputs/setup-all.sh

Deploy kubernetes using Kubespray

# Create and activate venv:
python3 -m venv ~/.venv/default
source ~/.venv/default/bin/activate

# Note: For Ubuntu 20.04 and RHEL/CentOS 8, you need to use python 3.9.
# python3.9 -m venv ~/.venv/default
# source ~/.venv/default/bin/activate

# Extract kubespray and apply patches:
./extract-kubespray.sh
cd kubespray-{version}

# For Ubuntu 22.04, you need to install build tools to build some python packages.
sudo apt install gcc python3-dev libffi-dev libssl-dev

# Install ansible:
pip install -U pip                # update pip
pip install -r requirements.txt   # Install ansible

Create offline.yml

Create and place offline.yml file to your group_vars/all/offline.yml of your inventory directory.
You need to change YOUR_HOST with your registry/nginx host IP.

http_server: "http://YOUR_HOST"
registry_host: "YOUR_HOST:35000"

# Insecure registries for containerd
containerd_registries_mirrors:
  - prefix: "{{ registry_host }}"
    mirrors:
      - host: "http://{{ registry_host }}"
        capabilities: ["pull", "resolve"]
        skip_verify: true

files_repo: "{{ http_server }}/files"
yum_repo: "{{ http_server }}/rpms"
ubuntu_repo: "{{ http_server }}/debs"

# Registry overrides
kube_image_repo: "{{ registry_host }}"
gcr_image_repo: "{{ registry_host }}"
docker_image_repo: "{{ registry_host }}"
quay_image_repo: "{{ registry_host }}"

# Download URLs: See roles/download/defaults/main.yml of kubespray.
kubeadm_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubeadm"
kubectl_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubectl"
kubelet_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubelet"
# etcd is optional if you **DON'T** use etcd_deployment=host
etcd_download_url: "{{ files_repo }}/kubernetes/etcd/etcd-{{ etcd_version }}-linux-amd64.tar.gz"
cni_download_url: "{{ files_repo }}/kubernetes/cni/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
crictl_download_url: "{{ files_repo }}/kubernetes/cri-tools/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
# If using Calico
calicoctl_download_url: "{{ files_repo }}/kubernetes/calico/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
# If using Calico with kdd
calico_crds_download_url: "{{ files_repo }}/kubernetes/calico/{{ calico_version }}.tar.gz"

runc_download_url: "{{ files_repo }}/runc/{{ runc_version }}/runc.{{ image_arch }}"
nerdctl_download_url: "{{ files_repo }}/nerdctl-{{ nerdctl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
containerd_download_url: "{{ files_repo }}/containerd-{{ containerd_version }}-linux-{{ image_arch }}.tar.gz"

Notes:

• runc_donwload_url differ from kubespray official document, and must include runc_version.
• The insecure registries configurations of containerd were changed from kubespray 2.23.0. You need to set containerd_registries_mirrors instead of containerd_insecure_registries.

Error

• Install requirement.txt error
  You can download whl and install offline

Deploy offline repo configurations

Deploy offline repo configurations which use your yum_repo/ubuntu_repo to all target nodes using ansible.
First, copy offline setup playbook to kubespray directory.

cp -r ${outputs_dir}/playbook ${kubespray_dir}

Then execute offline-repo.yml playbook.

cd ${kubespray_dir}
ansible-playbook -i ${your_inventory_file} offline-repo.yml

Run kubespray

Run kubespray ansible playbook.

# Example  
ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=root cluster.yml

Finish！！！