For the last few days I have been trying out deploying K8s with a CSI based storage so I can test Kasten K10.

I found this article:

Rook, Ceph, CSI, Kubernetes, and K10: An All-in-One Stateful Experience on your Laptop (kasten.io)

On how do to do this on Docker Desktop, so have modified the process slightly to working a standalone vanilla K8s cluster.

For cluster I have 4 x Ubuntu 20.04 VMware (6.7) VMs; one controller and three workers. Each have 2 vCPU and 4GB RAM, 16GB OS and 100GB data disk on the workers all thin provisioned.

I’ve installed k8s on the cluster using kubeadm with calico networking.

All nodes:

cat << EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/systemctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables    = 1
net.ipv4.ip_forward                   = 1
net.bridge.bridge-nf-call-ip6tables   = 1
EOF

sudo sysctl --system 

sudo apt-get update && sudo apt-get install -y containerd

sudo mkdir -p /etc/containerd

sudo containerd config default | sudo tee /etc/containerd/config.toml

sudo systemctl restart containerd

# sudo systemctl status containerd

sudo swapoff -a

sudo rm /swap.img

sudo vim /etc/fstab

# Press i
# Delete or comment out the below line
# /swap.img     none    swap    sw      0       0
# Press ESC, :, w, q, ENTER

sudo apt-get update && sudo apt-get install -y apt-transport-https curl

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF

sudo apt-get update

# 1.21.3 was most up-to-date version at time of writing
sudo apt-get install -y kubelet=1.21.3-00 kubeadm=1.21.3-00 kubectl=1.21.3-00

sudo apt-mark hold kubelet kubeadm kubectl

Then on Control node unless otherwise stated

sudo kubeadm init --pod-network-cidr 192.168.0.0/16 --kubernetes-version 1.21.3

# run the mkdir command shown on the screen

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

# check control node is showing ready
kubectl get nodes

# Run the result of this command on each of the workers
kubeadm token create --print-join-command

# This should show all nodes in read state
kubectl get nodes

Installing and configuring Rook.

First need to make sure that you have a sdb drive attached to the worker nodes.

sudo lsblk

Then follow the next steps, you’ll find most of them in the quick start but there are some modifications due to this being a cluster instead of local.

Rook Docs

The changes as well as the info on the toolbox came from the Linux Foundation.

Kubernetes – Getting Started With Rook – YouTube

git clone https://github.com/rook/rook.git

cd rook

git checkout v1.6.8

cd rook/cluster/examples/kubernetes/ceph

kubectl create -f crd.yaml -f common.yaml -f operator.yaml

vim cluster.yaml

You will need to make some changes to the cluster.yaml file

Awesome SSH Client: Release Alpha 148 · Eugeny/tabby (github.com)

storage.nodes I have added:

storage:
  useAllNodes: true
  useAllDevices: true
  config:
  nodes:
  - name: "k8s-worker1"
    devices: 
    - name: "sdb"
  - name: "k8s-worker2"
    devices:
    - name: "sdb"
  -name: "k8s-worker3"
    devices:
    - name: "sdb"

Make sure that your node names match up with your node names.

Next run:

kubectl apply -f cluster.yaml

kubectl apply -f csi/rbd/storageclass.yaml

kubectl apply -f toolbox.yaml

# check progress with 

kubectl -n rook-ceph get pods --watch

When you have three (or as many worker nodes as you have added ) of the below:

rook-ceph-osd-X-XXXXXXX-XXXX

Then you know that everything is up and running. But if you want a closer look we can use the toolbox pod created above.

kubectl -n rook-ceph exec -it <pod_name> bash

# Overall status
ceph status

# Information on the 
ceph osd status

exit

Next create the StorageClass

kubectl create -f csi/rbd/storageclass.yaml

I used the one above as opposed to the storageclass-test in the Kasten blog as it suites the multi-node cluster that has been deployed.

Now for some reason that I’m still working out I could not just create the VolumeSnapshotClass via:

kubectl create -f csi/rbd/snapshotclass.yaml

This throw up errors for me. Apparently this was an issue when it was still in beta but my understanding is that it has been included with snapshot.storage.k8s.io/v1 since version 1.20 and I’m running 1.21.3.

I suggest trying the command above and if you hit an error these are the steps that I took to fix it. These are also detailed on the https://github.com/kubernetes-csi/external-snapshotter page.

git clone https://github.com/kubernetes-csi/external-snapshotter.git

cd external-snapshotter

kubectl create -f client/config/crd

kubectl create -f deploy/kubernetes/snapshot-controller

Now you should be able to create the SnapshotClass shown above.

Next we need to make a few changes to the StorageClass and VolumeSnapshotClass so they’ll work in our set up.

# Without it being the default it has caused issues
kubectl patch storageclass rook-ceph-block \
    -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

kubectl annotate volumesnapshotclass csi-rbdplugin-snapclass k10.kasten.io/is-snapshot-class=true

# You can modify the snapshot class before applying but this is just as easy
kubectl patch volumesnapshotclass csi-rbdplugin-snapclass -p '{"deletionPolicy":"Retain"}' --type=merge

Right we’re now done with Rook 😀

Kasten Install

Kasten install is well documented on the site but I’m putting the steps below for ease.

# runs a check to see if everyone is good to go
curl https://docs.kasten.io/tools/k10_primer.sh | bash

# adds the kasten k10 repo
helm repo add kasten https://charts.kasten.io/

# installs k10
helm install k10 kasten/k10 -n=kasten-io

# monitor the progress
kubectl -n kasten-io get pods --watch

Now the bit that caught me off guard was how the heck to access the k10 Dashboard externally as the Kasten blog assumes you are running it locally so 127.0.0.1 will work.

So you will need to create a NodePort using the following.

# Expose the node port to the Gateway Service
kubectl expose service gateway -n kasten-io --type=NodePort --name=gateway-nodeport

# Get the Node port
kubectl get svc -n kasten-io gateway-nodeport

You can then access the interface externally via:

http://192.168.4.77:31013/k10/#/

Obviously replacing the IP and Port with your relative ones or add it to DNS and it give it a friendly name.

After that is should just work.

Thanks for reading!