In this article, we will guide you through the step-by-step process of installing the Mayastor Add-on. MicroK8s, a lightweight Kubernetes distribution, offers a convenient way to deploy Kubernetes clusters for development and testing purposes. If you’re looking to leverage containerized storage in your MicroK8s cluster, the Mayastor add-on can be a valuable addition. Mayastor provides high-performance, scalable, and persistent storage for containerized environments.
Please note that the following requirements are applicable to all nodes in a MicroK8s cluster. Make sure to execute the commands on each individual node.
To enable Mayastor Add-on, it is necessary to have HugePages enabled, with a minimum requirement of 1024 4MB HugePages. This can be accomplished by running the following commands on each host:
sudo sysctl vm.nr_hugepages=1024
echo 'vm.nr_hugepages=1024' | sudo tee -a /etc/sysctl.confAdditionally, the nvme_fabrics and nvme_tcp modules must be installed and enabled on all hosts. Use the following commands for installation:
sudo apt install -y linux-modules-extra-$(uname -r)And enable the modules with:
sudo modprobe nvme_tcp
echo 'nvme-tcp' | sudo tee -a /etc/modules-load.d/microk8s-mayastor.confIf MicroK8s is already installed, it is advised to restart MicroK8s at this point:
microk8s stop
microk8s startMicroK8s DNS and Helm3 addons are required and will be automatically installed if missing.
Install Mayastor Add-on
Assuming that the cluster has been configured as mentioned above, you can now enable Mayastor Add-on by following these steps:
Enable the Mayastor add-on:
sudo microk8s enable core/mayastor --default-pool-size 20GWait for the Mayastor Add-on control plane and data plane pods to start:
microk8s kubectl get pod -n mayastorMayastor Add-on will automatically create one DiskPool per node in the MicroK8s cluster, with each pool backed by a sparse image file. For information on using existing block devices, refer to the Mayastor documentation.
To verify that all disk pools are operational, use the following command:
microk8s kubectl get diskpool -n mayastorIn a 3-node cluster, the output should resemble the following:
NAME NODE STATUS CAPACITY USED AVAILABLE
microk8s-m2-pool m2 Online 21449670656 0 21449670656
microk8s-m1-pool m1 Online 21449670656 0 21449670656
microk8s-m3-pool m3 Online 21449670656 0 21449670656Mayastor Add-on is now successfully deployed!
To deploy a test workload, the Mayastor addon provides two storage classes:
mayastor: Suitable for single-node clusters.mayastor-3: Requires a minimum of 3 cluster nodes and replicates volume data across 3 storage pools to ensure data redundancy.
To create a pod using the mayastor storage class, use the provided YAML file (pod-with-pvc.yaml):
# pod-with-pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-pvc
spec:
storageClassName: mayastor
accessModes: [ReadWriteOnce]
resources: { requests: { storage: 5Gi } }
---
apiVersion: v1
kind: Pod
metadata:
name: test-nginx
spec:
volumes:
- name: pvc
persistentVolumeClaim:
claimName: test-pvc
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: pvc
mountPath: /usr/share/nginx/htmlCreate the pod using the following command:
microk8s kubectl create -f pod-with-pvc.yamlVerify the creation of the PVC and pod with the following command:
microk8s kubectl get pod,pvcTo configure custom storage classes for advanced use cases, such as defining the number of replicas and the underlying protocol, use the provided example command:
microk8s kubectl apply -f - <<EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: mayastor-2
parameters:
repl: '2'
protocol: 'nvmf'
ioTimeout: '60'
local: 'true'
provisioner: io.openebs.csi-mayastor
volumeBindingMode: WaitForFirstConsumer
EOFFor more information, refer to the documentation on creating Mayastor StorageClass(es).
To configure MayaStor pools, it is recommended to use designated disks instead of the default local image file per node. A helper script, accessible as microk8s mayastor-pools, is provided for easy creation, listing, and deletion of Mayastor pools within the cluster.
For example, the following commands demonstrate various pool operations:
# Get help
microk8s mayastor-pools --help
# Create a Mayastor pool using `/dev/sdb` on node `uk8s-1`
microk8s mayastor-pools add --node uk8s-1 --device /dev/sdb
# Create a Mayastor pool of 100GB using a sparse image file on node `uk8s-1`. The image file will be placed under `/var/snap/microk8s/common/mayastor/data`.
microk8s mayastor-pools add --node uk8s-1 --size 100GB
# List Mayastor pools
microk8s mayastor-pools list
# Delete a Mayastor pool. The --force flag removes the pool even if it is in use, while --purge removes the backing image file. The mayastor pool name is required as it appears in the output of the list command.
microk8s mayastor-pools remove microk8s-uk8s-1-pool --force --purgeFor additional information, consult the documentation on creating Mayastor Pool(s).
Troubleshooting
If you encounter difficulties starting the Mayastor data plane, and the mayastor data plane pods are stuck in the CrashLoopBackOff state, it may be due to a failure in initializing EAL. To verify this, check the logs of the mayastor daemonset using the following command:
microk8s kubectl logs -n mayastor daemonset/mayastorIf the logs contain an error message similar to the following, indicating IOVA exceeding limits or failed EAL initialization, follow the solution steps:
EAL: alloc_pages_on_heap(): couldn't allocate memory due to IOVA exceeding limits of current DMA mask
EAL: alloc_pages_on_heap(): Please try initializing EAL with --iova-mode=pa parameter
EAL: error allocating rte services array
EAL: FATAL: rte_service_init() failed
EAL: rte_service_init() failed
thread 'main' panicked at 'Failed to init EAL', mayastor/src/core/env.rs:543:13To resolve the issue, edit the manifest of the mayastor daemonset using the following command:
microk8s kubectl edit -n mayastor daemonset mayastorThen, add the argument --env-context=--iova-mode=pa to the command line of the mayastor pod. Save and exit the editor to apply the changes, and wait for the mayastor pods to restart.
0 Comments