Lab 1: Non-persistent storage with Kubernetes¶
Storing data in containers or worker nodes are considered as the non-persistent forms of data storage. In this lab, we will explore storage options on the IBM Kubernetes worker nodes. Follow this lab is you are interested in learning more about container-based storage.
The lab covers the following topics:
- Create and claim IBM Kubernetes non-persistent storage based on the primary and secondary storage available on the worker nodes.
- Make the volumes available in the
Guestbook
application. - Use the volumes to store application cache and debug information.
- Access the data from the guestbook container using the Kubernetes CLI.
- Assess the impact of losing a pod on data retention.
- Claim back the storage resources and clean up.
The primary storage maps to the volume type hostPath
and the secondary storage maps to emptyDir
. Learn more about Kubernetes volume types here.
Reserve Persistent Volumes¶
From the cloud shell prompt, run the following commands to get the guestbook application and the kubernetes configuration needed for the storage labs.
cd $HOME
git clone --branch fs https://github.com/IBM/guestbook-nodejs.git
git clone --branch storage https://github.com/rojanjose/guestbook-config.git
cd $HOME/guestbook-config/storage/lab1
Let's start with reserving the Persistent volume from the primary storage.
Review the yaml file pv-hostpath.yaml
. Note the values set for type
, storageClassName
and hostPath
.
apiVersion: v1
kind: PersistentVolume
metadata:
name: guestbook-primary-pv
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data"
Create the persistent volume as shown in the command below:
kubectl create -f pv-hostpath.yaml
persistentvolume/guestbook-primary-pv created
kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
guestbook-primary-pv 10Gi RWO Retain Available manual 13s
Next PVC yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: guestbook-local-pvc
spec:
storageClassName: manual
accessModes:
- ReadWriteMany
resources:
requests:
storage: 3Gi
Create PVC:
kubectl create -f pvc-hostpath.yaml
persistentvolumeclaim/guestbook-local-pvc created
❯ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
guestbook-local-pvc Bound guestbook-local-pv 10Gi RWX manual 6s
Guestbook application using storage¶
The application is the Guestbook App, which is a simple multi-tier web application built using the loopback framework.
Change to the guestbook application source directory:
cd $HOME/guestbook-nodejs/src
Review the source common/models/entry.js
. The application uses storage allocated using hostPath
to store data cache in the file data/cache.txt
. The file logs/debug.txt
records debug messages provisioned using the emptyDir
storage type.
module.exports = function(Entry) {
Entry.greet = function(msg, cb) {
// console.log("testing " + msg);
fs.appendFile('logs/debug.txt', "Received message: "+ msg +"\n", function (err) {
if (err) throw err;
console.log('Debug stagement printed');
});
fs.appendFile('data/cache.txt', msg+"\n", function (err) {
if (err) throw err;
console.log('Saved in cache!');
});
...
Run the commands listed below to build the guestbook image and copy into the docker hub registry:
docker build -t $DOCKERUSER/guestbook-nodejs:storage .
docker login -u $DOCKERUSER
docker push $DOCKERUSER/guestbook-nodejs:storage
Review the deployment yaml file guestbook-deplopyment.yaml
prior to deploying the application into the cluster.
cd $HOME/guestbook-config/storage/lab1
cat guestbook-deployment.yaml
Replace the first part of image
name with your docker hub user id.
The section spec.volumes
lists hostPath
and emptyDir
volumes. The section spec.containers.volumeMounts
lists the mount paths that the application uses to write in the volumes.
apiVersion: apps/v1
kind: Deployment
metadata:
name: guestbook-v1
labels:
app: guestbook
...
spec:
containers:
- name: guestbook
image: rojanjose/guestbook-nodejs:storage
imagePullPolicy: Always
ports:
- name: http-server
containerPort: 3000
volumeMounts:
- name: guestbook-primary-volume
mountPath: /home/node/app/data
- name: guestbook-secondary-volume
mountPath: /home/node/app/logs
volumes:
- name: guestbook-primary-volume
persistentVolumeClaim:
claimName: guestbook-primary-pvc
- name: guestbook-secondary-volume
emptyDir: {}
...
Deploy the Guestbook application:
kubectl create -f guestbook-deployment.yaml
deployment.apps/guestbook-v1 created
kubectl get pods
NAME READY STATUS RESTARTS AGE
guestbook-v1-6f55cb54c5-jb89d 1/1 Running 0 14s
kubectl create -f guestbook-service.yaml
service/guestbook created
Find the URL for the guestbook application by joining the worker node external IP and service node port.
HOSTNAME=`kubectl get nodes -o wide | tail -n 1 | awk '{print $7}'`
SERVICEPORT=`kubectl get svc guestbook -o=jsonpath='{.spec.ports[0].nodePort}'`
echo "http://$HOSTNAME:$SERVICEPORT"
Open the URL in a browser and create guest book entries.
Next, inspect the data. To do this, run a bash process inside the application container using kubectl exec
. Reference the pod name from the previous kubectl get pods
command. Once inside the container, use the subsequent comands to inspect the data.
kubectl exec -it [POD NAME] -- bash
root@guestbook-v1-6f55cb54c5-jb89d:/home/node/app# ls -al
total 256
drwxr-xr-x 1 root root 4096 Nov 11 23:40 .
drwxr-xr-x 1 node node 4096 Nov 11 23:20 ..
-rw-r--r-- 1 root root 12 Oct 29 21:00 .dockerignore
-rw-r--r-- 1 root root 288 Oct 29 21:00 .editorconfig
-rw-r--r-- 1 root root 8 Oct 29 21:00 .eslintignore
-rw-r--r-- 1 root root 27 Oct 29 21:00 .eslintrc
-rw-r--r-- 1 root root 151 Oct 29 21:00 .gitignore
-rw-r--r-- 1 root root 30 Oct 29 21:00 .yo-rc.json
-rw-r--r-- 1 root root 105 Oct 29 21:00 Dockerfile
drwxr-xr-x 2 root root 4096 Nov 11 03:40 client
drwxr-xr-x 3 root root 4096 Nov 10 23:04 common
drwxr-xr-x 2 root root 4096 Nov 11 23:16 data
drwxrwxrwx 2 root root 4096 Nov 11 23:44 logs
drwxr-xr-x 439 root root 16384 Nov 11 23:20 node_modules
-rw-r--r-- 1 root root 176643 Nov 11 23:20 package-lock.json
-rw-r--r-- 1 root root 830 Nov 11 23:20 package.json
drwxr-xr-x 3 root root 4096 Nov 10 23:04 server
root@guestbook-v1-6f55cb54c5-jb89d:/home/node/app# cat data/cache.txt
Hello Kubernetes!
Hola Kubernetes!
Zdravstvuyte Kubernetes!
Nǐn hǎo Kubernetes!
Goedendag Kubernetes!
root@guestbook-v1-6f55cb54c5-jb89d:/home/node/app# cat logs/debug.txt
Received message: Hello Kubernetes!
Received message: Hola Kubernetes!
Received message: Zdravstvuyte Kubernetes!
Received message: Nǐn hǎo Kubernetes!
Received message: Goedendag Kubernetes!
root@guestbook-v1-6f55cb54c5-jb89d:/home/node/app# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 98G 3.5G 90G 4% /
tmpfs 64M 0 64M 0% /dev
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/mapper/docker_data 98G 3.5G 90G 4% /etc/hosts
shm 64M 0 64M 0% /dev/shm
/dev/xvda2 25G 3.6G 20G 16% /home/node/app/data
tmpfs 7.9G 16K 7.9G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 7.9G 0 7.9G 0% /proc/acpi
tmpfs 7.9G 0 7.9G 0% /proc/scsi
tmpfs 7.9G 0 7.9G 0% /sys/firmware
While still inside the container, create a file on the container file system. This file will not persist when we kill the container. Then run /sbin/killall5
to terminate the container.
root@guestbook-v1-6f55cb54c5-jb89d:/home/node/app# touch dontdeletemeplease
root@guestbook-v1-6f55cb54c5-jb89d:/home/node/app# ls dontdeletemeplease
dontdeletemeplease
root@guestbook-v1-66798779d6-fqh2j:/home/node/app# /sbin/killall5
root@guestbook-v1-66798779d6-fqh2j:/home/node/app# command terminated with exit code 137
The killall5
command will kick you out of the container (which is no longer running), but the pod is still running. Verify this with kubectl get pods
. Not the 0/1 status indicating the application container is no longer running.
kubectl get pods
NAME READY STATUS RESTARTS AGE
guestbook-v1-66798779d6-fqh2j 0/1 CrashLoopBackOff 2 16m
After a few seconds, the Pod will restart the container:
kubectl get pods
NAME READY STATUS RESTARTS AGE
guestbook-v1-66798779d6-fqh2j 1/1 Running 3 16m
Run a bash process inside the container to inspect your data again:
kubectl exec -it [POD NAME] -- bash
root@guestbook-v1-6f55cb54c5-jb89d:/home/node/app# cat data/cache.txt
Hello Kubernetes!
Hola Kubernetes!
Zdravstvuyte Kubernetes!
Nǐn hǎo Kubernetes!
Goedendag Kubernetes!
root@guestbook-v1-6f55cb54c5-jb89d:/home/node/app# cat logs/debug.txt
Received message: Hello Kubernetes!
Received message: Hola Kubernetes!
Received message: Zdravstvuyte Kubernetes!
Received message: Nǐn hǎo Kubernetes!
Received message: Goedendag Kubernetes!
root@guestbook-v1-6f55cb54c5-jb89d:/home/node/app# ls dontdeletemeplease
ls: dontdeletemeplease: No such file or directory
Notice how the storage from the primary (hostPath
) and secondary (emptyDir
) storage types persisted beyond the lifecycle of the container, but the dontdeletemeplease
file, did not.
Next, we'll kill the pod to see the impact of deleting the pod on data.
kubectl get pods
NAME READY STATUS RESTARTS AGE
guestbook-v1-6f55cb54c5-jb89d 1/1 Running 0 12m
kubectl delete pod guestbook-v1-6f55cb54c5-jb89d
pod "guestbook-v1-6f55cb54c5-jb89d" deleted
kubectl get pods
NAME READY STATUS RESTARTS AGE
guestbook-v1-5cbc445dc9-sx58j 1/1 Running 0 86s
Enter new data:
Log into the pod to view the state of the data.
kubectl get pods
NAME READY STATUS RESTARTS AGE
guestbook-v1-5cbc445dc9-sx58j 1/1 Running 0 86s
kubectl exec -it guestbook-v1-5cbc445dc9-sx58j bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
root@guestbook-v1-5cbc445dc9-sx58j:/home/node/app# cat data/cache.txt
Hello Kubernetes!
Hola Kubernetes!
Zdravstvuyte Kubernetes!
Nǐn hǎo Kubernetes!
Goedendag Kubernetes!
Bye Kubernetes!
Aloha Kubernetes!
Ciao Kubernetes!
Sayonara Kubernetes!
root@guestbook-v1-5cbc445dc9-sx58j:/home/node/app# cat logs/debug.txt
Received message: Bye Kubernetes!
Received message: Aloha Kubernetes!
Received message: Ciao Kubernetes!
Received message: Sayonara Kubernetes!
root@guestbook-v1-5cbc445dc9-sx58j:/home/node/app#
This shows that the storage type emptyDir
loose data on a pod restart whereas hostPath
data lives until the worker node or cluster is deleted.
Storage Type | Persisted at which level | Example Uses |
---|---|---|
Container local storage | Container | ephermal state |
Secondary Storage (EmptyDir) | Pod | Checkpoint a long computation process |
Primary Storage (HostPath) | Node | Running cAdvisor in a container |
Normally Kubernetes clusters have multiple worker nodes in a cluster with replicas for a single application running across different worker nodes. In this case, only applications running on the same worker node will share data persisted with IKS Primary Storage (HostPath). More suitable solutions are available for cross worker nodes, cross availability-zone and cross-region storage.
Clean up¶
cd $HOME/guestbook-config/storage/lab1
kubectl delete -f .