IaC for Containers Registry

The steps in this pattern require the local workstation to be configured with the IBM Cloud CLI, CLI plugins for container-service, container-registry & schematics, the Terraform CLI, IBM Terrraform provider and a local installation of Docker . For more details on setting up the various CLI environments, see the Setup Environment chapter.

IBM Cloud Container Registry (ICR) is used to store, manage and deploy private container images in a highly available and scalable architecture. You can also set up your own image namespace and push container images to them. To learn more, see the Container Registry documentation. There are no specific IaC steps required to enable the Container Registry, this is a capablity that is available to an IBM Cloud account without performing a service creation task.

Container images for IBM Cloud follow the Open Container Initiative (OCI) standards to provide interoperability and flexibility in tooling for the container lifecycle. One well known tool for createing OCI-compliant images is docker which will be used for the examples in this pattern.

The docker command creates an image from a Dockerfile, which contains instructions to build the image. A Dockerfile might reference build artifacts in its instructions that are stored separately, such as an app, the app’s configuration, and its dependencies. Images are typically stored in a registry that can either be accessible by the public (public registry) or set up with limited access for a small group of users (private registry). By using IBM Cloud Container Registry, only users with access to your IBM Cloud account through IAM can access your images.

Continue using the same application from the previous patterns in order to have a simple container image that can be used with the IBM Container Registry and Kubernetes service. Create the first version of a Dockerfile with the following content in the directory docker/1.0/ on this project.

Copy to the data/v1 folder the JSON database file db.min.json from the previous patterns. Now, build and test the container image locally using the docker command, the Dockerfile in docker/1.0/ using the current directory as context because it needs the data/v1/ directory with the JSON database.

To create an Container Registry namespace, use the IBM Cloud CLI with the container-registry plugin. Make sure you have the latest version installed and you have setup the environment correctly. Namespace names (like Docker Hub and other container repositories) must be unique for a container registry region, so substitute the name shown here with a unique one of your choosing.

The sub-command namespace-add will create the new namespace. The examples that follow will use iac-registry as the namespace:

In order to push your local OCI image to the namespace registry, it must be tagged as: REGION.icr.io/NAMESPACE/IMAGE:TAG. Use the sub-command region to find the registry region you are targeting:

Continuing with the example, the region is us so the registry is us.icr.io. The namespace is iac-registry, the image name is movies and the version tag 1.0. The full tag would be: us.icr.io/iac-registry/movies:1.0. The image has already been created with the tag movies so to update, use the docker tag command:

Before pushing the image to the registry it’s required to login with the IBM Cloud CLI login sub-command:

This command will set up the local docker CLI with a credentials object that allows it to communicate to the namespaces defined for your account in the current container registry region. After logging in, push the image with the Docker command push:

You can check the image in the registry in different ways: (1) listing the images in the registry with the ibmcloud cr images command, or (2) using the docker command to pull the image, either from a different computer or locally deleting the image and pulling it down from the registry:

With the container image uploaded to the IBM Container Registry, you will be able to create Kubernetes deployments of the image by specifying the path to the fully qualified tag name us.icr.io/iac-registry/movies:1.0 . Before doing this, you will need to create an IKS cluster.

IBM Cloud Kubernetes Service (IKS) is a managed offering providing dedicated Kubernetes clusters to deploy and manage containerized apps. In this section you will create a Kubernetes cluster and deploy a simple API application. Examples will be provided using IBM Cloud CLI, Terraform and Schematics. The scope of this section is to cover creation of clusters and simple application deployment using IaC techniques. It will not cover deeper details for managing Kubernetes resources in general or broadly managing Kubernetes and deployments.

To create a Kubernetes cluster using the IBM Cloud CLI you need to specify parameters such as zone and worker node flavor. Discover these using the following commands. In this example, we are using Zone us-south-1 and worker node flavor mx2.4x32.

You also need a VPC and Subnet for the Kubernetes cluster. If they do not yet exist, they may be created using the IBM Cloud CLI:

After the VPC is created, the default security group will not have network access rules needed by the load balancers of the Kubernetes service to talk to the ingress controllers or other applications deployed as NodePort services. Update the default security group by adding the following rule.

If you already have a VPC and Subnets, get their IDs with the following ibmcloud ks sub-commands:

The available Kubernetes versions to install are listed with the command ibmcloud ks versions. For IKS on Gen2, use a kubernetes cluster version > 1.18. With all input parameters defined, including a name and Kubernetes veyou are ready to create the cluster using the cluster create sub-command, like this:

The default values for the optional parameters are:

To identify your Kubernetes cluster status use the command ibmcloud ks clusters, wait a few minutes to have it up and running.

When the Kubernetes cluster state is normal get the configuration to access the cluster using the following command:

Now you are ready to use the kubectl command, these are some initial commands:

You can obtain more information of the cluster with the commands:

To know more read the Kubernetes Service (IKS) documentation.

All the same actions executed with the IBM Cloud CLI has to be done with Terraform, lets create a new main.tf file with the IBM Provisioner using Gen 2, the given region and the data source to get the info of the user selected resource group.

The variables.tf file defines the required variables above, the project name and environment to use them as prefix to name the resources, the code would be like this:

To not have to enter the variables every time we execute terraform, lets add some variables value to the terraform.tfvars file. Make sure this file is appended to the .gitignore file.

The IKS clusters needs a VPC, Subnet(s) and Security Group Rules(s) added to the default security group of the VPC. Just like we did using the IBM Cloud CLI let’s create them allowing inbound traffic to ports 30000 - 32767 for the security group rules. Same as you did on Network and Compute the number of subnets is defined by the number of zones provided by the user. Lets code this in the network.tf file and append the following variables to variables.tf.

Last but not least, create the iks.tf file to define the IKS cluster using the ibm_container_vpc_cluster resource.

The above code also takes the Kubernetes version, worker nodes flavor and number from the variables k8s_version, flavor and workers_count respectively, so lets add them to the variables.tf file.

This will create a Kubernetes cluster of 3 worker nodes with 4 CPU and 32 Gb Memory. To know the available flavors in the zone, use the following IBM Cloud CLI command:

To sort them by CPU and memory, use the same command with sort:

The main input parameters of the ibm_container_vpc_cluster resource are listed in the following table:

The creation of a cluster can take some minutes to complete. To avoid long wait times, you can specify the stage when you want Terraform to mark the cluster resource creation as completed. The cluster creation might not be fully completed and continues to run in the background, however this can help you to continue with the code execution without waiting for the cluster to be fully created.

To set the waiting stage, use the wait_till with one of the following stages:

This would be enough to have an IKS cluster running. Just need to execute terraform apply, however lets create workers pools, one in each subnet or zone, using the resource ibm_container_vpc_worker_pool. Replace the code in iks.tf file for the following code and modify the variables used for the number of workers and its flavor.

The main input parameters for the ibm_container_vpc_worker_pool resource are similar to the parameters for ibm_container_vpc_cluster except for worker_pool_name which is used to name the pool, and cluster with the name or ID of the cluster set this pool.

Using a file output.tf helps us to get some useful information about the cluster through output variables, like so.

Now everything is ready to create the cluster with the wellknown Terraform commands:

After having the cluster ready, you can use the IBM Cloud CLI to get the cluster configuration to setup kubectl, like so:

Enjoy the new cluster, here are some basic initial commands to verify the cluster is working

For simplicity and creation speed, lets modify the terraform.tfvars to have a simpler cluster with one single node. This will help us to have the cluster quicker.

Remember to get a supported and the latest Kubernetes version from the output of the command ibmcloud ks versions, otherwise you may get an error like this one:

Executing terraform plan & terraform apply will get an IKS cluster up and running quicker than before.

Running this code with IBM Cloud Schematics is the same as with the other patterns. Create the workspace.json file adding the variables required for this code and replacing OWNER for your username or id, like this one:

To create the workspace using the IBM Cloud CLI execute the following commands:

Set the variable WORKSPACE_ID because it’ll be used several times. Then plan and apply the code like so.

Note the execution of apply will take some time, so check the logs either with the IBM Cloud CLI command or using the IBM Cloud Web Console. When the cluster is ready, you can use the IBM Cloud CLI to get the cluster configuration to setup kubectl and validate the cluster is accesible:

To deploy the previously built Docker image version 1.0 we use the Kubernetes API and resources. Lets create a deployment file either by getting it from the following example or generating it with kubectl generators, like so:

To deploy the application execute the kubectl apply command like this:

To validate the application you need to get the external IP or DNS to access the application executing the following code. You may have to wait a few minutes until the Load Balancer is ready. You can checkt thest status again using kubectl get svc movies.

In a real application, it’s quite common to have new or changing data. In this example, such a change to the JSON database would require a new image. If this happens very often it becomes very efficient. To address this inflexible model, you can put the JSON database in a ConfigMap. Create the cm.yaml file to define the ConfigMap with the content of the JSON file data/v1/db.min.json using this command

Or, edit the file yourself with the following content.

And apply the code to the cluster using kubectl, like this.

To make the pod access the JSON file you need to modify the Pod definition inside the deployment. Modify the deployment.yaml file to add the volumes and volumeMounts specifications, like so.

Update the new Pod applying the code, then verify it was sucessfuly applied using these commands.

The application should be running as usual:

To double check, modify the ConfigMap updating a movie or modifying the database, then access the application using curl. The instructions when the ConfigMap is modified are as follows.

In the version 1.0 of the application, the JSON database was in the ephemeral container, this may not be good practice in general so let’s migrate the database to a persistent storage such as IBM Cloud Block Storage for VPC. This storage provides hypervisor-mounted, high-performance data storage for your VSI or IKS nodes that you can provision within a VPC.

Let’s start creating the file pvc.yaml with the definition of a Persisten Volume Claim with 1Gb and the profile ibmc-vpc-block-5iops-tier.

Before use the Persistent Volume Claim (PVC) apply the changes, it has to be ready before being used.

To use this volume we need to modify the Pod specification in the deployment, open the kubernetes/deployment.yaml file to add the volumes and volumeMounts specifications.

However, these changes don’t put the initial JSON database into the volume yet. There may be different ways to do this, a possible option is to the use of Init Containers to dump the initial JSON database into the volume, but this option cannot be used because the volume access mode is ReadWriteOnce which only allows one container to access the volume at a time. Other option, and the one we will implement, is to make the Docker container copy the initial database into the volumen if there isn’t any yet. The initial JSON file is provided with a ConfigMap, let’s add it just like we did in the previous section.

The deployment.yaml file will be like this.

As you can see the image version is different, it is 1.1. In the new Dockerfile we remove the line COPY ./data/v1 /data and add the line VOLUME /data. Also, instead of executing the json-server command, it runs a script to copy the database to the right location. The new Dockerfile, to be tagged with version 1.1, is like this.

And the script to be executed as entrypoint executes the input with exec "$@", however if no command is passed in it’ll execute json-server after initialize the JSON database file. This script is like follows.

This new image has to be built, tagged and pushed to ICR very similar to like we did with the initial version and that’s what we will do in a moment. This time the context of docker build change to docker/1.1 because we don’t use the ./data directory and we use the docker/1.1/entrypoint.sh script.

Apply all files and verify the new changes executing the following commands.

Having the JSON Database in a persistent volume we can modify the database and the changes will persist the next time we deploy the application or restart the container. Having the following movie to add:

Lets add the new movie using curl, scale the deployment to zero containers, then back to one and verify the new movie is still there.

To learn more about the storage provided to the persistent volume claim, see the Block Storage for VPC documentation.

This section provides an example of deploying the Python API application used in the Cloud Databases pattern also in the GitHub repository https://github.com/IBM/cloud-enterprise-examples/ in the directory 08_cloud-services/app.

This change requires more major changes to the Docker container so it’s going to make sense to bump the tag to 2.0. In the following Dockerfile we use a multi-stage build to reduce the size of the final Docker image. The build stage use Python VirtualEnv to install all the required packages then they are copied to the app image which is used to execute the API application.

Just as with the previous versions, let’s build and push the container using the following commands:

We also need the IBM Cloud Database, with the following Terraform code in the db.tf file copied from the Cloud Databases pattern, like so.

This file also requires addition of the following input variables to the variables.tf file and output variables to the output.tf file:

To get it running, execute the plan and apply Terraform commands.

Before deploying the container to our Kubernetes cluster, do some local testing using just Docker. Execute the following commands to run the container locally, mounting the local directory ./data/v2 in a volume on the container directory /data/init/ so the application can reach the db.min.json file with the initial values of the database. The initial database db.min.json file is different to the one used for version 1 because the id field is not required. Also, to allow the container to reach the IBM Cloud MongoDB Database that was created, populate environment variables with values from the Terraform output variables.

The following commands will: create all the application input data, initialize the database, run the container with the API application and finally query the application with curl.

To wipe out the database use the same docker container but instead run the python import.py command with the --empty parameter, like so:

The current Database was created with a public endpoint (it’s public by default) and considering the current IKS cluster is private you may also want to migrate this database to be private as well. It was not created from the very begining because you may want to test the database from your computer, ilke we did running Docker locally.

To set this database with a private endpoint add the parameter service_endpoints = "private" to the ibm_database.iac_app_db_instance located in the db.tf file, Like so:

To apply this migration the database has to be deleted first, applying this change now with Terrafor will cause an error because it’s a parameter that cannot be set and modify the database. So, delete the database using the destroy Terraform command targetting the database, then apply the changes.

If the local docker container works, everything is ready to work on the Kubernetes deployment. A new ConfigMap is required with the initial data for MongoDB, another ConfigMap is required with the environment variables to have access to the database, and finally, two Secrets are required, the first Secret is used to store the database CA certificate and the second Secret stores the DB admin password. Create the ConfigMaps and Secrets with the following commands

The new deployment.yaml uses the ConfigMap to initialize the database but this time with an Init Container to execute the import.py python script. Both containers get the ConfigMap with the environment variables and the two Secrets.

All is set to apply the new Deployment, Services, Secrets and ConfigMaps with the following commands.

The PersistentVolumeClaim is not longer required for this version, so you may delete it with this command.

One of the differences with the version 1 is that this new architecture allows us to scale up the replicas of the pods. You can try it with these commands:

To verify the application is working, use the same curl commands we’ve been using.

There is more that you can do with this sample application, you can:

If you have any problem with the validation and want to debug or troubleshot it, use the following commands to identify the root cause.

If you need to login to a container replace the command in the deployment for command: ["/bin/sh", "-c", "while true; do sleep 1000;done"] so it doesn’t fail and you have time to execute a remote bash session.

If you need to connect to the database and it has a private endpoint, deploy a MongoDB container with the mongo client and the required ConfigMaps and Secrets to connect to the database. Push to ICR the official MongoDB image and execute the kubectl generator, then modify the output file to include the ConfigMap and Secrets:

Then login into the container and run the mongo client like so.

Or, instead, just execute this one-liner:

All the code used for this pattern is located and available to download in the GitHub repository https://github.com/IBM/cloud-enterprise-examples/ in the directory 09-containers. The main files for the latest version (version 2) of the application are:

When you are done with the Kubernetes cluster should destroy it.

If you want to keep the cluster running but remove everything you have done, you can execute:

If the cluster was created using the IBM Cloud CLI, execute the following commands:

If the cluster was created using Terraform, just need to execute the command:

And, if the cluster was created using IBM Cloud Schematics, execute the following commands: