Kubeflow on IKS – Documentation

Docs: Kubeflow Deployment Process

Mon, 01 Jan 0001 00:00:00 +0000

Understanding the Kubeflow deployment process

The deployment process is controlled by the following commands:

kustomize build - Use kustomize to generate configuration files defining the various resources for your deployment. .
kubectl apply - Apply the resources created by kustomize build to the kubenetes cluster

Repository layout

IBM manifests repository contains the following files and directories:

iks-single directory: A kustomize file for single-user deployment
iks-multi directory: A kustomize file for multi-user deployment
others Other files are used to compose Kubeflow resources

Kubeflow installation

Starting from Kubeflow 1.3, the official installation documentation uses a combination of kustomize and kubectl to install Kubeflow.

Install kubectl and kustomize

Next Steps

Check Kubeflow Compatibility
Go here for installing Kubeflow on IKS

Docs: Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

By default, Kubeflow Pipelines on IBM Cloud are running with the Tekton backend. In this guide you’ll learn how to use the Kubeflow Pipelines with the Tekton backend (kfp-tekton). This assumes you have deployed Kubeflow on IBM Cloud.

You can also do a standalone installation of Kubeflow Pipelines with Tekton if you don’t want whole of Kubeflow.

In this tutorial, we use the below single step pipeline as our example

from kfp import dsl
def echo_op():
    return dsl.ContainerOp(
        name='echo',
        image='busybox',
        command=['sh', '-c'],
        arguments=['echo "Got scheduled"']
    )

@dsl.pipeline(
    name='echo',
    description='echo pipeline'
)
def echo_pipeline(
):
    echo = echo_op()

Declare the Python Client for Kubeflow Pipelines

1. Single-user Kubeflow Pipelines deployment with the SDK

You will be using the Kubeflow Pipelines with Tekton SDK (kfp-tekton) v0.4.0 or above.
If you have deployed Kubeflow on IBM Cloud using the kfctl_ibm.v1.2.0.yaml manifest you can configure (kfp-tekton) SDK to list all your Kubeflow Pipelines experiments as follows:

from kfp_tekton import TektonClient

KUBEFLOW_PUBLIC_ENDPOINT_URL = 'http://<YOUR_KF_PUBLIC_ENDPOINT_URL>'
KUBEFLOW_PROFILE_NAME = None
client = TektonClient(host=f'{KUBEFLOW_PUBLIC_ENDPOINT_URL}/pipeline')

experiments = client.list_experiments(namespace=KUBEFLOW_PROFILE_NAME)

Note: <YOUR_KF_PUBLIC_ENDPOINT_URL> is the EXTERNAL_IP you exposed as a LoadBalancer following this instruction. If you have not done that step during Kubeflow setup, please include port 31380 because the Kubeflow endpoint is exposed with NodePort 31380.

2. Authenticating multi-user Kubeflow Pipelines with the SDK

You will be using the Kubeflow Pipelines SDK (kfp-tekton) v0.4.0 or above.
Note that this feature is available with multi-user, auth-enabled Kubeflow installation deployed from the kfctl_ibm_multi_user.v1.2.0.yaml manifest.

Note: You’re highly recommended enabling HTTPS for the public endpoint of Kubeflow because this method transports sensitive information like session cookie values over edge network.

It requires authentication via the public endpoint of Kubeflow deployment when using the Kubeflow Pipelines multi-user feature with Pipelines SDK.

You need to provide the following three variables if you’re using an in-cluster Jupyter notebook or a remote client machine:

KUBEFLOW_PUBLIC_ENDPOINT_URL - Kubeflow public endpoint URL. You can obtain it from command ibmcloud ks nlb-dns ls --cluster <your-cluster-name>.
SESSION_COOKIE - A session cookie starts with authservice_session=. You can obtain it from your browser after authenticated from Kubeflow UI. Notice that this session cookie expires in 24 hours, so you need to obtain it again after cookie expired.
KUBEFLOW_PROFILE_NAME - Your Kubeflow profile name

Once you provide the three variables, the SDK can use the following Python code to list all your Kubeflow Pipelines experiments:

from kfp_tekton import TektonClient

KUBEFLOW_PUBLIC_ENDPOINT_URL = 'https://xxxx.<region-name>.containers.appdomain.cloud'
# this session cookie looks like "authservice_session=xxxxxxx"
SESSION_COOKIE = 'authservice_session=xxxxxxx'
KUBEFLOW_PROFILE_NAME = '<your-profile-name>'

client = TektonClient(
    host=f'{KUBEFLOW_PUBLIC_ENDPOINT_URL}/pipeline',
    cookies=SESSION_COOKIE
)

experiments = client.list_experiments(namespace=KUBEFLOW_PROFILE_NAME)

Pipelines components like experiments and runs are isolated by Kubeflow profiles. A Kubeflow user can only see Pipelines experiments and runs belonging to this user’s Kubeflow profile.

Upload pipelines

Once you have declared the Python client, your Kubeflow pipelines can be uploaded using Python.

Run the following code inside a Python session to upload the pipelines. This example shows different versions of the pipeline using the Python client.

import os

# Initial version of the compiled pipeline
pipeline_file_path = 'echo_pipeline.yaml'
pipeline_name = 'echo_pipeline'

# For the purpose of this tutorial, we will be using the same pipeline for both version.
pipeline_version_file_path = 'echo_pipeline.yaml'
pipeline_version_name = 'new_echo_pipeline'

# Upload initial version of the pipeline
pipeline_file = os.path.join(pipeline_file_path)
pipeline = client.pipeline_uploads.upload_pipeline(pipeline_file, name=pipeline_name)

# Upload new version of the pipeline
pipeline_version_file = os.path.join(pipeline_version_file_path)
pipeline_version = client.pipeline_uploads.upload_pipeline_version(pipeline_version_file,
                                                                   name=pipeline_version_name,
                                                                   pipelineid=pipeline.id)

Run pipelines from the SDK

The TektonClient can run pipelines using one of the below sources:

Run pipelines from the Python DSL source code

To learn about executing pipelines using the Python DSL source code, try the code below in a Python session using the echo_pipeline example. The create_run_from_pipeline_func takes the DSL source code to compile and run it directly using the Kubeflow pipeline API without uploading it to the pipeline list. This method is recommended if you are doing quick experiments without version control.

# You can overwrite the pipeline default parameters by providing a dictionary of key-value arguments.
# If you don't want to overwrite the default parameters, then define the arguments as an empty dictionary.
arguments={}

client.create_run_from_pipeline_func(echo_pipeline, arguments=arguments, namespace=KUBEFLOW_PROFILE_NAME)

Run pipelines from the compiled pipeline file

Alternatively, you can also run the pipeline directly using a pre-compiled file.

EXPERIMENT_NAME = 'Demo Experiments'
experiment = client.create_experiment(name=EXPERIMENT_NAME, namespace=KUBEFLOW_PROFILE_NAME)
run = client.run_pipeline(experiment.id, 'echo-pipeline', 'echo_pipeline.yaml')

Run pipelines from the list of uploaded pipelines

Similarly, you can also run the pipeline from the list of uploaded pipelines using the same run_pipeline function.

EXPERIMENT_NAME = 'Demo Experiments'
experiment = client.create_experiment(name=EXPERIMENT_NAME, namespace=KUBEFLOW_PROFILE_NAME)

# Find the pipeline ID that you want to use.
client.list_pipelines()

run = client.run_pipeline(experiment.id, pipeline_id='925415d5-18e9-4e08-b57f-3b06e3e54648', job_name='echo_pipeline_run')

Docs: IBM Cloud Kubernetes and Kubeflow Compatibility

Mon, 01 Jan 0001 00:00:00 +0000

Compatibility

The following table relates compatibility between Kubernetes versions 1.22+ of IBM Cloud Kubernetes service and Kubeflow version 1.6.

IBM Cloud Kubernetes Versions	Kubeflow 1.7.0
1.22	Compatible
1.23	Compatible
1.24	Compatible

Incompatible: the combination is not known to work together
Compatible: all Kubeflow features have been tested and verified for the IKS Kubernetes version
No known issues: the combination has not been fully tested but there are no reported issues

Next Steps

1. Go here for installing Kubeflow on IKS

Docs: Install Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to use the kustomize + kubectl to deploy Kubeflow on IBM Cloud Kubernetes Service (IKS).

Prerequisites

Authenticate with IBM Cloud

Log into IBM Cloud using the IBM Cloud Command Line Interface (CLI) as follows:
```
ibmcloud login
```
Or, if you have federated credentials, run the following command:
```
ibmcloud login --sso
```
Create and access a Kubernetes cluster on IKS

To deploy Kubeflow on IBM Cloud, you need a cluster running on IKS. If you don’t have a cluster running, follow the Create an IBM Cloud cluster guide.

Run the following command to switch the Kubernetes context and access the cluster:
```
ibmcloud ks cluster config --cluster <cluster_name>
```
Replace <cluster_name> with your cluster name.
kustomize (version 3.2.0) (download link)
Python 3 with passlib and bcrypt packages installed

Storage setup for a Classic IBM Cloud Kubernetes cluster

Note: This section is only required when the worker nodes provider WORKER_NODE_PROVIDER is set to classic. For other infrastructures, IBM Cloud Storage with Group ID support is already set up as the cluster’s default storage class.

When you use the classic worker node provider of an IBM Cloud Kubernetes cluster, it uses the regular IBM Cloud File Storage based on NFS as the default storage class. File Storage is designed to run RWX (read-write multiple nodes) workloads with proper security built around it. Therefore, File Storage does not allow fsGroup securityContext unless it’s configured with Group ID, which is needed for the OIDC authentication service and Kubeflow Jupyter server.

Therefore, you’re recommended to set up the default storage class with Group ID support so that you can get the best experience from Kubeflow.

Set the File Storage with Group ID support as the default storage class.

NEW_STORAGE_CLASS=ibmc-file-gold-gid
OLD_STORAGE_CLASS=$(kubectl get sc -o jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io\/is-default-class=="true")].metadata.name}')
kubectl patch storageclass ${NEW_STORAGE_CLASS} -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

# List all the (default) storage classes
kubectl get storageclass | grep "(default)"

Example output:

ibmc-file-gold-gid (default)   ibm.io/ibmc-file    Delete          Immediate           false                  14h

Make sure ibmc-file-gold-gid is the only (default) storage class. If there are two or more rows in the above output, unset the previous (default) storage classes with the command below:
```
kubectl patch storageclass ${OLD_STORAGE_CLASS} -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
```

Storage setup for vpc-gen2 IBM Cloud Kubernetes cluster

Note: To deploy Kubeflow, you don’t need to change the storage setup for vpc-gen2 Kubernetes cluster.

Currently, there is no option available for setting up RWX (read-write multiple nodes) type of storages. RWX is not a mandatory requirement to run Kubeflow and most pipelines. It is required by certain sample jobs/pipelines where multiple pods write results to a common storage. A job or a pipeline can also write to a common object storage like minio, so the absence of this feature is not a blocker for working with Kubeflow. Examples of jobs/pipelines that will not work, are: Distributed training with Kubeflow TFJob

If you are on vpc-gen2 and still need RWX, you may try portworx enterprise product. To set it up on IBM Cloud use the portworx install with IBM Cloud guide.

Installation

Choose either single user or multi-user section based on your usage.

If you’re experiencing issues during the installation because of conflicts on your Kubeflow deployment, you can uninstall Kubeflow and install it again.

Single user

Using kustomize together with kubectl to deploy kubeflow:

Clone the manifest repo as follows:

git clone https://github.com/IBM/manifests.git -b v1.6-branch ibm-manifests-160

Change directory to ibm-manifests-160:
```
cd ibm-manifests-160
```
Generate password for default user: user@example.com
```
python3 -c 'from passlib.hash import bcrypt; import getpass; print(bcrypt.using(rounds=12, ident="2y").hash(getpass.getpass()))'
```
Type your password and press <Enter> after you see Password: prompt. Copy the hash code for next step.
Edit dist/stacks/ibm/application/dex-auth/custom-env.yaml and fill the relevant field with the hash code from previous step:
```
staticPasswords:
- email: user@example.com
  hash: <enter the generated hash here>
```
You can also change the email value if needed.

Apply the kustomize file under iks-single folder for single user deployment:

while ! kustomize build iks-single 2>/dev/null | awk '!/well-defined/' | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

Accessing your cluster

The Kubeflow endpoint is exposed with NodePort 30380. To get a static ip, you can expose the Kubeflow endpoint as a LoadBalancer and access the EXTERNAL_IP.

For single-user Kubeflow, IBM Cloud uses Dex authentication by default. You can access the cluster using the email and password you specified in step 3 and 4 of Single User

Multi-user, auth-enabled

Run the following steps to deploy Kubeflow with IBM Cloud AppID as an authentication provider.

The scenario is a Kubeflow cluster admin configures Kubeflow as a web application in AppID and manages user authentication with builtin identity providers (Cloud Directory, SAML, social log-in with Google or Facebook etc.) or custom providers.

Prerequisites

For authentication, IBM Cloud uses AppID

Follow the Creating an App ID service instance on IBM Cloud guide to learn about Kubeflow authentication. You can also learn how to use App ID with different authentication methods.
Follow the Registering your app section of the App ID guide to create an application with type regularwebapp under the provisioned AppID instance. Make sure the scope contains email. Then retrieve the following configuration parameters from your AppID:
- clientId
- secret
- oAuthServerUrl
You will be using these information in the subsequent sections.
Register the Kubeflow OIDC redirect page. The Kubeflow REDIRECT_URL URL is [http|https]://<kubeflow-FQDN>/authservice/oidc/callback, depends on if you enable the HTTPS or not. <kubeflow-FQDN> is the endpoint for accessing Kubeflow. By default, the <kubeflow-FQDN> on IBM Cloud is <worker_node_external_ip>:30380. To get a static ip, you can expose the Kubeflow endpoint as a LoadBalancer and use the EXTERNAL_IP for your <kubeflow-FQDN>. Or use ibmcloud ks nlb-dns command to map the EXTERNAL_IP to the generated FQDN for your cluster. In this case, you use the generated FQDN as kubeflow-FQDN. If you enable HTTPS, you shall use generated FQDN.
Then, you need to place the Kubeflow OIDC REDIRECT_URL under Manage Authentication > Authentication settings > Add web redirect URLs.

Example: https://my-kubeflow-442dbba0442be6c8c50f31ed96b00601-0000.sjc04.containers.appdomain.cloud/authservice/oidc/callback

Deploy: Using kustomize together with kubectl

Clone the manifest repo as follows:

git clone https://github.com/IBM/manifests.git -b v1.6-branch ibm-manifests-160

Change directory to ibm-manifests-160:
```
cd ibm-manifests-160
```
Update the dist/stacks/ibm/application/oidc-authservice-appid/params.env with values collected in Prereq section. You will need the following values:
- <oAuthServerUrl> - replace <APP_ID_oauthServerUrl>
- <kubeflow-FQDN> - fill in the FQDN of Kubeflow, if you don’t know yet, just give a dummy one like localhost. Then change it after you got one. Or get default FQDN of your cluster by this command: ibmcloud ks nlb-dns ls -c <cluster name> (replace <cluter name> with your cluster name)
Example:
```
OIDC_PROVIDER=https://us-south.appid.cloud.ibm.com/oauth/v4/f341ff8b-a088-497a-same-5da4628df7fd
REDIRECT_URL=https://my-kubeflow-442dbba0442be6c8c50f31ed96b00601-0000.sjc04.containers.appdomain.cloud/authservice/oidc/callback
OIDC_AUTH_URL=https://us-south.appid.cloud.ibm.com/oauth/v4/f341ff8b-a088-497a-same-5da4628df7fd/authorization
```
Update the dist/stacks/ibm/application/oidc-authservice-appid/secret_params.env with values collected in Prereq section. You will need the following values:
- <clientId> - replace the <APP_ID_clientId>
- <secret> - replace the <APP_ID_secret>
Example:
```
CLIENT_SECRET=NjNhZDA3ODAtM2I3MCSECRETLTkwN2QtNDdhYmU5ZGIyMTBl
CLIENT_ID=52b3e496-8888-8888-ABC9-c0da309cdf52
```

You can apply the kustomize file in iks-multi folder:

while ! kustomize build iks-multi 2>/dev/null | awk '!/well-defined/' | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

If at any point the values change and you have to change them, you can either patch the configmap and secret or change the content in the files and apply the kustomize again. You will need to restart authservice with kubectl delete pod -l app=authservice -n istio-system .

To apply just the oidc-authservice-appid you can use this command:
```
kustomize build dist/stacks/ibm/application/oidc-authservice-appid | kubectl apply -f -
kubectl delete pod -l app=authservice -n istio-system
```

Verify mutli-user installation

Check the pod authservice-0 is in running state in namespace istio-system:

kubectl get pod -l app=authservice -n istio-system

Extra network setup requirement for vpc-gen2 clusters only

Note: These steps are not required for classic clusters, i.e. where WORKER_NODE_PROVIDER is set to classic.

A vpc-gen2 cluster does not assign a public IP address to the Kubernetes master node by default. It provides access via a Load Balancer, which is configured to allow only a set of ports over public internet. Access the cluster’s resources in a vpc-gen2 cluster, using one of the following options,

Load Balancer method: To configure via a Load Balancer, go to Expose the Kubeflow endpoint as a LoadBalancer. This method is recommended when you have Kubeflow deployed with Multi-user, auth-enabled support — otherwise it will expose cluster resources to the public.
Socks proxy method: If you need access to nodes or NodePort in the vpc-gen2 cluster, this can be achieved by starting another instance in the same vpc-gen2 cluster and assigning it a public IP (i.e. the floating IP). Next, use SSH to log into the instance or create an SSH socks proxy, such as ssh -D9999 root@new-instance-public-ip.

Then, configure the socks proxy at localhost:9999 and access cluster services.

kubectl port-forward method: To access Kubeflow dashboard, run kubectl -n istio-system port-forward service/istio-ingressgateway 7080:http2. Then in a browser, go tohttp://127.0.0.1:7080/

Important notice: Exposing cluster/compute resources publicly without setting up a proper user authentication mechanism is very insecure and can have very serious consequences(even legal). If there is no need to expose cluster services publicly, Socks proxy method or kubectl port-forward method are recommended.

Next steps: secure the Kubeflow dashboard with HTTPS

Prerequisites

For both classic and vpc-gen2 cluster providers, make sure you have Multi-user, auth-enabled Kubeflow set up.

Setup

Follow the steps in Exposing the Kubeflow dashboard with DNS and TLS termination. Then, you will have the required DNS name as Kubeflow FQDN to enable the OIDC flow for AppID:

Follow the step Adding redirect URIs to fill a URL for AppID to redirect to Kubeflow. The URL should look like https://<kubeflow-FQDN>/authservice/oidc/callback.

Update the secret appid-application-configuration with the updated Kubeflow FQDN to replace <kubeflow-FQDN> in below command:

export REDIRECT_URL=https://<kubeflow-FQDN>/authservice/oidc/callback
export PATCH=$(printf '{"data": {"REDIRECT_URL": "%s"}}' "$REDIRECT_URL")

kubectl patch configmap/oidc-authservice-parameters -n istio-system -p="$PATCH"

Restart the pod authservice-0:

kubectl delete pod -l app=authservice -n istio-system

Then, visit https://<kubeflow-FQDN>/. The page should redirect you to AppID for authentication.

Troubleshooting

Expose the Kubeflow endpoint as a LoadBalancer

By default, the Kubeflow deployment on IBM Cloud only exposes the endpoint as NodePort 30380. If you want to expose the endpoint as a LoadBalancer, run:

kubectl patch svc istio-ingressgateway -n istio-system -p '{"spec": {"type": "LoadBalancer"}}'

Then, you can locate the LoadBalancer in the EXTERNAL_IP column when you run the following command:

kubectl get svc istio-ingressgateway -n istio-system

There is a small delay, usually ~5 mins, for above commands to take effect.

Authservice pod taking too long to restart

You might see the authservice-0 pod taking some time to restart. If that happens you can delete to pod which will kick off restart from kubernetes reconciler.

kubectl delete pod -l app=authservice -n istio-system

Docs: Using IBM Cloud Container Registry (ICR)

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install and configure the IBM Cloud CLI.
Install the CLI plug-in for the IBM Cloud Container Registry by running the command ibmcloud plugin install container-registry.
Create a namespace in ICR with the command ibmcloud cr namespace-add <my_namespace>, replace <my_namespace> with your preferred name.

Note: The ICR namespace is different from the Kubeflow Profile namespace. The ICR namespace is used to group container images stored in ICR, while a Kubeflow Profile namespace is a group of all Kubernetes clusters owned by a user.

Image pull secret

As a Kubernetes cluster uses the Secret of docker-registry type to authenticate with a container registry to pull a private image, it needs an image pull secret to pull container images from IBM Cloud Container Registry. You can use the default image pull secret set up by the cluster or your account’s IAM API key.

Using a default image pull secret

By default, the IBM Cloud Kubernetes cluster is set up to pull images from only your account’s namespace in IBM Cloud Container Registry by using the secret all-icr-io in the default namespace. A cluster admin can copy this secret to any Kubernetes namespace used as Kubeflow profile. For example, run below command to copy the secret all-icr-io to the anonymous namespace:

kubectl get secret all-icr-io -n default -o yaml \
| sed 's/namespace: default/namespace: anonymous/g' \
| kubectl -n anonymous create -f -

Once this secret is ready in your Kubeflow profile, a data scientist can use it to pull container images from ICR.

See details and FAQs from the official guide Setting up an image registry.

Getting an IAM API Key

You will need an IBM Cloud IAM API Key to work with ICR if you:

Have no access to the default image pull secret all-icr-io from the default namespace.
Need to access container images in other IBM Cloud accounts.
Need customized IAM policy by using a separate IAM service ID.

If you don’t have an IBM Cloud IAM API Key, follow the official guide Create an API Key.

Once you get your IBM Cloud IAM API Key, run the following command:

kubectl -n <my_namespace> create secret docker-registry <secret_name> \
--docker-server=<registry_domain_name> \
--docker-username=iamapikey \
--docker-password=<ibm_cloud_iam_api_key> \
--docker-email=<docker_email>

Notes:

<my_namespace>: your namespace to use with ICR to create an image pull secret.
<ibm_cloud_iam_api_key>: your IBM Cloud API Key.
<secret_name>: a unique name for the pull image secret, such as us-icr-io, for example.
<registry_domain_name>: the image registry where your registry namespace is set up. Use regional domain name when storing container images in specific region, such as us.icr.io when using region en-us and uk.icr.io when using region eu-gb. See full list of regional domain names from the About IBM Cloud Container Registry page.
<docker_email>: your docker email address or any fictional email address, such as a@b.c.

Scenarios

Using container images from ICR in Kubeflow Pipelines

The pull image secret may be set in Kubeflow Pipelines SDK’s PipelineConf. Refer to this imagepullsecrets.py sample in Kubeflow Pipelines project for usage.

Using notebook images from ICR in a Jupyter Notebook

When a namespace is created for Kubeflow with its profile controller, a default service account default-editor is created in that namespace. Before creating a Notebook Server, run following command to patch the service account. Replace <secret_name> with the ICR pull image secret name and <my_namespace> with the Kubeflow profile namespace.

Replace <my_namespace> with your namespace then run below command to patch the service account default-editor with this image pull secret:

kubectl patch serviceaccount default-editor \
-p '{"imagePullSecrets": [{"name": "<secret-name>"}]}' \
-n <my_namespace>

The service account should be updated. Then, when you create the Notebook Server through Kubeflow dashboard, you should be able to choose a Custom Image. Afterwards, set the notebook image path from the ICR as follows:

Docs: End-to-end Kubeflow on IBM Cloud

Mon, 01 Jan 0001 00:00:00 +0000

This is a guide for an end-to-end example of Kubeflow on IBM Cloud Kubernetes Service (IKS). The core steps will be to take a base Tensorflow model, modify it for distributed training, serve the resulting model with TFServing, and deploy a web application that uses the trained model.

Introduction

Overview of IKS

IBM Cloud Kubernetes Service (IKS) enables the deployment of containerized applications in Kubernetes clusters with specialized tools for management of the systems.

The IBM Cloud CLI can be used for creating, developing, and deploying cloud applications.

Here’s a list of IBM Cloud services you will use:

The model and the data

This tutorial trains a TensorFlow model on the MNIST dataset, which is the hello world for machine learning.

The MNIST dataset contains a large number of images of hand-written digits in the range 0 to 9, as well as the labels identifying the digit in each image.

After training, the model can classify incoming images into 10 categories (0 to 9) based on what it’s learned about handwritten images. In other words, you send an image to the model, and the model does its best to identify the digit shown in the image.

In the above screenshot, the image shows a hand-written 7. This image was the input to the model. The table below the image shows a bar graph for each classification label from 0 to 9, as output by the model. Each bar represents the probability that the image matches the respective label. Judging by this screenshot, the model seems pretty confident that this image is a 7.

The overall workflow

The following diagram shows what you accomplish by following this guide:

In summary:

Setting up Kubeflow on IKS.
Training the model:
- Packaging a Tensorflow program in a container.
- Submitting a Tensorflow training (tf.train) job.
Using the model for prediction (inference):
- Saving the trained model to IBM Cloud Object Storage.
- Using Tensorflow Serving to serve the model.
- Running the simple web app to send prediction request to the model and display the result.

It’s time to get started!

Run the MNIST Tutorial on IKS

Follow the IKS instructions to deploy Kubeflow.
Launch a Jupyter notebook.
Launch a terminal in Jupyter and clone the Kubeflow examples repo.
```
git clone https://github.com/kubeflow/examples.git git_kubeflow-examples
```
- Tip: When you start a terminal in Jupyter, run the command bash to start a bash terminal which is much more friendly than the default shell.
- Tip: You can change the URL for your notebook from ‘/tree’ to ‘/lab’ to switch to using Jupyterlab.
Open the notebook mnist/mnist_ibm.ipynb.
Follow the notebook to train and deploy MNIST on Kubeflow.

Docs: Securing the Kubeflow authentication with HTTPS

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to secure the Kubeflow authentication with HTTPS. You can enable HTTPS for Kubeflow dashboard (and other web UIs) using the network load balancer (NLB) feature of the IBM Cloud Kubernetes service—choose the classic worker nodes provider in the Setting environment variables section of the Create an IBM Cloud cluster guide.

Note: For details on NLB, go to the official Classic: About network load balancers guide.

Prerequisites

Install and configure the IBM Cloud CLI.
Install multi-user, auth-enabled Kubeflow.

Setting up an NLB

To set up an NLB for your Kubernetes cluster, follow the official Classic: Setting up basic load balancing with an NLB 1.0 guide. Notice that the setup process for a multi-zone cluster differs from that of a single-zone cluster. For details, go to Setting up an NLB 1.0 in a multi-zone cluster.

To use the existing Istio ingress gateway (instead of creating a new service), you need to update the service type of istio-ingressgateway to LoadBalancer from NodePort. Run the following command:
```
kubectl patch svc istio-ingressgateway -n istio-system -p '{"spec":{"type":"LoadBalancer"}}'
```
Verify that the NLB was created successfully. It might take a few minutes for the service to be created and an IP address to be made available. Run the command below and check if you can see the LoadBalancer Ingress IP address:
```
kubectl describe service istio-ingressgateway -n istio-system | grep "LoadBalancer Ingress"
```

Store the external IP of the istio-ingressgateway service in an environment variable:

export INGRESS_GATEWAY_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Exposing the Kubeflow dashboard with DNS and TLS termination

The following instructions use the Kubeflow dashboard as an example. However, they apply to other web UI applications, since they all go through the Istio ingress gateway.

Store the Kubernetes cluster name in an environment variable by running the following command:
```
export CLUSTER_NAME=<cluster_name>
```

Create a DNS domain and certificates for the IP of the service istio-ingressgateway in namespace istio-system:

ibmcloud ks nlb-dns create classic --cluster $CLUSTER_NAME --ip $INGRESS_GATEWAY_IP --secret-namespace istio-system

List the registered domain names:

ibmcloud ks nlb-dns ls --cluster $CLUSTER_NAME

Wait until the status of the certificate—the fourth field—of the new domain name becomes created. Then, save the value of the column SSL Cert Secret Name in environment variables by running these commands (replace {SECRET_NAME} with the secret’s name as shown in the SSL Cert Secret Name column):
```
export INGRESS_GATEWAY_SECRET={SECRET_NAME}
```
Note: If there is more than one entry in the output, choose the one that matches the IP address from LoadBalancer Ingress (step 2) of service istio-ingressgateway.

Create a secret named istio-ingressgateway-certs for the istio-ingressgateway pods in namespace istio-system:

kubectl get secret $INGRESS_GATEWAY_SECRET -n istio-system -o yaml > istio-ingressgateway-certs.yaml

Update the istio-ingressgateway-certs.yaml file by changing the value of metadata.name to istio-ingressgateway-certs and the value of metadata.namespace to istio-system. Then, run the following commands:
```
kubectl apply -f istio-ingressgateway-certs.yaml -n istio-system
kubectl rollout restart deploy istio-ingressgateway -n istio-system
rm istio-ingressgateway-certs.yaml
```

Update the gateway kubeflow-gateway to expose port 443. Create a resource file kubeflow-gateway.yaml as follows by replacing <hostname> with the value of the column Hostname in step 4:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: kubeflow
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '<hostname>'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt

Verify that the traffic is routed via HTTPS by using the value of above-mentioned Hostname in your browser. It should redirect traffic from an HTTP address to HTTPS address automatically.

Note: The certificates for the NLB DNS host secret expire every 90 days. The secret in the default namespace is automatically renewed by IBM Cloud Kubernetes Service 37 days before it expires. After this secret is updated, you must manually copy it to the istio-ingressgateway-certs secret by repeating commands in step 5 and 6.

Optional - KServe configuration

With this HTTPS setup, you need to make additional changes to get KServe to work.

First, update the Knative domain that is used for the KServe routes to the hostname that you used when updating kubeflow-gateway.

kubectl edit configmap config-domain -n knative-serving

This will open your default text editor, and you will see something like:

apiVersion: v1
data:
  _example: |
    ################################
    #                              #
    #    EXAMPLE CONFIGURATION     #
    #                              #
    ################################
    # ...
    example.com: |    
kind: ConfigMap
...

Add a line above the _example key with your hostname as the key and an empty string value. Be sure to update <hostname>:

apiVersion: v1
data:
  <hostname>: ""
  _example: |
   ...   
kind: ConfigMap
...

Then, save and exit. The routes for your InferenceServices will start using this new domain.

Since the certificates provided by IBM Cloud only allow for a single level of domain name added to the base domain, the domain template for Knative needs to be adjusted so that the certificates will be valid for the InferenceService routes.
```
kubectl edit configmap config-network -n knative-serving
```
This will open your default text editor, and you will again see something like:
```
apiVersion: v1
data:
  _example: |
    ################################
    #                              #
    #    EXAMPLE CONFIGURATION     #
    #                              #
    ################################
    # ...    
kind: ConfigMap
...
```
Add a line above the _example key with the domainTemplate key like the following:
```
apiVersion: v1
data:
  domainTemplate: "{{.Name}}-{{.Namespace}}.{{.Domain}}"
  _example: |
   ...   
kind: ConfigMap
...
```
Save and exit. The default template uses a two-level subdomain (i.e. {{.Name}}.{{.Namespace}}.{{.Domain}}), so this just adjusts it to use one.

Adjust kubeflow-gateway one more time, adding a wildcard host in the HTTPS hosts section.

kubectl edit gateway kubeflow-gateway -n kubeflow

This will open your default text editor, but you can also optionally edit kubeflow-gateway.yaml file you created previously. Here, just add another entry to the HTTPS hosts list containing your hostname prepended with a *. so that the Knative subdomains are properly routed.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: kubeflow
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '<hostname>'
    - '*.<hostname>'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt

Save and exit.

After these adjustments, InferenceServices should now be reachable via HTTPS. To test out an external prediction, you can use the authservice_session cookie for the Kubeflow dashboard site from the browser. Once the content of the cookie is retrieved from the browser, it can be added as a header in your request (e.g. "Cookie: authservice_session=MTYwNz..."). For example:

curl -v https://sklearn-iris-kserve-test.kf-dev-442dbba0442be6c8c50f31ed96b00532-0001.sjc03.containers.appdomain.cloud/v1/models/sklearn-iris:predict -d '{"instances": [[6.8,  2.8,  4.8,  1.4],[6.0,  3.4,  4.5,  1.6]]}' -H "Cookie: authservice_session=MTYwODMyODk5M3xOd3dBTkVzeU5VSlJRazlQVnpWT1dGUldWa0ZXVDBRMVFsY3pVVFZHVVVGV01rWkRORmd6VmxCVVNsQkVSa2xaUlZVMFRUVldVMEU9fJBHfRCAvs6nSh_J04VlBEq_yqhkUvc5Z1Mqahe9klOd"

Docs: Kubernetes Cluster on Classic Infra

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to create a Kubernetes cluster with IBM Cloud Kubernetes Service.

IBM Cloud Kubernetes Service provides powerful tools and services to help deploy highly available containerized apps in Kubernetes clusters and to automate, isolate, secure, manage, and monitor your workloads across zones or regions.

Prerequisites

IBMid

To get started, first go to IBM Cloud to create your IBMid if you do not have one.
Installing the IBM Cloud CLI

Follow the instructions in this Getting started with the IBM Cloud CLI guide to install the IBM Cloud CLI.
Installing the IBM Cloud Kubernetes Service plug-in with the command
```
ibmcloud plugin install container-service
```
Refer to this link for more info on IBM Cloud Kubernetes Service CLI.
Authenticating with IBM Cloud
```
ibmcloud login
```
Use your registered email and password for your IBMid to log in to IBM Cloud.

Connecting to an existing cluster

If you have an existing cluster, use it to install Kubeflow as far as it meets the minimum system requirement.

Get the Kubeconfig file:

ibmcloud ks cluster config --cluster $CLUSTER_NAME

From here on, go to Install Kubeflow on IKS for more information.

Create and setup a new cluster

Use a classic provider if you want to try out Kubeflow.
Use a vpc-gen2 provider if you are familiar with Cloud networking and want to deploy Kubeflow on a secure environment.

A classic provider exposes each cluster node to the public internet and therefore has a relatively simpler networking setup. Services exposed using Kubernetes NodePort need to be secured using authentication mechanism.

To create a cluster with vpc-gen2 provider, follow the Create a cluster on IKS with a vpc-gen2 provider guide.

The next section will explain how to create and set up a new IBM Cloud Kubernetes Service (IKS)

Setting environment variables

Choose the region and the worker node provider for your cluster, and set the environment variables.

export KUBERNETES_VERSION=1.21
export CLUSTER_ZONE=dal13
export WORKER_NODE_PROVIDER=classic
export CLUSTER_NAME=kubeflow

where:

KUBERNETES_VERSION specifies the Kubernetes version for the cluster. Run ibmcloud ks versions to see the supported Kubernetes versions. If this environment variable is not set, the cluster will be created with the default version set by IBM Cloud Kubernetes Service. Refer to Minimum system requirements and choose a Kubernetes version compatible with the Kubeflow release to be deployed.
CLUSTER_ZONE identifies the regions or location where cluster will be created. Run ibmcloud ks locations to list supported IBM Cloud Kubernetes Service locations. For example, choose dal13 to create your cluster in the Dallas (US) data center.
WORKER_NODE_PROVIDER specifies the kind of IBM Cloud infrastructure on which the Kubernetes worker nodes will be created. The classic type supports worker nodes with GPUs. There are other worker nodes providers including vpc-classic and vpc-gen2 where zone names and worker flavors will be different. Run ibmcloud ks zones --provider classic to list zone names for classic provider and set the CLUSTER_ZONE accordingly.
CLUSTER_NAME must be lowercase and unique among any other Kubernetes clusters in the specified ${CLUSTER_ZONE}.

Notice: Refer to Creating clusters in the IBM Cloud documentation for additional information on how to set up other providers and zones in your cluster.

Choosing a worker node flavor

The worker node flavor name varies from zones and providers. Run ibmcloud ks flavors --zone ${CLUSTER_ZONE} --provider ${WORKER_NODE_PROVIDER} to list available flavors.

For example, the following are some worker node flavors supported in the dal13 zone with a classic node provider.

ibmcloud ks flavors --zone dal13 --provider classic

Example output:

OK
For more information about these flavors, see 'https://ibm.biz/flavors'
Name                      Cores   Memory   Network Speed   OS             Server Type   Storage      Secondary Storage   Provider
b2c.16x64                 16      64GB     1000Mbps        UBUNTU_16_64   virtual       25GB         100GB               classic
b2c.32x128                32      128GB    1000Mbps        UBUNTU_16_64   virtual       25GB         100GB               classic
b2c.4x16                  4       16GB     1000Mbps        UBUNTU_16_64   virtual       25GB         100GB               classic
b2c.56x242                56      242GB    1000Mbps        UBUNTU_16_64   virtual       25GB         100GB               classic
b2c.8x32                  8       32GB     1000Mbps        UBUNTU_16_64   virtual       25GB         100GB               classic
b3c.16x64                 16      64GB     1000Mbps        UBUNTU_18_64   virtual       25GB         100GB               classic
b3c.32x128                32      128GB    1000Mbps        UBUNTU_18_64   virtual       25GB         100GB               classic
b3c.4x16                  4       16GB     1000Mbps        UBUNTU_18_64   virtual       25GB         100GB               classic
b3c.56x242                56      242GB    1000Mbps        UBUNTU_18_64   virtual       25GB         100GB               classic
b3c.8x32                  8       32GB     1000Mbps        UBUNTU_18_64   virtual       25GB         100GB               classic
...

Choose a flavor that will work for your applications. For the purpose of the Kubeflow deployment, the recommended configuration for a cluster is at least 8 vCPU cores with 16GB memory. Hence you can either choose the b3c.8x32 flavor to create a one-worker-node cluster or choose the b3c.4x16 flavor to create a two-worker-node cluster. Keep in mind that you can always scale the cluster by adding more worker nodes should your application scales up.

Now, set the environment variable with the worker node flavor of your choice:

export WORKER_NODE_FLAVOR=b3c.4x16

Creating an IBM Cloud Kubernetes cluster

Run with the following command to create a cluster:

Replace the workers parameter above with the desired number of worker nodes.

If you’re starting in a fresh account with no public and private VLANs, they are created automatically for you when creating a Kubernetes cluster with worker nodes provider classic for the first time. If you already have VLANs configured in your account, retrieve them via ibmcloud ks vlans --zone ${CLUSTER_ZONE} and include the public and private VLAN ids (set in the PUBLIC_VLAN_ID and PRIVATE_VLAN_ID environment variables) in the command.

ibmcloud ks cluster create ${WORKER_NODE_PROVIDER} \
  --name=$CLUSTER_NAME \
  --zone=$CLUSTER_ZONE \
  --version=${KUBERNETES_VERSION} \
  --flavor ${WORKER_NODE_FLAVOR} \
  --workers=2 \
  --private-vlan ${PRIVATE_VLAN_ID} \
  --public-vlan ${PUBLIC_VLAN_ID}

Wait until the cluster is deployed and configured. It can take a while for the cluster to be ready. Run with following command to periodically check the state of your cluster. Your cluster is ready when the state is normal.

ibmcloud ks clusters --provider ${WORKER_NODE_PROVIDER} |grep ${CLUSTER_NAME} |awk '{print "Name:"$1"\tState:"$3}'

Verifying the cluster

To use the created cluster, switch the Kubernetes context to point to the cluster with the command

ibmcloud ks cluster config --cluster ${CLUSTER_NAME}

Make sure all worker nodes are up with the command below

kubectl get nodes

and make sure all the nodes are in Ready state.

Delete the cluster

Delete the cluster including it’s storage:

ibmcloud ks cluster rm --force-delete-storage -c ${CLUSTER_NAME}

Docs: Uninstall Kubeflow

Mon, 01 Jan 0001 00:00:00 +0000

Uninstall Kubeflow on your IBM Cloud IKS cluster.

Go to your Kubeflow deployment directory where you download the IBM manifests repository: https://github.com/IBM/manifests.git
```
cd ibm-manifests-160
```
Run the following command to get Kubeflow Profiles:
```
kubectl get profile
```
Delete all Kubeflow Profiles manually:
```
kubectl delete profile --all
```
Use the following command to check all namespaces for Kubeflow Profiles are removed properly:
```
kubectl get ns
```
Make sure no namespace is in the Terminating state.

Remove Kubeflow:

For single-user deployment:

kustomize build iks-single 2>/dev/null | awk '!/well-defined/' | kubectl delete -f -

For multi-user deployment:

kustomize build iks-multi 2>/dev/null | awk '!/well-defined/' | kubectl delete -f -

Docs: Kubernetes cluster on a VPC

Mon, 01 Jan 0001 00:00:00 +0000

Create and setup a new cluster

Follow these steps to create and setup a new IBM Cloud Kubernetes Service(IKS) cluster on vpc-gen2 provider.

A vpc-gen2 cluster does not expose each node to the public internet directly and thus has more secure and more complex network setup. It is recommended setup for secured production use cases of Kubeflow.

Setting environment variables

Choose the region and the worker node provider for your cluster, and set the environment variables.

export KUBERNERTES_VERSION=1.18
export CLUSTER_ZONE=us-south-3
export CLUSTER_NAME=kubeflow-vpc

where:

KUBERNETES_VERSION: Run ibmcloud ks versions to see the supported Kubernetes versions. Refer to Supported version matrix.
CLUSTER_ZONE: Run ibmcloud ks locations to list supported zones. For example, choose us-south-3 to create your cluster in the Dallas (US) data center.
CLUSTER_NAME must be lowercase and unique among any other Kubernetes clusters in the specified ${CLUSTER_ZONE}.

Notice: Refer to Creating clusters in the IBM Cloud documentation for additional information on how to set up other providers and zones in your cluster.

Choosing a worker node flavor

The worker nodes flavor name varies from zones and providers. Run ibmcloud ks flavors --zone ${CLUSTER_ZONE} --provider vpc-gen2 to list available flavors.

Below are some examples of flavors supported in the us-south-3 zone with vpc-gen2 node provider:

ibmcloud ks flavors --zone us-south-3 --provider vpc-gen2

Example output:

For more information about these flavors, see 'https://ibm.biz/flavors'
Name         Cores   Memory   Network Speed   OS             Server Type   Storage   Secondary Storage   Provider   
bx2.16x64    16      64GB     16Gbps          UBUNTU_18_64   virtual       100GB     0B                  vpc-gen2   
bx2.2x8†     2       8GB      4Gbps           UBUNTU_18_64   virtual       100GB     0B                  vpc-gen2   
bx2.32x128   32      128GB    16Gbps          UBUNTU_18_64   virtual       100GB     0B                  vpc-gen2   
bx2.48x192   48      192GB    16Gbps          UBUNTU_18_64   virtual       100GB     0B                  vpc-gen2   
bx2.4x16     4       16GB     8Gbps           UBUNTU_18_64   virtual       100GB     0B                  vpc-gen2   
...

The recommended configuration for a cluster is at least 8 vCPU cores with 16GB memory. Hence, we recommend bx2.4x16 flavor to create a two-worker-node cluster. Keep in mind that you can always scale the cluster by adding more worker nodes should your application scales up.

Now set the environment variable with the flavor you choose.

export WORKER_NODE_FLAVOR=bx2.4x16

Create an IBM Cloud Kubernetes cluster for `vpc-gen2` infrastructure

Creating a vpc-gen2 based cluster needs a VPC, a subnet and a public gateway attached to it. Fortunately, this is a one time setup. Future vpc-gen2 clusters can reuse the same VPC/subnet(with attached public-gateway).

Begin with installing a vpc-infrastructure plugin:
```
ibmcloud plugin install vpc-infrastructure
```
Refer to this link, for more information.
Target vpc-gen 2 to access gen 2 VPC resources:
```
ibmcloud is target --gen 2
```
Verify that the target is correctly set up:
```
ibmcloud is target
```
Example output:
```
Target Generation: 2
```

Create or use an existing VPC:

a) Use an existing VPC:

ibmcloud is vpcs

Example output:

Listing vpcs for generation 2 compute in all resource groups and region ...
ID                                          Name                Status      Classic access   Default network ACL                                    Default security group                                 Resource group   
r006-hidden-68cc-4d40-xxxx-4319fa3gxxxx   my-vpc1              available   false            husker-sloping-bee-resize                              blimp-hasty-unaware-overflow                           kubeflow

If the above list contains the VPC that can be used to deploy your cluster - make a note of its ID.

b) To create a new VPC, proceed as follows:

ibmcloud is vpc-create my-vpc

Example output:

Creating vpc my-vpc in resource group kubeflow under account IBM as ...

ID                                             r006-hidden-68cc-4d40-xxxx-4319fa3fxxxx   
Name                                           my-vpc   
...

Save the ID in a variable VPC_ID as follows, so that we can use it later.

export VPC_ID=r006-hidden-68cc-4d40-xxxx-4319fa3fxxxx

Create or use an existing subnet:

a) To use an existing subnet:

ibmcloud is subnets

Example output:

Listing subnets for generation 2 compute in all resource groups and region ...
ID                                          Name                      Status      Subnet CIDR       Addresses     ACL                                                    Public Gateway                             VPC                 Zone         Resource group   
0737-27299d09-1d95-4a9d-a491-a6949axxxxxx   my-subnet                 available   10.240.128.0/18   16373/16384   husker-sloping-bee-resize                              my-gateway                                 my-vpc              us-south-3   kubeflow

If the above list contains the subnet corresponding to your VPC, that can be used to deploy your cluster - make sure you note it’s ID.

b) To create a new subnet:

List address prefixes and note the CIDR block corresponding to a Zone; in the below example, for Zone: us-south-3 the CIDR block is : 10.240.128.0/18.

ibmcloud is vpc-address-prefixes $VPC_ID

Example output:

Listing address prefixes of vpc r006-hidden-68cc-4d40-xxxx-4319fa3fxxxx under account IBM as user new@user-email.com...
ID                                          Name                                CIDR block        Zone         Has subnets   Is default   Created   
r006-xxxxxxxx-4002-46d2-8a4f-f69e7ba3xxxx   rising-rectified-much-brew          10.240.0.0/18     us-south-1   false         true         2021-03-05T14:58:39+05:30   
r006-xxxxxxxx-dca9-4321-bb6c-960c4424xxxx   retrial-reversal-pelican-cavalier   10.240.64.0/18    us-south-2   false         true         2021-03-05T14:58:39+05:30   
r006-xxxxxxxx-7352-4a46-bfb1-fcbac6cbxxxx   subfloor-certainly-herbal-ajar      10.240.128.0/18   us-south-3   false         true         2021-03-05T14:58:39+05:30

Now create a subnet as follows:

ibmcloud is subnet-create my-subnet $VPC_ID $CLUSTER_ZONE --ipv4-cidr-block "10.240.128.0/18"

Example output:

Creating subnet my-subnet in resource group kubeflow under account IBM as user new@user-email.com...

ID                  0737-27299d09-1d95-4a9d-a491-a6949axxxxxx   
Name                my-subnet

Make sure you export the subnet IDs follows:

export SUBNET_ID=0737-27299d09-1d95-4a9d-a491-a6949axxxxxx

Create a vpc-gen2 based Kubernetes cluster:

ibmcloud ks cluster create vpc-gen2 \
--name $CLUSTER_NAME \
--zone $CLUSTER_ZONE \
--version ${KUBERNETES_VERSION} \
--flavor ${WORKER_NODE_FLAVOR} \
--vpc-id ${VPC_ID} \
--subnet-id ${SUBNET_ID} \
--workers 2

Attach a public gateway

This step is mandatory for Kubeflow deployment to succeed, because pods need public internet access to download images.

First, check if your cluster is already assigned a public gateway:

ibmcloud is pubgws

Example output:

Listing public gateways for generation 2 compute in all resource groups and region ...
ID                                          Name                                       Status      Floating IP      VPC                 Zone         Resource group   
r006-xxxxxxxx-5731-4ffe-bc51-1d9e5fxxxxxx   my-gateway                                 available   xxx.xxx.xxx.xxx       my-vpc              us-south-3   default

In the above run, the gateway is already attached for the vpc: my-vpc. In case no gateway is attached, proceed with the rest of the setup.

Next, attach a public gateway by running the following command:

ibmcloud is public-gateway-create my-gateway $VPC_ID $CLUSTER_ZONE

Example output:

ID: r006-xxxxxxxx-5731-4ffe-bc51-1d9e5fxxxxxx

Save the above generated gateway ID as follows:

export GATEWAY_ID="r006-xxxxxxxx-5731-4ffe-bc51-1d9e5fxxxxxx"

Finally, attach the public gateway to the subnet:

ibmcloud is subnet-update $SUBNET_ID --public-gateway-id $GATEWAY_ID

Example output:

Updating subnet 0737-27299d09-1d95-4a9d-a491-a6949axxxxxx under account IBM as user new@user-email.com...

ID                  0737-27299d09-1d95-4a9d-a491-a6949axxxxxx   
Name                my-subnet   
...

Verifying the cluster

To use the created cluster, switch the Kubernetes context to point to the cluster:

ibmcloud ks cluster config --cluster ${CLUSTER_NAME}

Make sure all worker nodes are up with the command below:

kubectl get nodes

and verify that all the nodes are in Ready state.

Delete the cluster

Delete the cluster including it’s storage:

ibmcloud ks cluster rm --force-delete-storage -c ${CLUSTER_NAME}

Kubeflow on IKS – Documentation

Docs: Kubeflow Deployment Process

Understanding the Kubeflow deployment process

Repository layout

Kubeflow installation

Install kubectl and kustomize

Next Steps

Docs: Pipelines

Declare the Python Client for Kubeflow Pipelines

1. Single-user Kubeflow Pipelines deployment with the SDK

2. Authenticating multi-user Kubeflow Pipelines with the SDK

Upload pipelines

Run pipelines from the SDK

Run pipelines from the Python DSL source code

Run pipelines from the compiled pipeline file

Run pipelines from the list of uploaded pipelines

Docs: IBM Cloud Kubernetes and Kubeflow Compatibility

Compatibility

Next Steps

Docs: Install Kubeflow

Prerequisites

Storage setup for a Classic IBM Cloud Kubernetes cluster

Storage setup for vpc-gen2 IBM Cloud Kubernetes cluster

Installation

Single user

Accessing your cluster

Multi-user, auth-enabled

Prerequisites

Deploy: Using kustomize together with kubectl

Verify mutli-user installation

Extra network setup requirement for vpc-gen2 clusters only

Next steps: secure the Kubeflow dashboard with HTTPS

Prerequisites

Setup

Troubleshooting

Expose the Kubeflow endpoint as a LoadBalancer

Authservice pod taking too long to restart

Docs: Using IBM Cloud Container Registry (ICR)

Prerequisites

Image pull secret

Using a default image pull secret

Getting an IAM API Key

Scenarios

Using container images from ICR in Kubeflow Pipelines

Using notebook images from ICR in a Jupyter Notebook

Docs: End-to-end Kubeflow on IBM Cloud

Introduction

Overview of IKS

The model and the data

The overall workflow

Run the MNIST Tutorial on IKS

Docs: Securing the Kubeflow authentication with HTTPS

Prerequisites

Setting up an NLB

Exposing the Kubeflow dashboard with DNS and TLS termination

Optional - KServe configuration

Docs: Kubernetes Cluster on Classic Infra

Prerequisites

Connecting to an existing cluster

Create and setup a new cluster

Setting environment variables

Choosing a worker node flavor

Creating an IBM Cloud Kubernetes cluster

Verifying the cluster

Delete the cluster

Docs: Uninstall Kubeflow

Docs: Kubernetes cluster on a VPC

Create and setup a new cluster

Setting environment variables

Choosing a worker node flavor

Create an IBM Cloud Kubernetes cluster for vpc-gen2 infrastructure

Verifying the cluster

Delete the cluster

Create an IBM Cloud Kubernetes cluster for `vpc-gen2` infrastructure