Connections

A connection is an object used to store connection-related data for a single datasource (Kafka, Azure, etc.), such as credentials, secrets, and URLs. Each connection is defined by exactly one datasource.

Tip

To read more about datasources see Retrieving a DatasourceType.

The SDK provides functionality to interact with connections.

This includes operations such as:

• Creating a connection

• Retrieving connection(s)

• Updating a connection

• Deleting a connection

Creating a Connection

In the UI, you can create a new Connection by navigating to Assets -> New asset -> Connect to a datasource.

You will need to choose the desired datasource from the list.

and provide a Name and other additional configuration (depending on the selected datasource).

In the SDK, you can create a new Connection object within a Project, by selecting the appropriate project from the Platform and then using the Project.create_connection() method to instantiate the connection.

You must provide a name and datasource_type to create a connection. DatasourceType is an object that defines the data source for which the connection will be created.

By default, the server validates a connection before creating it. If the connection cannot be established with the provided parameters, you will get an error and the connection will not be saved. To skip validation and create the connection anyway, set the test parameter of Project.create_connection() to False.

Note

To see available datasources, refer to Retrieving a DatasourceType.

>>> datasource_type = platform.datasources.get(name='http')
>>> connection = project.create_connection(
...     name='Connection Name',
...     datasource_type=datasource_type,
...     description='Description ...',
...     properties={'url': 'https://my/connection/url'},
... )
>>> connection
Connection(name='Connection Name')

Important

Each datasource_type defines its own set of required properties.

To see the list of available properties, refer to Retrieving a DatasourceType connection properties.

Retrieving an existing Connection

In the UI, you can view all Connections by navigating to Assets -> Data access -> Connections.

In the SDK, a Connection can be retrieved using the Project.connections property. You can also filter the returned connections based on attributes including name, connection_id, context, properties, and datasource_type.

This property returns a Connections object.

>>> # Returns the first connection matching given `name`
>>> connection = project.connections.get(name='Connection Name')
>>> connection
Connection(name='Connection Name')

>>> # Get the connection id
>>> connection_id = connection.connection_id

>>> # Returns a connection by its connection_id
>>> project.connections.get(connection_id=connection_id)
Connection(name='Connection Name')

>>> # Return a list of all connections that match `properties`
>>> project.connections.get_all(
...     properties={'url': 'https://my/connection/url'}
... )
[Connection(name='Connection Name')]

>>> # Return a list of all connections
>>> project.connections 
[Connection(name='Connection Name')]

Tip

For detailed information about parameters and values refer to https://cloud.ibm.com/apidocs/data-ai-common-core Connections section.

Updating a Connection

In the UI, you can update a Connection by navigating to Assets -> Data access -> Connections and clicking the edit button on the Connection you want to edit. This button is visible only when the cursor is over the Connection object.

To update a connection in the SDK, first modify the properties of the connection and then pass the instance to the Project.update_connection() method.

This method returns an HTTP response indicating the status of the update operation.

Similar to Creating a Connection, you can skip validation by setting the test parameter of Project.update_connection() to False.

>>> connection.name = 'New Connection Name'
>>> project.update_connection(connection)
<Response [200]>
>>> project.connections.get(name='New Connection Name') 
Connection(name='New Connection Name')

Using a Connection

In the UI, you can use a Connection in a flow by adding a stage that takes a Connection and choosing the Connection in the view connection or connection property. For batch flows, this would be any connector stage.

For streaming flows, this would be select source and target stages.

In the SDK, to use a Connection instance for batch flows, you can pass it to the StageNode.use_connection() method of the connector stage. To use a Connection instance for streaming flows, you can pass it to the Stage.use_connection() method of the target or destination stage.

>>> # Batch Flow
>>> new_flow = project.create_flow(name='New flow', environment=None, flow_type='batch')
>>> connection = project.connections.get(name='New Connection Name')
>>> connection 
Connection(name='New Connection Name')
>>> http = new_flow.add_stage(type='HTTP', label='HTTP_Stage_1')
>>> http.use_connection(connection) 
>>> # Streaming Flow
>>> connection_to_use
Connection(name='dummy_connection ...')
>>> origin = flow.add_stage('JDBC Query Consumer')
>>> origin.use_connection(connection_to_use)

Using a local Connection (Batch Only)

An alternative way to use a connector stage is to put connection parameters directly into the stage’s properties. In the UI, this can be done under the Connection details section of the stage.

If you are already Using a Connection, the Connection details section will not be visible.

In the SDK, each connector stage’s configuration has a connection property. To use a local connection, you can directly edit the desired fields of this connection property.

>>> http = new_flow.add_stage(type='HTTP', label='HTTP_Stage_1')
>>> http.configuration.connection.url = 'yoururl.com'
>>> http.configuration.connection.ssl_certificate = 'yoursslcertificate'

Deleting a Connection

In the UI, you can delete an existing Connection by navigating to the Assets -> Data access -> Connections.

In the SDK, to delete a connection instance, you can pass it to the Project.delete_connection() method.

This method returns an HTTP response indicating the status of the delete operation.

>>> project.delete_connection(connection)
<Response [204]>