streamsx.toolkits package

Utility functions for IBM Streams topology applications

For details of implementing applications in Python for IBM Streams including IBM Cloud Pak for Data and the Streaming Analytics service running on IBM Cloud see:

Overview

Provides provides some utility functions to work with SPL toolkits, for example for use in a Python Notebook.

streamsx.toolkits.create_keystore(client_cert, client_private_key, store_filepath, store_passwd=None)

Creates a JKS type keystore for a client certificate and its private RSA key.

The aliases of the certificate in the keystore will be ‘client_cert’. The private key will be encrypted with the store password.

Example:

client_cert = """
-----BEGIN CERTIFICATE-----
MIICtjCCAZ4CCQCBrBBIDMiWTTANBgkqhkiG9w0BAQsFADAqMRMwEQYDVQQKDApp
by5zdHJpbXppMRMwEQYDVQQDDApjbGllbnRzLWNhMB4XDTE5MDgxNjA5MzAzOVoX
DTIwMDgwNjA5MzAzOVowEDEOMAwGA1UEAwwFcm9sZWYwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQC4rmdUhW0ZE0O7mwaYilIWb8Lqy/g4G7zYKa+5nVNI
pM8Y9DLM6UmXFmO5SyiYWNzJCvbrj1u20deOrLetTpid6/LactoekVF6Gwkcy7uh
FvkMvvzO00zU/7HGdzMRqwIWHZFVFQ7eNJcF5ITTPSyjfuXKCqVlTN70U6pVR1Kt
jvh95U4qi6UxfrhfPSgX5rL1UgmMnYemeFWW9AmROk5FNOrXpiH9pRM9e034J/Lx
X4fSArMc/HB2qmFkat5RSYNdYE47Upv/ufE0bl1CXvZ4hlKdQ2nyFV0ZkZ/IdtNV
wXRczXmVKKEvCISTZWri1qFG2zs5ZibQXt2TWi/Hmb7rAgMBAAEwDQYJKoZIhvcN
AQELBQADggEBABhDMkkXfGXagGY6EgZ4cev1h18jVm0uNknlUw6UXgLfoX7OBUmr
yjfYWIqdNOewY3IZ1iTQHoEWXuFmuEUalb+CfYXdoZuHx/s211sLBy0S9pod3h+f
VcwovrXqfE0V/Of5+3svE2p+O2N+UschjH/Ha9ljJ4CbVN0b/K8KkOt2dd6lMn2l
jLyzjCBtnxBYnS1tbyILK6Mdb0Q6EeYBQw5K6x+f9gRvcyNpBcMMFZFk0n0Hr+zW
4MuwSg7eM4OU1hExqUSILEfaVtsj4IgCimUN5RGEYBp8YFzT0GYnEm+c4iaEb4Uh
1aDgpwpHly2t0dI7m3i2VYN4/UeRi2tXtS8=
-----END CERTIFICATE-----
"""

private_key = """
-----BEGIN PRIVATE KEY-----
MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQC4rmdUhW0ZE0O7
mwaYilIWb8Lqy/g4G7zYKa+5nVNIpM8Y9DLM6UmXFmO5SyiYWNzJCvbrj1u20deO
rLetTpid6/LactoekVF6Gwkcy7uhFvkMvvzO00zU/7HGdzMRqwIWHZFVFQ7eNJcF
5ITTPSyjfuXKCqVlTN70U6pVR1Ktjvh95U4qi6UxfrhfPSgX5rL1UgmMnYemeFWW
9AmROk5FNOrXpiH9pRM9e034J/LxX4fSArMc/HB2qmFkat5RSYNdYE47Upv/ufE0
bl1CXvZ4hlKdQ2nyFV0ZkZ/IdtNVwXRczXmVKKEvCISTZWri1qFG2zs5ZibQXt2T
Wi/Hmb7rAgMBAAECggEBAJyes1zXprUcRFXm3AojTAhhEaGEB2a2B0oe2DsGtCmo
M0XLWwWyVkl+oGX02mGRAvSwisUsl7AMd800prSgMgfDoPonatl/jeSK5wh3sxCj
ZeSsV8OjKMZ+biqMzk+Ogj91JdpM+IXzfZxut0JZ3/Br3r6glrG5Jl94ULzvtqse
YBXzi+9wbDjhzUrZea8GBSEgm4er8nGVlvl3cGA6B05KCkndSrpzJmD/1O1yHl2w
jWcxnWbMjgJ+ZgTkWxNx1AACP1XnUubuHtH5X6CdyyexJVIcvSou29LE4AEgskzf
N5f9g4WQqm/wqJouA6g2O2bYuH6RqPBR6LvRiWN+XCkCgYEA9W+xqD3EadT+9bkq
SFkAHntFMnkmiu78azr5CSqXGZWiT12BloZg44Y4ouHkz0Q7UIOkuKnt8XUT+XTx
cMJXdM/L8A7XadbLRGm+jQkPkamxzbhNha3RPqNcMmuSl+bus0/Y4CPlRzqmVFM2
6i+7AJdSovt7ELPyB5VRoFTjqM8CgYEAwKFJ5C0DPbtm8lwf42Vg9lKReR2lhrdh
70rIvuBwLpvmDEjT4E5XlCQZvmwCJhYQLeR7pdWTZUIUY3PRg6saUPt6mseQaY5G
UMfd8eKNmYDq1dzw66L/rXVfLXL/U965mlchgrv1aYxFvS2oaXc06FGWf7HiX70r
AYL3s4NsVyUCgYAntVFt4chkQvSY9sMzf0q4o8L3PoiQ08d85caiyAQ4PHBNHNi5
bLZCOVXybE+jhOt/xDyjL8LyXLnFkzj4wEh7+p6JES2izwCF6gHNOZkOPQEqQlYw
eYWv96gT9Dl8X/1gf1ucL9KJaqC8ZdYtn1lG6DSTb9JiSmqu9WaGZRWMGwKBgQCz
Y4DUpbam/qbYfUKVGgw6WJpxGFLWKmFh31sjpw/R2A1c01W9Rjf5FcWDuEIoBdP7
aRRdU5B4dBlYpLswy18um4OKMN8j7gM9p2PhpwIHOi94CtGyH5HLj8qbIMT27iwa
t5nVjgT1zlQD4Uk9Lm5vC8wEDzypvBkYx0a2PmXeoQKBgQCdoz1xPj5Ltrocn992
ci3mu3a5OxozeJVi36XSoWwr9HrK0arTaTEkbqcQxnjuCx/pIxM+FQQrVVjpAdop
N1lLtfM0/X9cXohp2CwQGFldjOd7pF0t28ewN0wRij6oOdVa/u3EsVqijnhSEhOl
YmR4uwOY4uG/ecALiF/2BBz5OA==
-----END PRIVATE KEY-----
"""
key_store_password = create_keystore(client_cert, private_key, "/tmp/keystore.jks")

# read key and certificate from existing files
store_passwd = "Y3456tugzh"
create_keystore("/tmp/client_cert.pem", "/tmp/client_priv_key.pem", "/tmp/keystore.jks", store_passwd)
Parameters
  • client_cert (str) – the filename with a client certificate in PEM format or the certificate itself

  • client_private_key (str) – the filename with a private key in PEM format or the private key itself

  • store_filepath (str) – the filename of the keystore file. The location must be writable. Intermediate directories are not created.

  • store_passwd (str) – the password for the private key and the keystore. When None, a 16 characters password is generated.

Returns

the generated or given password of the keystore and the private key

Return type

str

See also: create_truststore(), extend_keystore()

New in version 1.1.

streamsx.toolkits.extend_keystore(client_cert, client_private_key, store_filepath, store_passwd)

Extends a JKS type keystore with a client certificate and its private RSA key.

The alias of the certificate in the keystore will be client_cert-<n>. The private key will be encrypted with the store password.

Example:

store_file = '/tmp/keystore.jks'
store_passwd = create_keystore(client_cert='/tmp/db_client.crt',
                               client_private_key='/tmp/db_client.key',
                               store_filepath=store_file)
extend_keystore(client_cert='/tmp/mqtt_client.crt',
                client_private_key='/tmp/mqtt_client.key',
                store_filepath=store_file,
                store_passwd=store_passwd)
Parameters
  • client_cert (str) – the filename with a client certificate in PEM format or the certificate itself

  • client_private_key (str) – the filename with a private key in PEM format or the private key itself

  • store_filepath (str) – the filename of the keystore file. The file must exist and be writable.

  • store_passwd (str) – the password for the keystore. It is also used to encrypt the given private key.

Returns

the aliases of the added entries (currently a list with one element)

Return type

list

See also: create_keystore()

New in version 1.2.

streamsx.toolkits.create_truststore(trusted_cert, store_filepath, store_passwd=None)

Creates a JKS type keystore from a single trusted certificate or a list of trusted certificates. A keystore with trusted certificates is therefore referred to as a truststore. Truststores are typically used to store root certificates or self-signed certificates that servers present to clients. The aliases of the certificates in the keystore will be ‘ca_cert-i’, where i is a number starting at 0.

Example:

trusted_cert = """
-----BEGIN CERTIFICATE-----
MIIDJzCCAg+gAwIBAgIJAJhu1pO2qbj7MA0GCSqGSIb3DQEBCwUAMCoxEzARBgNV
BAoMCmlvLnN0cmltemkxEzARBgNVBAMMCmNsdXN0ZXItY2EwHhcNMTkwODE2MDc1
ODI3WhcNMjAwODE1MDc1ODI3WjAqMRMwEQYDVQQKDAppby5zdHJpbXppMRMwEQYD
VQQDDApjbHVzdGVyLWNhMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
poC8jaqI9izS96CS39m4aOFruWDPxcrDsfU4gIbFveMHOV57F/LWkiC4GHWrcujj
YHEKShMyP2rMfmQp148lxEZMc84zigB89ZUjWiFFR8PZBeX7syUmvTb/sEwWswOP
epMUwqoNwDJ49HLQKbrSuZ+T4XymcPAmHU+Osm77hpJRGsloN1uVCBwzJJgxf9oz
m0qLDMI4sw9GD3B8gjPZowZ70LtDnKkmG3hGXZwW1cvL2vuECdKWm77FUe+L2NVx
c6cnZ5htNX0wjATf5hTN6uXsNUK/Xf6lKwcRgatYQVGT+kd+tDraHwhkID8cPyFi
F+vbIhDzhKSBHGwYIGn+SwIDAQABo1AwTjAdBgNVHQ4EFgQUIYRTcQDd8rskiN2Q
cV101xbmI9MwHwYDVR0jBBgwFoAUIYRTcQDd8rskiN2QcV101xbmI9MwDAYDVR0T
BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAQtuxhqeT0z5fNB3Vs13TvVLfXPiG
YSTwYvxfkrxGKtDiLjIZ6Q6LJjMcWLG4x3I0WmGgvytTs04S4B+1vp721jmqRKm9
xCJ+Obi/eWmN4uu1rr0fR7fXr7wWLFBiDr8dZX3EnzjWeVWeNESoZTpaMKScuYTZ
ocAT5iL3ZDUj/lwqJRptmzGFcdko+woFae68HRx1ygSgROls7bXy/CwgME0LFFQp
B+2YAhUw1sPU410JUxU3/v6R5vJfI9imE75aha3U7UbeOX8+1+Cu3HOT1QMn80k2
6LnZeMCCgCBp+Yz3YNeUMRejMU6x4WlhTPO7bBq3tKGgwCoyGIX25wMM1Q==
-----END CERTIFICATE-----
"""
generated_truststore_password = create_truststore(trusted_cert, "/tmp/truststore.jks")

# multiple certificates in existing files
store_passwd = "Y3456tugzh"
create_truststore(["/tmp/cert1.pem", "/tmp/cert2.pem"], "/tmp/truststore.jks", store_passwd)
Parameters
  • trusted_cert (list|str) – a list of filenames with trusted certificates in PEM format or the certificates literally

  • store_filepath (str) – the filename of the truststore file. The location must be writable. Intermediate directories are not created.

  • store_passwd (str) – the password for the keystore. When None, a 16 characters password is generated.

Returns

the generated or given password of the truststore

Return type

str

See also: create_keystore(), extend_truststore()

New in version 1.1.

streamsx.toolkits.extend_truststore(trusted_cert, store_filepath, store_passwd)

Extends an existing truststore by trusted certificate entries.

Example:

store_file = '/tmp/truststore.jks'
store_passwd = create_truststore('/tmp/db.crt', store_filepath=store_file)
# add two more certificates:
extend_truststore(['/tmp/mqtt.crt', '/tmp/kafka.crt'],
                  store_filepath=store_file,
                  store_passwd=store_passwd)
Parameters
  • trusted_cert (list|str) – a list of filenames with trusted certificates in PEM format or the certificates literally

  • store_filepath (str) – the filename of the truststore file. The truststore must exist and be writable.

  • store_passwd (str) – the password for the given truststore.

Returns

the aliases of the added entries

Return type

list

See also: create_truststore()

New in version 1.2.

streamsx.toolkits.download_toolkit(toolkit_name, repository_name=None, url=None, target_dir=None)

Downloads the latest SPL toolkit from GitHub for the given toolkit name.

Example for adding the com.ibm.streams.nlp toolkit with latest toolkit from GitHub:

import streamsx.toolkits as tkutils
# download toolkit from GitHub
location = tkutils.download_toolkit('com.ibm.streamsx.nlp')
# add toolkit to topology
streamsx.spl.toolkit.add_toolkit(topo, location)
Parameters
  • toolkit_name (str) – the toolkit directory in the archive (where toolkit.xml is located), for example “com.ibm.streamsx.nlp”

  • repository_name (str) – name of the GitHub repository at “github.com/IBMStreams”, for example “streamsx.nlp”. Set this parameter if repository name is not part of toolkit name without “com.ibm.”.

  • url (str) – the download URL, apply link to toolkit archive (*.tgz) to be downloaded. ‘https://github.com/IBMStreams/REPOSITORY-NAME/releases/latest’ is used per default.

  • target_dir (str) – the directory where the toolkit is unpacked to. If a relative path is given, the target_dir is appended to the system temporary directory, for example to /tmp on Unix/Linux systems.

Returns

the absolute path of the toolkit directory

Return type

str

Note

This function requires an outgoing Internet connection

streamsx.toolkits.get_pypi_packages(package_name=None)

Discover the latest Python packages available on pypi.org.

Parameters
  • package_name (str) – the name of the Python package to be searched for. If None is given,

  • function searches for all available streamsx Python packages. (the) –

Returns

A dictionary with mappings from Python package name to the package version

Return type

dict

Note

This function requires an outgoing Internet connection

streamsx.toolkits.get_installed_packages()

Discover installed streamsx python packages in your Python environment.

Returns

A dictionary with mappings from Python package name to the package version

Return type

dict

streamsx.toolkits.get_build_service_toolkits(streams_cfg=None, verify=False)

Discover toolkits on IBM Streams build service.

Parameters
  • streams_cfg (dict) – Service instance details of the IBM Streams instance. If value is None then the external build service endpoint is used.

  • verify (bool) – Set to True if SSL verification shall be enabled.

Example, use the function in IBM Cloud Pak for Data with service instance details retrieved from icpd_util.get_service_details:

from icpd_core import ipcd_util
cfg = icpd_util.get_service_details(name='instanceName', instance_type='streams')

import streamsx.toolkits as tkutils
build_service_toolkits = tkutils.get_build_service_toolkits(cfg)
print(build_service_toolkits)
Returns

A dictionary with mappings from toolkit name to the toolkit version

Return type

dict

streamsx.toolkits.get_github_toolkits(toolkit_name=None)

Discover the latest releases of product toolkits available on public GitHub.

Parameters
  • toolkit_name (str) – the name of the toolkit to be searched for. If None is given,

  • function searches for all product toolkits. (the) –

Returns

A dictionary with mappings from toolkit name to the toolkit version

Return type

dict

Note

This function requires an outgoing Internet connection

Indices and tables