.. _examples.text.tokenization:

Tokenize text data
==================

Runs tokenization asynchronously in parallel to achieve better performance

.. literalinclude:: ../../../examples/text/tokenization.py
	:language: python
	:caption: See `tokenization.py <https://github.com/IBM/ibm-generative-ai/blob/main/examples/text/tokenization.py>`_ on GitHub.
	:lines: 7-