You're reading an old version of this documentation. If you want up-to-date information, please have a look at v3.0.0.

Hide navigation sidebar

Hide table of contents sidebar

Toggle site navigation sidebar

IBM Generative AI Python SDK (Tech Preview)

Toggle table of contents sidebar

IBM Generative AI Python SDK (Tech Preview)

Getting Started
V2 Migration Guide
Examples
Toggle navigation of Examples
- Text
  Toggle navigation of Text
- Models
  Toggle navigation of Models
  - Show information about supported models
- Tunes
  Toggle navigation of Tunes
  - Tune a custom model (Prompt Tuning)
- Prompts
  Toggle navigation of Prompts
  - Create a custom prompt with variables
- Files
  Toggle navigation of Files
  - Working with files
- Users
  Toggle navigation of Users
  - Show information about current user
- Requests
  Toggle navigation of Requests
  - Working with your requests
- Extra
  Toggle navigation of Extra
- Extensions
  Toggle navigation of Extensions
  - Transformers (HuggingFace)
    Toggle navigation of Transformers (HuggingFace)
    - Run Transformers Agents
  - LangChain
    Toggle navigation of LangChain
  - LLamaIndex
    Toggle navigation of LLamaIndex
    - Use a model through LLamaIndex
  - LocalServer
    Toggle navigation of LocalServer
    - Customize behavior of local client
    - Use a local server with a custom model
FAQ

Versions

main (unreleased)
v3.0.0
v2.3.0
v2.2.0
v2.1.1
v2.1.0
v2.0.0

Toggle table of contents sidebar

genai.text.tokenization.schema module#

Copyright © 2024, IBM Research

Made with Sphinx and @pradyunsg's Furo