genai.extensions.localserver package#

Extension for running local simplified inference API compatible with current version of SDK

class genai.extensions.localserver.LocalLLMServer[source]#

Bases: object

__init__(models, port=8080, interface='0.0.0.0', api_key=None, insecure_api=False)[source]#
Parameters:

  • models (list[type[LocalModel]]) –

  • port (int) –

  • interface (str) –

  • api_key (str | None) –

  • insecure_api (bool) –

get_credentials()[source]#
run_locally()[source]#
start_server()[source]#
stop_server()[source]#
class genai.extensions.localserver.LocalModel[source]#

Bases: ABC

abstract generate(input_text, parameters)[source]#

Generate a response from your llm using the provided input text and parameters

Parameters:

  • input_text (str) – The input prompt chat

  • parameters (TextGenerationParameters) – The parameters that the user code wishes to be used

Raises:

NotImplementedError – If you do not implement this function.

Returns:

The result to be sent back to the client

Return type:

TextGenerationResult

property model_id#

Model ID This is the ID that you would use when defining the model you want to use

example: “google/flan-t5-base”

Raises:

NotImplementedError – If you do not implement this property.

abstract tokenize(input_text, parameters)[source]#

Tokenize the input text with your model and return the output

Parameters:

  • input_text (str) – The input prompt chat

  • parameters (TextTokenizationParameters) – The parameter that the user code wishes to be used

Raises:

NotImplementedError – If you do not implement this function.

Returns:

The result to be sent back to the client

Return type:

TextTokenizationCreateResults

Submodules#