Skip to content

Experiment API

AI4RAGExperiment

AI4RAGExperiment(
    documents: list[DoclingDocument],
    benchmark_data: DataFrame,
    search_space: AI4RAGSearchSpace,
    optimizer_settings: OptimizerSettings,
    event_handler: BaseEventHandler,
    vector_store_config: BaseVectorStoreConfig,
    optimization_metric: RAGMetric = Metrics.OVERALL_SCORE,
    **kwargs
)

Class responsible for conducting AutoRAG experiment, that consists of finding the best hyperparameters for several steps/stages.

AI4RAGExperiment is essentially an orchestrator for the RAG Patterns hyperparameters optimization for the desired metric. It requires from user to provide fully defined search space on which the experiment will be executed.

AI4RAG uses 'BaseRAGTemplate' inheriting classes as definitions on how to build and utilize RAG Pattern with the given search space nodes.

Template selection uses the categorical template_id search parameter.

Parameters:

  • documents (list[DoclingDocument]) –

    List of parsed docling documents to embed in vector db and use as context in RAG.

  • benchmark_data (DataFrame | BenchmarkData) –

    Structure with 3 columns: 'question', 'correct_answers' and 'correct_answer_document_keys'.

  • search_space (AI4RAGSearchSpace) –

    Grid of parameters used during hyperparameter optimization.

  • optimizer_settings (OptimizerSettings) –

    Settings for the optimizer to be used during the experiment.

  • vector_store_config (BaseVectorStoreConfig) –

    Connection config for the vector store backend. Its type (via config.provider) determines which vector store implementation is used for indexing and retrieval.

  • event_handler (BaseEventHandler) –

    Instance satisfying BaseEventHandler's interface to stream pattern evaluation results and intermediate status updates. EventHandler is an entrypoint to configure custom logging and assets handling.

  • optimization_metric (RAGMetric, default: Metrics.OVERALL_SCORE ) –

    Metric used for calculating the final score that drives optimization. Must be a RAGMetric instance selected from :class:Metrics.

Other Parameters:

  • metrics (Sequence[RAGMetric]) –

    Metrics evaluated during the AutoRAG experiment, each a RAGMetric instance selected from :class:Metrics. Not all of these metrics are used to calculate the final score, but they are included in the evaluation results. When omitted, defaults are derived from the configured evaluators.

  • evaluators (list[BaseEvaluator] | None) –

    Evaluator instances used to score RAG patterns during optimization. When None, defaults to [UnitxtEvaluator()]. To enable LLM-as-a-Judge evaluation, pass both a UnitxtEvaluator and a LLMaJEvaluator configured with a judge model.

  • n_mps_foundation_models (int) –

    Amount of foundation models to be further used in experiment post pre-selection.

  • n_mps_embedding_models (int) –

    Amount of embedding models to be further used in experiment post pre-selection.

  • inference_max_threads (int) –

    Defines the number of threads to use during generation model inference.

Attributes:

  • results (ExperimentResults) –

    Instance holding information about each iteration during the experiment. It consists of statuses, RAG pattern objects, scores and settings.

Source code in ai4rag/core/experiment/experiment.py
def __init__(
    self,
    documents: list[DoclingDocument],
    benchmark_data: pd.DataFrame,
    search_space: AI4RAGSearchSpace,
    optimizer_settings: OptimizerSettings,
    event_handler: BaseEventHandler,
    vector_store_config: BaseVectorStoreConfig,
    optimization_metric: RAGMetric = Metrics.OVERALL_SCORE,
    **kwargs,
):
    self.documents = documents
    self.benchmark_data = BenchmarkData(benchmark_data)
    self.search_space = search_space
    self.vector_store_config = vector_store_config
    self.optimizer_settings = optimizer_settings
    self.event_handler = event_handler
    self.optimization_metric = optimization_metric

    self.evaluators: list[BaseEvaluator] = kwargs.pop("evaluators", None)
    self.metrics: Sequence[RAGMetric] | None = kwargs.pop(
        "metrics", None
    )  # resolved in _resolve_metrics_and_validate
    self.n_mps_foundation_models = kwargs.pop(
        "n_mps_foundation_models", PreSelectorConstants.DEFAULT_N_FOUNDATION_MODELS
    )
    self.n_mps_embedding_models = kwargs.pop(
        "n_mps_embedding_models", PreSelectorConstants.DEFAULT_N_EMBEDDING_MODELS
    )
    self.known_observations: list[dict] | None = kwargs.pop("known_observations", None)
    self.inference_max_threads: int = kwargs.pop("inference_max_threads", 10)
    self.kg_extraction_config: dict[str, Any] | None = kwargs.pop("kg_extraction_config", None)
    graph_retrieval_config = kwargs.pop("graph_retrieval_config", None)
    if graph_retrieval_config is None:
        self.graph_retrieval_config: Neo4jGraphRetrievalConfig | None = (
            Neo4jGraphRetrievalConfig() if self.vector_store_config.provider == "neo4j" else None
        )
    elif isinstance(graph_retrieval_config, Neo4jGraphRetrievalConfig):
        self.graph_retrieval_config = graph_retrieval_config
    elif isinstance(graph_retrieval_config, Mapping):
        self.graph_retrieval_config = Neo4jGraphRetrievalConfig.from_mapping(graph_retrieval_config)
    else:
        raise TypeError("graph_retrieval_config must be a mapping or Neo4jGraphRetrievalConfig.")
    self.results: ExperimentResults = ExperimentResults()
    self._exception_handler = ExperimentExceptionHandler(self.event_handler)
    self._optimization_phase: str | None = None

    if kwargs:
        logger.warning("Unknown parameters: %s", kwargs)

    self._resolve_metrics_and_validate()

Attributes

documents property writable

documents: list[DoclingDocument]

Get list of documents.

optimization_metric property writable

optimization_metric: RAGMetric

Get optimization metrics used for the experiment.

benchmark_data property writable

benchmark_data: BenchmarkData

Get benchmark data.

evaluators property writable

evaluators: list[BaseEvaluator]

Get experiment evaluators.

metrics property writable

metrics: Sequence[RAGMetric] | None

Get evaluation metrics.

Methods:

run_pre_selection

run_pre_selection(
    foundation_models: list[BaseFoundationModel],
    embedding_models: list[BaseEmbeddingModel],
    n_records: int = 5,
    random_seed: int = 17,
) -> dict[str, list[BaseEmbeddingModel | BaseFoundationModel]]

Run models pre-selection using ModelsPreSelector and sample of the data.

Parameters:

  • embedding_models (list[BaseEmbeddingModel]) –

    Embedding models to be considered during pre-selection process.

  • foundation_models (list[BaseFoundationModel]) –

    Foundation models to be evaluated during pre-selection process.

  • n_records (int, default: 5 ) –

    Amount of records that should be used during models pre-selection.

  • random_seed (int, default: 17 ) –

    Random seed value used for sampling benchmark data records.

Returns:

  • dict[str, list[BaseFoundationModel | EmbeddingModel]] –

    Best embedding models and foundation models found in pre-selection.

Source code in ai4rag/core/experiment/experiment.py
def run_pre_selection(
    self,
    foundation_models: list[BaseFoundationModel],
    embedding_models: list[BaseEmbeddingModel],
    n_records: int = 5,
    random_seed: int = 17,
) -> dict[str, list[BaseEmbeddingModel | BaseFoundationModel]]:
    """
    Run models pre-selection using ModelsPreSelector and sample
    of the data.

    Parameters
    ----------
    embedding_models : list[BaseEmbeddingModel]
        Embedding models to be considered during pre-selection process.

    foundation_models : list[BaseFoundationModel]
        Foundation models to be evaluated during pre-selection process.

    n_records : int, default=5
        Amount of records that should be used during models pre-selection.

    random_seed : int, default=17
        Random seed value used for sampling benchmark data records.

    Returns
    -------
    dict[str, list[BaseFoundationModel | EmbeddingModel]]
        Best embedding models and foundation models found in pre-selection.
    """
    _log_start_mps = (
        f"Starting foundation models pre-selection with following "
        f"foundation models: {[str(fm) for fm in foundation_models]} "
        f"and following embedding models: {[str(em) for em in embedding_models]}."
    )
    logger.info(_log_start_mps)
    self.event_handler.on_status_change(
        level=LogLevel.INFO,
        message=_log_start_mps,
        step=ExperimentStep.MODEL_SELECTION,
    )

    from ai4rag.core.experiment.mps import ModelsPreSelector

    mps = ModelsPreSelector(
        benchmark_data=self.benchmark_data.get_random_sample(n_records=n_records, random_seed=random_seed),
        documents=self.documents.copy(),
        foundation_models=foundation_models,
        embedding_models=embedding_models,
        metric=Metrics.OVERALL_SCORE,
    )
    mps.evaluate_patterns()

    selected_models = mps.select_models(
        n_embedding_models=self.n_mps_embedding_models,
        n_foundation_models=self.n_mps_foundation_models,
    )

    logger.info(
        "Models pre-selection has been finished. Selected foundation models: %s and selected embedding models: %s.",
        [str(model) for model in selected_models["foundation_models"]],
        [str(model) for model in selected_models["embedding_models"]],
    )

    return selected_models

run_single_evaluation

run_single_evaluation(rag_params: RAGParamsType, publish_pattern: bool = True) -> float

Evaluate a single RAG configuration and return its score using provided documents.

Parameters:

  • rag_params (RAGParamsType) –

    A dictionary containing rag parameters as keys and their values.

  • publish_pattern (bool, default: True ) –

    Whether to send the evaluated pattern to the event handler immediately. GAM optimization suppresses this only for warm-start candidates; GAM candidates are published as they complete.

Returns:

  • float –

    A single evaluation score obtained by the executed rag pattern.

Source code in ai4rag/core/experiment/experiment.py
def run_single_evaluation(self, rag_params: RAGParamsType, publish_pattern: bool = True) -> float:
    """
    Evaluate a single RAG configuration and return its score using provided documents.

    Parameters
    ----------
    rag_params : RAGParamsType
        A dictionary containing rag parameters as keys and their values.

    publish_pattern : bool, default=True
        Whether to send the evaluated pattern to the event handler immediately.
        GAM optimization suppresses this only for warm-start candidates;
        GAM candidates are published as they complete.

    Returns
    -------
    float
        A single evaluation score obtained by the executed rag pattern.
    """
    template_id = rag_params[AI4RAGParamNames.TEMPLATE_ID]
    template_spec = get_template_spec(template_id)
    start_time = time.time()

    chunking_params = get_chunking_params(rag_params)
    chunking_params["include_metadata"] = chunking_params.get(AI4RAGParamNames.CHUNKING_METHOD) == "hybrid"

    retrieval_params = get_retrieval_params(rag_params)

    foundation_model = rag_params.get(AI4RAGParamNames.FOUNDATION_MODEL)
    embedding_model = rag_params.get(AI4RAGParamNames.EMBEDDING_MODEL)
    search_mode = retrieval_params.get(AI4RAGParamNames.SEARCH_MODE, "vector")

    embedding_params_dict = (
        asdict(embedding_model.params) if is_dataclass(embedding_model.params) else embedding_model.params
    )
    indexing_params = {
        "chunking": chunking_params,
        "embedding": {
            "model_id": embedding_model.model_id,
            "embedding_params": embedding_params_dict,
        },
    }

    # A Neo4j collection contains LLM-extracted entities and relationships,
    # not only chunk embeddings.  Those graph contents vary with the model
    # and extraction settings, so they must participate in the collection
    # reuse key as well.
    if self.vector_store_config.provider == "neo4j":
        kg_extraction_config = self.kg_extraction_config or {"mode": "constrained"}
        indexing_params["knowledge_graph"] = {
            "model_id": foundation_model.model_id,
            "model_params": {
                "temperature": foundation_model.params.temperature,
                "max_completion_tokens": kg_extraction_config.get(
                    "max_completion_tokens", DEFAULT_KG_EXTRACTION_MAX_COMPLETION_TOKENS
                ),
            },
            "extraction_config": kg_extraction_config,
        }

    logger.info("Using indexing params: %s", indexing_params)

    retrieval_method = retrieval_params[AI4RAGParamNames.RETRIEVAL_METHOD]
    number_of_chunks = retrieval_params[AI4RAGParamNames.NUMBER_OF_CHUNKS]

    context_template_text = foundation_model.context_template_text
    system_message_text = foundation_model.system_message_text
    user_message_text = foundation_model.user_message_text

    rag_params = {
        "template_id": template_id,
        "retrieval": retrieval_params,
        "generation": {
            "model_id": foundation_model.model_id,
            "temperature": foundation_model.params.temperature,
            "max_completion_tokens": foundation_model.params.max_completion_tokens,
            "context_template_text": context_template_text,
            "user_message_text": user_message_text,
            "system_message_text": system_message_text,
            "language": foundation_model.language.to_dict(),
        },
    }
    if search_mode == "graph" and self.graph_retrieval_config:
        rag_params["retrieval"].update(self.graph_retrieval_config.to_search_kwargs())

    logger.info("Using retrieval and generation params: %s", rag_params)

    result_score = self.results.evaluation_explored_or_cached(
        indexing_params=indexing_params, rag_params=rag_params
    )
    if result_score is not None:
        return result_score

    pattern_name = self._create_pattern_name()
    logger.info("Using name '%s' for the currently evaluated pattern.", pattern_name)

    collection_name = self._get_reusable_collection_name(indexing_params=indexing_params)

    vector_store_config = self.vector_store_config
    if isinstance(vector_store_config, PGVectorConfig):
        # Size the connection pool to this run's actual query concurrency so a
        # fully concurrent query_rag() call never queues for a slot (see
        # PGVectorConfig.pool_max_size). Never shrink below a user-set ceiling:
        # a caller who deliberately raised pool_max_size (e.g. to share the store
        # with other concurrent work) must keep that headroom, so take the larger
        # of the configured size and this run's inference concurrency.
        pool_max_size = max(vector_store_config.pool_max_size, self.inference_max_threads)
        if pool_max_size != vector_store_config.pool_max_size:
            logger.info(
                "Raising PGVector pool_max_size from %d to %d to match inference_max_threads (%d).",
                vector_store_config.pool_max_size,
                pool_max_size,
                self.inference_max_threads,
            )
        else:
            logger.info(
                "Keeping configured PGVector pool_max_size %d (>= inference_max_threads %d).",
                vector_store_config.pool_max_size,
                self.inference_max_threads,
            )
        vector_store_config = replace(vector_store_config, pool_max_size=pool_max_size)

    try:
        vector_store = get_vector_store(
            embedding_model=embedding_model,
            collection_name=collection_name,
            config=vector_store_config,
            foundation_model=foundation_model if search_mode == "graph" else None,
            kg_extraction_config=self.kg_extraction_config if search_mode == "graph" else None,
        )
    except Exception as exc:
        raise VectorStoreInitializationError(
            exc,
            embedding_model_id=embedding_model.model_id,
            vector_store_provider_id=self.vector_store_config.provider,
        ) from exc

    collection_name = vector_store.collection_name

    with vector_store:
        if not self._collection_exists(collection_name=collection_name):
            chunking_method = chunking_params.get(AI4RAGParamNames.CHUNKING_METHOD)
            chunk_size = chunking_params.get(AI4RAGParamNames.CHUNK_SIZE)
            chunk_overlap = chunking_params.get(AI4RAGParamNames.CHUNK_OVERLAP)

            if chunking_method == "hybrid":
                chunker = DoclingChunker(max_tokens=chunk_size)
            else:
                chunker = LangChainChunker(
                    method=chunking_method,
                    chunk_size=chunk_size,
                    chunk_overlap=chunk_overlap,
                )
            chunked_documents = chunker.split_documents(self.documents)

            if self.event_handler:
                self.event_handler.on_status_change(
                    level=LogLevel.INFO,
                    message=(
                        f"Chunking documents using the {chunking_method} method, chunk_size: {chunk_size} "
                        f"and chunk_overlap: {chunk_overlap}."
                    ),
                    step=ExperimentStep.CHUNKING,
                )

            self.event_handler.on_status_change(
                level=LogLevel.INFO,
                message=(
                    f"Embedding chunks using the {embedding_model.model_id} model. "
                    f"Building index: {collection_name}."
                ),
                step=ExperimentStep.EMBEDDING,
            )

            try:
                vector_store.add_documents(chunked_documents)
            except Exception as exc:
                raise IndexingError(exc, collection_name, embedding_model.model_id) from exc

        else:
            self.event_handler.on_status_change(
                level=LogLevel.INFO,
                message=f"Using index {collection_name}.",
                step=ExperimentStep.EMBEDDING,
            )

        logger.info("Using retriever with parameters: %s", retrieval_params)

        retriever = Retriever(
            vector_store=vector_store,
            number_of_chunks=number_of_chunks,
            method=retrieval_method,
            search_mode=search_mode,
            ranker_strategy=retrieval_params.get(AI4RAGParamNames.RANKER_STRATEGY),
            ranker_k=retrieval_params.get(AI4RAGParamNames.RANKER_K),
            ranker_alpha=retrieval_params.get(AI4RAGParamNames.RANKER_ALPHA),
            search_kwargs=(self.graph_retrieval_config.to_search_kwargs() if search_mode == "graph" else None),
        )

        rag_pattern = template_spec.template_class(
            foundation_model=foundation_model,
            retriever=retriever,
        )

        _rag_log = (
            f"Retrieval and generation using collection: '{collection_name}' and "
            f"foundation model: '{foundation_model.model_id}'."
        )
        logger.info(_rag_log)
        self.event_handler.on_status_change(
            level=LogLevel.INFO,
            message=_rag_log,
            step=ExperimentStep.GENERATION,
        )

        inference_response = query_rag(
            rag=rag_pattern,
            questions=list(self.benchmark_data.questions),
            max_threads=self.inference_max_threads,
        )

    conversations = self._conversations_from_response(inference_response)
    result_scores, evaluation_data = self._evaluate_response(
        inference_response=inference_response,
        pattern_name=pattern_name,
    )

    stop_time = time.time()
    execution_time = stop_time - start_time

    final_score = self._resolve_optimization_score(result_scores, pattern_name)

    logger.info("Calculated optimization score for '%s': %s", pattern_name, final_score)

    evaluation_result = EvaluationResult(
        pattern_name=pattern_name,
        collection=collection_name,
        indexing_params=indexing_params,
        rag_params=rag_params,
        scores=result_scores,
        execution_time=execution_time,
        final_score=final_score,
        conversations=conversations or None,
    )

    evaluation_results_json = self.results.create_evaluation_results_json(
        evaluation_data=evaluation_data,
        evaluation_result=evaluation_result,
    )

    logger.info(
        "Evaluation scores: %s",
        {el.get("question"): el.get("metrics") for el in evaluation_results_json if isinstance(el, dict)},
    )

    iteration = len(self.results) + (len(self.known_observations) if self.known_observations else 0)
    if publish_pattern:
        try:
            self._stream_finished_pattern(
                evaluation_result=evaluation_result,
                evaluation_results_json=evaluation_results_json,
                iteration=iteration,
            )
        except Exception as exc:
            raise AssetSaveError(exc) from exc

    self.results.add_evaluation(
        evaluation_data=evaluation_data,
        evaluation_result=evaluation_result,
    )

    return final_score

search

search(**kwargs) -> None

Prepare and execute experiment to find the best RAG parameters.

Result of the search() can be reviewed via self.results as this object stores results of each evaluation or via self.event_handler with custom implementation.

Source code in ai4rag/core/experiment/experiment.py
def search(self, **kwargs) -> None:
    """
    Prepare and execute experiment to find the best RAG parameters.

    Result of the search() can be reviewed via self.results as this object
    stores results of each evaluation or via self.event_handler with custom
    implementation.
    """

    logger.info("Starting RAG optimization process...")

    optimizer_class: type[BaseOptimizer] = kwargs.get("optimizer", GAMOptimizer)
    is_gam_optimizer = issubclass(optimizer_class, GAMOptimizer)

    def objective_function(space: RAGParamsType) -> float | None:
        """Function passed to the optimizer."""
        self._optimization_phase = getattr(optimizer, "current_phase", None)
        try:
            return self.run_single_evaluation(
                space,
                publish_pattern=not is_gam_optimizer or self._optimization_phase != "warm_start",
            )
        except AI4RAGError as err:
            msg = self._exception_handler.handle_exception(err)
            raise FailedIterationError(msg) from err
        finally:
            self._optimization_phase = None

    # MPS - models pre-selection based on sample evaluation.
    # Run if there are more than 3 foundation models or more than 2 embedding models.
    foundation_models = list(self.search_space[AI4RAGParamNames.FOUNDATION_MODEL].values)
    embedding_models = list(self.search_space[AI4RAGParamNames.EMBEDDING_MODEL].values)

    if (
        len(embedding_models) > self.n_mps_embedding_models or len(foundation_models) > self.n_mps_foundation_models
    ) and not kwargs.get("skip_mps", False):
        selected_models = self.run_pre_selection(
            foundation_models=foundation_models, embedding_models=embedding_models
        )
        self.search_space[AI4RAGParamNames.FOUNDATION_MODEL] = Parameter(
            name=AI4RAGParamNames.FOUNDATION_MODEL,
            param_type="C",
            values=selected_models["foundation_models"],
        )
        self.search_space[AI4RAGParamNames.EMBEDDING_MODEL] = Parameter(
            name=AI4RAGParamNames.EMBEDDING_MODEL,
            param_type="C",
            values=selected_models["embedding_models"],
        )

    optimizer_kwargs = {}
    if self.known_observations is not None and is_gam_optimizer:
        optimizer_kwargs["known_observations"] = self.known_observations

    optimizer = optimizer_class(
        objective_function=objective_function,
        search_space=self.search_space,
        settings=self.optimizer_settings,
        **optimizer_kwargs,
    )
    logger.info(
        "Using optimizer: %s with optimizer settings: %s",
        optimizer_class.__name__,
        self.optimizer_settings.to_dict(),
    )

    try:
        _ = optimizer.search()
    except OptimizationError as err:
        final_error_msg = self._exception_handler.get_final_error_msg()
        raise RAGExperimentError(final_error_msg) from err

    if is_gam_optimizer:
        self._publish_best_warm_start_pattern()

    self.event_handler.on_status_change(
        level=LogLevel.INFO,
        message="Experiment optimization process finished.",
    )

Experiment Results

results

Classes

EvaluationResult dataclass

EvaluationResult(
    pattern_name: str,
    collection: str,
    indexing_params: dict[str, Any],
    rag_params: dict[str, Any],
    scores: EvaluationMetricsResult,
    execution_time: float,
    final_score: float,
    conversations: list[dict[str, Any]] | None = None,
)

Class holding data from single evaluation.

Parameters:

  • pattern_name (str) –

    Name of the RAG pattern created. Next names should be i.e. PatternA, PatternB, ..., PatternZ, PatternAA

  • collection (list[str]) –

    Name of the indexes/collections in the VectorStores for single configuration.

  • indexing_params (dict[str, Any] | None) –

    Subspace of hyperparameters used during indexing stage in the Retrieval Augmented Generation.

  • rag_params (dict[str, Any]) –

    Subspace of hyperparameters used during inference stage in the Retrieval Augmented Generation.

  • scores (EvaluationMetricsResult) –

    Evaluation output with aggregate metrics and per-question scores.

  • execution_time (float) –

    Time in seconds how long did experiment take to run.

  • final_score (float) –

    Single score calculated for optimization process as the value to be minimized or maximized.

  • conversations (list[dict[str, Any]] | None, default: None ) –

    Agent trajectories for the evaluated cases. Full trajectories are kept once per pattern and are written to a dedicated artifact rather than embedded in every evaluation record.

Methods:
to_dict
to_dict() -> dict[str, Any]

Cast instance to dict

Source code in ai4rag/core/experiment/results.py
def to_dict(self) -> dict[str, Any]:
    """Cast instance to dict"""
    return asdict(self)

ExperimentResults

ExperimentResults()

Class holding information about each run in the optimization process.

Attributes:

  • evaluations (list[EvaluationResult]) –

    Instances of each evaluation.

  • evaluation_data (list[list[EvaluationData]]) –

    Data for evaluation, questions, ground truth answers and context references.

Methods:

  • add_evaluation –

    Extend space with another evaluation step from the objective function.

  • sorted_ –

    Return sorted version of the evaluations.

  • get_best_evaluations –

    Return k-best evaluations. If k is None all evaluations are returned sorted based on optimization metric score.

Source code in ai4rag/core/experiment/results.py
def __init__(self):
    self.evaluations: list[EvaluationResult] = []
    self.evaluation_data: list[list[EvaluationData]] = []
Attributes
collection_names property
collection_names: list[str]

Return list of collection_names from the HPO run.

Returns:

  • list[str] –

    Collections/indexes names created during all HPO runs.

Methods:
add_evaluation
add_evaluation(evaluation_data: list[EvaluationData], evaluation_result: EvaluationResult) -> None

This method is responsible for adding searched point in the parameters space.

Parameters:

  • evaluation_data (list[EvaluationData]) –

    Data used to make evaluation. It is a list, as one instance of EvaluationData is per 1 question and answer.

  • evaluation_result (EvaluationResult) –

    Instance containing information about single HPO run.

Source code in ai4rag/core/experiment/results.py
def add_evaluation(
    self,
    evaluation_data: list[EvaluationData],
    evaluation_result: EvaluationResult,
) -> None:
    """
    This method is responsible for adding searched point in the parameters space.

    Parameters
    ----------
    evaluation_data : list[EvaluationData]
        Data used to make evaluation. It is a list, as one instance
        of EvaluationData is per 1 question and answer.

    evaluation_result : EvaluationResult
        Instance containing information about single HPO run.
    """
    logger.info("Adding evaluation results and evaluation data to training results.")
    self.evaluations.append(evaluation_result)
    self.evaluation_data.append(evaluation_data)
evaluation_explored_or_cached
evaluation_explored_or_cached(indexing_params: dict[str, Any], rag_params: dict[str, Any]) -> float | None

This method checks if an evaluation to certain params already exists.

Parameters:

  • indexing_params (dict[str, Any]) –

    Dictionary containing keys and values that are compared with previously used ones, to establish if the evaluation already done.

  • rag_params (dict[str, Any]) –

    Dictionary containing keys and values that are compared with previously used ones, to establish if the evaluation already done.

Returns:

  • float | None –

    The final score of the evaluation if it exists, otherwise None.

Source code in ai4rag/core/experiment/results.py
def evaluation_explored_or_cached(
    self, indexing_params: dict[str, Any], rag_params: dict[str, Any]
) -> float | None:
    """
    This method checks if an evaluation to certain params already exists.

    Parameters
    ----------
    indexing_params : dict[str, Any]
        Dictionary containing keys and values that are compared with
        previously used ones, to establish if the evaluation already done.

    rag_params : dict[str, Any]
        Dictionary containing keys and values that are compared with
        previously used ones, to establish if the evaluation already done.

    Returns
    -------
    float | None
        The final score of the evaluation if it exists, otherwise None.
    """
    for evaluation in self.evaluations:
        if evaluation.indexing_params == indexing_params and evaluation.rag_params == rag_params:
            return evaluation.final_score
    return None
get_existing_collection
get_existing_collection(indexing_params: dict[str, Any]) -> str | None

This method checks if a collection to certain params already exists. If so, returns the collection name.

Parameters:

  • indexing_params (dict[str, Any]) –

    Dictionary containing keys and values that are compared with previously used ones, to establish if the index already created.

Returns:

  • str | None –

    The collection name of the index if it exists, otherwise None.

Source code in ai4rag/core/experiment/results.py
def get_existing_collection(self, indexing_params: dict[str, Any]) -> str | None:
    """
    This method checks if a collection to certain params already exists.
    If so, returns the collection name.

    Parameters
    ----------
    indexing_params : dict[str, Any]
        Dictionary containing keys and values that are compared with
        previously used ones, to establish if the index already created.

    Returns
    -------
    str | None
        The collection name of the index if it exists, otherwise None.
    """
    for evaluation in self.evaluations:
        if evaluation.indexing_params == indexing_params:
            return evaluation.collection
    return None
get_best_evaluations
get_best_evaluations(k: int | None = None) -> tuple[EvaluationResult]

Return k-best evaluations.

Parameters:

  • k (int, default: None ) –

    how many evaluations to return

Returns:

Source code in ai4rag/core/experiment/results.py
def get_best_evaluations(self, k: int | None = None) -> tuple[EvaluationResult]:
    """
    Return k-best evaluations.

    Parameters
    ----------
    k : int
        how many evaluations to return

    Returns
    -------
    tuple[EvaluationResult]
        the best evaluations from the AutoRAG experiment
    """
    evaluations = list(self.evaluations)
    sorted_evs = tuple(
        sorted(
            evaluations,
            key=lambda x: float("-inf") if x.final_score is None else x.final_score,
            reverse=True,
        )
    )
    return sorted_evs[:k]
create_evaluation_results_json staticmethod
create_evaluation_results_json(
    evaluation_data: list[EvaluationData], evaluation_result: EvaluationResult
) -> list[dict[str, Any]]

Create json made of evaluation results with proper data.

Example file content::

[
    {
        "question_id": "0",
        "answer": "<model's answer>",
        "answer_contexts": [
            {"text": "<content1_text>", "document_key": "document_1.pdf"},
        ],
        "metrics": [
            {"name": "answer_correctness", "evaluator": "unitxt", "score": 0.79},
            {"name": "overall_score", "evaluator": "custom", "score": 0.79},
        ]
    },
]

Parameters:

  • evaluation_data (list[EvaluationData]) –

    Evaluation data to be used for the json generation.

  • evaluation_result (EvaluationResult) –

    Evaluation result to be used for the json generation.

Returns:

  • list[dict[str, Any]] –

    json-like object with evaluation results.

Source code in ai4rag/core/experiment/results.py
@staticmethod
def create_evaluation_results_json(
    evaluation_data: list[EvaluationData],
    evaluation_result: EvaluationResult,
) -> list[dict[str, Any]]:
    """
    Create json made of evaluation results with proper data.

    Example file content::

        [
            {
                "question_id": "0",
                "answer": "<model's answer>",
                "answer_contexts": [
                    {"text": "<content1_text>", "document_key": "document_1.pdf"},
                ],
                "metrics": [
                    {"name": "answer_correctness", "evaluator": "unitxt", "score": 0.79},
                    {"name": "overall_score", "evaluator": "custom", "score": 0.79},
                ]
            },
        ]

    Parameters
    ----------
    evaluation_data : list[EvaluationData]
        Evaluation data to be used for the json generation.

    evaluation_result : EvaluationResult
        Evaluation result to be used for the json generation.

    Returns
    -------
    list[dict[str, Any]]
        json-like object with evaluation results.
    """
    scores_by_qid = {q["question_id"]: q["metrics"] for q in evaluation_result.scores["question_scores"]}

    conversations = evaluation_result.conversations
    conversation_ids: list[str | None] = (
        [conversation.get("conversation_id") for conversation in conversations]
        if conversations is not None and len(conversations) == len(evaluation_data)
        else []
    )
    if conversations is not None and not conversation_ids:
        logger.warning(
            "Conversation count (%s) does not match evaluation data count (%s); omitting conversation IDs.",
            len(conversations),
            len(evaluation_data),
        )

    data = []
    for index, ev in enumerate(evaluation_data):
        record = {
            "question": ev.question,
            "correct_answers": ev.ground_truths,
            "answer": ev.answer,
            "answer_contexts": [
                {"text": text, "document_key": doc_id} for text, doc_id in zip(ev.contexts, ev.context_ids)
            ],
            "metrics": [
                {"name": m["name"], "evaluator": m["evaluator"], "score": m["value"]}
                for m in scores_by_qid.get(ev.question_id, [])
            ],
        }
        if conversation_ids and conversation_ids[index] is not None:
            record["conversation_id"] = conversation_ids[index]
        data.append(record)

    return data