Experiment API¶
AI4RAGExperiment ¶
AI4RAGExperiment(
documents: list[DoclingDocument],
benchmark_data: DataFrame,
search_space: AI4RAGSearchSpace,
optimizer_settings: OptimizerSettings,
event_handler: BaseEventHandler,
vector_store_config: BaseVectorStoreConfig,
optimization_metric: RAGMetric = Metrics.OVERALL_SCORE,
**kwargs
)
Class responsible for conducting AutoRAG experiment, that consists of finding the best hyperparameters for several steps/stages.
AI4RAGExperiment is essentially an orchestrator for the RAG Patterns hyperparameters optimization for the desired metric. It requires from user to provide fully defined search space on which the experiment will be executed.
AI4RAG uses 'BaseRAGTemplate' inheriting classes as definitions on how to build and utilize RAG Pattern with the given search space nodes.
Template selection uses the categorical template_id search parameter.
Parameters:
-
documents(list[DoclingDocument]) –List of parsed docling documents to embed in vector db and use as context in RAG.
-
benchmark_data(DataFrame | BenchmarkData) –Structure with 3 columns: 'question', 'correct_answers' and 'correct_answer_document_keys'.
-
search_space(AI4RAGSearchSpace) –Grid of parameters used during hyperparameter optimization.
-
optimizer_settings(OptimizerSettings) –Settings for the optimizer to be used during the experiment.
-
vector_store_config(BaseVectorStoreConfig) –Connection config for the vector store backend. Its type (via
config.provider) determines which vector store implementation is used for indexing and retrieval. -
event_handler(BaseEventHandler) –Instance satisfying BaseEventHandler's interface to stream pattern evaluation results and intermediate status updates. EventHandler is an entrypoint to configure custom logging and assets handling.
-
optimization_metric(RAGMetric, default:Metrics.OVERALL_SCORE) –Metric used for calculating the final score that drives optimization. Must be a
RAGMetricinstance selected from :class:Metrics.
Other Parameters:
-
metrics(Sequence[RAGMetric]) –Metrics evaluated during the AutoRAG experiment, each a
RAGMetricinstance selected from :class:Metrics. Not all of these metrics are used to calculate the final score, but they are included in the evaluation results. When omitted, defaults are derived from the configured evaluators. -
evaluators(list[BaseEvaluator] | None) –Evaluator instances used to score RAG patterns during optimization. When
None, defaults to[UnitxtEvaluator()]. To enable LLM-as-a-Judge evaluation, pass both aUnitxtEvaluatorand aLLMaJEvaluatorconfigured with a judge model. -
n_mps_foundation_models(int) –Amount of foundation models to be further used in experiment post pre-selection.
-
n_mps_embedding_models(int) –Amount of embedding models to be further used in experiment post pre-selection.
-
inference_max_threads(int) –Defines the number of threads to use during generation model inference.
Attributes:
-
results(ExperimentResults) –Instance holding information about each iteration during the experiment. It consists of statuses, RAG pattern objects, scores and settings.
Source code in ai4rag/core/experiment/experiment.py
Attributes¶
optimization_metric property writable ¶
Get optimization metrics used for the experiment.
Methods:¶
run_pre_selection ¶
run_pre_selection(
foundation_models: list[BaseFoundationModel],
embedding_models: list[BaseEmbeddingModel],
n_records: int = 5,
random_seed: int = 17,
) -> dict[str, list[BaseEmbeddingModel | BaseFoundationModel]]
Run models pre-selection using ModelsPreSelector and sample of the data.
Parameters:
-
embedding_models(list[BaseEmbeddingModel]) –Embedding models to be considered during pre-selection process.
-
foundation_models(list[BaseFoundationModel]) –Foundation models to be evaluated during pre-selection process.
-
n_records(int, default:5) –Amount of records that should be used during models pre-selection.
-
random_seed(int, default:17) –Random seed value used for sampling benchmark data records.
Returns:
-
dict[str, list[BaseFoundationModel | EmbeddingModel]]–Best embedding models and foundation models found in pre-selection.
Source code in ai4rag/core/experiment/experiment.py
run_single_evaluation ¶
Evaluate a single RAG configuration and return its score using provided documents.
Parameters:
-
rag_params(RAGParamsType) –A dictionary containing rag parameters as keys and their values.
-
publish_pattern(bool, default:True) –Whether to send the evaluated pattern to the event handler immediately. GAM optimization suppresses this only for warm-start candidates; GAM candidates are published as they complete.
Returns:
-
float–A single evaluation score obtained by the executed rag pattern.
Source code in ai4rag/core/experiment/experiment.py
413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 | |
search ¶
Prepare and execute experiment to find the best RAG parameters.
Result of the search() can be reviewed via self.results as this object stores results of each evaluation or via self.event_handler with custom implementation.
Source code in ai4rag/core/experiment/experiment.py
743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 | |
Experiment Results¶
results ¶
Classes¶
EvaluationResult dataclass ¶
EvaluationResult(
pattern_name: str,
collection: str,
indexing_params: dict[str, Any],
rag_params: dict[str, Any],
scores: EvaluationMetricsResult,
execution_time: float,
final_score: float,
conversations: list[dict[str, Any]] | None = None,
)
Class holding data from single evaluation.
Parameters:
-
pattern_name(str) –Name of the RAG pattern created. Next names should be i.e. PatternA, PatternB, ..., PatternZ, PatternAA
-
collection(list[str]) –Name of the indexes/collections in the VectorStores for single configuration.
-
indexing_params(dict[str, Any] | None) –Subspace of hyperparameters used during indexing stage in the Retrieval Augmented Generation.
-
rag_params(dict[str, Any]) –Subspace of hyperparameters used during inference stage in the Retrieval Augmented Generation.
-
scores(EvaluationMetricsResult) –Evaluation output with aggregate metrics and per-question scores.
-
execution_time(float) –Time in seconds how long did experiment take to run.
-
final_score(float) –Single score calculated for optimization process as the value to be minimized or maximized.
-
conversations(list[dict[str, Any]] | None, default:None) –Agent trajectories for the evaluated cases. Full trajectories are kept once per pattern and are written to a dedicated artifact rather than embedded in every evaluation record.
ExperimentResults ¶
Class holding information about each run in the optimization process.
Attributes:
-
evaluations(list[EvaluationResult]) –Instances of each evaluation.
-
evaluation_data(list[list[EvaluationData]]) –Data for evaluation, questions, ground truth answers and context references.
Methods:
-
add_evaluation–Extend space with another evaluation step from the objective function.
-
sorted_–Return sorted version of the evaluations.
-
get_best_evaluations–Return k-best evaluations. If k is None all evaluations are returned sorted based on optimization metric score.
Source code in ai4rag/core/experiment/results.py
Attributes¶
collection_names property ¶
Return list of collection_names from the HPO run.
Returns:
-
list[str]–Collections/indexes names created during all HPO runs.
Methods:¶
add_evaluation ¶
This method is responsible for adding searched point in the parameters space.
Parameters:
-
evaluation_data(list[EvaluationData]) –Data used to make evaluation. It is a list, as one instance of EvaluationData is per 1 question and answer.
-
evaluation_result(EvaluationResult) –Instance containing information about single HPO run.
Source code in ai4rag/core/experiment/results.py
evaluation_explored_or_cached ¶
evaluation_explored_or_cached(indexing_params: dict[str, Any], rag_params: dict[str, Any]) -> float | None
This method checks if an evaluation to certain params already exists.
Parameters:
-
indexing_params(dict[str, Any]) –Dictionary containing keys and values that are compared with previously used ones, to establish if the evaluation already done.
-
rag_params(dict[str, Any]) –Dictionary containing keys and values that are compared with previously used ones, to establish if the evaluation already done.
Returns:
-
float | None–The final score of the evaluation if it exists, otherwise None.
Source code in ai4rag/core/experiment/results.py
get_existing_collection ¶
This method checks if a collection to certain params already exists. If so, returns the collection name.
Parameters:
-
indexing_params(dict[str, Any]) –Dictionary containing keys and values that are compared with previously used ones, to establish if the index already created.
Returns:
-
str | None–The collection name of the index if it exists, otherwise None.
Source code in ai4rag/core/experiment/results.py
get_best_evaluations ¶
Return k-best evaluations.
Parameters:
-
k(int, default:None) –how many evaluations to return
Returns:
-
tuple[EvaluationResult]–the best evaluations from the AutoRAG experiment
Source code in ai4rag/core/experiment/results.py
create_evaluation_results_json staticmethod ¶
create_evaluation_results_json(
evaluation_data: list[EvaluationData], evaluation_result: EvaluationResult
) -> list[dict[str, Any]]
Create json made of evaluation results with proper data.
Example file content::
[
{
"question_id": "0",
"answer": "<model's answer>",
"answer_contexts": [
{"text": "<content1_text>", "document_key": "document_1.pdf"},
],
"metrics": [
{"name": "answer_correctness", "evaluator": "unitxt", "score": 0.79},
{"name": "overall_score", "evaluator": "custom", "score": 0.79},
]
},
]
Parameters:
-
evaluation_data(list[EvaluationData]) –Evaluation data to be used for the json generation.
-
evaluation_result(EvaluationResult) –Evaluation result to be used for the json generation.
Returns:
-
list[dict[str, Any]]–json-like object with evaluation results.