Optimizers¶

Optimizers are the algorithms that explore your search space to find the best RAG configuration. Instead of testing every possible combination (which could be millions), optimizers use intelligent strategies to converge on high-performing configurations with far fewer evaluations.

What Optimizers Do¶

Given a search space with parameters like:

• chunk_size: [200, 400, 800, 1000]
• chunk_overlap: [0, 50, 100, 200]
• retrieval_method: ["simple", "window"]
• number_of_chunks: [3, 5, 7, 10]

There are 4 × 4 × 2 × 4 = 128 possible combinations (before validation rules).

Testing all 128 would be expensive and time-consuming. Optimizers find near-optimal configurations by testing only a fraction of the space (e.g., 10-20 evaluations).

Available Optimizers¶

ai4rag provides two optimizer implementations:

GAM Optimizer (Recommended)¶

The GAM Optimizer uses Generalized Additive Models to predict which configurations are likely to perform well, then evaluates those promising candidates.

How It Works¶

The optimization process has two phases:

Phase 1: Warm-Start Exploration

• Select n_random_nodes configurations from the search space, ordered according to warm_start_strategy:
  • "random" (default) — shuffle the candidates and take the first n_random_nodes as-is; no coverage guarantee
  • "greedy" — greedily pick configurations so every discrete parameter value (categorical or numeric) appears at least twice, auto-adjusting n_random_nodes upward (with a logged notice) when it is too small for full coverage
  • "balanced" — round-robin across the unique value combinations of fields_to_balance, guaranteeing every combination appears at least once, with other discrete values covered as a secondary goal
• Evaluate each using the objective function (your chosen metric)
• Build an initial dataset of (configuration, score) pairs

Phase 2: Model-Guided Search

• Train a LinearGAM model on the evaluated configurations, using factor terms for categorical parameters and spline terms for numeric ones (a categorical column falls back to a spline term if the warm-start sample didn't observe every one of its values)
• Use the model to predict scores for all remaining (unevaluated) configurations
• Select the top evals_per_trial configurations with highest predicted scores
• Evaluate those configurations
• Repeat until max_evals total evaluations are reached, or max_iterations published patterns have been produced

This approach balances exploration (warm-start phase) with exploitation (model-guided phase).

Configuration¶

from ai4rag.core.hpo.gam_opt import GAMOptSettings

optimizer_settings = GAMOptSettings(
    max_evals=20,                  # Total number of configurations to evaluate
    n_random_nodes=4,              # Number of warm-start evaluations before using the model
    evals_per_trial=1,             # Number of configurations to evaluate per iteration (default: 1)
    warm_start_strategy="random",  # "random" (default), "greedy", or "balanced"
    random_state=64                # Random seed for reproducibility (default: 64)
)

Parameters Explained¶

max_evals¶

What it controls: Total number of RAG configurations to evaluate.

Typical values:

• 10-15: Quick exploration (small search spaces)
• 20-30: Balanced optimization (medium search spaces)
• 50+: Thorough search (large, complex search spaces)

Trade-offs:

• Higher = better results, but slower and more expensive
• Lower = faster, but may miss optimal configurations

n_random_nodes¶

What it controls: Number of configurations to evaluate in the warm-start phase, before training the GAM model. How they are selected is controlled by warm_start_strategy.

Default: 4

Why it matters: The warm-start phase builds the initial (configuration, score) dataset the GAM model trains on. With warm_start_strategy="greedy" or "balanced", it also determines categorical coverage — the more diverse the sample, the less likely the model is to be biased toward over-represented parameter values.

Typical values:

• 3-4: Small search spaces, quick convergence
• 5-8: Medium search spaces, balanced exploration
• 10+: Large search spaces, high diversity needed

Trade-offs:

• Higher = more exploration, better model training data, but fewer model-guided evaluations
• Lower = more model-guided iterations, but risk of poor model predictions if initial samples aren't diverse

warm_start_strategy¶

What it controls: How the n_random_nodes warm-start configurations are selected.

Default: "random"

Options:

• "random" — shuffle the candidate list and take the first n_random_nodes. No coverage guarantee; fastest and simplest.
• "greedy" — greedily select configurations so every discrete parameter value (categorical or numeric) appears at least twice, maximizing coverage before falling back to model-guided search.
• "balanced" — round-robin across the unique value combinations of fields_to_balance (e.g. ensure every (foundation_model, embedding_model) pair is tried at least once), with other discrete values covered as a secondary objective. Requires fields_to_balance to be set.

Why it matters: "greedy" and "balanced" produce a more representative warm-start sample for the GAM model — especially useful when some parameter values dominate the search space or when a specific set of fields (e.g. model choices) must each get a fair trial.

fields_to_balance¶

What it controls: The parameter names balanced by round-robin when warm_start_strategy="balanced".

Default: None

Why it matters: Each unique combination of values across these fields is guaranteed at least one warm-start evaluation before GAM training begins. Required (and must be non-empty) when warm_start_strategy="balanced"; ignored otherwise.

GAMOptSettings(
    max_evals=20,
    warm_start_strategy="balanced",
    fields_to_balance=["foundation_model", "embedding_model"],
)

max_iterations¶

What it controls: The maximum number of evaluated patterns retained as results and published to the event handler.

Default: None (falls back to the effective max_evals)

Why it matters: max_evals bounds how many objective-function calls are made; max_iterations separately bounds how many of those evaluations become published Pattern* results. This lets you spend a larger evaluation budget on warm-start coverage or GAM refinement without flooding your event handler with every intermediate evaluation. It cannot exceed max_evals, and is clamped to the number of available search-space combinations.

evals_per_trial¶

What it controls: How many configurations to evaluate in each model-guided iteration.

Default: 1

Why it matters: Evaluating multiple candidates per iteration speeds up optimization but reduces the number of times the model is retrained.

Typical values:

• 1: Maximum model refinement (model is retrained after every evaluation)
• 2-3: Faster convergence with slight trade-off in model accuracy

Trade-offs:

• Higher = fewer iterations, faster wall-clock time, but less adaptive model
• Lower = more iterations, slower, but model adapts more frequently

random_state¶

What it controls: Random seed for reproducibility.

Default: 64

Why it matters: Using the same random state across runs ensures identical random exploration phases, making results reproducible.

Usage Example¶

from ai4rag.core.experiment.experiment import AI4RAGExperiment
from ai4rag.core.hpo.gam_opt import GAMOptSettings
from ai4rag.search_space.src.search_space import AI4RAGSearchSpace
from ai4rag.search_space.src.parameter import Parameter

# Define search space
search_space = AI4RAGSearchSpace(
    params=[
        # ... model parameters
        Parameter(name="chunk_size", param_type="C", values=[200, 400, 800, 1000]),
        Parameter(name="chunk_overlap", param_type="C", values=[0, 50, 100, 200]),
        Parameter(name="number_of_chunks", param_type="C", values=[3, 5, 7, 10]),
    ]
)

# Conservative exploration: more random sampling
optimizer_settings = GAMOptSettings(
    max_evals=20,
    n_random_nodes=8  # 40% of evaluations are random
)

experiment = AI4RAGExperiment(
    # ... other parameters
    search_space=search_space,
    optimizer_settings=optimizer_settings,
)

best_pattern = experiment.search()

Resuming from Known Observations¶

If you've already run experiments and want to continue optimization with new settings or a refined search space:

from ai4rag.core.hpo.gam_opt import GAMOptimizer, GAMOptSettings

# Previous results (from earlier experiments)
known_observations = [
    {
        "chunk_size": 512,
        "chunk_overlap": 64,
        "number_of_chunks": 5,
        "score": 0.72
    },
    {
        "chunk_size": 1024,
        "chunk_overlap": 128,
        "number_of_chunks": 7,
        "score": 0.68
    },
    # ... more observations
]

optimizer = GAMOptimizer(
    objective_function=my_objective_function,
    search_space=search_space,
    settings=GAMOptSettings(max_evals=20, n_random_nodes=4),
    known_observations=known_observations  # Warm-start from previous runs
)

best_result = optimizer.search()

Benefits of warm-starting:

• Skip re-evaluating configurations you've already tested
• Build on previous optimization runs
• Useful when expanding search space or changing objective metric
• With warm_start_strategy="greedy" or "balanced", coverage already provided by known observations is accounted for automatically, so early slots are not wasted on redundant coverage

When to Use GAM Optimizer¶

Use GAM when:

• You have medium to large search spaces (50+ combinations)
• You want efficient optimization (minimize evaluations)
• You're willing to invest in 20+ evaluations for quality results
• You need a balance between exploration and exploitation

Avoid GAM when:

• Search space is tiny (<10 combinations) - use Random Optimizer instead
• You want absolute certainty of finding the global optimum - exhaustive search is better

Random Optimizer¶

The Random Optimizer is a simple baseline that randomly shuffles all possible combinations and evaluates them up to max_evals.

How It Works¶

1. Generate all valid combinations from the search space (after applying validation rules)
2. Shuffle the list randomly
3. Evaluate the first max_evals configurations
4. Return the configuration with the highest score

No model training, no prediction - just random sampling.

Configuration¶

from ai4rag.core.hpo.random_opt import RandomOptSettings

optimizer_settings = RandomOptSettings(
    max_evals=10,       # Number of configurations to evaluate (required)
    random_state=64     # Random seed for reproducibility (default: 64)
)

Usage Example¶

from ai4rag.core.experiment.experiment import AI4RAGExperiment
from ai4rag.core.hpo.random_opt import RandomOptSettings
from ai4rag.search_space.src.search_space import AI4RAGSearchSpace

search_space = AI4RAGSearchSpace(
    params=[
        # ... parameters
    ]
)

optimizer_settings = RandomOptSettings(max_evals=10)

experiment = AI4RAGExperiment(
    # ... other parameters
    search_space=search_space,
    optimizer_settings=optimizer_settings,
)

best_pattern = experiment.search()

When to Use Random Optimizer¶

Use Random when:

• You have a very small search space (<20 combinations)
• You want a quick baseline to compare against GAM
• You're debugging and want deterministic behavior
• You're running a sensitivity analysis across the full search space

Avoid Random when:

• Search space is large (>50 combinations) - you'll waste evaluations
• You need efficient optimization - GAM will find better results faster

Optimizer Comparison¶

Configuration Examples¶

Example 1: Conservative Exploration (More Random)¶

When you have a large, complex search space and want thorough exploration:

GAMOptSettings(
    max_evals=30,
    n_random_nodes=10  # 33% random exploration
)

Use case: First-time optimization on a new dataset, unfamiliar search space.

Example 2: Aggressive Exploitation (More Model-Guided)¶

When you have a well-understood search space and want fast convergence:

GAMOptSettings(
    max_evals=20,
    n_random_nodes=3  # 15% random exploration
)

Use case: Refinement runs, similar datasets, known parameter relationships.

Example 3: Quick Baseline (Random)¶

When you just want a rough estimate:

RandomOptSettings(
    max_evals=10
)

Use case: Initial exploration, proof-of-concept, small search spaces.

Example 4: Warm-Start from Previous Run¶

When you're expanding your search space or changing the objective metric:

import json

# Load previous results
with open("./results/experiment_results.json") as f:
    previous_results = json.load(f)

known_observations = [
    {
        "chunk_size": result["settings"]["chunking"]["chunk_size"],
        "chunk_overlap": result["settings"]["chunking"]["chunk_overlap"],
        "number_of_chunks": result["settings"]["retrieval"]["number_of_chunks"],
        "score": result["final_score"]
    }
    for result in previous_results
]

optimizer = GAMOptimizer(
    objective_function=objective_fn,
    search_space=expanded_search_space,
    settings=GAMOptSettings(max_evals=15, n_random_nodes=4),
    known_observations=known_observations
)

Use case: Iterative optimization, budget-constrained experiments.

Tips for Choosing Optimizer Settings¶

1. Start with Defaults¶

For most use cases, these defaults work well:

GAMOptSettings(
    max_evals=20,
    n_random_nodes=4
)

2. Scale with Search Space Size¶

3. Balance Exploration and Exploitation¶

Rule of thumb: n_random_nodes should be 15-30% of max_evals.

# Good balance
GAMOptSettings(max_evals=20, n_random_nodes=5)  # 25% random

# Too much exploration
GAMOptSettings(max_evals=20, n_random_nodes=15)  # 75% random - might as well use Random Optimizer

# Too little exploration
GAMOptSettings(max_evals=20, n_random_nodes=1)  # 5% random - risk of poor model training

4. Consider Evaluation Cost¶

If each evaluation is expensive (large documents, slow models):

• Lower max_evals to reduce total runtime
• Use GAM Optimizer to make every evaluation count
• Increase evals_per_trial to 2-3 to speed up wall-clock time

If evaluations are cheap:

• Higher max_evals for thorough search
• Use Random Optimizer for simple baselines

5. Use Reproducible Seeds¶

Always set random_state for reproducible experiments:

GAMOptSettings(
    max_evals=20,
    n_random_nodes=4,
    random_state=42  # Any integer; ensures reproducibility
)

Troubleshooting¶

Optimization Converges Too Quickly¶

Symptom: Best configuration found in first few evaluations, no improvement afterward.

Cause: Search space is too small or too constrained.

Actions:

• Expand your search space with more parameter values
• Check validation rules - are they too restrictive?
• Verify max_evals is appropriate for your search space size

All Evaluations Fail¶

Symptom: OptimizationError: Number of evaluations has reached limit. All iterations have failed.

Cause: Objective function is raising exceptions for every configuration.

Actions:

• Check logs for error messages during evaluation
• Verify models are accessible (Llama Stack connection, API keys)
• Test a single configuration manually to isolate the issue
• Ensure benchmark data and documents are properly formatted

Random Phase Takes Forever¶

Symptom: Optimizer gets stuck during random exploration.

Cause: n_random_nodes is set too high relative to evaluation speed.

Actions:

• Reduce n_random_nodes to speed up convergence
• Optimize evaluation pipeline (smaller models, fewer chunks, smaller benchmark)

Model-Guided Phase Doesn't Improve Results¶

Symptom: Scores during GAM phase are no better than random phase.

Cause: GAM model can't find meaningful patterns in the search space.

Actions:

• Increase n_random_nodes to give the model more training data
• Check if your search space has too few parameters or values
• Verify that your objective metric is sensitive to parameter changes

Related Topics¶

• Search Space: Define parameters for optimization
• Evaluation: Understanding objective metrics
• Event Handlers: Monitor optimization progress
• Quick Start: Complete optimization example

Summary¶

Optimizers in ai4rag:

• GAM Optimizer: Recommended for most use cases, balances exploration and exploitation
• Random Optimizer: Simple baseline, best for small search spaces
• Key settings: max_evals (total evaluations), n_random_nodes (warm-start evaluations), warm_start_strategy (random/greedy/balanced), max_iterations (published results cap)
• Resuming: Reuse previous results with known_observations
• Trade-offs: More evaluations = better results but higher cost

Start with the default GAM settings (max_evals=20, n_random_nodes=4) and adjust based on your search space size, evaluation cost, and optimization goals.