CLI Reference ============= This section provides a comprehensive reference for the ARES command-line interface (CLI). Overview -------- The ARES CLI provides commands for running evaluations, viewing available modules, and managing configurations. Main Commands ------------- evaluate ^^^^^^^^ Execute a red-teaming evaluation using a configuration file. **Syntax:** .. code-block:: bash ares evaluate [OPTIONS] **Arguments:** - ``config_file``: Path to the YAML configuration file (required) **Options:** .. list-table:: :header-rows: 1 :widths: 20 15 65 * - Option - Type - Description * - ``--limit`` - Flag - Run ARES in test mode with limited number of seeds (typically 5). Useful for quick validation. * - ``--first N`` - Integer - Use only the first N seeds from the provided source. Allows precise control over evaluation size. * - ``--dashboard`` - Flag - Launch the interactive dashboard after evaluation completes. Opens automatically in browser. * - ``-v, --verbose`` - Flag - Show detailed configuration information during execution. Displays resolved configs and parameters. **Examples:** Basic evaluation: .. code-block:: bash ares evaluate example_configs/minimal.yaml Test mode with limited seeds: .. code-block:: bash ares evaluate example_configs/minimal.yaml --limit Use first 10 seeds only: .. code-block:: bash ares evaluate example_configs/minimal.yaml --first 10 Run with dashboard: .. code-block:: bash ares evaluate example_configs/minimal.yaml --dashboard Verbose mode for debugging: .. code-block:: bash ares evaluate example_configs/minimal.yaml --verbose ares evaluate example_configs/minimal.yaml -v Combined options: .. code-block:: bash ares evaluate example_configs/minimal.yaml --first 5 --dashboard --verbose show ^^^^ Display information about available ARES modules and configurations. **Subcommands:** show modules """""""""""" List all available ARES modules (goals, strategies, evaluators, connectors). .. code-block:: bash ares show modules **Output:** Displays a comprehensive list of all registered modules in ARES. show connectors """"""""""""""" List all available connector implementations. .. code-block:: bash ares show connectors **With name filter:** .. code-block:: bash ares show connectors -n huggingface **Output:** Shows connector details including type, configuration parameters, and usage examples. show goals """""""""" List all available goal implementations. .. code-block:: bash ares show goals **With name filter:** .. code-block:: bash ares show goals -n generic_attack_goal **Output:** Displays goal types, their purposes, and configuration templates. show strategies """"""""""""""" List all available attack strategy implementations. .. code-block:: bash ares show strategies **With name filter:** .. code-block:: bash ares show strategies -n direct_requests ares show strategies -n crescendo **Output:** Shows strategy details including attack methods, parameters, and example configurations. show evals """""""""" List all available evaluator implementations. .. code-block:: bash ares show evals **With name filter:** .. code-block:: bash ares show evals -n keyword ares show evals -n harmbench_eval **Output:** Displays evaluator types, evaluation methods, and configuration examples. Common Usage Patterns ---------------------- Quick Validation ^^^^^^^^^^^^^^^^ Test your configuration with a small subset of data: .. code-block:: bash ares evaluate my_config.yaml --limit --verbose This runs with ~5 seeds and shows detailed output for debugging. Production Evaluation ^^^^^^^^^^^^^^^^^^^^^ Run a full evaluation with results dashboard: .. code-block:: bash ares evaluate my_config.yaml --dashboard Controlled Testing ^^^^^^^^^^^^^^^^^^ Test with specific number of seeds: .. code-block:: bash ares evaluate my_config.yaml --first 20 --verbose Exploring Available Options ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Before creating a configuration, explore available modules: .. code-block:: bash ares show modules ares show strategies ares show evals ares show connectors Get specific module details: .. code-block:: bash ares show strategies -n crescendo ares show connectors -n huggingface Environment Variables --------------------- ARES_HOME ^^^^^^^^^ Set the base directory for ARES file resolution. **Usage:** .. code-block:: bash export ARES_HOME=/path/to/ares/directory ares evaluate config.yaml **Behavior:** - ARES searches for configuration files relative to ``ARES_HOME`` - ``connectors.yaml`` is loaded from the same directory as the config file - If not found, searches up to ``ARES_HOME`` directory - Default: Current working directory **Example:** .. code-block:: bash # Set ARES_HOME to project root export ARES_HOME=/home/user/ares-project # Run evaluation - files resolved relative to ARES_HOME ares evaluate configs/my_eval.yaml Connector-Specific Variables ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Different connectors require specific environment variables. These should be set in a ``.env`` file or exported in your shell. **HuggingFace:** .. code-block:: bash # Login via CLI (for gated models) huggingface-cli login **WatsonX:** .. code-block:: bash export WATSONX_URL=https://your-watsonx-instance.com export WATSONX_API_KEY=your-api-key export WATSONX_PROJECT_ID=your-project-id **WatsonX AgentLab:** .. code-block:: bash export WATSONX_AGENTLAB_API_KEY=your-agent-api-key **OpenAI (via LiteLLM):** .. code-block:: bash export OPENAI_API_KEY=your-openai-key Exit Codes ---------- The ARES CLI uses standard exit codes: - ``0``: Success - ``1``: General error (configuration error, file not found, etc.) - ``2``: Invalid command or arguments Troubleshooting --------------- Configuration Not Found ^^^^^^^^^^^^^^^^^^^^^^^ **Error:** ``FileNotFoundError: config.yaml not found`` **Solution:** 1. Check file path is correct 2. Use absolute path or set ``ARES_HOME`` 3. Verify file exists: ``ls -la config.yaml`` Module Not Found ^^^^^^^^^^^^^^^^ **Error:** ``ModuleNotFoundError: No module named 'ares_plugin'`` **Solution:** 1. Install the required plugin: ``pip install ares-plugin`` 2. Verify installation: ``pip list | grep ares`` 3. Check plugin is in correct location Invalid Configuration ^^^^^^^^^^^^^^^^^^^^^ **Error:** ``KeyError: 'target'`` or ``Invalid configuration`` **Solution:** 1. Use ``--verbose`` to see full config 2. Validate YAML syntax 3. Check required fields are present 4. Use ``ares show modules`` to verify module names Connector Errors ^^^^^^^^^^^^^^^^ **Error:** ``OSError: API key not found`` **Solution:** 1. Check environment variables are set 2. Verify ``.env`` file is in correct location 3. For HuggingFace: run ``huggingface-cli login`` 4. Check API key permissions Getting Help ------------ For additional help: .. code-block:: bash ares --help ares evaluate --help ares show --help Visit the `ARES Documentation `_ for detailed guides and examples. Report issues on `GitHub `_.