.. _examples.extensions.lm_eval.lm_eval_cli:

lm-evaluation-harness CLI usage
===============================

The recommended way to run benchmarks is through CLI.
In your python environment with 'ibm-generative-ai[lm-eval]' installed:

Example::

    python -m genai.extensions.lm_eval \
          --model="ibm_genai" \
          --model_args="model_id=tiiuae/falcon-40b,temperature=0" \
          --task="hellaswag" \
          --num_fewshot=10 \
          --output_path="falcon-40b_hellaswag.json"

.. literalinclude:: ../../../examples/extensions/lm_eval/lm_eval_cli.py
	:language: python
	:caption: See `lm_eval_cli.py <https://github.com/IBM/ibm-generative-ai/blob/v3.0.0/examples/extensions/lm_eval/lm_eval_cli.py>`_ on GitHub.
	:lines: 17-