Risk identification

This notebook explains how to uncover risks related to your usecase based on a given taxonomy.¶

Import libraries¶

In [1]:

from risk_atlas_nexus.blocks.inference import (
    RITSInferenceEngine,
    WMLInferenceEngine,
    OllamaInferenceEngine,
    VLLMInferenceEngine,
)
from risk_atlas_nexus.blocks.inference.params import (
    InferenceEngineCredentials,
    RITSInferenceEngineParams,
    WMLInferenceEngineParams,
    OllamaInferenceEngineParams,
    VLLMInferenceEngineParams,
)
from risk_atlas_nexus.library import RiskAtlasNexus

from risk_atlas_nexus.blocks.inference import ( RITSInferenceEngine, WMLInferenceEngine, OllamaInferenceEngine, VLLMInferenceEngine, ) from risk_atlas_nexus.blocks.inference.params import ( InferenceEngineCredentials, RITSInferenceEngineParams, WMLInferenceEngineParams, OllamaInferenceEngineParams, VLLMInferenceEngineParams, ) from risk_atlas_nexus.library import RiskAtlasNexus

/Users/dhaval/Projects/Usage-Governance/risk-atlas-nexus/src/risk_atlas_nexus/toolkit/job_utils.py:2: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)
  from tqdm.autonotebook import tqdm

Risk Atlas Nexus uses Large Language Models (LLMs) to infer risks dimensions. Therefore requires access to LLMs to inference or call the model.¶

Available Inference Engines: WML, Ollama, vLLM, RITS. Please follow the Inference APIs guide before going ahead.

Note: RITS is intended solely for internal IBM use and requires TUNNELALL VPN for access.

In [ ]:

inference_engine = WMLInferenceEngine(
    model_name_or_path="ibm/granite-20b-code-instruct",
    credentials={
        "api_key": "WML_API_KEY",
        "api_url": "WML_API_URL",
        "project_id": "WML_PROJECT_ID",
    },
    parameters=WMLInferenceEngineParams(
        max_new_tokens=1000, decoding_method="greedy", repetition_penalty=1
    ),
)

# inference_engine = OllamaInferenceEngine(
#     model_name_or_path="hf.co/ibm-granite/granite-20b-code-instruct-8k-GGUF",
#     credentials=InferenceEngineCredentials(api_url="OLLAMA_API_URL"),
#     parameters=OllamaInferenceEngineParams(
#         num_predict=1000, num_ctx=8192, temperature=0.7, repeat_penalty=1
#     ),
# )

# inference_engine = VLLMInferenceEngine(
#     model_name_or_path="ibm-granite/granite-3.1-8b-instruct",
#     credentials=InferenceEngineCredentials(
#         api_url="VLLM_API_URL", api_key="VLLM_API_KEY"
#     ),
#     parameters=VLLMInferenceEngineParams(max_tokens=1000, temperature=0.7),
# )

# inference_engine = RITSInferenceEngine(
#     model_name_or_path="ibm/granite-20b-code-instruct",
#     credentials={
#         "api_key": "RITS_API_KEY",
#         "api_url": "RITS_API_URL",
#     },
#     parameters=RITSInferenceEngineParams(max_tokens=1000, temperature=0.7),
# )

inference_engine = WMLInferenceEngine( model_name_or_path="ibm/granite-20b-code-instruct", credentials={ "api_key": "WML_API_KEY", "api_url": "WML_API_URL", "project_id": "WML_PROJECT_ID", }, parameters=WMLInferenceEngineParams( max_new_tokens=1000, decoding_method="greedy", repetition_penalty=1 ), ) # inference_engine = OllamaInferenceEngine( # model_name_or_path="hf.co/ibm-granite/granite-20b-code-instruct-8k-GGUF", # credentials=InferenceEngineCredentials(api_url="OLLAMA_API_URL"), # parameters=OllamaInferenceEngineParams( # num_predict=1000, num_ctx=8192, temperature=0.7, repeat_penalty=1 # ), # ) # inference_engine = VLLMInferenceEngine( # model_name_or_path="ibm-granite/granite-3.1-8b-instruct", # credentials=InferenceEngineCredentials( # api_url="VLLM_API_URL", api_key="VLLM_API_KEY" # ), # parameters=VLLMInferenceEngineParams(max_tokens=1000, temperature=0.7), # ) # inference_engine = RITSInferenceEngine( # model_name_or_path="ibm/granite-20b-code-instruct", # credentials={ # "api_key": "RITS_API_KEY", # "api_url": "RITS_API_URL", # }, # parameters=RITSInferenceEngineParams(max_tokens=1000, temperature=0.7), # )

[2025-02-17 21:30:35:969] - INFO - RiskAtlasNexus - Created WML inference engine.

Create an instance of RiskAtlasNexus¶

Note: (Optional) You can specify your own directory in RiskAtlasNexus(base_dir=<PATH>) to utilize custom AI ontologies. If left blank, the system will use the provided AI ontologies.

In [3]:

risk_atlas_nexus = RiskAtlasNexus()

risk_atlas_nexus = RiskAtlasNexus()

[2025-02-17 21:30:36:58] - INFO - RiskAtlasNexus - Created RiskAtlasNexus instance. Base_dir: None

Risk Identification API¶

RiskAtlasNexus.identify_risks_from_usecases()

Params:

• usecases (List[str]): A List of strings describing AI usecases
• inference_engine (InferenceEngine): An LLM inference engine to infer risks from the usecase.
• taxonomy (str, optional): The label for a taxonomy. If not specified, If not provided, the default is ibm-risk-atlas.

Risk Identification using IBM AI Risk taxonomy¶

In [ ]:

usecase = "Generate personalized, relevant responses, recommendations, and summaries of claims for customers to support agents to enhance their interactions with customers."

risks = risk_atlas_nexus.identify_risks_from_usecases(
    usecases=[usecase],
    inference_engine=inference_engine,
    taxonomy="ibm-risk-atlas",
)

risks

usecase = "Generate personalized, relevant responses, recommendations, and summaries of claims for customers to support agents to enhance their interactions with customers." risks = risk_atlas_nexus.identify_risks_from_usecases( usecases=[usecase], inference_engine=inference_engine, taxonomy="ibm-risk-atlas", ) risks

[2025-02-17 21:30:36:63] - INFO - RiskAtlasNexus - Selected taxonomy is IBM AI Risk Atlas. For more info: https://www.ibm.com/docs/en/watsonx/saas?topic=ai-risk-atlas

Out[ ]:

[[Risk(id='atlas-data-acquisition', name='Data acquisition restrictions', description='Laws and other regulations might limit the collection of certain types of data for specific AI use cases.', url='https://www.ibm.com/docs/en/watsonx/saas?topic=SSYOK8/wsj/ai-risk-atlas/data-acquisition.html', dateCreated=datetime.date(2024, 3, 6), dateModified=datetime.date(2025, 2, 11), hasRelatedAction=None, isDefinedByTaxonomy='ibm-risk-atlas', isPartOf='ibm-risk-atlas-data-laws', closeMatch=None, exactMatch=None, broadMatch=None, narrowMatch=None, relatedMatch=None, tag='data-acquisition', type='training-data', phase=None, descriptor='amplified', concern='"There are several ways of collecting data for building a foundation models: web scraping, web crawling, crowdsourcing, and curating public datasets. Data acquisition restrictions can also impact the availability of the data that is required for training an AI model and can lead to poorly represented data."'),
  Risk(id='atlas-data-usage-rights', name='Data usage rights restrictions', description='Terms of service, license compliance, or other IP issues may restrict the ability to use certain data for building models.', url='https://www.ibm.com/docs/en/watsonx/saas?topic=SSYOK8/wsj/ai-risk-atlas/data-usage-rights.html', dateCreated=datetime.date(2024, 3, 6), dateModified=datetime.date(2025, 2, 10), hasRelatedAction=None, isDefinedByTaxonomy='ibm-risk-atlas', isPartOf='ibm-risk-atlas-intellectual-property', closeMatch=None, exactMatch=None, broadMatch=['nist-data-privacy', 'nist-value-chain-and-component-integration'], narrowMatch=None, relatedMatch=None, tag='data-usage-rights', type='training-data', phase=None, descriptor='amplified', concern='Laws and regulations concerning the use of data to train AI are unsettled and can vary from country to country, which creates challenges in the development of models.'),
  Risk(id='atlas-harmful-code-generation', name='Harmful code generation', description='Models might generate code that causes harm or unintentionally affects other systems.', url='https://www.ibm.com/docs/en/watsonx/saas?topic=SSYOK8/wsj/ai-risk-atlas/harmful-code-generation.html', dateCreated=datetime.date(2024, 3, 6), dateModified=datetime.date(2025, 2, 10), hasRelatedAction=None, isDefinedByTaxonomy='ibm-risk-atlas', isPartOf='ibm-risk-atlas-value-alignment', closeMatch=None, exactMatch=None, broadMatch=['nist-dangerous-violent-or-hateful-content', 'nist-information-security'], narrowMatch=None, relatedMatch=None, tag='harmful-code-generation', type='output', phase=None, descriptor='specific', concern='The execution of harmful code might open vulnerabilities in IT systems.'),
  Risk(id='atlas-data-contamination', name='Data contamination', description='Data contamination occurs when incorrect data is used for training. For example, data that is not aligned with model’s purpose or data that is already set aside for other development tasks such as testing and evaluation.', url='https://www.ibm.com/docs/en/watsonx/saas?topic=SSYOK8/wsj/ai-risk-atlas/data-contamination.html', dateCreated=datetime.date(2024, 9, 24), dateModified=datetime.date(2025, 2, 10), hasRelatedAction=None, isDefinedByTaxonomy='ibm-risk-atlas', isPartOf='ibm-risk-atlas-accuracy', closeMatch=None, exactMatch=None, broadMatch=['nist-information-security', 'nist-value-chain-and-component-integration'], narrowMatch=None, relatedMatch=None, tag='data-contamination', type='training-data', phase=None, descriptor='amplified', concern='Data that differs from the intended training data might skew model accuracy and affect model outcomes.'),
  Risk(id='atlas-untraceable-attribution', name='Untraceable attribution', description='The content of the training data used for generating the model’s output is not accessible.', url='https://www.ibm.com/docs/en/watsonx/saas?topic=SSYOK8/wsj/ai-risk-atlas/untraceable-attribution.html', dateCreated=datetime.date(2024, 3, 6), dateModified=datetime.date(2025, 2, 10), hasRelatedAction=None, isDefinedByTaxonomy='ibm-risk-atlas', isPartOf='ibm-risk-atlas-explainability', closeMatch=None, exactMatch=None, broadMatch=['nist-information-integrity'], narrowMatch=None, relatedMatch=None, tag='untraceable-attribution', type='output', phase=None, descriptor='amplified', concern='Without the ability to access training data content, the possibility of using source attribution techniques can be severely limited or impossible. This makes it difficult for users, model validators, and auditors to understand and trust the model.'),
  Risk(id='atlas-over-or-under-reliance', name='Over- or under-reliance', description='In AI-assisted decision-making tasks, reliance measures how much a person trusts (and potentially acts on) a model’s output. Over-reliance occurs when a person puts too much trust in a model, accepting a model’s output when the model’s output is likely incorrect. Under-reliance is the opposite, where the person doesn’t trust the model but should.', url='https://www.ibm.com/docs/en/watsonx/saas?topic=SSYOK8/wsj/ai-risk-atlas/over-or-under-reliance.html', dateCreated=datetime.date(2024, 3, 6), dateModified=datetime.date(2025, 2, 10), hasRelatedAction=None, isDefinedByTaxonomy='ibm-risk-atlas', isPartOf='ibm-risk-atlas-value-alignment', closeMatch=None, exactMatch=None, broadMatch=None, narrowMatch=None, relatedMatch=None, tag='over-or-under-reliance', type='output', phase=None, descriptor='amplified', concern='In tasks where humans make choices based on AI-based suggestions, over/under reliance can lead to poor decision making because of the misplaced trust in the AI system, with negative consequences that increase with the importance of the decision.'),
  Risk(id='atlas-incorrect-risk-testing', name='Incorrect risk testing', description='A metric selected to measure or track a risk is incorrectly selected, incompletely measuring the risk, or measuring the wrong risk for the given context.', url='https://www.ibm.com/docs/en/watsonx/saas?topic=SSYOK8/wsj/ai-risk-atlas/incorrect-risk-testing.html', dateCreated=datetime.date(2024, 9, 24), dateModified=datetime.date(2024, 9, 24), hasRelatedAction=None, isDefinedByTaxonomy='ibm-risk-atlas', isPartOf='ibm-risk-atlas-governance', closeMatch=None, exactMatch=None, broadMatch=['nist-value-chain-and-component-integration'], narrowMatch=None, relatedMatch=None, tag='incorrect-risk-testing', type='non-technical', phase=None, descriptor='amplified', concern='If the metrics do not measure the risk as intended, then the understanding of that risk will be incorrect and mitigations might not be applied. If the model’s output is consequential, this might result in societal, reputational, or financial harm.'),
  Risk(id='atlas-membership-inference-attack', name='Membership inference attack', description="A membership inference attack repeatedly queries a model to determine whether a given input was part of the model’s training. More specifically, given a trained model and a data sample, an attacker samples the input space, observing outputs to deduce whether that sample was part of the model's training. ", url='https://www.ibm.com/docs/en/watsonx/saas?topic=SSYOK8/wsj/ai-risk-atlas/membership-inference-attack.html', dateCreated=datetime.date(2024, 3, 6), dateModified=datetime.date(2025, 2, 10), hasRelatedAction=None, isDefinedByTaxonomy='ibm-risk-atlas', isPartOf='ibm-risk-atlas-privacy', closeMatch=None, exactMatch=None, broadMatch=['nist-data-privacy'], narrowMatch=None, relatedMatch=None, tag='membership-inference-attack', type='inference', phase=None, descriptor='amplified', concern='Identifying whether a data sample was used for training data can reveal what data was used to train a model. Possibly giving competitors insight into how a model was trained and the opportunity to replicate the model or tamper with it. Models that include publicly-available data are at higher risk of such attacks.'),
  Risk(id='atlas-hallucination', name='Hallucination', description='Hallucinations generate factually inaccurate or untruthful content with respect to the model’s training data or input. This is also sometimes referred to lack of faithfulness or lack of groundedness.', url='https://www.ibm.com/docs/en/watsonx/saas?topic=SSYOK8/wsj/ai-risk-atlas/hallucination.html', dateCreated=datetime.date(2024, 3, 6), dateModified=datetime.date(2025, 2, 10), hasRelatedAction=None, isDefinedByTaxonomy='ibm-risk-atlas', isPartOf='ibm-risk-atlas-robustness', closeMatch=None, exactMatch=['nist-confabulation'], broadMatch=['owasp-llm062025-excessive-agency'], narrowMatch=None, relatedMatch=None, tag='hallucination', type='output', phase=None, descriptor='specific', concern='Hallucinations can be misleading. These false outputs can mislead users and be incorporated into downstream artifacts, further spreading misinformation. False output can harm both owners and users of the AI models. In some uses, hallucinations can be particularly consequential.'),
  Risk(id='atlas-output-bias', name='Output bias', description='Generated content might unfairly represent certain groups or individuals.', url='https://www.ibm.com/docs/en/watsonx/saas?topic=SSYOK8/wsj/ai-risk-atlas/output-bias.html', dateCreated=datetime.date(2024, 3, 6), dateModified=datetime.date(2025, 2, 10), hasRelatedAction=None, isDefinedByTaxonomy='ibm-risk-atlas', isPartOf='ibm-risk-atlas-fairness', closeMatch=None, exactMatch=None, broadMatch=['nist-harmful-bias-or-homogenization'], narrowMatch=None, relatedMatch=None, tag='output-bias', type='output', phase=None, descriptor='specific', concern='Bias can harm users of the AI models and magnify existing discriminatory behaviors.'),
  Risk(id='atlas-toxic-output', name='Toxic output', description='Toxic output occurs when the model produces hateful, abusive, and profane (HAP) or obscene content. This also includes behaviors like bullying.', url='https://www.ibm.com/docs/en/watsonx/saas?topic=SSYOK8/wsj/ai-risk-atlas/toxic-output.html', dateCreated=datetime.date(2024, 3, 6), dateModified=datetime.date(2025, 2, 10), hasRelatedAction=None, isDefinedByTaxonomy='ibm-risk-atlas', isPartOf='ibm-risk-atlas-value-alignment', closeMatch=['nist-dangerous-violent-or-hateful-content', 'nist-obscene-degrading-and-or-abusive-content'], exactMatch=None, broadMatch=['owasp-llm052025-improper-output-handling'], narrowMatch=None, relatedMatch=None, tag='toxic-output', type='output', phase=None, descriptor='specific', concern='Hateful, abusive, and profane (HAP) or obscene content can adversely impact and harm people interacting with the model.'),
  Risk(id='atlas-unreliable-source-attribution', name='Unreliable source attribution', description="Source attribution is the AI system's ability to describe from what training data it generated a portion or all its output. Since current techniques are based on approximations, these attributions might be incorrect.", url='https://www.ibm.com/docs/en/watsonx/saas?topic=SSYOK8/wsj/ai-risk-atlas/unreliable-source-attribution.html', dateCreated=datetime.date(2024, 3, 6), dateModified=datetime.date(2024, 9, 24), hasRelatedAction=None, isDefinedByTaxonomy='ibm-risk-atlas', isPartOf='ibm-risk-atlas-explainability', closeMatch=None, exactMatch=None, broadMatch=['nist-information-security'], narrowMatch=None, relatedMatch=None, tag='unreliable-source-attribution', type='output', phase=None, descriptor='specific', concern='Low-quality attributions make it difficult for users, model validators, and auditors to understand and trust the model.'),
  Risk(id='atlas-harmful-output', name='Harmful output', description='A model might generate language that leads to physical harm The language might include overtly violent, covertly dangerous, or otherwise indirectly unsafe statements.', url='https://www.ibm.com/docs/en/watsonx/saas?topic=SSYOK8/wsj/ai-risk-atlas/harmful-output.html', dateCreated=datetime.date(2024, 3, 6), dateModified=datetime.date(2025, 2, 10), hasRelatedAction=None, isDefinedByTaxonomy='ibm-risk-atlas', isPartOf='ibm-risk-atlas-value-alignment', closeMatch=['nist-dangerous-violent-or-hateful-content'], exactMatch=None, broadMatch=['owasp-llm052025-improper-output-handling'], narrowMatch=None, relatedMatch=['nist-cbrn-information-or-capabilities', 'nist-data-privacy', 'nist-obscene-degrading-and-or-abusive-content'], tag='harmful-output', type='output', phase=None, descriptor='specific', concern='A model generating harmful output can cause immediate physical harm or create prejudices that might lead to future harm.'),
  Risk(id='atlas-confidential-information-in-data', name='Confidential information in data', description='Confidential information might be included as part of the data that is used to train or tune the model.', url='https://www.ibm.com/docs/en/watsonx/saas?topic=SSYOK8/wsj/ai-risk-atlas/confidential-information-in-data.html', dateCreated=datetime.date(2024, 3, 6), dateModified=datetime.date(2025, 2, 10), hasRelatedAction=None, isDefinedByTaxonomy='ibm-risk-atlas', isPartOf='ibm-risk-atlas-intellectual-property', closeMatch=None, exactMatch=None, broadMatch=['nist-intellectual-property'], narrowMatch=None, relatedMatch=None, tag='confidential-information-in-data', type='training-data', phase=None, descriptor='amplified', concern='If confidential data is not properly protected, there could be an unwanted disclosure of confidential information. The model might expose confidential information in the generated output or to unauthorized users.'),
  Risk(id='atlas-unrepresentative-data', name='Unrepresentative data', description='Unrepresentative data occurs when the training or fine-tuning data is not sufficiently representative of the underlying population or does not measure the phenomenon of interest.', url='https://www.ibm.com/docs/en/watsonx/saas?topic=SSYOK8/wsj/ai-risk-atlas/unrepresentative-data.html', dateCreated=datetime.date(2024, 9, 24), dateModified=datetime.date(2025, 2, 10), hasRelatedAction=None, isDefinedByTaxonomy='ibm-risk-atlas', isPartOf='ibm-risk-atlas-accuracy', closeMatch=None, exactMatch=None, broadMatch=['nist-harmful-bias-or-homogenization'], narrowMatch=None, relatedMatch=['nist-value-chain-and-component-integration'], tag='unrepresentative-data', type='training-data', phase=None, descriptor='traditional', concern='If the data is not representative, then the model will not work as intended.'),
  Risk(id='atlas-spreading-disinformation', name='Spreading disinformation', description='Generative AI models might be used to intentionally create misleading or false information to deceive or influence a targeted audience.', url='https://www.ibm.com/docs/en/watsonx/saas?topic=SSYOK8/wsj/ai-risk-atlas/spreading-disinformation.html', dateCreated=datetime.date(2024, 3, 6), dateModified=datetime.date(2025, 2, 10), hasRelatedAction=None, isDefinedByTaxonomy='ibm-risk-atlas', isPartOf='ibm-risk-atlas-misuse', closeMatch=None, exactMatch=None, broadMatch=['nist-information-integrity'], narrowMatch=None, relatedMatch=None, tag='spreading-disinformation', type='output', phase=None, descriptor='specific', concern='Spreading disinformation might affect human’s ability to make informed decisions. A model that has this potential must be properly governed.')]]

Risk Identification using NIST AI taxonomy¶

In [ ]:

usecase = "Generate personalized, relevant responses, recommendations, and summaries of claims for customers to support agents to enhance their interactions with customers."

risks = risk_atlas_nexus.identify_risks_from_usecases(
    usecases=[usecase],
    inference_engine=inference_engine,
    taxonomy="nist-ai-rmf",
)

risks

usecase = "Generate personalized, relevant responses, recommendations, and summaries of claims for customers to support agents to enhance their interactions with customers." risks = risk_atlas_nexus.identify_risks_from_usecases( usecases=[usecase], inference_engine=inference_engine, taxonomy="nist-ai-rmf", ) risks

[2025-02-17 21:32:55:113] - INFO - RiskAtlasNexus - Selected taxonomy is NIST AI Risk Management Framework (AI RMF). For more info: https://www.nist.gov/itl/ai-risk-management-framework

Out[ ]:

[[Risk(id='nist-confabulation', name='Confabulation', description='The production of confidently stated but erroneous or false content (known colloquially as “hallucinations” or “fabrications”) by which users may be misled or deceived.', url=None, dateCreated=None, dateModified=None, hasRelatedAction=['GV-1.3-002', 'GV-4.1-001', 'GV-5.1-002', 'MS-2.3-001', 'MS-2.3-002', 'MS-2.3-004', 'MS-2.5-001', 'MS-2.5-003', 'MS-2.6-005', 'MS-2.9-001', 'MS-2.13-001', 'MS-3.2-001', 'MS-4.2-002', 'MG-2.2-009', 'MG-3.2-009', 'MG-4.1-002', 'MG-4.1-004', 'MG-4.3-002'], isDefinedByTaxonomy='nist-ai-rmf', isPartOf=None, closeMatch=None, exactMatch=None, broadMatch=None, narrowMatch=None, relatedMatch=None, tag=None, type=None, phase=None, descriptor=None, concern=None),
  Risk(id='nist-harmful-bias-or-homogenization', name='Harmful Bias or Homogenization', description='Amplification and exacerbation of historical, societal, and systemic biases; performance disparities8 between sub-groups or languages, possibly due to non-representative training data, that result in discrimination, amplification of biases, or incorrect presumptions about performance; undesired homogeneity that skews system or model outputs, which may be erroneous, lead to ill-founded decision-making, or amplify harmful biases.', url=None, dateCreated=None, dateModified=None, hasRelatedAction=None, isDefinedByTaxonomy='nist-ai-rmf', isPartOf=None, closeMatch=None, exactMatch=None, broadMatch=None, narrowMatch=None, relatedMatch=None, tag=None, type=None, phase=None, descriptor=None, concern=None),
  Risk(id='nist-information-integrity', name='Information Integrity', description='Lowered barrier to entry to generate and support the exchange and consumption of content which may not distinguish fact from opinion or fiction or acknowledge uncertainties, or could be leveraged for large-scale dis- and mis-information campaigns.', url=None, dateCreated=None, dateModified=None, hasRelatedAction=['GV-1.2-001', 'GV-1.3-001', 'GV-1.3-006', 'GV-1.3-007', 'GV-1.5-001', 'GV-1.5-003', 'GV-1.6-003', 'GV-4.3-001', 'GV-4.3-003', 'GV-6.1-003', 'GV-6.1-004', 'GV-6.1-005', 'GV-6.1-006', 'GV-6.1-008', 'GV-6.2-006', 'MP-2.1-001', 'MP-2.2-001', 'MP-2.2-002', 'MP-2.3-001', 'MP-2.3-003', 'MP-2.3-004', 'MP-3.4-001', 'MP-3.4-002', 'MP-3.4-003', 'MP-3.4-005', 'MP-3.4-006', 'MP-5.1-001', 'MP-5.1-002', 'MP-5.1-004', 'MS-1.1-001', 'MS-1.1-002', 'MS-1.1-003', 'MS-1.1-005', 'MS-1.1-007', 'MS-1.1-009', 'MS-2.2-001', 'MS-2.2-002', 'MS-2.2-003', 'MS-2.3-004', 'MS-2.5-005', 'MS-2.6-005', 'MS-2.7-001', 'MS-2.7-002', 'MS-2.7-003', 'MS-2.7-004', 'MS-2.7-005', 'MS-2.7-006', 'MS-2.7-008', 'MS-2.8-003', 'MS-2.9-002', 'MS-2.10-001', 'MS-2.10-002', 'MS-2.13-001', 'MS-3.3-002', 'MS-3.3-004', 'MS-3.3-005', 'MS-4.2-001', 'MS-4.2-003', 'MS-4.2-004', 'MG-2.2-002', 'MG-2.2-003', 'MG-2.2-007', 'MG-2.2-009', 'MG-3.1-005', 'MG-3.2-002', 'MG-3.2-003', 'MG-3.2-005', 'MG-3.2-006', 'MG-3.2-007', 'MG-4.1-001', 'MG-4.1-006', 'MG-4.3-002'], isDefinedByTaxonomy='nist-ai-rmf', isPartOf=None, closeMatch=None, exactMatch=None, broadMatch=None, narrowMatch=None, relatedMatch=None, tag=None, type=None, phase=None, descriptor=None, concern=None)]]

Risk Identification using MIT AI taxonomy¶

In [ ]:

usecase = "Generate personalized, relevant responses, recommendations, and summaries of claims for customers to support agents to enhance their interactions with customers."

risks = risk_atlas_nexus.identify_risks_from_usecases(
    usecases=[usecase],
    inference_engine=inference_engine,
    taxonomy="mit-ai-risk-repository",
)

risks

usecase = "Generate personalized, relevant responses, recommendations, and summaries of claims for customers to support agents to enhance their interactions with customers." risks = risk_atlas_nexus.identify_risks_from_usecases( usecases=[usecase], inference_engine=inference_engine, taxonomy="mit-ai-risk-repository", ) risks

[2025-02-17 21:33:15:706] - INFO - RiskAtlasNexus - Selected taxonomy is The AI Risk Repository. For more info: https://airisk.mit.edu/

Out[ ]:

[[Risk(id='mit-ai-risk-subdomain-7.3', name='Lack of capability or robustness', description='AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.', url=None, dateCreated=None, dateModified=None, hasRelatedAction=None, isDefinedByTaxonomy='mit-ai-risk-repository', isPartOf='mit-ai-risk-domain-7', closeMatch=None, exactMatch=None, broadMatch=None, narrowMatch=None, relatedMatch=None, tag=None, type=None, phase=None, descriptor=None, concern=None),
  Risk(id='mit-ai-risk-subdomain-7.4', name='Lack of transparency or interpretability', description='Challenges in understanding or explaining the decision-making processes of AI systems, which can lead to mistrust, difficulty in enforcing compliance standards or holding relevant actors accountable for harms, and the inability to identify and correct errors.', url=None, dateCreated=None, dateModified=None, hasRelatedAction=None, isDefinedByTaxonomy='mit-ai-risk-repository', isPartOf='mit-ai-risk-domain-7', closeMatch=None, exactMatch=None, broadMatch=None, narrowMatch=None, relatedMatch=None, tag=None, type=None, phase=None, descriptor=None, concern=None)]]

Risk Identification using Granite Guardian taxonomy¶

In [ ]:

usecase = "Generate personalized, relevant responses, recommendations, and summaries of claims for customers to support agents to enhance their interactions with customers."

risks = risk_atlas_nexus.identify_risks_from_usecases(
    usecases=[usecase],
    inference_engine=inference_engine,
    taxonomy="ibm-granite-guardian",
)

risks

usecase = "Generate personalized, relevant responses, recommendations, and summaries of claims for customers to support agents to enhance their interactions with customers." risks = risk_atlas_nexus.identify_risks_from_usecases( usecases=[usecase], inference_engine=inference_engine, taxonomy="ibm-granite-guardian", ) risks

[2025-02-17 21:33:33:796] - INFO - RiskAtlasNexus - Selected taxonomy is IBM Granite Guardian. For more info: https://arxiv.org/abs/2412.07724

Out[ ]:

[[Risk(id='granite-relevance', name='Context Relevance', description="This occurs in when the retrieved or provided context fails to contain information pertinent to answering the user's question or addressing their needs. Irrelevant context may be on a different topic, from an unrelated domain, or contain information that doesn't help in formulating an appropriate response to the user.", url='https://www.ibm.com/docs/en/watsonx/saas?topic=gg-relevance', dateCreated=datetime.date(2024, 12, 10), dateModified=datetime.date(2024, 12, 10), hasRelatedAction=None, isDefinedByTaxonomy='ibm-granite-guardian', isPartOf='granite-guardian-rag-safety-group', closeMatch=None, exactMatch=None, broadMatch=['atlas-hallucination'], narrowMatch=None, relatedMatch=None, tag='relevance', type=None, phase=None, descriptor=None, concern=None),
  Risk(id='granite-answer-relevance', name='Answer Relevance', description="This occurs when the LLM response fails to address or properly respond to the user's input. This includes providing off-topic information, misinterpreting the query, or omitting crucial details requested by the User. An irrelevant answer may contain factually correct information but still fail to meet the User's specific needs or answer their intended question.", url='https://www.ibm.com/docs/en/watsonx/saas?topic=gg-answer-relevance', dateCreated=datetime.date(2024, 12, 10), dateModified=datetime.date(2024, 12, 10), hasRelatedAction=None, isDefinedByTaxonomy='ibm-granite-guardian', isPartOf='granite-guardian-rag-safety-group', closeMatch=None, exactMatch=None, broadMatch=['atlas-hallucination'], narrowMatch=None, relatedMatch=None, tag='answer-relevance', type=None, phase=None, descriptor=None, concern=None),
  Risk(id='granite-function-call', name='Function Calling Hallucination', description='This occurs when the LLM response contains function calls that have syntax or semantic errors based on the user query and available tool definition. For instance, if an AI agent purportedly queries an external information source, this capability monitors for fabricated information flows.', url='https://www.ibm.com/docs/en/watsonx/saas?topic=gg-function-call', dateCreated=datetime.date(2024, 12, 10), dateModified=datetime.date(2024, 12, 10), hasRelatedAction=None, isDefinedByTaxonomy='ibm-granite-guardian', isPartOf='granite-guardian-agentic-safety-group', closeMatch=None, exactMatch=None, broadMatch=['atlas-hallucination'], narrowMatch=None, relatedMatch=None, tag='function-call', type=None, phase=None, descriptor=None, concern=None)]]