NIST AI RMF in practice: the best framework nobody is implementing

What NIST AI RMF actually says

NIST AI 100-1 (January 2023) defines a voluntary framework for managing risks associated with AI systems. It is organized around four core functions: Govern, Map, Measure, Manage.¹

graph TB
    G[GOVERN<br/>Policies, roles, culture, accountability]
    G --> M1[MAP<br/>Context, scope, risk identification]
    G --> M2[MEASURE<br/>Analysis, metrics, monitoring]
    G --> M3[MANAGE<br/>Response, mitigation, communication]
    M1 --> M2
    M2 --> M3
    M3 --> M1

    style G fill:#1a1917,stroke:#00c8a0,color:#e8e5de
    style M1 fill:#1a1917,stroke:#c9a84c,color:#e8e5de
    style M2 fill:#1a1917,stroke:#c9a84c,color:#e8e5de
    style M3 fill:#1a1917,stroke:#c9a84c,color:#e8e5de

The structure is sound. Govern sits at the top and wraps the other three functions because governance is the precondition for everything else. Map identifies risks in context. Measure quantifies them. Manage responds to them. The cycle is continuous. The framework explicitly acknowledges that AI risk is not a one-time assessment.

The companion document, the AI RMF Playbook (NIST AI 100-1 Playbook), provides suggested actions and references for each subcategory. The NIST AI RMF Profiles document allows organizations to select relevant subcategories based on their specific use case.

The framework is well-designed. The problem is not the framework. The problem is what happens when the framework meets an organization.

Where the Govern function goes to die

Govern is where every AI RMF implementation either succeeds or fails. It never fails spectacularly. It fails quietly, through delegation without authority, through committees that meet quarterly, through policies that exist on SharePoint and nowhere else.

The result is a governance structure where every function can point to someone else. Security says the model output is a data quality issue. Data says the model behavior is an operations issue. Operations says the model risk is a legal issue. Legal says the model selection was a technology decision.

This is not hypothetical. This is what the conversation looks like in most organizations that have attempted AI RMF implementation. The framework’s GOVERN 1.1 subcategory explicitly calls for “legal and regulatory requirements involving AI are understood, managed, and documented.” But understanding, managing, and documenting cross-functional requirements requires a single point of accountability. The framework assumes that point exists. Most organizations do not have it.

GOVERN 1 is the load-bearing wall. If nobody owns AI risk as a unified concern, every other function in the framework operates without a foundation.

Map: context is harder than it sounds

MAP is the identification function. MAP 1 says: understand the context in which the AI system operates. MAP 2 says: categorize the AI system. MAP 3 says: identify known and foreseeable risks.

Where MAP breaks down in practice: organizations map what they know. AI systems produce risks from what organizations do not know. The emergent behavior problem is real. A model performing well on benchmarks can produce harmful output on edge cases that nobody included in the test set. MAP assumes you can identify risks before they materialize. For AI systems, some risks only become visible in production.

The agentic AI problem specifically. MAP was written before agentic AI became a deployment pattern. An AI agent that takes actions (makes API calls, writes files, sends messages) has a fundamentally different risk profile than an AI model that produces text. The risk is not just “wrong output.” The risk is “wrong action.” MAP’s risk identification categories do not cleanly accommodate the difference between a model that says something incorrect and a model that does something incorrect. The action surface is not in the framework.

Measure: the metrics maturity gap

MEASURE is where the AI RMF should shine. This is the quantification function. MEASURE 1 says: select appropriate metrics. MEASURE 2 says: assess AI systems against those metrics. MEASURE 3 says: track metrics over time.

The problem: AI metrics are immature. For most of the framework’s subcategories, there is no industry-accepted metric, no established measurement methodology, and no benchmark against which to evaluate. The framework says “measure bias.” With what instrument? Against what standard? Measured when, by whom, and validated how?

The Measure function describes what a mature AI risk program would measure. It does not describe how to measure it. For most organizations, that gap is the entire problem.

Specific measurement gaps that practitioners encounter:

Fairness metrics. MEASURE 2.6 requires assessment of “the AI system’s fairness and bias.” Multiple competing fairness definitions exist (demographic parity, equalized odds, predictive parity). They are mathematically incompatible. You cannot satisfy all of them simultaneously. The framework does not specify which definition to use. This is arguably correct (context should determine the definition) but in practice it means organizations pick whichever definition produces the best-looking numbers.

Explainability metrics. MEASURE 2.8 references “the interpretability and explainability of AI system outputs.” There is no agreed-upon metric for explainability. SHAP values, LIME, attention visualization, counterfactual explanations, all measure different things. The framework requires explainability without defining what explainability means in operational terms.

Behavioral drift. MEASURE 4.2 references monitoring for “changes and degradation.” Drift detection in production LLMs is a real and unsolved measurement problem. Existing ML monitoring assumes access to the model’s internals (weights, activations, loss curves). Prompted LLMs running on third-party inference APIs give you none of that. You are measuring behavioral output, not model state. The measurement methodology for this does not exist at scale.²

Manage: the strongest function in the framework

MANAGE is where the AI RMF is most practical. MANAGE 1 says: prioritize risks. MANAGE 2 says: plan responses. MANAGE 3 says: manage risks from third-party AI. MANAGE 4 says: document and communicate.

The weakness of MANAGE: it assumes MAP and MEASURE have done their jobs. If risk identification was incomplete (MAP) and risk quantification was aspirational (MEASURE), then risk response (MANAGE) is operating on incomplete data. You cannot manage what you have not measured. The framework’s linear dependency chain means that weaknesses in upstream functions propagate forward.

What the checkbox version looks like

Here is how most organizations “implement” the AI RMF:

This is not a caricature. This is the median outcome. The framework is treated as a compliance artifact, not an operational practice. The governance committee provides cover. The risk register provides documentation. The annual assessment provides a date stamp. None of it changes how AI systems are built, deployed, or monitored.

What a real implementation looks like

I have seen organizations get this right. Not many, but enough to know what the pattern looks like.

graph LR
    subgraph Continuous
        G[Govern<br/>Single AI risk owner<br/>Monthly cadence]
        M1[Map<br/>Per-system risk cards<br/>Updated at deploy]
        M2[Measure<br/>Automated metrics<br/>Dashboard + alerts]
        M3[Manage<br/>Runbook per risk<br/>Response tested quarterly]
    end

    G --> M1
    M1 --> M2
    M2 --> M3
    M3 -->|Feedback| M1
    M3 -->|Escalation| G
    M2 -->|Threshold breach| M3

    style G fill:#1a1917,stroke:#00c8a0,color:#e8e5de

Govern: a single human owns AI risk. Not a committee. A named person with authority to stop a deployment, require a re-evaluation, or escalate to the board. The committee advises. The person decides.

Map: risk cards per AI system, not per category. Each AI system in production has its own risk card that identifies the specific risks to that system in its specific deployment context. “Bias” at the category level is not useful. “Model X producing disparate rejection rates for protected class Y in use case Z” is useful. The risk card is updated every time the system is modified.

Measure: automated, continuous, dashboarded. Whatever metrics exist (and they are few), they run automatically in the deployment pipeline. Fairness metrics on batch inference. Drift detection on production outputs. Latency and error rate monitoring. The dashboard exists. Thresholds produce alerts. Alerts produce responses.

Manage: runbooks, not reports. Each identified risk has a response runbook. The runbook says: who is notified, what is the initial response, what is the escalation path, what is the rollback procedure. Runbooks are tested. “We have documented the risk” is not the same as “we know what to do when the risk materializes.”

Where the framework does not account for agentic AI

The AI RMF was published in January 2023. Agentic AI, where models take actions through tool use, was not a mainstream deployment pattern at that time. The framework assumes a model that produces output. Agentic AI produces actions.

Agentic AI does not need a new framework. It needs the existing framework extended with an authorization function that the original authors did not anticipate. The question is not “should we manage AI risk?” It is “how do we manage the risk of AI that acts?”

What the framework gets right that nobody is using

Despite the criticism above, the AI RMF contains several genuinely excellent ideas that most implementations ignore:

What I recommend to organizations

Three things. Not twelve. Twelve means none of them happen.

First: name the AI risk owner. A human. Not a committee. Someone who can say “no” and have it stick. This single action will do more for your AI governance posture than the rest of the framework combined.

Second: create risk cards per AI system, not per category. Stop managing “AI bias” as an abstract concept. Start managing “Model X’s rejection rate disparity in Use Case Y.” Specific, measurable, tied to a system, updated when the system changes.

Third: test the runbooks. If your AI system produces a harmful output tomorrow, do you know what happens? Not in theory. In practice. Run the drill. Find out if the escalation path works, if the rollback procedure is documented, if the person who is supposed to get the call actually gets the call. The drill will teach you more about your AI risk posture than the assessment ever will.