Last Updated: September 2024

Prepared by: Charles Worthington, Veterans Affairs Chief Artificial Intelligence Officer (CAIO) and Chief Technology Officer (CTO)

1. Strengthening AI Governance

General

Describe any planned or current efforts within your agency to update any existing internal AI principles, guidelines, or policy to ensure consistency with M-24-10.

Prior to the release of OMB Memo M-24-10 (PDF, 33-pages, 518KB), the Department of Veterans Affairs (VA) had a few types of guidance related to AI. All guidance has been reviewed for consistency with M- 24-10.

• VA Trustworthy AI Framework: VA developed and subsequently adopted the Trustworthy AI Framework in July 2023. The six Trustworthy AI principles unify and clarify multiple Federal requirements, mandates, and frameworks to streamline VA AI governance activities, which are applicable to M-24-10.
  • Internal generative AI guidance: VA has reviewed its internal generative AI guidance published in July 2023, which reinforces its current information technology (IT) policies, and found it to be consistent with M-24-10.
  • Miscellaneous policies including VA’s privacy assessments, authority to operate (ATO) policies, and general IT polices all still apply and are consistent with M-24-10.
  • Where relevant, existing processes (e.g., ATO, privacy, Federal Information Technology Acquisition Reform Act (FITARA)) are in the process of being updated such that AI projects that come through them are appropriately routed through the AI inventory and governance process.
  • VA is exploring ways to integrate its VA Trustworthy AI Framework and AI guidance into key offices that evaluate equitable outcomes (Federal civil rights compliance) in the delivery of healthcare and benefits, as well as the impact on employees (e.g., Office of Health Equity, Office of Equity Assurance, Center for Minority Veterans, Center for Women Veterans, and Office of Resolution Management Diversity Equity and Inclusion).
  • VA has developed an AI Ethics Toolkit, including an AI Ethics Assessment Tool and Quick Start Guide, that is consistent with M-24-10 and that advances VA’s ability to anticipate and mitigate risks to rights and safety from the use of AI.
  • Several Veterans Integrated Service Networks (VISN) have established (or are establishing) AI Oversight Subcommittees to assess Trustworthy AI practices.
  • VA has a robust AI research program that has existed for many years due to VA’s partnerships with academic medical institutions across the country, as well as the National Artificial Intelligence Institute (NAII) in the Veterans Health Administration (VHA). Research programs have associated policies and systems, including Institutional Review Boards (IRB) that ensure the rights, welfare, and well-being of human research participants. These have been reviewed and found to be consistent with M-24-10. Note that most AI research is considered out-of-scope of M-24-10 according to section 2(b)(iv). A subset of AI research, prospective studies with real world impact, are subject to both research and M-24-10 requirements.
  • VHA has a long history of patient safety systems and policies that ensure patient safety issues are identified and remediated quickly, which apply to AI systems as well.
  • VA has developed contracting language for responsible AI, which is being piloted in a few contracts.
  • VA Handbook 0900, Open Data – Managing Information as an Asset, requires Administration and Staff Office Open Data Liaisons to form an internal committee or team to review data assets that meet Open Data criteria. The committee should consist of representatives with the following subject matter expertise: Privacy Act, Artificial Intelligence, Ethics, Freedom of Information Act (FOIA), and Information System Security Officers (ISSO). The structure of this committee differs and generally resides under the oversight of existing governance structures.

AI Governance Bodies

Identify the offices that are represented on your agency’s AI governance body.

VA’s AI Governance Council is chaired by the VA Deputy Secretary and co-vice-chaired by VA’s Chief AI Officer and VHA’s Chief Digital Health Officer. The council has broad VA Administration and Staff Office representation for the purpose of engaging VA senior officials from all relevant functional disciplines in enterprise-level decisions on the development and use of AI. The following are member organizations of the council: Office of the Secretary, Office of Information and Technology (OIT), VHA, Veterans Benefits Administration, National Cemetery Administration, Office of Human Resources and Administration/Operations, Security and Preparedness (HRA/OSP), Office of Acquisitions, Logistics and Construction, Office of Enterprise Integration, Office of Management, Veterans Experience Office (VEO), Board of Veterans Appeals, Office of General Counsel, Office of the Inspector General (OIG), Office of Public and Intergovernmental Affairs, and the Office of Small and Disadvantaged Business Utilization.

From its member organizations, the council derives representatives in key roles related to AI adoption and risk management, including the Chief Data Officer (CDO); Chief Information Security Officer; Chief Risk Officer; Chief Human Capital Officer; Chief Diversity Officer; Deputy Chief Financial Officer; Deputy Executive Director, Office of Acquisitions, Logistics and Construction; Deputy Chief Veterans Experience Officer; Director, Center for Minority Veterans; Director, Center for Women Veterans; Executive Director; Office of Small and Disadvantaged Business Utilization; Deputy General Counsel; VA Statistical Official, and Enterprise Chief Privacy Officer. Many of these officials are also responsible for implementing AI within their respective program office organization.

Describe the expected outcomes for the AI governance body and your agency’s plan to achieve them.

The purpose of the VA AI Governance Council is to serve as VA’s principal governing body for AI safety, privacy, and security oversight. The council’s immediate goal is to oversee the implementation of OMB M-24-10. Expected outcomes include transparency, wide participation, shared understanding, and risk management for AI across the agency.

Some specific responsibilities of the council include:

• Approve enterprise strategy on advancing the responsible use of AI.
• Create and approve policies and processes to review AI safety, privacy, security, and testing measures for compliance with Federal standards.
• Provide input for, or participate in, critical stages of pre-implementation testing and performance evaluation of AI systems.
• Review post-implementation evaluations to ensure AI services and programs meet forecasted benefits and outcomes.
• Provide direction and counsel to the AI Operations Directorate (AIOD), which is a group under the Chief AI Officer that leads the day-to-day implementation of OMB M-24-10 and operationalization of AI across VA.
• Resolve disputes between the AIOD and program offices involved in AI development, procurement, testing, or use.
• Establish cross-team collaboration to provide guidance, identify common agency challenges, establish best practices, and share solutions.
• Leverage the experience and business context of other governance entities in VA for strategy recommendations, insights, and lessons learned in the areas of risk management and mitigation, impact of decisions on program executors, and improved buy-in across agency.
• Review and consult with senior executives responsible for AI program status reports to oversight agencies, e.g., OMB, OIG, and GAO.

Describe how, if at all, your agency’s AI governance body plans to consult with external experts as appropriate and consistent with applicable law. External experts are characterized as individuals outside your agency, which may include individuals from other agencies, federally funded research and development centers, academic institutions, think tanks, industry, civil society, or labor unions.

VA regularly engages with external experts on AI topics. Some examples of this include:

• VA has partnerships with universities and academic medical institutions across the country, with many academic experts holding cross-appointments at VA. These arrangements result in significant cross-pollination of ideas between VA and academics conducting cutting edge research.
  • The AI Governance Council is setting up a mailing list for members to communicate external and internal AI-related events and opportunities.

• Members of the AI governance body:
  • Attend and present at industry conferences, such as the Healthcare Information and Management Systems Society (HIMSS), ViVE and the American Council for Technology-Industry Advisory Council (ACT-IAC).
  • Meet with other agencies regularly, including a recurring series on health AI with the Department of Defense (DoD) and Health and Human Services (HHS).
  • Engage with various interagency groups and councils including the CDO council, the CAIO council, the Chief Human Capital Officer (CHCO) council, the Office of Personnel Management (OPM), the AI Talent Task Force, and the three interagency working groups coordinated by the CAIO council on Generative AI, AI-related acquisitions, and AI risk management.
  • Participate in the General Services Administration (GSA) AI Community of Practice.
  • Engage with external experts on health AI through organizations such as the Coalition for Health AI (CHAI) and the Health AI Partnership (HAIP).
  • Engage with external experts on specific AI use cases, such as VEO’s engagement with external healthcare providers on change management related to VA’s ambient AI scribe pilot.
  • Consult with other large enterprises to share experiences integrating AI features into software.
  • Serve as members of the Data and Model subcommittee of the National Artificial Intelligence Research Resource (NAIRR).
  • Plan to engage with VA’s labor union on AI and AI workforce topics.

AI Use Case Inventories

Describe your agency’s process for soliciting and collecting AI use cases across all sub- agencies, components, or bureaus for the inventory. In particular, address how your agency plans to ensure your inventory is comprehensive, complete, and encompasses updates to existing use cases.

VA has issued an agency-wide memorandum through its Veterans Information Systems and Technology Architecture Web Enabled System (VIEWS) detailing the 2024 inventory requirements and use case review process, with a call to action for all in-scope AI use case owners to submit details on their use cases through a linked form. This communication was followed by communication from the AI Governance Council executive leads on the need for all in-scope AI use cases from their component office to be submitted.

VA’s CAIO has begun to integrate into existing processes and intakes, such as ATOs, FITARA, software request intake processes, and innovation-related intake processes, to enable the identification of use cases not submitted into the intake.

VA has also contacted all AI use case owners from the prior 2022 and 2023 use case inventories for updates on their existing use cases and with the new requirements from M- 24-10.

Reporting on AI Use Cases Not Subject to Inventory

Describe your agency’s process for soliciting and collecting AI use cases that meet the criteria for exclusion from being individually inventoried, as required by Section 3(a)(v) of M-24-10. In particular, explain the process by which your agency determines whether a use case should be excluded from being individually inventoried and the criteria involved for such a determination.

Identify how your agency plans to periodically revisit and validate these use cases. In particular, describe the criteria that your agency intends to use to determine whether an AI use case that previously met the exclusion criteria for individual inventorying should subsequently be added to the agency’s public inventory.

VA does not expect to have many use cases where sharing would be inconsistent with applicable law and governmentwide policy. VA intends to assess and triage AI use cases according to applicable law and policy, and exclusions in this category will be handled on a case-by-case basis.

2. Advancing Responsible AI Innovation

AI Strategy [OPTIONAL]

VA will release an updated AI strategy as a separate document by March 2025. Broadly, VA sees significant opportunity for AI, when implemented responsibly, to contribute to our mission and has been executing across four main AI workstreams: AI policy and risk management, AI workforce development, AI infrastructure, and AI priority use cases.

Removing Barriers to the Responsible Use of AI

Describe any barriers to the responsible use of AI that your agency has identified, as well as any steps your agency has taken (or plans to take) to mitigate or remove these identified barriers. In particular, elaborate on whether your agency is addressing access to the necessary software tools, open-source libraries, and deployment and monitoring capabilities to rapidly develop, test, and maintain AI applications.

• VA has identified and is addressing several barriers to responsible building of AI, including access to authoritative data sources for training, testing and validation of AI models and ensuring that these data sources have documentation describing how they are cleaned and refined to support model audits.
• VA currently supports several enterprise data platforms and is implementing an enterprise data catalog to play a core role in building in-house AI systems. These data platforms are also crucial to secure access of Personal Identifiable Information (PII) and Protected Health Information (PHI). An example of a data platform is the Summit Data Platform, which provides cloud access to authoritative and refined health and customer experience data assets. These cloud platforms have access to modern data science and engineering tools, including AI and generative AI services. These service offerings have expanded over the past year, particularly as new services achieve FedRAMP approval.
• VA’s cloud platforms also have monitoring capabilities that are important for model testing and deployment, and utilize many open-source libraries.
• VA currently blocks most commercial generative AI chat interface websites, as they are not approved IT systems. As an alternative, VA is piloting an on-network generative AI chat interface.

Identify whether your agency has developed (or is in the process of developing) internal guidance for the use of generative AI. In particular, elaborate on how your agency has established adequate safeguards and oversight mechanisms that allow generative AI to be used in the agency without posing undue risk.

Yes, VA has developed guidance for the use of AI and posted it on an internal website:

• No web-based, publicly available generative AI service has been approved for use with VA-sensitive data. Examples of these include OpenAI’s ChatGPT and GPT4, Google’s Gemini, Anthropic’s Claude, Microsoft’s Bing Copilot, and AskSage.ai. VA follows existing Federal requirements and processes to ensure VA data is protected. When users enter information into an unapproved web-based tool, VA loses control of the data. Some public Large Language Model (LLM) web services have terms of service that explicitly allow them to use the data entered into the tool for other purposes.
• No PII, PHI, or VA-sensitive data should be entered into these unapproved services. VA sensitive data includes financial, budgetary, research, quality assurance, confidential commercial, critical infrastructure, and investigatory and law enforcement information. The entire definition of VA Sensitive Data can be found at 38 U.S.C. § 5727 (23).
• Where possible, limit the sharing and saving of data in unapproved services.
• VA staff should carefully evaluate the output of any LLM tool for accuracy before using the output in VA work. LLMs are known to generate inaccurate information that sounds plausibly true, and VA staff are responsible for the accuracy of their work products.

AI Talent

Describe any planned or in-progress initiatives from your agency to increase AI talent. In particular, reference any hiring authorities that your agency is leveraging, describe any AI-focused teams that your agency is establishing or expanding, and identify the skillsets or skill-levels that your agency is looking to attract. If your agency has designated an AI Talent Lead, identify which office they are assigned to.

VA is increasing AI and AI-enabling talent through a multi-pronged approach. VA is particularly focused on recruiting talent that can contribute to our goal of operationalizing trustworthy AI across VA.

• VA has created an AI workforce working group, with representatives from across the agency led by HRA/OSP, OIT, and VHA. This group is developing a comprehensive AI workforce hiring and training strategy.
• VA is actively investing in recruiting AI and AI-enabling talent, including by:
  • Developing continuous recruitment pipelines using the White House Presidential Innovation Fellowship, White House Presidential Management Fellowship, United States Digital Corps, Health Professional Trainees, Technical Career Field Programs, Science and Technology Policy Fellowships with the American Association for the Advance of Science, pooled hiring actions and Tech to Gov career fairs.
  • Utilizing Direct Hire Authority (DHA) when appropriate for the following approved occupations and series: 1560 Data Scientist, 1515 Operations Research Analyst, 2210 IT Specialist (Artificial Intelligence), 1550 Computer Scientist (Artificial Intelligence), 0854 Computer Engineer (Artificial Intelligence), and 0343 Management and Program Analyst (AI-related design and development of systems using machine learning).
  • Exploring additional hiring authorities including new Schedule A(r) authority under 5 CFR 213.3102(i)(3) when appropriate, and hybrid Title 38 and Title 38 hiring authorities for occupational series where appropriate.
• VA is also exploring various incentives including Student Loan Repayment Program (SLRP), and other recruitment, relocation, and retention (3R) incentives, and the new authorities contained in the Promise to Address (PACT) Act.
• VA has classified AI position descriptions under IT Specialist (AI).
• VA has designated an AI Talent Lead from the HRA/OSP to serve on OPM’s AI talent interagency working group and to be accountable for reporting to agency leadership and tracking AI hiring.
• VA has released an AI Workforce Blueprint, which describes a strategic framework for recruiting, hiring, training, and retaining top AI talent.
• VA has developed an AI talent construct that describes the essential roles and skills needed to effectively implement AI. This talent construct has been shared widely across agencies and is now being used as the basis for OPM’s talent construct.
• Under the Chief AI Officer, VA has established a small AI team to lead AI risk management and innovation at the enterprise level.
• VHA has established a new Digital Health Office (DHO) to integrate and expand the application of data solutions for healthcare applications. The NAII under the AI and Emerging Technology section of DHO aims to add AI talent to speed development of innovative products and move them into operations for the benefit of Veterans and other VA stakeholders.

If applicable, describe your agency’s plans to provide any resources or training to develop AI talent internally and increase AI training opportunities for Federal employees. In particular, reference any role-based AI training tracks that your agency is interested in, or actively working to develop (e.g., focusing on leadership, acquisition workforce, hiring teams, software engineers, administrative personnel, or others).

VA has developed AI training to support the workforce performing AI functions and general overview training to provide baseline awareness. In April 2024, VA released role-based training for AI accessible to all employees. The below describes the three personas and the AI course names:

• All Employees
  • Building an AI Powered Workforce

• Data Bias and Ethical Considerations in AI
  • Leaders and Managers
    • Embracing Risk and Learning from Setbacks with AI Projects
    • Data Analytics and Data Ethics
    • Generative AI and its Impact on Everyday Business
    • Planning AI Implementation
  • Executive Leaders
    • Navigating AI Ethical Challenges and RisksLeading in the Age of Generative AI
    • AI Enterprise Planning

In addition to the above, the following efforts are underway:

• AI 101 training for all staff is an asynchronous training opportunity in development with the NAII and the Institute for Learning, Education and Development (ILEAD).
• AI 101 for Leadership is a pilot of a synchronous training presentation with an opportunity for question and answers. The leadership training has been piloted in VHA with VISN and facility level leadership.
• The AI@VA Community is an online community of practice designed to share AI information, news, and training; encourage collaboration; and provide a platform for inquiry for AI practitioners and those interested in AI.
• The Talent Management System (TMS 2.0) has 247 active AI related courses listed, the majority of which are third-party, asynchronous options available to all employees for their professional development.
• VA’s ASPIRE (All Services Personnel and Institutional Readiness Engine) is content agnostic, personnel assessment and upskilling platform designed with trustworthiness, accessibility, and equity in mind.

AI Sharing and Collaboration

Describe your agency’s process for ensuring that custom-developed AI code— including models and model weights — for AI applications in active use is shared consistent with Section 4(d) of M-24-10.

Elaborate on your agency’s efforts to encourage or incentivize the sharing of code, models, and data with the public. Include a description of the relevant offices that are responsible for coordinating this work.

VA’s AI inventory and review process will point AI use case owners to both VA’s Open Data Initiative process and to resources for open-sourcing their software code, in the event their project is suitable for open-sourcing. The in-house AI inventory also includes tethering model cards and data sheets on its development roadmap which will increase the transparency and reusability of models internally and foster the internal developer ecosystem.

VA has had several prominent open-source projects for many years, including the Veterans Health Information Systems and Technology Architecture (VistA) health record system and the VA.gov digital experience website and platform.

VA’s Open Data portal is available and the Open Data lead reports to the VA CDO, within in the Office of Enterprise Integration. VA does expect that a large proportion of the data and models related to AI at VA will contain PII, PHI, or otherwise sensitive information that result in them not being available through open source. VA also has other mechanisms for sharing data with approved parties, such as the VA Data Commons which provides researchers access to relevant de-identified VA data for medical research purposes.

Harmonization of Artificial Intelligence Requirements

Explain any steps your agency has taken to document and share best practices regarding AI governance, innovation, or risk management. Identify how these resources are shared and maintained across the agency.

VA documents and shares best practices on AI via a variety of mechanisms. These include:

• Centralized web resources:
  • VA has created a centralized internal hub for AI-related information. This includes an overview of VA’s AI program, an overview of our governance process, our AI guidance documentation, and Frequently Asked Questions related to AI at VA.
  • VA’s Trustworthy AI Framework
• Asynchronous training: VA has released asynchronous AI training agency wide as described in the training section of this document.
• Synchronous training and presentations: AI experts regularly give presentations and meet with groups across the agency to present on best practices.

3. Managing Risks from the Use of Artificial Intelligence

Determining Which Artificial Intelligence Is Presumed to Be Safety-Impacting or Rights-Impacting

Explain the process by which your agency determines which AI use cases are rights- impacting or safety-impacting. In particular, describe how your agency is reviewing or planning to review each current and planned use of AI to assess whether it matches the definition of safety-impacting AI or rights-impacting AI, as defined in Section 6 of M-24-10. Identify whether your agency has created additional criteria for when an AI use is safety-impacting or rights-impacting and describe such supplementary criteria.

VA has adopted the OMB definitions of safety-impacting and rights-impacting AI. VA has elaborated upon these definitions in a document that has been reviewed by the VA AI Governance Council. In the document VA identified a representative set of potential AI use cases across VA and jointly determined whether VA identifies them as safety-impacting and/or rights-impacting, providing a written rationale for each use case. It is important to note that this representative set should not be assumed to be exhaustive. This document serves as the primary reference to refer to for making safety-impacting and rights-

impacting decisions at VA. As new regulations emerge and additional use cases are identified, VA will iterate and refine the document accordingly.

If your agency has developed its own distinct criteria to guide a decision to waive one or more of the minimum risk management practices for a particular use case, describe the criteria.

VA has not developed this.

Describe your agency’s process for issuing, denying, revoking, tracking, and certifying waivers for one or more of the minimum risk management practices.

Currently, VA has not issued any waivers. Waivers will be centrally tracked by the CAIO’s office. Proposed waivers will be determined after an AI use case owner has answered all required questions and the team has engaged with the use case.

Required documentation includes: all OMB-required questions and answers, a written description from the AI use case owner on why the requirements cannot be met, and the reasoning for requesting a waiver. The reasoning must be that fulfilling the requirement would increase risks to safety or rights overall or would create an unacceptable impediment to critical agency operations.

Waivers will be tracked in a central repository. They will be reviewed on at least an annual basis where use case owners will be contacted for updates and changes in the use case.

Implementation of Risk Management Practices and Termination of Non-Compliant AI

Elaborate on the controls your agency has put in place to prevent non-compliant safety-impacting or rights-impacting AI from being deployed to the public.

Describe your agency’s intended process to terminate, and effectuate that termination of, any non-compliant AI.

VA will leverage its existing applicable release and oversight workflows, business processes and tools by integrating the AI use case review process into them. Examples include: the Unified System Registry for system initiation, Risk Management Framework for ATO, FITARA, and various software and data platform implementation processes. This ensures capture of AI use cases occurring at VA at key points in the product lifecycle.

For use cases that are found to be non-compliant, the first step is to work with use case owners to remediate any identified issues. If it is not possible to remediate issues, VA will determine whether to issue an extension, a waiver or remove the AI use case from operations. This determination will be made by balancing the benefits and risks of removing an active AI project from production. If it is determined that the non-compliant AI must be terminated, VA can achieve this through a few possible pathways, including through the Risk Management Framework (RMF) Denial Authority to Operate (DATO) and blocking the connection at the VA boundary.

Minimum Risk Management Practices

Identify how your agency plans to document and validate implementation of the minimum risk management practices. In addition, discuss how your agency assigns responsibility for the implementation and oversight of these requirements.

VA’s AI inventorying and use case review process is a three-step process as described below. Use case reviews will be documented in a centralized repository managed by the CAIO team. Use cases will be reviewed based on a high-level rubric currently in development. It is expected that the first year of use case review will be primarily manual – a qualitative review of the evidence that a use case is meeting the requirements. VA places emphasis on evaluating AI use cases holistically – evaluating the system as a whole and how it compares with the non-AI alternative system – when making determinations.

In subsequent years, VA plans to develop additional standardization and automation for the review process and determinations. AI standards are still being developed across the world and particularly in healthcare.

Step 1: Initial intake

• AI use case owners submit short AI intake form.
  • Use cases reviewers will determine whether the AI use case meets the M-24-10 inclusion criteria.
  • If a use case is determined to be out of scope, the process is complete. Reasons a use case may be identified as out-of-scope include if it does not meet the definition of AI or is part of M-24-10’s exclusion criteria for non-operational research.

Step 2: Inventory questionnaire

• If the use case is determined to be in scope, an additional OMB-mandated questionnaire will be sent to the use case owner to complete.
• The questionnaire will ask for required information to assist reviewers in determining whether an AI use case is considered safety and/or rights impacting.
• For use cases determined to not be safety or rights impacting, the process is complete.

Step 3: Additional review and requirements for safety and rights impacting AI

• For use cases determined to be safety and/or rights impacting, use case owners will be sent an additional questionnaire and reviewers will determine whether the use case is meeting M-24-10 requirements.
• Use cases in operations will have until December 1, 2024, to meet the requirements. For use cases that are unable to meet the requirements by that date, reviewers will determine whether to submit a one-year extension, a waiver, or to remove the use case from operations until the requirements can be met. One-year extensions must be submitted to OMB by October 15, 2024.
• For use cases not yet in operations, M-24-10 requirements must be met before the use case moves into operations unless an extension or waiver is granted.

Inventory Submission and Completion

• As stated in M-24-10, the CAIO has the primary responsibility for coordinating their agency’s use of AI, promoting AI innovation, and managing risks from the use of AI. This includes overseeing agency compliance with requirements to manage risks from the use of AI, as well as the creation and maintenance of the annual AI use case inventory.
• Use case reviewers will include the CAIO team and individuals appointed by VA administrations and offices. It is expected that the majority of VA’s AI use cases will reside in VHA. VHA NAII is developing a distributed governance model for VHA that will involve representatives from the VISNs participating as reviewers and in oversight of AI use cases within their VISN. All use case reviewers will participate in training by the CAIO team and/or the VHA NAII team.
• After the completion of the review process, each administration or office will certify their portion of the inventory is complete and correct to the AI Governance Council.
• The AI Governance Council will act as the final approval body for the annual AI inventory and use case review determinations.