1. AI Governance and Risk
A. AI Models, Considerations, and Requirements
Types of AI
- Generative
- Predictive
- Narrow
- General
Machine Learning/AI Models
- Basic models
- Neural networks
Algorithms
- Classes of algorithms
- Additional AI considerations (technical terms and
- concepts relevant to the IS auditor)
AI Lifecycle Overview
- Plan and design
- Collect and process data
- Build and/or adapt model(s)
- Test, evaluate, verify, and validate
- Make available for use/deploy
- Operate and monitor
- Retire/decommission
Business Considerations
- Business use cases, needs, scope, and objectives
- Cost-benefit analysis
- Return on investment
- Internal vs. cloud hosting
- Vendors
- Shared responsibility
B. AI Governance and Program Management
AI Strategy
- Strategies
- Opportunities
- Vision and mission
- Value alignment
AI-Related Roles and Responsibilities
- Categories, focuses, and common examples
AI-Related Policies and Procedures
AI-Related Policies and Procedures
- Skills, knowledge, and competencies
Program Metrics
- Examples of metrics with objectives and definitions
C. AI Risk Management
AI-Related Risk Identification
- AI threat landscape
- AI risks
- Challenges for AI risk management
Risk Assessment
- Risk assessment
- Risk appetite and tolerance
- Risk mitigation and prioritization
- Remediation plans/best practices
Risk Monitoring
- Continuous improvement
- Risk and performance metrics
D. Privacy and Data Governance Programs
Data Governance
- Data classification
- Data clustering
- Data licensing
- Data cleansing and retention
Privacy Considerations
- Data privacy
- Data ownership (governance and privacy)
Privacy Regulatory Considerations
- Data consent
- Collection, use, and disclosure
E. Leading Practices, Ethics, Regulations, and Standards for AI Standards, Frameworks, and Regulations Related to AI
- Best practices
- Industry standards and frameworks
- Laws and regulations
Ethical Considerations
- Ethical use
- Bias and fairness
- Transparency and explainability
- Trust and safety
- IP considerations
- Human rights
2. AI Operations
A. Data Management Specific To AI
Data Collection
- Consent
- Fit for purpose
- Data lag
Data Classification
Data Confidentiality
Data Quality
Data Balancing
Data Scarcity
Data Security
- Data encoding
- Data access
- Data secrecy
- Data replication
- Data backup
B. AI Solution Development Methodologies and Lifecycle
AI Solution Development Life Cycle
- Use case development
- Design
- Development
- Deployment
- Monitoring and maintenance
- Decommission
Privacy and Security by Design
- Explainability
- Robustness
C. Change Management Specific To AI
Change Management Considerations
- Data dependency
- AI model
- Regulatory and societal impact
- Emergency changes
- Configuration management
D. Supervision of AI Solutions
AI Agency
- Logging and monitoring
- AI observability
- Human in the Loop (HITL)
- Hallucination
E. Testing Techniques for AI Solutions
Conventional Software Testing Techniques
- A/B testing
- Unit and integration testing
- Objective verification
- Code reviews
- Black box testing
AI-Specific Testing Techniques
- Model cards
- Bias testing
- Adversarial testing
F. Threats and Vulnerabilities Specific To AI
Types of AI-Related Threats
- Training data leakage
- Data poisoning
- Model poisoning
- Model theft
- Prompt injections
- Model evasion
- Model inversion
- Threats for using vendor supplied AI
- AI solution disruption
Controls for AI-Related Threats
- Threat and vulnerability identification
- Prompt templates
- Defensive distillation
- Regularization
G. Incident Response Management Specific To AI
Prepare
- Policies, procedures, and model documentation
- Incident response team
- Tabletop exercises
Identify and Report
Assess
Respond
- Containment
- Eradication
- Recovery
Post-Incident Review
3. AI Auditing Tools and Techniques
A. Audit Planning and Design
Identification of AI Assets and Controls
- Inventory objective and procedure
- Inventory and data gathering methods
- Documentation
- Surveys
- Interviews
Types of AI Controls
- Examples including control categories, controls, and explanations
Audit Use Cases
- Large language models
- Audit process improvement
- Generative AI
- Audit-specific AI applications
Internal Training for AI Use
- Key components for auditor knowledge
- Practical skills development
B. Audit Testing and Sampling Methodologies
Designing an AI Audit
- AI audit objectives
- Audit scoping and resources
AI Audit Testing Methodologies
- AI systems overall testing
- Financial models
AI Sampling
- Judgmental sampling
- AI sampling
Outcomes of AI Testing
- Reduce false positives
- Reduce workforce needs
- Outliers
C. Audit Evidence Collection Techniques
Data Collection
- Training and testing data
- Unstructured and structured data collection
- Extract, transform, and load
- Data manipulation
- Scraping
Walkthroughs and Interviews
- Design interview questions
AI Collection Tools
- Using AI to collect logs
- AI agents to create outputs
- Voice to speech
- Optimal character recognition
D. Audit Data Quality and Data Analytics
Data Quality
Data Analytics
- Sentiment analysis
- Run data analytics
Data Reporting
E. AI Audit Outputs and Reports
Reports
- Report types (examples and details)
- Advisory reports
- Charts and visualizations
Audit Follow-up
Quality Assurance