As Artificial Intelligence (AI) and Machine Learning (ML) become core drivers of modern software, traditional quality assurance approaches are no longer sufficient. Evaluating an AI-based system requires moving beyond conventional functional testing into a multi-dimensional framework that addresses non-determinism, adaptability, and complex safety challenges.

In the ISTQB® Certified Tester AI Testing (CT-AI) v2.0 Syllabus, Chapter 2 establishes a standardized quality model based on ISO/IEC 25059—an extension of the widely known ISO/IEC 25010 software quality standard.

1. What is ISO/IEC 25059?

Traditional quality models (like ISO/IEC 25010) measure standard software attributes such as performance efficiency, usability, and reliability. However, ISO/IEC 25059 extends these models specifically for AI-based systems.

It evaluates AI systems from two main perspectives:

1. Product Quality: The internal and structural characteristics of the system.
2. Quality in Use: How the system impacts users, stakeholders, and society during real-world operation.

2. Key AI-Specific Quality Characteristics

ISO/IEC 25059 introduces modified and brand-new sub-characteristics tailored to AI behavior:

🛡️ AI Robustness (Product Quality – Sub-characteristic of Reliability)

• Definition: The degree to which an AI system maintains its required level of AI functional correctness under challenging or adverse conditions.
• Key Focus: Handling biased data, adversarial attacks, noisy inputs, hardware/environmental disruptions, and operator errors.

🔄 Functional Adaptability (Product Quality – Sub-characteristic of Functionality)

• Definition: The ability of an AI system to learn, adjust, and optimize its functional performance based on continuous operational data or dynamic environments.

🕹️ User Controllability (Product Quality – Sub-characteristic of Usability/Interaction)

• Definition: The extent to which users or human operators can guide, override, or manage the behavior of the AI system during its execution.

🔍 Transparency (Quality in Use – Sub-characteristic of Satisfaction)

• Definition: The degree to which adequate, clear, and context-appropriate information about the AI system’s internal workings, limitations, and predictions is communicated to stakeholders.

🛑 Intervenability (Product Quality – Sub-characteristic of Security)

• Definition: The extent to which a human operator can safely and promptly intervene in the AI system’s operation to prevent harm, hazards, or catastrophic failures.

⚖️ Societal and Ethical Risk Mitigation (Quality in Use – Sub-characteristic of Freedom from Risk)

• Definition: Strategies and mechanisms designed to reduce harmful societal impacts and maintain alignment with human values.
• Covers: Fairness and non-discrimination, privacy protection, accountability, human-centered design, labor practices, and environmental sustainability.

3. AI and Safety: Unique Challenges in High-Risk Domains

When AI is deployed in safety-critical domains (such as autonomous driving, healthcare, or industrial robotics), failure can lead to severe injury, environmental harm, or loss of human life.

While traditional safety engineering relies heavily on predictable, deterministic software behavior, AI-based systems introduce critical challenges:

• Vague Specifications: AI models rely heavily on data rather than strict logic rules, making it difficult to write complete formal specifications.
• Non-Determinism & Unpredictability: Probabilistic outputs make system behavior harder to verify completely across all possible conditions.
• Self-Learning Behavior: Continuous update mechanisms post-deployment mean system logic evolves over time, shifting the operational definition of correctness.
• Limited Explainability: “Black box” deep neural network models make root-cause analysis and safety validation challenging.
• Regulatory Landscape: Strict frameworks like the EU AI Act explicitly label AI used in safety components as high-risk, establishing demanding standards for verification, validation, and risk management.

Conclusion: Rethinking the Tester’s Role

Testing AI systems requires a fundamental shift in mindset. Quality assurance is no longer just about pass/fail assertion checks—it involves defining statistical threshold acceptance criteria, evaluating bias and ethics, testing against adversarial conditions, and ensuring rapid human intervenability.