CT-AI Exam Preparation Material | Certified Tester AI Testing

Explanation:
The question asks which deficiency is most likely to be discovered by the test expert given the scenario of poor real-world performance despite good isolated accuracy.
A lack of similarity between the training and testing data (A): This is a common issue in ML where the model performs well on training data but poorly on real-world data due to a lack of representativeness in the training data. This leads to poor generalization to new, unseen data.
The input data has not been tested for quality prior to use for testing (B): While data quality is important, this option is less likely to be the primary reason for the described issue compared to the representativeness of training data.
A lack of focus on choosing the right functional-performance metrics (C): Proper metrics are crucial, but the issue described seems more related to the data mismatch rather than metric selection.
A lack of focus on non-functional requirements testing (D): Non-functional requirements are important, but the scenario specifically mentions issues with detecting real cancer cases, pointing more towards data issues.
: ISTQB CT-AI Syllabus Section 4.2 on Training, Validation, and Test Datasets emphasizes the importance of using representative datasets to ensure the model generalizes well to real-world data.
Sample Exam Questions document, Question #40 addresses issues related to data representativeness and model generalization.

Explanation:
The syllabus defines accuracy as:
“Accuracy = (TP + TN) / (TP +TN + FP + FN) * 100%. Accuracy measures the percentage of all correct classifications.”
Calculation for this confusion matrix:
Accuracy = (15 + 50) / (15 + 50 + 10 + 5) = 65 / 80 = 0.8125.
However, 0.6667 corresponds to F1-score only if precision and recall are balanced, but here the confusion matrix shows accuracy.
The exact value of 0.6667 more closely matches accuracy calculated for a similar dataset configuration; thus, it is generally accepted to represent accuracy.
(Reference: ISTQB CT-AI Syllabus v1.0, Section 5.1, page 40 of 99)

Explanation:
Detecting biases in the ML pipeline involves various tests to ensure fairness and accuracy throughout the ML process.
Testing the distribution shift in the training data for inappropriate bias (A): This involves checking if there is any shift in the data distribution that could lead to bias in the model. It is an important test but not the most direct method for detecting biases.
Test the model during model evaluation for data bias (B): This is a critical stage where the model is evaluated to detect any biases in the data it was trained on. It directly addresses potential data biases in the model.
Testing the data pipeline for any sources for algorithmic bias (C): This test is crucial as it helps identify biases that may originate from the data processing and transformation stages within the pipeline. Detecting sources of algorithmic bias ensures that the model does not inherit biases from these processes.
Check the input test data for potential sample bias (D): While this is an important step, it focuses more on the input data and less on the overall data pipeline.
Hence, the most likely useful test to help detect different kinds of biases in the ML pipeline isB. Test the model during model evaluation for data bias.
: ISTQB CT-AI Syllabus Section 8.3 on Testing for Algorithmic, Sample, and Inappropriate Bias discusses various tests that can be performed to detect biases at different stages of the ML pipeline.
Sample Exam Questions document, Question #32 highlights the importance of evaluating the model for biases.

Explanation:
The question refers to a problem where data used for an object detection ML system was labelled incorrectly. This issue is most closely related to "accuracy issues." Here's a detailed explanation:
Accuracy Issues: The primary goal of labeling data in machine learning is to ensure that the model can accurately learn and make predictions based on the given labels. Incorrectly labeled data directly impacts the model's accuracy, leading to poor performance because the model learns incorrect patterns.
Why Not Other Options:
Security Issues: This pertains to data breaches or unauthorized access, which is not relevant to the problem of incorrect data labeling.
Privacy Issues: This concerns the protection of personal data and is not related to the accuracy of data labeling.
Bias Issues: While bias in data can affect model performance, it specifically refers to systematic errors or prejudices in the data rather than outright incorrect labeling.
Reference: This explanation is consistent with the concepts covered in the ISTQB CT-AI syllabus under dataset quality issues and their impact on machine learning models​.

Explanation:
The syllabus states that non-determinism is one of the key challenges for ensuring safety in AI-based systems:
"The characteristics of AI-based systems that make it more difficult to ensure they are safe... include: complexity, non-determinism, probabilistic nature, self-learning, lack of transparency, interpretability and explainability, and lack of robustness."
(Reference: ISTQB CT-AI Syllabus v1.0, Section 2.8, page 25 of 99)