⚡ Free Classes and Scholarships Available for Underprivileged Students -

Monitoring, testing, and troubleshooting ML solutions

Monitoring ML Solutions

Set 1
10 Questions
No time limit
Practice Mode
0%
Score
0
Correct
0
Incorrect
10
Total Questions
Back to Topics
Question 1 of 10
You deployed a credit scoring model to Vertex AI Predictions two months ago. Recent customer complaints suggest the model is rejecting valid applications. You need to identify whether the model's input data distribution has changed since training. What should you do?
Explanation
Option C (Enable Vertex AI Model Monitoring with skew detection) is correct because skew detection specifically compares the statistical distribution of serving data against the baseline training data to identify distribution shifts. This directly addresses the need to detect whether input data has changed since training. Option D is incorrect because while Cloud Logging captures requests, manually analyzing patterns with SQL doesn't provide automated statistical comparison of distributions between training and serving data. Option A is incorrect because Explainable AI helps understand individual predictions but doesn't detect distribution changes across the entire serving dataset. Option B is incorrect because latency and error rate metrics don't detect input data distribution changes—the model could be performing normally from a technical standpoint while receiving different data.
Question 2 of 10
Your fraud detection model shows declining precision in production despite maintaining high accuracy. The training dataset was balanced, but production data has only 2% fraud cases. Which monitoring approach best identifies this issue?
Explanation
Option B (Analyze model performance separately for each class) is correct because class-imbalanced scenarios require per-class metrics to identify problems. A model predicting mostly negative cases can have high accuracy but poor precision on the minority class, which becomes evident only when examining class-specific confusion matrix metrics like true positives, false positives, and per-class precision/recall. Option C is incorrect because while baseline comparison is valuable, the key issue is understanding class-specific performance, not just comparing aggregate metrics to a baseline. Option D is incorrect because feature importance drift tracks which features are influential, not whether the model performs differently across imbalanced classes. Option A is incorrect because skew detection identifies input distribution changes, but the problem here is about evaluating model performance on imbalanced classes, not detecting distribution shifts.
Question 3 of 10
You need to detect when your image classification model's feature importance changes over time in production. The model was trained with integrated gradients. Which Vertex AI capability should you configure?
Explanation
Option A (Model Monitoring with feature attribution drift detection) is correct because Vertex AI Model Monitoring includes feature attribution drift detection that automatically monitors changes in feature importance over time by computing attribution scores on serving traffic and comparing them to baseline attributions. This directly addresses the requirement to detect feature importance changes in production. Option D is incorrect because prediction drift monitors output distributions, not feature importance or attribution changes. Option B is incorrect because while batch Explainable AI provides attributions, it requires manual scheduling and comparison rather than automated continuous monitoring of attribution drift. Option C is incorrect because confidence scores per class monitor prediction uncertainty and distribution, not feature importance or which features drive predictions.
Question 4 of 10
Your regression model predicts housing prices. After deployment, you notice predictions are consistently 15% higher than actual values, though training metrics showed good performance. What type of issue are you experiencing?
Explanation
Option D (Training-serving skew caused by preprocessing differences) is correct because when a model performs well during training but shows systematic prediction errors immediately after deployment, this typically indicates training-serving skew—differences in how data is processed or transformed between training and serving environments. The consistent 15% overestimation suggests a systematic preprocessing issue. Option B is incorrect because concept drift refers to changes in the underlying relationship between features and target over time, not immediate post-deployment issues. Option C is incorrect because feature attribution drift tracks changes in feature importance, not systematic prediction bias. Option A is incorrect because data drift refers to changes in input feature distributions over time, but the immediate systematic bias points to environmental differences rather than evolving data distributions.
Question 5 of 10
You're configuring Vertex AI Model Monitoring for a deployed model. You want to detect both distribution changes in input features and shifts in prediction outputs. What monitoring capabilities should you enable?
Explanation
Option B (Enable both skew and drift detection) is correct because skew detection monitors input feature distribution changes against training data baselines, while drift detection monitors both input and prediction distributions over time in production. Together, they provide comprehensive monitoring for both input distribution changes and prediction output shifts. Option A is incorrect because drift detection alone monitors changes over time but doesn't compare against training baselines, missing potential training-serving skew. Option C is incorrect because skew detection only compares serving to training data but doesn't monitor ongoing temporal changes in production, missing drift over time. Option D is incorrect because monitoring only prediction drift misses input feature distribution changes, which can be early indicators of problems before predictions are affected.
Question 6 of 10
Your deployed TensorFlow model returns inconsistent predictions for identical inputs when called multiple times. Training validation showed stable results. What is the most likely cause you should investigate first?
Explanation
Option D (Dropout or batch normalization not in inference mode) is correct because these layers behave differently during training versus inference. Dropout randomly drops neurons during training but should be disabled during serving, and batch normalization uses batch statistics during training but population statistics during inference. Failure to properly set inference mode causes non-deterministic predictions for identical inputs. Option C is incorrect because while preprocessing issues can cause problems, they typically produce consistent wrong results rather than varying results for truly identical inputs. Option A is incorrect because checkpoint corruption would cause consistent errors or failures, not varying predictions. Option B is incorrect because the question specifies identical inputs, so distribution shifts between requests don't apply.
Question 7 of 10
You need to establish continuous evaluation metrics for a model that predicts customer churn. The business requires alerting when model performance degrades below acceptable thresholds. Which approach should you implement?
Explanation
Option C (Configure Model Monitoring with ground truth and thresholds) is correct because Vertex AI Model Monitoring can continuously evaluate model performance when ground truth labels become available, computing metrics like accuracy, precision, and recall, and triggering alerts when thresholds are breached. This directly satisfies the requirement for continuous evaluation with alerting. Option B is incorrect because comparing predictions against historical distributions detects drift but doesn't evaluate actual model performance, which requires ground truth labels. Option D is incorrect because champion-challenger testing compares models but doesn't provide continuous evaluation of a single model's performance over time. Option A is incorrect because Explainable AI tracks feature attributions, not model performance metrics like accuracy or precision.
Question 8 of 10
During training, your model achieves 95% accuracy, but in production it only reaches 78% accuracy. Training and serving use identical preprocessing code. What should you investigate as the most likely cause?
Explanation
Option A (Distribution shift between training and production data) is correct because when preprocessing is identical and the model performs well in training but poorly in production, the most common cause is that the production data comes from a different distribution than the training data. This is especially true when the gap is significant (95% vs 78%). Option D is incorrect because data leakage would typically be caught during validation with held-out test sets showing poor performance, not just in production. Option C is incorrect because overfitting usually shows up in training vs validation metrics, and the question implies training accuracy includes validation. Option B is incorrect because underfitting would show poor performance during training, not high training accuracy followed by production degradation.
Question 9 of 10
You want to compare your complex ensemble model's production performance against a simpler logistic regression baseline to justify the additional complexity. Which monitoring strategy should you implement?
Explanation
Option B (Deploy both models and track metrics side-by-side) is correct because comparing performance against baseline models requires running both in production (or simulating the baseline on production data) and monitoring their respective performance metrics on identical data. Custom dashboards allow direct comparison of metrics like accuracy, latency, and business KPIs. Option C is incorrect because drift detection monitors one model's changes over time, not comparative performance between different models. Option A is incorrect because comparing feature attributions shows how models make decisions differently but doesn't compare their actual performance metrics. Option D is incorrect because A/B testing with traffic splitting serves different data to each model, making direct performance comparison less reliable; parallel evaluation on the same data is preferred for baseline comparison.
Question 10 of 10
Your model serving endpoint returns HTTP 500 errors for 5% of requests. Cloud Logging shows 'InvalidArgument: Feature name mismatch' errors. What troubleshooting step should you take first?
Explanation
Option C (Compare feature schema against expected input signature) is correct because the error message 'Feature name mismatch' explicitly indicates that the features provided in serving requests don't match what the model expects. The first troubleshooting step is to verify the schema by comparing request features against the model's input signature to identify missing, renamed, or extra features. Option D is incorrect because resource constraints would cause timeout or overload errors, not feature name mismatches. Option A is incorrect because while preprocessing updates might eventually be needed, you must first identify the specific mismatch before knowing what to fix. Option B is incorrect because validation would reject requests but doesn't help troubleshoot the root cause of the mismatch—you need to identify what's different first.