⚡ Free Classes and Scholarships Available for Underprivileged Students -

Scaling online model serving

Serving and Scaling Models

Set 1
10 Questions
No time limit
Practice Mode
0%
Score
0
Correct
0
Incorrect
10
Total Questions
Back to Topics
Question 1 of 10
You deployed a real-time recommendation model on Vertex AI Prediction that requires 50+ features per request. Users are experiencing high latency because features are computed on-demand from BigQuery tables. You need to reduce prediction latency while maintaining feature freshness within 15 minutes. What should you do?
Explanation
Option A (Enable Vertex AI Feature Store with online serving and scheduled batch ingestion) is correct because Feature Store is specifically designed for this use case - it provides low-latency feature retrieval through online serving while maintaining freshness through scheduled batch ingestion from BigQuery. The 15-minute freshness requirement aligns well with batch ingestion schedules. Option D is incorrect because while Memorystore reduces latency, it requires custom engineering to implement feature refresh logic and doesn't provide the feature management capabilities needed for ML serving at scale. Option B is incorrect because denormalizing into a single BigQuery table doesn't solve the fundamental latency problem - BigQuery is designed for analytical queries, not sub-100ms feature lookups required for real-time predictions. Option C is incorrect because running feature computation in a sidecar still requires on-demand calculation from BigQuery, which doesn't reduce the latency problem and adds operational complexity.
Question 2 of 10
Your team deployed a TensorFlow image classification model on Vertex AI Prediction with CPU-only infrastructure. The model serves 100 predictions per second with 200ms average latency. Traffic is expected to increase 10x. Your SLA requires maintaining latency below 150ms. What is the most cost-effective approach?
Explanation
Option D (Enable autoscaling and switch to NVIDIA T4 GPU machine types) is correct because GPUs excel at parallel matrix operations in image classification, providing significantly lower per-prediction latency than CPUs. T4 GPUs offer excellent price-performance for inference workloads, can handle the 10x throughput increase while reducing latency, and autoscaling ensures you only pay for needed capacity. Option A is incorrect because while larger CPU machines increase throughput through parallelism, they don't reduce per-prediction latency for compute-intensive image operations and are more expensive than GPU-based solutions at this scale. Option C is incorrect because TensorFlow Lite is optimized for edge/mobile devices with limited resources; converting and redeploying to Cloud Run adds unnecessary complexity and Cloud Run's CPU-based infrastructure won't achieve the latency requirements for complex image classification at high throughput. Option B is incorrect because multi-region deployment improves availability and geographic latency but doesn't reduce model inference latency, and it significantly increases costs without addressing the computational performance requirement.
Question 3 of 10
You are serving a BERT-based NLP model on Vertex AI Prediction. The model uses 4GB of memory and handles 50 requests per second. You need to reduce serving costs by 40% while maintaining throughput and keeping cold start latency under 10 seconds. What should you do?
Explanation
Option C (Apply model distillation to create a smaller student model with similar accuracy) is correct because distillation can reduce model size by 3-5x while maintaining 95%+ of the original accuracy, directly reducing memory footprint and compute requirements, which lowers costs by reducing machine size needs. The smaller model loads quickly (under 10 seconds) and can maintain the same throughput with fewer resources. Option A is incorrect because switching to batch prediction changes the serving paradigm entirely - it cannot handle real-time requests of 50 QPS and is designed for periodic bulk processing, not online serving. Option B is incorrect because scaling to zero introduces cold start latency much greater than 10 seconds (typically 1-3 minutes for loading a 4GB model), which violates the stated requirement even though it would reduce costs during idle periods. Option D is incorrect because while quantization reduces model size, int8 quantization typically achieves only 4x compression and requires careful validation to avoid accuracy degradation; additionally, TensorFlow Lite is more suitable for edge deployment than cloud serving, and the conversion effort is higher than distillation for BERT models.
Question 4 of 10
Your fraud detection model on Vertex AI Prediction must process credit card transactions with sub-50ms latency. The model receives sensitive PII data that cannot traverse the public internet. You need to configure the deployment to meet security and performance requirements. What should you do?
Explanation
Option B (Deploy the model to a private endpoint within your VPC network) is correct because Vertex AI private endpoints ensure traffic never leaves your VPC, meeting the PII security requirement while providing the low-latency performance needed for fraud detection. Private endpoints integrate directly with your VPC network and support sub-50ms latency requirements. Option D is incorrect because public endpoints, even with VPC Service Controls, still expose traffic to the public internet during the final hop to the endpoint, which doesn't meet the requirement that PII cannot traverse public networks. Option C is incorrect because while Cloud Run with Binary Authorization provides security, it doesn't specifically address the requirement that traffic must not traverse the public internet; Cloud Run endpoints are typically publicly accessible even with authentication. Option A is incorrect because while GKE with Private Google Access keeps cluster traffic private, it requires significantly more operational overhead than Vertex AI private endpoints for model serving, and the question specifically asks for a deployment configuration, not infrastructure management.
Question 5 of 10
You deployed a PyTorch recommendation model that serves 5,000 predictions per second during peak hours and 200 predictions per second during off-peak hours. Currently, you run 50 replicas continuously. You need to optimize costs while ensuring scaling occurs before traffic increases. What should you do?
Explanation
Option D (Configure autoscaling with custom metrics based on request queue depth) is correct because queue-based autoscaling is a leading indicator that scales proactively before latency degrades. When requests queue up, new replicas are provisioned before the existing replicas become overloaded, ensuring the system scales before traffic increases impact users. This is the recommended approach for prediction workloads with variable traffic. Option C is incorrect because CPU-based autoscaling is a lagging indicator - by the time CPU reaches 60%, requests are already being processed slowly. It scales reactively rather than proactively, and the minimum 5 replicas may not handle peak 5,000 QPS effectively. Option B is incorrect because while scheduled scaling based on patterns can reduce costs during known off-peak periods, it cannot respond to unexpected traffic spikes and doesn't scale 'before traffic increases' in dynamic scenarios, making it fragile for production ML serving. Option A is incorrect because memory utilization for inference workloads is typically stable (model weights don't change), making it a poor scaling metric. Memory-based HPA wouldn't trigger scaling based on request volume and wouldn't scale before traffic increases.
Question 6 of 10
You need to deploy a time-series forecasting model that processes IoT sensor data from 10,000 edge devices. Each device sends data every 5 seconds, and predictions must be returned within 100ms. The model is 500MB and requires GPU acceleration. What deployment strategy should you use?
Explanation
Option B (Deploy the model to Vertex AI Prediction with A2 GPU instances and autoscaling) is correct because centralized GPU-based serving on Vertex AI can efficiently handle 10,000 devices × 0.2 requests/second = 2,000 QPS with managed autoscaling, provides the GPU acceleration required by the model, and meets the 100ms latency requirement through optimized infrastructure without the operational complexity of managing edge deployments. Option C is incorrect because deploying a 500MB model (even quantized) to 10,000 edge devices is operationally complex, expensive, and difficult to update. Edge devices may not have GPU capabilities, and model updates require coordinating deployments across all devices. Option D is incorrect because the model explicitly requires GPU acceleration, and Cloud Run only supports CPU instances. CPU-based inference would not meet the 100ms latency requirement for a GPU-optimized time-series model at this scale. Option A is incorrect because while GKE with GPU nodes would work technically, it requires significantly more operational overhead (managing clusters, nodes, deployments) compared to the fully managed Vertex AI Prediction, and the question doesn't indicate a need for the additional customization that would justify GKE's complexity.
Question 7 of 10
Your ensemble model combines predictions from three separate models (XGBoost, LightGBM, and a neural network) hosted on different Vertex AI endpoints. The ensemble logic runs in Cloud Functions, causing 300ms latency. You need to reduce latency to under 100ms. What should you do?
Explanation
Option C (Deploy all models in a single custom container with ensemble logic on Vertex AI) is correct because co-locating all models and ensemble logic in one container eliminates network latency between components, allows models to run in parallel on the same instance, and keeps all computation in a single process. This can reduce latency from 300ms to under 100ms by avoiding multiple network hops. Option D is incorrect because Cloud Run still requires separate network calls to three different Vertex AI endpoints, which is the primary source of latency. Cloud Run may execute slightly faster than Cloud Functions, but the network overhead between services remains and won't achieve the under-100ms requirement. Option B is incorrect because Vertex AI Pipelines are designed for workflow orchestration and training pipelines, not low-latency online serving. Pipelines add orchestration overhead and are inappropriate for real-time prediction scenarios requiring sub-100ms latency. Option A is incorrect because caching predictions defeats the purpose of real-time inference (predictions become stale) and asynchronous combination means predictions aren't available immediately, which doesn't meet the latency requirement for online serving scenarios.
Question 8 of 10
You are serving a computer vision model on Vertex AI Prediction that requires preprocessing images to 224x224 pixels. The preprocessing currently happens client-side, causing inconsistent results. You need to ensure consistent preprocessing while minimizing latency. What should you do?
Explanation
Option A (Add preprocessing as a TensorFlow layer in the SavedModel before deployment) is correct because embedding preprocessing directly in the model ensures consistency, reduces latency by keeping all operations in a single inference call, and leverages GPU/TPU acceleration for preprocessing operations. TensorFlow's preprocessing layers become part of the model graph and export with the SavedModel. Option B is incorrect because while custom prediction routines can include preprocessing, they typically run in Python outside the optimized inference graph, which is slower than preprocessing layers compiled into the model graph and doesn't leverage hardware acceleration as effectively. Option C is incorrect because adding a separate preprocessing service introduces network latency between services, increases operational complexity, and creates an additional point of failure without providing benefits over in-model preprocessing for this use case. Option D is incorrect because Dataflow is designed for batch processing, not real-time online inference. Preprocessing images in Dataflow and storing them doesn't support dynamic, real-time prediction requests where new images arrive continuously and need immediate processing.
Question 9 of 10
Your language model deployment on Vertex AI serves 1,000 requests per second with high variance in input text length (10 to 2,000 tokens). You notice GPU utilization varies between 20% and 95%, and some requests timeout during peaks. You need to improve throughput stability. What should you do?
Explanation
Option D (Implement dynamic batching with variable batch sizes based on token count) is correct because dynamic batching can group requests with similar token lengths together, maximizing GPU utilization while preventing timeouts. By adjusting batch sizes based on total token count rather than request count, you ensure consistent GPU memory usage and processing time, which stabilizes throughput and prevents the underutilization seen with short inputs and overload with long inputs. Option B is incorrect because while A100s have more memory than T4s, the problem isn't insufficient memory but rather inefficient utilization due to varying input sizes. Upgrading hardware is more expensive and doesn't address the root cause of variable GPU utilization. Option C is incorrect because routing to separate endpoints based on input length adds complexity, requires maintaining multiple model replicas, and doesn't actually solve the GPU utilization problem - each endpoint would still experience utilization variance unless combined with batching strategies. Option A is incorrect because preprocessing (tokenization) is typically CPU-bound already, and the GPU utilization variance is caused by the variable-length model inference itself, not preprocessing. This wouldn't address the core issue of inconsistent GPU load during transformer attention operations.
Question 10 of 10
You deployed a model to Vertex AI Prediction that experiences traffic spikes from 100 to 5,000 requests per second within 30 seconds. The current autoscaling configuration has a 3-minute scale-up time, causing request failures during spikes. You need to handle traffic spikes without over-provisioning. What should you do?
Explanation
Option A (Implement request queuing with Cloud Tasks to buffer requests during scale-up) is correct because queuing requests during traffic spikes prevents request failures while autoscaling catches up. Cloud Tasks provides reliable request buffering, allows the system to process requests as capacity becomes available, and avoids over-provisioning for short spikes. This provides a buffer between traffic bursts and serving capacity. Option D is incorrect because setting a high minimum replica count to handle 5,000 QPS means continuously running infrastructure for peak capacity even though spikes are temporary, which violates the 'without over-provisioning' requirement and significantly increases costs during the normal 100 QPS periods. Option C is incorrect because while reducing cool-down and lowering target utilization helps scale faster, it cannot overcome the inherent 3-minute startup time for new prediction replicas (model loading, container initialization). The physical scale-up time remains insufficient for 30-second spikes. Option B is incorrect because GKE doesn't fundamentally solve the scale-up time problem - model containers still need time to initialize and load. Preemptible nodes actually increase risk during spikes due to potential interruptions, and GKE adds operational complexity without addressing the core buffering need.