What should the data scientist do to identify and address training issues with the LEAST development effort?
Use CPU utilization metrics that are captured in Amazon CloudWatch. Configure a CloudWatch alarm to stop the training job early if low CPU utilization occurs.
Use high-resolution custom metrics that are captured in Amazon CloudWatch. Configure an AWS Lambda function to analyze the metrics and to stop the training job early if issues are detected.
Use the SageMaker Debugger vanishing_gradient and LowGPUUtilization built-in rules to detect issues and to launch the StopTrainingJob action if issues are detected.
Use the SageMaker Debugger confusion and feature_importance_overweight built-in rules to detect issues and to launch the StopTrainingJob action if issues are detected.
Explanations:
CPU utilization metrics do not directly indicate GPU or model training issues. CPU utilization may not provide relevant insights into GPU-bound training problems, so using it to stop the training job early is not an optimal solution.
High-resolution custom metrics and AWS Lambda functions add unnecessary complexity. The built-in rules in SageMaker Debugger provide a simpler and more direct approach to identifying training issues.
The SageMaker Debugger’s built-in rules, such as vanishing_gradient and LowGPUUtilization, are specifically designed to detect common issues like training not converging or suboptimal GPU usage. The StopTrainingJob action automatically stops the job when these issues are detected, requiring minimal development effort.
Confusion and feature_importance_overweight are not directly relevant to the issue of training convergence or GPU utilization. These rules focus on other aspects like model evaluation and feature importance, not on the GPU utilization or training convergence.
From my point of view, the answer is:
Use the SageMaker Debugger vanishing_gradient and LowGPUUtilization built-in rules to detect issues and to launch the StopTrainingJob action if issues are detected.
If I had to guess, I’d say the answer is:
Use the SageMaker Debugger vanishing_gradient and LowGPUUtilization built-in rules to detect issues and to launch the StopTrainingJob action if issues are detected.