Announcing Fine-Grained Compute and Memory Quota Allocation with HyperPod Task Governance

Optimize Amazon SageMaker HyperPod Cluster Utilization

Enhancing Resource Management for Teams and Projects in Amazon EKS

Introduction to Compute Quota Management

Utilize GPUs Efficiently Across Teams

Key Features of HyperPod Task Governance

Maximizing Resource Utilization: Insights from Industry Leaders

A Deep Dive into Quota Allocation Strategies

Getting Started: Prerequisites for Implementation

Step-by-Step Guide for AWS Console Resource Allocation

Using AWS CLI for Compute Quota Management

Understanding HyperPod Task Governance and Kueue Integration

Submitting Tasks: Your Options Explained

Common Use Cases for Granular Resource Allocation

Conclusion: Leveraging HyperPod for Optimized ML Workloads

About the Authors: Meet the Team Behind HyperPod Task Governance

Unlocking the Power of Fine-Grained Compute and Memory Quota Allocation with HyperPod Task Governance

We are thrilled to announce the general availability of fine-grained compute and memory quota allocation through HyperPod task governance. This new capability allows organizations to optimize the utilization of Amazon SageMaker HyperPod clusters on Amazon Elastic Kubernetes Service (EKS). With this feature, customers can achieve fair usage distribution and support efficient resource allocation across diverse teams or projects.

What is Compute Quota Management?

Compute quota management is an administrative tool that controls resource limits across users, teams, and projects. It ensures fair distribution of computational resources, preventing any single entity from monopolizing these shared capabilities. This is especially crucial in a budget-sensitive environment, where multiple teams may require access to limited compute resources.

For example, a data scientist may need four H100 GPUs for model development but not the entire instance’s compute capacity. In other instances, a company may want to distribute limited resources across various teams to ensure no idle capacity goes unused.

Key Features of HyperPod Task Governance

Granular Resource Allocation

With HyperPod task governance, administrators gain the ability to allocate granular GPU, vCPU, and memory resources to teams and projects based on their specific needs. Notable features include:

• GPU-level quota allocation: Allocate quotas based on instance type, family, or hardware type, supporting both Trainium and NVIDIA GPUs.
• Fine-tuned CPU and memory allocation: Control CPU and memory usage to meet unique project requirements.
• Priority levels: Set team priorities for fair-share idle compute allocation, ensuring critical workloads receive the necessary resources first.

Quote from Industry Expert:
“With a wide variety of frontier AI data experiments and production pipelines, maximizing SageMaker HyperPod Cluster utilization is extremely high impact. This requires fair and controlled access to shared resources like state-of-the-art GPUs, granular hardware allocation, and more. HyperPod task governance is exactly what this is built for.”
— Daniel Xu, Director of Product at Snorkel AI

Workflow Overview

Prerequisites

To implement this feature, you need:

1. A local environment (machine or cloud-based) to run HyperPod CLI and kubectl commands.
2. The HyperPod Training Operator installed in your cluster.

Allocating Quota via the AWS Console

Administrators will use the AWS Management Console for managing cluster compute allocation.

1. Log in to the AWS Management Console and navigate to Cluster Management under HyperPod Clusters in the Amazon SageMaker AI console.
2. Select your HyperPod cluster and choose the Policies tab.
3. Go to the Compute allocations section and click Create.

You can enable task prioritization and resource sharing through cluster policies, allowing for critical workload prioritization and idle compute distribution among teams.

Allocating Quota via AWS CLI

The compute quota can also be set using the AWS CLI. Here’s a simple command example to create a GPU-only quota:

aws sagemaker create-compute-quota \
--region <your-region> \
--name "only-gpu-quota" \
--cluster-arn "arn:aws:sagemaker:your-cluster-arn" \
--description "test description" \
--compute-quota-config "ComputeQuotaResources=[{InstanceType=ml.g6.12xlarge,Accelerators=2}]" \
--activation-state "Enabled" \
--compute-quota-target "TeamName=onlygputeam,FairShareWeight=10"

This allows for mixed quota types and specific settings according to your organization’s requirements.

Under the Hood: How HyperPod Task Governance Works

HyperPod task governance integrates with Kueue, a Kubernetes-native system designed for job queueing. Kueue interacts with the kube-scheduler to admit workloads based on resource quotas, while managing pod placement across nodes effectively.

When a compute allocation policy is established, Kueue creates ClusterQueues and defines resource quotas alongside scheduling policies. This integration ensures that compute resources are efficiently governed and allocated according to the defined framework.

Submitting Tasks: The Data Scientist’s Perspective

Data scientists can seamlessly submit tasks using either the HyperPod CLI or kubectl. Here’s a quick overview of both methods:

Using HyperPod CLI

The HyperPod CLI simplifies command submission:

hyp create hyp-pytorch-job \
--job-name sample-job1 \
--image <your-image>.dkr.ecr..amazonaws.com/: \
--instance-type ml.g5.8xlarge \
--vcpu 4 \
--memory 1 \
--namespace hyperpod-ns-team1

Using Kubectl

Alternatively, a job can be submitted using kubectl with a specified namespace and priority class:

apiVersion: batch/v1
kind: Job
metadata:
  name: gpu-training-job
  namespace: hyperpod-ns-team1
spec:
  ...

Common Use Cases

Fine-grained allocation proves especially useful for tasks such as:

1. Fine-tuning Language Models: Small to medium-sized models don’t always require a full instance. Allocating individual GPUs enhances utilization.
2. Efficient Inference Workloads: Teams can run their models without monopolizing entire instances, maximizing resource availability.
3. Experimentation: Data scientists can utilize IDEs hosted on HyperPod for quicker iterations without needing full-instance power.

Conclusion

The roll-out of fine-grained compute quota allocation within SageMaker HyperPod represents a transformative leap in machine learning infrastructure management. Organizations can now tailor their resource distribution to meet specific team needs while avoiding waste.

Whether fine-tuning models, conducting experiments, or running inference workloads, this enhanced capability ensures resources are allocated just in time for maximum efficiency. To explore these features, visit the SageMaker HyperPod product page and dive into the HyperPod task governance documentation.

Ready to optimize your AI projects? Try it out in the Amazon SageMaker AI console and share your experiences with us!