Enabling Amazon SageMaker Projects with Terraform Cloud

Overview of Amazon SageMaker Projects

AWS Service Catalog Engine for Terraform Cloud

Prerequisites for Deployment

Deployment Steps

Further Customization

Cleanup

Conclusion

About the Author

Empower Your Data Science Teams: Enabling Amazon SageMaker Projects with Terraform Cloud

Amazon SageMaker Projects provide a robust solution for data scientists, enabling them to leverage Amazon Web Services (AWS) tools and infrastructure throughout all stages of the machine learning (ML) lifecycle. By standardizing resources and offering pre-packaged templates, organizations can enhance collaboration among their data science teams. However, for enterprises relying on Terraform to define and manage their infrastructure as code (IaC), integrating SageMaker Projects traditionally required AWS CloudFormation—a dependency that raises governance concerns for many organizations.

This guide outlines a streamlined approach to enable SageMaker Projects directly with Terraform Cloud without the CloudFormation dependency.

AWS Service Catalog Engine for Terraform Cloud

At the heart of integrating SageMaker Projects with Terraform Cloud is the AWS Service Catalog. To eliminate the necessity of CloudFormation, products must be mapped as Terraform products using the AWS Service Catalog Engine (SCE) for Terraform Cloud. This actively maintained module from Hashicorp integrates AWS-native infrastructure, allowing your Service Catalog products to be deployed seamlessly through the Terraform Cloud platform.

By following the steps outlined in this blog, you will learn how to deploy SageMaker Projects directly from Terraform Cloud.

Prerequisites

Before getting started, ensure you have:

• An AWS account: With permissions to create and manage SageMaker Projects and Service Catalog products. Check the Service Catalog documentation for permission details.

• A SageMaker Studio domain: An existing domain linked with a SageMaker user profile, with SageMaker Projects enabled. Use Quick Setup for Amazon SageMaker AI for guidance.

• A Unix terminal: With the AWS Command Line Interface (CLI) and Terraform installed. Refer to installing or updating the AWS CLI and installing Terraform for setup instructions.

• A Terraform Cloud account: Ensure you have the necessary permissions to create and manage workspaces. Get started with HCP Terraform for account setup.

Deployment Steps

1. Clone the Repository
   Clone the sagemaker-custom-project-templates repository from the AWS Samples GitHub to your local machine:

   git clone https://github.com/aws-samples/sagemaker-custom-project-templates.git
   cd sagemaker-custom-project-templates
   git submodule update --init --recursive
   cd mlops-terraform-cloud

   The above code will create a Service Catalog portfolio, add the SageMaker Project template to the portfolio, allow SageMaker Studio roles to access this product, and include necessary tags for visibility in SageMaker Studio. For further details, see the Create Custom Project Templates documentation.

2. Login to Terraform Cloud
   Sign into your HCP account, generating a security token. Copy this token back into your terminal.

3. Retrieve the SageMaker User Role ARN
   Navigate to your AWS account to retrieve the ARN of the SageMaker user profile linked to your SageMaker Studio domain. To do this:

   • In the AWS Management Console for Amazon SageMaker, click on "Domains" in the navigation pane.
   • Select your studio domain.
   • Under "User Profiles," click on your user profile.
   • Copy the ARN from the User Details section.

4. Create a tfvars File
   Create a tfvars file for your Terraform Cloud workspace:

   cp terraform.tfvars.example terraform.tfvars

   Update the file with the required values:

   tfc_organization = "my-tfc-organization"
   tfc_team = "aws-service-catalog"
   token_rotation_interval_in_days = 30
   sagemaker_user_role_arns = ["arn:aws:iam::XXXXXXXXXXX:role/service-role/AmazonSageMaker-ExecutionRole"]

   Ensure your Terraform Cloud organization has the appropriate entitlements and that your tfc_team is unique.

5. Initialize and Apply the Terraform Cloud Workspace
   Run commands to initialize the workspace and apply your changes.

6. Create the Project
   Go back to the SageMaker console using the user profile linked to the SageMaker user role ARN. Navigate to "Projects" under "Deployments," then choose "Create project." Select the mlops-tf-cloud-example product and provide a unique name and optional description for your new project.

7. Verify Workspace Provisioning
   Open another tab for your Terraform Cloud account’s Workspaces. You should see a workspace provisioning directly from your SageMaker Project deployment.

Further Customization

The example provided can be tailored to include custom Terraform configurations in your SageMaker Project template. Modify your Terraform in the mlops-product/product directory and compress it for deployment:

cd mlops-product
tar -czf product.tar.gz product

Cleanup

To delete the resources deployed through this example, run the applicable command from the project directory.

Conclusion

Congratulations! You have successfully defined, deployed, and provisioned a SageMaker Project custom template entirely in Terraform, free from dependencies on other IaC tools. Now, you can enable SageMaker Projects strictly within your Terraform Enterprise infrastructure, fostering a more organized and compliant data science practice.

About the Author

Max Copeland is a Machine Learning Engineer for AWS, specializing in customer engagements spanning MLOps, data science, data engineering, and generative AI.

By following this guide, you’re taking a significant step towards enhancing the efficiency and governance of your ML projects with Terraform Cloud and Amazon SageMaker. Happy coding!