Introducing SageMaker Core SDK: Simplifying Machine Learning Lifecycle Management

Introducing SageMaker Core SDK: Streamlining the ML Lifecycle

We’re thrilled to announce the launch of SageMaker Core, a new Python SDK from Amazon SageMaker that aims to simplify the management of the machine learning (ML) lifecycle. This new SDK offers an object-oriented approach, streamlining data processing, training, and inference tasks with features like resource chaining, intelligent defaults, and enhanced logging capabilities. With SageMaker Core, handling ML workloads on SageMaker becomes more efficient and user-friendly. The SageMaker Core SDK is included as part of the SageMaker Python SDK version 2.231.0 and above.

In this blog post, we will explore how the SageMaker Core SDK enhances the developer experience by providing APIs for executing various steps in a general ML lifecycle. We will also discuss the key benefits of using this SDK and provide resources for further learning.

The Challenge of Traditional Approaches

Historically, developers had two primary options when working with SageMaker: the AWS SDK for Python (boto3) or the SageMaker Python SDK. While both these options provided robust APIs for ML lifecycle management, they often required the use of loosely typed constructs like hard-coded constants and JSON dictionaries that mimicked a REST interface. For example, creating a training job using boto3 involved crafting a verbose JSON dictionary with potential for errors or typos.

Similarly, the SageMaker Python SDK required the creation of an estimator object and invoking the fit() method, which might not be intuitive for all developers. This approach lacked the benefits of object-oriented programming, making it challenging to map estimator concepts to the actions needed to train a model.

Introducing SageMaker Core SDK

SageMaker Core SDK addresses these challenges by replacing lengthy dictionaries with object-oriented interfaces, allowing developers to work with abstractions that are easier to manage and utilize. The key features of SageMaker Core include:

• Object-oriented interface: Providing classes for processing, training, and deployment tasks with strong type checking
• Resource chaining: Passing SageMaker resources as objects to different resources
• Abstraction of low-level details: Handling resource state transitions and polling logic
• Intelligent defaults: Setting default values for parameters like AWS roles and VPC configurations
• Auto code completion: Offering real-time suggestions in IDEs
• Full parity with SageMaker APIs: Access to all SageMaker capabilities including generative AI
• Comprehensive documentation and type hints

Using SageMaker Core SDK for a generative AI lifecycle involving data preparation, fine-tuning, and deployment showcases the efficiency and readability it brings to the development process.

Getting Started with SageMaker Core SDK

To begin using SageMaker Core, ensure Python 3.8 or later is installed. Depending on your current setup, you can either install the sagemaker-core SDK or upgrade your existing SageMaker Python SDK to version 2.231.0 or higher.

The core SDK simplifies ML workload management with steps like data preparation, training, and deployment, showcasing the benefits of resource chaining, intelligent defaults, and streamlined development.

Conclusion

SageMaker Core SDK revolutionizes the ML development experience by providing a user-friendly, object-oriented interface for managing ML workloads on SageMaker. With features like resource chaining, intelligent defaults, and enhanced readability, developers can focus on building and deploying models without the burden of complex JSON structures.

Explore SageMaker Core SDK today and discover a more efficient and streamlined approach to ML lifecycle management.

About the Authors

Vikesh Pandey is a Principal GenAI/ML Specialist Solutions Architect at AWS with over a decade and a half of experience in ML and software engineering. He enjoys exploring different cuisines and playing outdoor sports in his free time.

Shweta Singh is a Senior Product Manager in the Amazon SageMaker Machine Learning platform team at AWS, leading the SageMaker Python SDK. With a background in Computer Engineering and Financial Engineering, she has extensive experience in product roles at Amazon.