Enhancements in Amazon SageMaker Feature Store: New Capabilities for Cost-Efficient and Secure ML Feature Management

Introduction

Explore the latest features in Amazon SageMaker Feature Store, designed to streamline machine learning feature management while addressing cost and security challenges.

Key Challenges Addressed

Discuss the operational hurdles organizations face as they scale ML platforms, particularly around data security and storage costs.

New Features in SageMaker Python SDK v3.8.0

Highlight the three brand-new capabilities that tackle the challenges identified earlier.

Native AWS Lake Formation Integration

Learn how to effortlessly enforce access control on feature data during feature group creation.

Advanced Apache Iceberg Table Properties

Discover how to manage metadata accumulation and control costs with new table properties.

Feature Store Support in SDK v3

Understand the modular improvements and capabilities introduced in the modernized SageMaker Python SDK v3.8.0.

Prerequisites for Implementation

Outline the requirements to effectively utilize the new features.

Solution Overview

Provide an overview of how the new parameters in the SDK facilitate automatic access control and metadata lifecycle management.

Features of SageMaker Python SDK v3

Delve into the comprehensive capabilities of the improved SDK.

Quick Start with SDK v3

Get started quickly by creating a feature group with Lake Formation and Iceberg parameters.

Governance with Lake Formation Integration

Explain the streamlined process for enabling Lake Formation access control.

Code Example

Provide a code snippet showcasing the activation of Lake Formation access.

Key Considerations

Discuss important points to keep in mind when implementing Lake Formation access control.

Managing Offline Store with Iceberg Table Properties

Gain insights into managing your offline store’s metadata lifecycle effectively.

The Solution

Present the new Iceberg properties that enhance metadata management.

Code Example

Showcase how to create an Iceberg-format feature group with lifecycle settings.

Best Practices

Offer tips for optimizing storage and performance while working with high-frequency writes.

Putting It All Together

Demonstrate how to create a feature group that is both governed and cost-optimized with one command.

Cleanup

Remind users to delete test feature groups and deregister resources to avoid charges.

Conclusion

Summarize the benefits of the new features that make Amazon SageMaker Feature Store more efficient, secure, and easier to integrate into ML workflows.

Further Reading

Provide links to additional resources for deeper insights into the new capabilities.

About the Authors

Highlight the expertise of the team behind these enhancements, providing insights into their professional backgrounds.

Enhancements to Amazon SageMaker Feature Store: Streamlining ML Operations

Amazon SageMaker Feature Store has taken a giant leap forward, emerging as a fully managed, purpose-built repository designed for seamless storage, sharing, and management of features in machine learning (ML) models. The latest version introduces significant updates, including support for Apache Iceberg table format, streaming ingestion capabilities, scalable batch processing, and enhanced fine-grained access control through AWS Lake Formation.

Addressing Operational Challenges in ML

As organizations transition their machine learning platforms from experimental models to full production, they often encounter two persistent challenges:

1. Secure Access to Sensitive Data: Managing access to sensitive feature data can be labor-intensive, particularly when numerous feature groups are involved.
2. Predictable Storage Costs: High-frequency streaming workloads often lead to exponential growth in Apache Iceberg metadata, resulting in unexpected storage costs. For example, one retail analytics team saw over 50 TB of metadata accumulate within a year, significantly increasing their Amazon Simple Storage Service (Amazon S3) charges.

Introducing New Capabilities

To tackle these challenges, Amazon has rolled out three new capabilities in SageMaker Python SDK v3.8.0:

1. Native AWS Lake Formation Integration: You can easily register your offline store with Lake Formation at feature group creation time, enforcing access controls automatically without requiring manual setup.

2. Enhanced Apache Iceberg Table Properties: Control metadata retention and snapshot lifecycle policies either at feature group creation or on existing feature groups, helping prevent excessive metadata accumulation and thus reducing storage costs.

3. Revamped Feature Store Support in SDK v3: The optimized SageMaker Python SDK v3.8.0 provides a modular, performance-oriented framework, incorporating comprehensive Feature Store capabilities.

Prerequisites for Implementation

A few prerequisites are necessary to leverage these new features:

• An AWS account with permissions to create Amazon SageMaker AI resources.
• An execution role in Amazon SageMaker that has access to Amazon S3, AWS Glue, and AWS Lake Formation.
• Installation of SageMaker Python SDK v3.8.0 or later (use pip install --upgrade "sagemaker>=3.8.0").
• At least one Data Lake Administrator configured in your AWS account for Lake Formation integration.
• An existing Amazon S3 bucket designated for offline store data.

Solution Overview

These capabilities leverage new parameters in the SDK’s FeatureGroupManager.create() and FeatureGroupManager.update() calls. The LakeFormationConfig facilitates automatic access control setup, while IcebergProperties addresses metadata lifecycle management. Both configurations can be applied at the point of feature group creation or to existing ones.

Operating with the New SDK v3

The SageMaker Python SDK v3.8.0 marks a significant upgrade, enabling a more modular architecture and improved performance. This version allows for:

• Feature group lifecycle management, including creation, description, and updates.
• Record operations such as PutRecord, GetRecord, and BatchGetRecord.
• Efficient training dataset extraction with point-in-time correctness.

The API for Feature Store aims for consistent operation with SDK v2, ensuring minimal disruption to existing code.

Quick Start with SDK v3

Here’s a code snippet to create a feature group utilizing the latest Lake Formation and Iceberg parameters:

fg = FeatureGroupManager.create(
    feature_group_name="my-features",
    record_identifier_feature_name="user_id",
    event_time_feature_name="event_time",
    feature_definitions=df,
    role_arn=role,
    online_store_config={"EnableOnlineStore": True},
    offline_store_config=OfflineStoreConfig(
        s3_storage_config=S3StorageConfig(s3_uri=f"s3://{bucket}/feature-store/"),
        table_format="Iceberg",
    ),
    lake_formation_config=LakeFormationConfig(
        enabled=True,
        hybrid_access_mode_enabled=True,
        acknowledge_risk=True,
    ),
    iceberg_properties=IcebergProperties(
        properties={
            "write.metadata.delete-after-commit.enabled": "true",
            "write.metadata.previous-versions-max": "10",
        }
    ),
)

Native Lake Formation Integration

The integration of AWS Lake Formation into Feature Store simplifies access control. Previously, the manual registration process involved various steps like setting data filters and revoking IAM permissions, making it tedious and prone to error.

Now, you can enable Lake Formation access control during feature group creation, streamlining the entire setup process:

fg = FeatureGroupManager.create(
    feature_group_name="governed-customer-features",
    record_identifier_feature_name="customer_id",
    event_time_feature_name="event_time",
    feature_definitions=customer_df,
    role_arn=role,
    online_store_config={"EnableOnlineStore": True},
    offline_store_config=OfflineStoreConfig(
        s3_storage_config=S3StorageConfig(s3_uri=f"s3://{bucket}/feature-store/"),
        table_format="Iceberg",
    ),
    lake_formation_config=LakeFormationConfig(
        enabled=True,
        hybrid_access_mode_enabled=True,
        acknowledge_risk=True,
    ),
)

Metadata Management with Iceberg Properties

With the integration of Apache Iceberg, managing metadata lifecycle becomes essential, especially for high-frequency writing pipelines that risk excessive metadata growth.

You can now configure Iceberg properties at feature group creation:

fg = FeatureGroupManager.create(
    feature_group_name="streaming-click-features",
    record_identifier_feature_name="session_id",
    event_time_feature_name="event_time",
    feature_definitions=clicks_df,
    role_arn=role,
    offline_store_config=OfflineStoreConfig(
        s3_storage_config=S3StorageConfig(s3_uri=f"s3://{bucket}/feature-store/"),
        table_format="Iceberg",
    ),
    iceberg_properties=IcebergProperties(
        properties={
            "write.metadata.delete-after-commit.enabled": "true",
            "write.metadata.previous-versions-max": "10",
        }
    ),
)

Best Practices

To optimize your usage of new capabilities:

• Configure metadata cleanup proactively, especially for streaming workloads.
• Perform regular compaction and cleanup operations for enhanced query performance.
• Set properties designed for metadata management during feature group creation.

Conclusion

The enhancements to Amazon SageMaker Feature Store vastly simplify the security, cost-efficiency, and integration of feature management in machine learning workflows. The new capabilities not only automate tedious processes like access control but also provide necessary tools for efficient metadata handling.

By adopting SageMaker Python SDK v3.8.0, organizations can ensure that their ML models are backed by robust and cost-effective feature management systems, empowering teams to accelerate their data-driven decisions.

For hands-on experience, refer to the complete documentation, including guides on Lake Formation and Iceberg metadata management.

This post was written by Dhaval Shah, Siamak Nariman, Bassem Halim, and Alex Young from AWS, who bring diverse expertise in machine learning, product management, and software engineering to enhance the SageMaker Feature Store.