Elevating Security Standards with Mantle: Amazon’s Next-Generation Inference Engine for Generative AI

A Commitment to Transparency and Innovation in Customer Data Protection

About the Authors

Anthony Liguori, AWS VP and Distinguished Engineer for Amazon Bedrock, and the lead engineer for Mantle.

Elevating AI Security: Introducing Mantle, Amazon’s Next-Generation Inference Engine

At Amazon, we pride ourselves on fostering a culture of open dialogue that emphasizes growth and innovation. This ethos allows us to continually raise the bar on how we provide value to our customers. Most recently, we unveiled Mantle, our next-generation inference engine for Amazon Bedrock. As the landscape of generative AI inferencing and fine-tuning continues to evolve, so too must our strategies for optimizing how we deliver these critical services to our customers.

Prioritizing Security in Generative AI

When we embarked on reimagining the architecture of our inferencing engine, placing security at the forefront was non-negotiable. AWS has consistently aligned with our customers’ stringent demands for security and data privacy. This was a critical focus even in the initial stages of Amazon Bedrock, where we recognized that generative AI workloads provide a powerful opportunity for businesses to harness the untapped potential of their data. However, with such opportunities come heightened responsibilities to ensure compliance, privacy, and security, particularly when dealing with sensitive data.

Amazon Bedrock is built on the robust operational security standards inherent to AWS. We adhere to a least privilege access model, enabling AWS operators to interact only with the minimum set of systems necessary for their tasks. Access to systems handling sensitive data is meticulously logged, monitored for anomalies, and carefully audited. Through these measures, AWS safeguards against actions that could compromise security protocols, ensuring that customer data remains protected.

Importantly, customer data is never used to train models on Amazon Bedrock. Model providers lack access to customer data, ensuring inferencing occurs exclusively within an environment controlled by Amazon Bedrock—this robust security paradigm empowers our customers to unlock the potential of generative AI in processing critical information.

Introducing Zero Operator Access with Mantle

With Mantle, we’ve taken our security measures a step further. Drawing inspiration from the AWS Nitro System, Mantle’s architecture is designed around a Zero Operator Access (ZOA) model. This means we have deliberately excluded any technical means for AWS operators to access customer data. Administrative tasks are managed solely through secure APIs and automation, thereby fortifying the security of customer information.

In Mantle, there is no avenue for AWS operators to sign into the underlying compute systems or access customer data such as inference prompts or outputs. Interactive communication tools like Secure Shell (SSH), AWS Systems Manager Session Manager, and serial consoles have been intentionally omitted from the Mantle environment. Additionally, all inference software updates are meticulously signed and verified prior to deployment, ensuring that only vetted code is executed.

Enhanced Security Capabilities

Mantle leverages a sophisticated, hardened, constrained, and immutable compute environment for processing customer data, utilizing the newly released EC2 instance attestation capability. The services within Mantle that manage model weights and execute inference operations are further supported by cryptographically signed attestation measurements from the Nitro Trusted Platform Module (NitroTPM).

When interacting with a Mantle endpoint—such as those serving the Responses API on Amazon Bedrock—customer data, or prompts, transition from the customer’s environment to the Mantle service via TLS, ensuring end-to-end encryption. Crucially, during this process, no operator—whether from AWS, the customer, or a model provider—has the capability to access the customer data.

Looking Forward

The ZOA design of Mantle embodies AWS’s long-term commitment to safeguarding our customers’ data. Our relentless focus on security has empowered our teams to continually enhance protective measures, ensuring an unwavering commitment to customer trust.

Moreover, we’re thrilled to announce that the confidential computing capabilities, such as NitroTPM Attestation, that we utilize internally, will soon be available for all customers to harness through Amazon Elastic Compute Cloud (Amazon EC2). This initiative reflects our dedication to advancing security and transparency further.

We are excited about the future and look forward to continually enhancing the security of your data. As we invest in these advancements, we promise to maintain transparency about how we achieve these ambitious goals.

About the Authors

Anthony Liguori is an AWS VP and Distinguished Engineer for Amazon Bedrock, and he serves as the lead engineer for Mantle.

In a world where data is currency, maintaining the highest security standards is not just important; it’s essential. With Mantle, we’re committed to driving innovation while protecting what matters most: our customers’ data.