Streamlining AI Deployment: Optimizing Large Language Models with Amazon SageMaker and BentoML

Introduction to Self-Hosting LLMs vs API Integration

Managing Infrastructure Complexity with Amazon SageMaker AI

Performance Optimization Challenges in LLM Deployment

Systematic Benchmarking with BentoML’s LLM-Optimizer

Overview of the Implementation Process

Defining Constraints in SageMaker AI Studio

Conducting Theoretical and Empirical Benchmarks

Generating and Deploying the Optimized Configuration

Importance of Inference Optimization in Real-World Applications

Understanding LLM Performance Metrics and Trade-offs

Key Performance Metrics Explained

The Roofline Model for Performance Visualization

The Throughput-Latency Trade-off

Practical Application: Deploying Qwen3-4B on Amazon SageMaker

Prerequisites for the Example Deployment

Running the LLM-Optimizer

Estimating Performance through Benchmarking

Conducting Actual Performance Benchmarks

Visualizing Benchmark Results with Pareto Analysis

Deploying the Optimized Model to Amazon SageMaker AI

Conclusion: A Data-Driven Approach to LLM Deployment

Additional Resources

About the Authors

Navigating the Future of AI: Self-Hosting vs. API Integration with Amazon SageMaker and BentoML

The rapid evolution of powerful large language models (LLMs) has transformed the landscape of artificial intelligence (AI). These models, accessible via API, have simplified the integration of AI capabilities into applications. However, many enterprises opt to self-host their models, accepting the challenges of infrastructure management, GPU costs, and model updates. This decision often hinges on two main factors: data sovereignty and model customization.

The Drive for Self-Hosting

Organizations increasingly prioritize data sovereignty, ensuring that sensitive information stays within their infrastructure due to regulatory, competitive, or contractual obligations. Additionally, model customization allows for fine-tuning on proprietary data sets to meet industry-specific needs, a feat often unattainable through generalized APIs.

The Amazon SageMaker Advantage

Amazon SageMaker AI simplifies the complexities associated with self-hosting. It abstracts the operational burdens of GPU resource management, enabling teams to focus on enhancing model performance rather than juggling infrastructure. The system features inference-optimized containers, such as the Large Model Inference (LMI) v16 container image, which supports advanced hardware like the Blackwell/SM100 generation.

Despite this managed service, optimal performance still necessitates careful configuration. Parameters like tensor parallelism degree and batch size can profoundly influence latency and throughput, making it essential to find a balance tailored to your workload.

Streamlining Optimization with BentoML

The challenge of configuration can be daunting. Enter BentoML’s LLM-Optimizer, which automates the benchmarking process. By systematically exploring different parameter configurations, this tool replaces the tedious trial-and-error method, allowing users to define constraints related to latency and throughput. The optimizer enables teams to efficiently identify the best settings, transitioning seamlessly to production.

Understanding Performance Metrics

Before diving into the practicalities of optimization, it’s crucial to grasp key performance metrics:

• Throughput: The number of requests processed per second.
• Latency: The time from request to response.
• Arithmetic Intensity: The ratio of computation performed to data moved, which helps categorize workloads as memory-bound or compute-bound.

The roofline model can visualize performance by plotting throughput against arithmetic intensity, revealing bottlenecks in memory bandwidth or computational capacity.

Practical Application: Deploying Qwen3-4B on Amazon SageMaker AI

In this blog post, we’ll go through the process of deploying the Qwen3-4B model, showcasing how to optimize LLM configurations for production.

Step-by-Step Workflow

1. Define Constraints: Start by using SageMaker AI Studio to outline deployment goals, such as desired latency and throughput.

2. Run Benchmarks with LLM-Optimizer: Use the optimizer to execute theoretical and empirical tests across various parameter combinations to identify the most efficient serving configuration.

3. Generate Deployment Configuration: Once benchmarking is complete, the optimal parameter values are compiled into a JSON configuration file, ready for deployment on a SageMaker endpoint.

Starting the Optimization Process

To kick off the optimization, use the LLM-Optimizer to run an initial estimate based on your defined constraints.

llm-optimizer estimate \
--model Qwen/Qwen3-4B \
--input-len 1024 \
--output-len 512 \
--gpu L40 \
--num-gpus 4

This estimate will yield insights into latency and throughput, informing the next steps involving real-world benchmarking.

Running the Benchmark

Transitioning from theoretical predictions to practical performance, the benchmarking phase involves running tests across various configurations. The following code snippet illustrates this:

llm-optimizer \
--framework vllm \
--model Qwen/Qwen3-4B \
--server-args "tensor_parallel_size=[1,2,4];max_num_batched_tokens=[4096,8192,16384]" \
--client-args "max_concurrency=[32,64,128];num_prompts=1000;dataset_name=sharegpt" \
--output-json vllm_results.json

This step produces artifacts such as an HTML visualization that highlights the trade-offs between latency and throughput across configurations.

Deploying to Amazon SageMaker AI

Once the optimal parameters are identified, the final step is deploying the tuned model.

1. Set Up the Environment: Define the required environment variables for the deployment.

env = {
    "HF_MODEL_ID": "Qwen/Qwen3-4B",
    "OPTION_ASYNC_MODE": "true",
    "OPTION_ROLLING_BATCH": "disable",
    "OPTION_ENTRYPOINT": "djl_python.lmi_vllm.vllm_async_service",
    "OPTION_MAX_ROLLING_BATCH_PREFILL_TOKENS": "8192",
    "OPTION_TENSOR_PARALLEL_DEGREE": "4",
}

2. Create and Activate the Endpoint: Use the following code to deploy the model into a managed SageMaker endpoint.

create_model = sm_client.create_model(
  ModelName=model_name,
  ExecutionRoleArn=role,
  PrimaryContainer={
      "Image": image_uri,
      "Environment": env,
  },
)

3. Handling Live Traffic: Post-deployment, the endpoint is ready for inference.

response_model = smr_client.invoke_endpoint(
    EndpointName=endpoint_name,
    Body=json.dumps(request),
    ContentType="application/json",
)

Conclusion: A New Era of AI Deployment

The transition from API-based models to self-hosted deployments does not have to be fraught with complexity. By leveraging BentoML’s LLM-Optimizer and Amazon SageMaker AI’s robust infrastructure, organizations can navigate the nuances of deployment effectively. This synergy fosters data-driven, automated optimization processes, enabling businesses to focus on innovation rather than resource management.

With systematic optimization and seamless deployment capabilities, enterprises can accurately balance cost, performance, and user satisfaction, ultimately transforming their AI ambitions into reality.

Additional Resources

For those interested in further exploring the world of LLMs and their deployment, check out the following:

• BentoML Documentation: BentoML
• Amazon SageMaker Resources: SageMaker
• Roofline Model Insights: AWS Neuron Batching Documentation

About the Authors

• Josh Longenecker: A Generative AI/ML Specialist Solutions Architect at AWS, passionate about empowering customers with advanced AI solutions.

• Mohammad Tahsin: A seasoned AI/ML Specialist Solutions Architect at AWS, dedicated to driving continuous learning and innovation in AI technologies.