Addressing Labor Shortages in Construction Through Autonomous Systems and Vision-Language Models

Bedrock Robotics: A Case Study in Accelerating Autonomous Construction

From Unstructured Video Data to a Strategic Asset Using VLMs

Accelerating AI Deployment Through Strategic Model Optimization

The Path Forward: Addressing Labor Shortages Through Automation

About the Authors

Tackling Labor Shortages with AI: A Look at Bedrock Robotics

Labor shortages are hitting some of the most vital sectors in our economy, including manufacturing, logistics, construction, and agriculture. The construction industry, in particular, faces a pressing challenge, with nearly 500,000 open positions in the United States and 40% of the workforce nearing retirement within the next decade. This shortage leads to delayed projects, increased costs, and postponed development plans. To mitigate these issues, organizations are turning to autonomous systems, capable of performing key tasks and improving productivity around the clock.

Data Preparation: The Bottleneck in AI Implementation

Building these autonomous systems starts with a crucial step: creating large, annotated datasets to train AI models. The effectiveness of this training can determine if these systems deliver real business value. However, a significant challenge lies in data preparation—especially the meticulous task of labeling video data. This involves identifying information about equipment, tasks, and environmental conditions, and it often becomes a roadblock that delays the deployment of AI-powered solutions.

For construction companies that manage millions of hours of video footage, manual data preparation becomes impractical. Enter Vision-Language Models (VLMs)—these cutting-edge technologies can interpret images and video, respond to natural language queries, and generate descriptions at scales and speeds far beyond manual processes, providing a cost-effective solution.

Case Study: Bedrock Robotics and the AWS Fellowship

Bedrock Robotics, a startup dedicated to developing autonomous construction equipment, has embraced this challenge head-on. Partnering with the AWS Generative AI Innovation Center through the AWS Physical AI Fellowship, Bedrock Robotics used VLMs to analyze construction video footage, extract operational details, and generate labeled training datasets efficiently.

Bedrock Operator: Revolutionizing Construction Equipment

Since its inception in 2024, Bedrock Robotics has been developing groundbreaking autonomous systems for construction machinery. Their flagship product, Bedrock Operator, is a retrofit solution that combines hardware and AI models to enable excavators and other machinery to operate with minimal human intervention. Tasks like digging, grading, and material handling can now be performed with exceptional precision.

Training these models requires substantial amounts of video data that portrays a wide range of operational scenarios. Traditionally, this data preparation process is resource-intensive and hinders scalability. However, VLMs streamline this process by analyzing visual data and generating text descriptions, making them perfect for annotation tasks—critical for teaching AI systems how to link visual patterns with human language.

By using VLM technology, Bedrock Robotics facilitated a dramatic improvement in tool identification accuracy from 34% to 70%. This not only transformed a labor-intensive, manual process into an automated, scalable solution but also accelerated the deployment of autonomous equipment.

Optimizing Model Performance for Construction Data

While off-the-shelf VLMs have shown promise, they struggle with specific challenges in construction video data. Unlike general images, operator footage presents unique angles, visibility issues due to dust or weather, and the need for domain-specific knowledge to differentiate similar-looking tools.

Bedrock Robotics tackled this issue through strategic model selection and prompt optimization. Collaborating with the AWS Innovation Center, they evaluated various VLMs and refined prompts to include detailed visual descriptions, guidance for confusing tool pairs, and systematic instructions for analyzing video frames. This led to significant advancements in classification accuracy, achieving 70% accuracy on a test set of 130 videos while reducing costs to just $10 per hour of processing.

The Future: Leveraging Automation to Address Labor Shortages

Bedrock Robotics’ innovative approach offers a replicable framework for other organizations facing similar challenges. By utilizing VLMs, companies can efficiently analyze and annotate vast datasets, essential for deploying autonomous systems. This strategic use of AI can transform workforce constraints into opportunities for growth and efficiency, ultimately leading to reduced operational costs and faster project delivery.

With a cost-effective, scalable annotation pipeline that adapts to operational needs, Bedrock Robotics sets a powerful example for other companies in manufacturing and industrial automation.

Conclusion

As labor shortages impact crucial sectors, investing in AI-driven solutions like those from Bedrock Robotics can provide significant competitive advantages. By streamlining data preparation, organizations can accelerate the deployment of autonomous systems and turn challenges into opportunities for innovation and success.

To learn more about how Bedrock Robotics is transforming the construction industry through AI, delve into their offerings or explore the physical AI resources available on AWS.

Meet the Authors

• Laura Kulowski: Senior Applied Scientist at the AWS Generative AI Innovation Center, focusing on developing cutting-edge AI solutions.
• Alla Simoneau: Emerging Technology Physical AI Lead at AWS, specializes in turning innovative technologies into real-world applications.
• Parmida Atighehchian: Senior Data Scientist with deep expertise in AI and customer-focused solutions, particularly in computer vision.
• Dan Volk: Senior Data Scientist at AWS, passionate about leveraging AI to transform business challenges into opportunities.
• Paul Amadeo: Technical Lead for Physical AI in AWS, with over 30 years of experience in AI and machine learning.
• Sri Elaprolu: Director of the AWS Generative AI Innovation Center, leading global AI innovation efforts for enterprises and government organizations.

By investing in the power of AI, we can tackle the critical labor shortages affecting our industries and pave the way for a more efficient future.