From prototype to production: A practical guide to running AI at the edge

Building an AI proof of concept is easier than ever. Deploying that same solution across hundreds of remote devices in production is where the real challenge begins. Resource constraints, device heterogeneity, cybersecurity requirements, and operational complexity often become the primary obstacles of Edge AI initiatives.  

Traditional cloud-based AI deployment comes with inference performance limitations caused by data transmission time. They also introduce data privacy challenges by exposing sensitive enterprise data to a cloud environment. In cloud-based deployments, inference costs typically scale with usage. As the number of requests grow, operational expenses can increase significantly.

Edge AI, which involves running AI workloads on devices closer to the data source, is an emerging technology to solve these issues. Since AI inference runs on the organization's own edge devices, the organization can avoid recurring inference charges, leading to  more predictable and often lower operational costs. The edge device can sit near the data source, providing real-time performance while remaining within the local enterprise network, thereby reducing data privacy concerns.

This blog explores the key challenges organizations face when moving Edge AI solutions from prototype to production. 

Edge and Edge AI 

The edge refers to the location where data is generated, processed, or consumed. It could be a device connected to PLCs (Programmable Logic Controllers) in a shop floor in a smart factory, the ECU (Electronic Control Unit) in a connected car or a local server connected to the digital Building Management System (BMS) in a smart building. Processing the data closer to the source is called edge computing. Running AI inference at the edge is referred to as Edge AI. 

       NVIDIA Jetson Orin                       Advantech EPC R7300               Advantech AIR-150         

Many AI models need specialized hardware such as a GPU (Graphics Processing Unit), TPU (Tensor Processing Unit), or NPU (Neural Processing Unit) for high-performance inference. To run AI on the edge, we need devices with specialized compute capabilities. Purchasing an AI-capable edge device is only the beginning. For production deployments, Edge AI comes with its own set of challenges. 

Challenges with Edge AI 

Like all AI projects, Edge AI works great as a prototype or in a lab, but taking it to production brings challenges ranging from hardware constraints and performance bottlenecks to security, governance, operations, and observability concerns. Some of the key challenges are:  

Model performance in resource-constrained environments 

Unlike the cloud, edge environments are not elastic. Once components are deployed on the edge device, they must operate within a  fixed amount of RAM, compute capacity, and disk space. A deployment might consist of one or multiple models along with multiple services. All software services and AI models compete for the same resources on the edge device.  

Consider the following example of a security system. 

An edge camera has a single NPU capable of processing up to 30 AI inferences per second. 

Two models are deployed: 

• Person Detection: identifies people in each video frame. 

• Loitering Detection: tracks detected people over time to identify suspicious lingering. 

When a crowded scene appears, Person Detection consumes most of the NPU cycles while processing many objects. As a result, Loitering Detection receives fewer compute resources, misses tracking updates, and fails to recognize loitering behavior. 

Conversely, if Loitering Detection is given priority and performs intensive tracking and behavior analysis, Person Detection's frame rate drops, causing missed person detections.  

While techniques such as quantization can improve model performance while preserving accuracy, managing multiple models and services that compete for the same resources while maintaining their health across hundreds or thousands of edge devices remains a significant challenge. 

Device heterogeneity

Edge devices come with a variety of configurations, different operating systems and technical specifications such as RAM, CPU, and GPU capacity. Ideally, a use case should be deployed on one type of edge device across all sites, simplifying operations. In reality, however, this is often not possible. A team might have to reuse old edge devices for some sites. They might have to procure different edge devices to avoid vendor lock-in. In different locations, organizations may choose different edge devices based on vendor’s ability to provide better SLAs(Service-Level Agreements). The need to run AI across a variety of hardware platforms brings challenges such as maintaining different build packages, configurations, deployment and testing pipelines, and deployment strategies.

Monitoring and observability 

Just like any enterprise system, monitoring and observability remain critical to maintaining high uptime, resolving issues quickly, minimizing and eliminating data loss and ensuring high performance and other SLOs (Service Level Objectives). But unlike traditional systems, in Edge AI, the AI models and related software are deployed in remote locations with intermittent internet connectivity. The devices not only have to be monitored and troubleshot remotely, but they must also store  telemetry data (logs, metrics, and traces) on the device locally. Device constraints such as limited disk space and limited network bandwidth also make managing the lifecycle of those signals crucial.  

Security risks 

Cybersecurity is fundamental to any edge deployment. The edge device might be inside a corporate network with access to confidential data. While network security might lie in the purview of the corporation’s IT team, securing the edge device itself presents additional challenges. Organizations must address vulnerabilities across multiple layers, including the OS, containers, and applications. Protecting data, managing cryptographic keys and secrets, ensuring physical security, controlling device access, and implementing Zero Trust principles become increasingly challenging, especially across distributed edge deployments. 

Lifecycle management  

Every AI project starts with data. Each model deployed to remote edge devices would have been trained on a dataset maintained on a centralized repository in the cloud or on-premises. This data is usually collected through the edge devices themselves, and models are tuned to solve real-world problems. However, over time, model performance often declines due to data drift caused by changes in operating conditions, user behavior, equipment, or environmental factors. Therefore, there is a need to continuously collect data for retraining purposes. While a monitoring system can be used to detect drift and trigger retraining, the complete lifecycle of data, models, and edge devices must be managed to scale the deployment of AI from a single device to hundreds or thousands of edge devices while ensuring that use-case performance and effectiveness do not degrade over time. 

Connectivity and offline operations 

In many industrial environments, connectivity cannot be assumed. Factories, warehouses, remote infrastructure sites, and vehicles may experience temporary outages or operate with limited bandwidth. Edge AI systems must be designed to continue making decisions locally while buffering data and synchronizing it with centralized systems once connectivity becomes available.

Summary 

Edge AI offers significant advantages over cloud-only AI deployments, including lower latency, improved data privacy, reduced bandwidth consumption, and more predictable operational costs. However, moving an AI model from a proof of concept environment to production introduces challenges that extend far beyond model development. 

Organizations must address resource constraints, device heterogeneity, observability, cybersecurity, connectivity, and the end-to-end lifecycle of data and models. Successfully overcoming these challenges requires a holistic approach that combines hardware, software, operations, and governance. While these challenges may seem fragmented, they are deeply interconnected. Solving them requires a cohesive platform approach.

Stay tuned for Part 2, where we'll explore the reference architecture, engineering best practices, and platform capabilities required to build reliable, scalable Edge AI solutions.