Bench Talk

(Source: sdecoret/Shutterstock.com)

The Impact of Moving AI to the Edge

In a previous blog post, we explored strong arguments for moving artificial intelligence (AI) to the network’s edge. In this installment, we discuss which AI applications benefit this approach. As a starting point, reviewing reasons to implement AI at the network edge gives some strong hints. Check whether any of the following apply to the project:

• No access to a fast, stable network connection
• Product operates in a restricted environment
• The project requires delivery of real-time AI
• Limited budget availability

Given these factors, what specific AI projects might be made easier by running the machine-learning (ML) models at the edge? Here, we will examine the benefits of moving AI and ML models such as virtual assistants, facial recognition, and real-time monitoring applications closer to the edge.

As it has so often, Apple set a trend with Siri’s launch in 2010. This paved the way for many other virtual assistants, most famously Amazon’s Alexa and the Google Assistant. Virtual assistants make science-fiction-style voice control into a reality and work as follows:

1. Start by saying a wake word or launching the assistant. For free-standing devices, such as Amazon Echo, the device continuously listens for the wake word and processes this locally, using simple speech pattern matching. This is why Alexa only recognizes certain wake words.
2. The device now connects to a cloud-based server and sends the recording of what it has heard.
3. The cloud server runs a voice-to-text ML model to convert the recorded speech into a natural language text block.
4. The text is parsed using natural-language processing to extract the meaning. 
5. The server works out what was asked for and sends the appropriate commands or content back to the device. 

It is easy to see how moving the ML models to the edge enhances the experience: The voice assistant would be more responsive, wouldn’t need an internet connection, and voice control could be embedded. The application being called for might itself require a network connection, for example, music streaming services.

Facial recognition is one of AI’s fastest-growing applications. The technology is still evolving, and with a few issues along the way. In 2016, Amazon’s Rekognition was mired in controversy and accusations of racism. The system incorrectly identified 28 ethnic minority US Congressional members as known criminals after being trained on a set of 25,000 images. 

In 2019, an early trial of facial recognition technology by the Metropolitan Police, the largest police force in the UK, showed the technology to be inaccurate 81 percent of the time. However, the latest facial-recognition systems are becoming far more accurate. Earlier this year, the Met announced it was adopting the technology to scan for known troublemakers at large events. 

Many use cases calling for facial recognition need the technology to work in near real-time. As a result, applications rely on moving ML models to the edge of the network. The system adopted by the Met is based on NEC NeoFace^® Watch, which is entirely standalone and works in real-time. NEC targets its technology at several other markets, including retail, corporate events, festivals and other mega-events, and transportation. 

Heavy industry and mining rely on extremely large and expensive machinery. Companies can potentially lose millions if this machinery suffers an unplanned breakdown. For instance, many mining operations rely on huge high-power pumps that keep the workings free from water and pump the mined slurry to the processing plant. The whole operation comes to a halt if one of these pumps suffers a catastrophic failure. As a result, mining companies invest significant resources into AI systems designed to predict potential failures before they happen. 

Currently, these systems are often based on transmitting data from Internet of Things (IoT) sensors attached to the equipment. This data is then processed at a central location, and any warning necessary is sent back to the appropriate operator. However, mines and construction sites can be tens of kilometers across, often in hostile terrain, so integrating the ML model directly into the edge device would simplify the whole process. 

Moving AI to the network edge requires three things:

• Suitable hardware
• New tools
• A new paradigm for creating ML models

Let’s look at each of these requirements.

Optimized Hardware

As already discussed, ML models often rely on large numbers of parallel operations. Bluntly, they need raw computing power. However, there is always a trade-off between computing power and the actual power drawn by the device. For ML models to move to the edge, devices that draw as little power as possible are required. This is even more true when the device needs to be embedded. Fortunately, a wide range of high-performance, low-power MCUs is now available.

Suitable Tools

The next thing needed is a suitable toolchain for running ML models on microcontrollers. The overwhelming majority of ML frameworks are designed to run on 64-bit Intel family CPUs or graphic processing units (GPUs). By contrast, all the suitable microcontrollers have a 32-bit reduced instruction-set architecture, such as the ARM^® Cortex^® series MCUs. However, ML frameworks such as TensorFlow Lite enable ML to run on such MCUs. 

Model Once, Run Anywhere 

The final piece of the puzzle is a different paradigm for creating and running ML models. This can be summed up with the phrase “Model once, run anywhere.” Essentially, this means exactly what it suggests: Create your model, typically using a high-power ML-optimized machine, then use your toolchain to convert it into code that can run on any microcontroller. Unfortunately, this eliminates the ability to benefit from continual learning or reinforcement learning.

The Trade-off 

The following table captures some of the trade-offs made when ML models run at the edge (Table1). Hopefully, it offers some pointers that will help decide whether to move your next AI project to the edge.

Table 1: Trade-offs of Running ML Models at the Edge

Conclusion

Moving ML models to the edge enable new use cases for AI, which promises to bring about an embeddable AI revolution. Developments in both MCU hardware and the tools required to run ML models on those MCU’s provide the basis for the expansion of such technology. To learn more about AI-capable MCUs, check out the Mouser TensorFlow Lite article.