Undoubtedly, you have read or heard about it, as the number of network edge devices has proliferated, a huge amount of data flow has been continuously increased. These devices include self-driving cars, Internet of Things devices, consumer electronics, and even laptops and personal computer. According to multiple estimates, by 2025, there will be tens of billions of IoT devices in operation. These devices send various forms of data to the cloud in the form of continuous data streams, and the data rates vary widely. In general, these devices will generate a large amount of raw data, and the amount of data will continue to increase over time.
Video recorders in security cameras, self-driving cars, and PCs generate high-rate, high-resolution video streams. IoT devices generate medium bit rate data and aggregate them into big data streams. Many other types of IoT sensors (measure temperature, pressure, location, light level, etc.) will generate low bit rate data streams, but soon the number of such sensors will reach billions. Therefore, even these low bit rate data streams can be aggregated into larger, high bit rate data streams before entering the cloud.
The rise of 5G wireless networks and other high-speed network technologies, including picocells, long-distance IoT networks (such as LoRaWAN), and global networking satellite networks (such as SpaceX’s expanding Starlink broadband network and Swarm Technologies’ satellite-based IoT Internet), which provides extensive and fast cloud access (Note: Starlink acquired Swarm Technologies in August 2021).These communications and network technologies accelerate the emerging network edge
. Real-time applications at the edge of the network usually do not tolerate high latency, so processing, analysis, and decision-making must be transferred to the device itself. These network edge devices include autonomous vehicles, IoT sensors, security cameras, smart phones, laptops, and personal computers. Therefore, the potential of network edge computing is huge.
Under the weight of data, the cloud cannot do everything
The exponential growth of smartphones and IoT devices has promoted the development of edge computing on the network. These devices are ubiquitous and must be connected to the Internet to send or receive information to and from the cloud. Some IoT devices (such as cameras) generate large amounts of data during operation.
Other IoT devices, such as temperature sensors, generate a small amount of data, but since the number of such sensors can reach billions, it brings a great burden to cloud processing. Therefore, processing based on the edge of the network is very necessary, not only to reduce the cost of network communication and cloud storage in the cloud, but also to avoid the overload of the cloud data channel.
Developers of network edge products and applications are increasingly adopting artificial intelligence and machine learning (AI/ML) algorithms to match and recognize complex patterns to help analyze data and make decisions based on this. In fact, the use of AI/ML technology has grown extremely rapidly.
Nowadays, AI/ML algorithms are regarded as necessary means to process raw data efficiently, because they can identify complex and multi-dimensional data patterns that are difficult to parse and recognize by traditional algorithm programs. Some specific AI/ML applications include detection, recognition, identification and counting of people or objects; asset and inventory tracking, environmental perception, sound and voice detection and recognition, system health monitoring, and system maintenance scheduling. The development of computing devices and applications.
Emerging network edge devices and applications include autonomous vehicles, robots, automated production, remote monitoring, supply chain and logistics systems, and video surveillance to ensure public and private security. The market’s demand for these network edge systems is growing rapidly because they can increase efficiency, reduce operating costs, and improve user experience. But no matter how much wireless and wired communication infrastructure we build, the turmoil of excess data may overwhelm or block these data pipelines to the cloud.
Localized processing at the edge of the network helps to unblock the data pipeline
These trends indicate that it is now necessary to do more processing where data is generated at the edge of the network and reduce the amount of data transmitted to the cloud. The explosive growth of the Internet of Things and other network-connected devices is the main driving force for the development of new network edge devices, which further stimulates the development of new applications, thereby transforming raw data into useful and actionable information to support rapid decision-making , Respond to changing situations in real time.
In the early stages of the development of network edge computing, companies focused on the cost of transmitting data to data centers over long distances. Initially, a major feature of network edge applications was the need to access data stored in the cloud and other computers connected to the cloud. These early applications are usually not real-time applications; response times of hundreds of milliseconds or even seconds are acceptable. However, the development of Internet of Things devices and the growing demand for real-time processing, analysis and response at the network edge have promoted the powerful development of network edge technology, along with greater design challenges.
Network edge processing makes computing and data storage closer and closer to the device that collects data, instead of analyzing and making decisions in a data center thousands of miles away
Figure 1. Trends in network edge computing (Image source: Lattice)
Many network edge applications that can take advantage of AI/ML capabilities need to operate under extremely stringent power consumption constraints. These widely distributed devices usually rely on battery power. Such applications abound in various network edge environments, including factories, farms, office buildings, retail stores, hospitals, warehouses, streets, and residences. As their number increases, these devices need to operate for a long time, even months or years, with only a single charge or only relying on the collection and storage of energy.
Therefore, many devices need to be in a sleep or hibernation state most of the time, and most of the circuits should be in a low-power standby mode when the device is in an inactive state. Then the activation event will start the device when needed. In this type of application, the basic circuit system operating at ultra-low power consumption must remain on standby, waiting for an activation event, and then power the rest of the device as needed.
FPGA realizes AL/ML with low power consumption
The requirements for low operating power consumption and AI/ML algorithm implementation seem to conflict with the requirements for low-power network edge device design. However, these two complex design requirements are not contradictory. Lattice’s latest FPGA-the low-power, small-size, high-performance CertusPro-NX series of devices-is tailored to meet the many design requirements of low-power network edge devices. These FPGAs can support multiple sensors, displays, high-resolution video, network connectivity, and AI/ML processing at the edge of the network.
At the same time, Lattice’s newly released version 4.1 of the sensAI solution collection provides ready-to-use AI/ML tools, IP cores, hardware platforms, reference designs and demonstrations, and customized design services to help design teams develop new Network edge equipment and quickly bring it to the market. The latest version of sensAI supports CertusPro-NX FPGA.
The Lattice sensAI solution collection accelerates end-to-end AI/ML model training, verification and compilation. Lattice added the sensAI Studio design environment to sensAI 4.0 released in early 2021, which is a graphical user interface (GUI)-based tool that helps developers quickly build machine learning applications. When using the tools in Lattice sensAI 4.1 to set up the network edge computing design, and using Lattice iCE40 UltraPlus, CrossLink-NX, ECP5 and CertusPro-NX FPGA, real-time AI/ML functions can be realized at ultra-low power consumption ――The power consumption is as low as 1mW to 1W.
Figure 2. Lattice’s sensAI Studio design environment accelerates end-to-end AI/ML model training, verification, and compilation. (Image source: Lattice)
With sensAI 4.1 supporting Lattice CertusPro-NX FPGA series products, the performance of sensAI has also been greatly improved. In addition to the existing object detection and tracking applications, applications such as real-time classification of multiple objects have also been added. The sensAI 4.1 solution set includes an updated neural network compiler and is also compatible with other widely used machine learning platforms, including the latest versions of Caffe, Keras, TensorFlow and TensorFlow Lite.
The IP cores in the Lattice sensAI 4.1 solution set include three types of convolutional neural network (CNN) accelerators-CNN, CNN Plus and CNN Compact-and a CNN coprocessor engine. The CNN IP core allows developers to use various widely used CNNs released by others, such as Mobilenet v1/v2, Resent, SSD and VGG, or customize the CNN model as needed. The sensAI 4.1 CNN accelerator utilizes the parallel processing capabilities, distributed memory and DSP resources of Lattice FPGAs to greatly simplify the implementation of ultra-low power AI designs. The accelerator core uses FPGA programmable logic to implement low-power neural networks, including extremely efficient binary neural networks (BNN), which can implement CNN with ultra-low power consumption in the milliwatt range.
Figure 3. Lattice sensAI solution collection can develop AI/ML devices based on Lattice FPGA. (Image source: Lattice)
Lattice sensAI 4.1 reference design
Lattice FPGAs provide programmable I/O, which can be configured to support a variety of electrical interface standards commonly used in sensor interfaces. The company also provides many hard-core and soft-core IP modules to support different sensor communication protocols. Because FPGAs have long had significant advantages in sensor fusion, the design of Lattice sensAI 4.1 aims to simplify the development of AI/ML inference functions based on multiple sensors in network edge devices and realize intelligent sensor fusion. The sensAI 4.1 solution collection includes many reference design examples, demonstrating a variety of smart sensor fusion application cases, which can run at the same time to achieve in-depth situational awareness. These reference designs include:
• Gesture detection
This reference design uses an IR image sensor to implement an AI-based low-power gesture detection system. The reference design provides a training data set, scripts that can be trained with commonly used neural network training tools, and a neural network model for users to modify.
•Keyword detection
This reference design uses a digital MEMS microphone to continuously detect keyword utterances. Designers can use deep learning frameworks (such as Caffe, Tensorflow, or Keras) to update the provided training data set to add wake word functions to the system. The reference design includes a training data set, scripts that can be trained using commonly used neural network training tools, and a neural network model for users to modify.
•Face Detection
This reference design uses image sensors to implement CNN-based face recognition, and can modify the training database to recognize other types of targets.
•Personnel detection
This reference design uses CMOS image sensors to continuously detect the presence of people. The AI system based on this design can use a deep learning framework (such as Caffe or Tensorflow) to update the provided training model to detect and locate any target of interest. The reference design includes a neural network model, a training data set, and scripts that can be trained using common training tools.
•Target detection, classification, tracking and counting
This reference design provides examples of target detection, classification, tracking, and counting. It has a complete design, including FPGA RTL for Lattice development boards, neural network models, example training data sets, and re-creation and update of the design script.
Common and potential network edge applications where AI can be used
Using AI/ML algorithms to improve the performance of many network edge devices (such as autonomous robots, environmental control, and video security cameras) has obvious advantages, while other types of network edge devices can also benefit from it, such as PCs and laptops. Lattice is working with partners and customers to use multi-mode, smart sensor fusion and AI/ML technology to continuously improve the experience of PC/laptop users and significantly reduce the operating power consumption of laptops. In some applications, The battery life has increased by up to 28%.
Which equipment features have potential value?
The usage of PCs and laptops within 24 hours is very different, and they are generally used intensively during working hours during the day. However, even during working hours, they will have a rest state. People take occasional breaks and have meals at noon. During these times, they usually keep the computer running to ensure that the various applications they open are not closed.
Combine AI/ML analysis and decision-making with the computer’s existing sensors (cameras and microphones) to achieve smart sensor fusion, allowing PCs or laptops to perceive the surrounding environment to decide when to turn off the display and CPU, and when to give They re-powered.
The simplest use of presence detection is to shut down the computer when no one is around. When the user is away from the Screen for a long time, the attention tracking function can dim the computer screen and activate the low-power mode. The low-power, small-size FPGA that acts as the center of the smart sensor can receive input from computer sensors, and then decide which components to supply power according to the situation.
Solve privacy and security issues
Similarly, these functions can also enhance the privacy and security of the computer. The computer’s built-in conference camera can be used to monitor the background behind the user and detect whether someone is peeping from behind the user’s shoulder. If the computer is configured to protect privacy, when someone behind an authorized user is suspected of peeping on the computer screen, it can pop up a warning to remind the user and even automatically dim the screen. It should be noted that, with these solutions, all inference data are stored locally in the FPGA. Only the metadata is passed to the SoC, which further enhances privacy and improves security.
Optimize user experience
AI/ML functions can also enhance the overall experience of computer users. For example, the AI/ML-based facial viewfinder function can use the higher resolution of the built-in video conference camera to crop and center the user’s avatar, providing a better picture for the video conference. Participants can also move during the meeting while their images remain centered. Similarly, gesture recognition can add non-contact operation capabilities to laptops or PCs or any other video-enabled IoT devices.
Health benefits
Many companies now make it clear that they want to protect the health of their employees. AI/ML-based perception functions can help avoid repetitive stress injuries through pop-up reminders and other measures, and use computer video sensors to ensure that employees have actually adopted the given rest recommendations.
AI/ML applications can also be used to detect the user’s posture, which may be another factor causing repetitive stress injuries. These features using active sensor feedback can be used to develop health applications, which is significantly better than the simple timing reminders currently used in enterprises, and can effectively deal with stress-related work injuries.
All of these functions implemented through AI/ML can help suppliers create PCs and laptops that are more attractive to corporate buyers, and all these functions can be achieved through the sensAI 4.1 solution collection and Lattice low-power FPGAs Features to achieve.
The use of this kind of FPGA surpasses the iconic function of FPGA development for a long time-sensor connection and fusion, and based on mature AI/ML algorithms, it adds sensor signal analysis and decision making functions. The addition of AI/ML makes FPGA a low-power system controller that can manage system functions, enhance user experience, and greatly extend battery life by reducing overall system operating power consumption.
Conclusion: A huge market of billions of network edge devices is yet to be developed
With its multiple low-power FPGA series products and a collection of sensAI 4.1 solutions that support these product series, Lattice is committed to bringing AI/ML technology to billions of network edge devices. Therefore, network edge applications are a highly potential target market.
According to various estimates, tens of billions of network edge devices are needed in a wide area of the world to meet the needs of a large number of network edge markets, which is very attractive for the FPGA business-of course this scale is suitable for any industry in this way. Lattice’s release of sensAI 4.1 solution set and its low-power, small-size FPGA series is directly aimed at network edge applications and markets. Lattice’s sensAI 4.1 solution collection is an innovative development tool for network edge applications, which allows system developers to develop flexible, application-specific, FPGA-based AI/ML inference solutions for various markets.