NeurOSmart: Development of a smart LiDAR sensor

Key visual of the NeurOSmart project
© Fraunhofer ISIT / Concrete Brandbuilding

… with integrated neuromorphic data processing picks up speed.

In the Fraunhofer lead project NeurOSmart, five institutes (ISIT, IPMS, IMS, IWU, IAIS)* are jointly researching a particularly energy-efficient and intelligent sensor for the next generation of autonomously acting systems. After just under a year of project work, the Fraunhofer researchers will present the expected sensor system and the hybrid, neuromorphic data processing pipeline at Electronica.

The current trend for complex, demanding applications, such as autonomous driving, is toward mobile supercomputers with significantly increasing energy consumption. Especially for mobile systems, this leads to shorter operating times or ranges and, according to current forecasts, will even reach the limits of global energy production in the next decades. To counteract this escalation, the Fraunhofer researchers involved in NeurOSmart are relying on decentralized intelligence that is tailored to the respective sensor and new approaches to energy-efficient data processing.

The basic idea behind the ambitious project is to integrate data processing directly into the sensor as far as possible and to become as energy-efficient as possible through coordinated components.

NeurOSmart researchers collecting initial application-specific training data for the system’s AI
© Fraunhofer

The desired result is a range sensor (LiDAR) that detects and classifies objects on the basis of individual pixels in a field of view of 90° x 90° at working distances of up to 12 meters. All this in real time with frame rates of over 20 Hz and with an energy consumption of a few 10 W. This system can thus be used as a safety system for collaboration between humans and industrial robots and, in the future, also for mobile applications.
This performance is made possible by a variety of innovations in data processing and sensor technology, which are combined in this project.

We would like to particularly highlight the raw data preparation and pre-processing, which provides an integrated solution for the energy-efficient computation of AI algorithms on embedded hardware in real-time via the connection of the RISC-V Manycore cluster AIRISC-N with the embedded AI framework AIfES. This way, the amount of data to be interpreted is reduced by a factor of 10 and allow to observe even only parts of the scene (region of interest), which again reduces energy consumption and increases response time.

Best of all – the Fraunhofer technologies of AIRISC Core and AI Framework AIfES are available for free via GitHub and can recently also be easily downloaded as a pre-configured system from the Intel Pathfinder Shop and integrated into own designs. In addition to image processing applications (LiDAR, thermal imaging, Visible), time series analysis (acoustic monitoring, medical technology) and AI-based predictions (predictive maintenance, failure prediction), among others, are also supported via preconfigured extensions.

Advanced data interpretation is implemented on a novel mixed-signal in-memory accelerator that outperforms current alternatives by at least two orders of magnitude with an expected energy efficiency of more than 1000 TOPS/W. At the heart of the accelerator are so-called ferroelectric field-effect transistors (FeFETs), which can both store and process information (in-memory compute). This way, the energy-intensive bottleneck in the communication between memory and computational core is resolved and complex AI processes can also be realized in mobile end devices.

The advantages of the new approach are underlined by the parallel development of particularly small and efficient models for object recognition and classification. They are specifically adapted to the sensor, to new possibilities of directly integrated electronics and to their applications. The result is fast response time, increased data protection and significant energy savings compared to the current trend of off-practice or cloud-based solutions that prefer to rely on ever larger, more energy-intensive models.

At Electronica 2022, you will find the project’s researchers at the Fraunhofer-Gesellschaft booth in Hall B4, Booth 258.

*The Fraunhofer Institutes involved in the lead project are:

Fraunhofer ISIT
Fraunhofer ISIT is the coordinator of the NeurOSmart project: research competencies include the development and integration of piezoelectric and ferroelectric materials for microelectronic and electromechanical applications. Within the project, Fraunhofer ISIT is involved in the development and fabrication of the scanning LiDAR system as well as AIScN-based ferroelectric multibit memories.

Fraunhofer IPMS
One of the primary research competences is the development of memory technologies in advanced node CMOS realizations. For this purpose, Fraunhofer IPMS has an innovative ferroelectric memory technology (FeFET). In the context of NeurOSmart, the memory emulation is used, among other things, for software blocks to control the in-memory hardware blocks and to manage data streams.

Fraunhofer IMS
The core competencies of Fraunhofer IMS are the development of embedded software and AI, Smart Sensor Systems. The institute has developed AIfES (Artificial Intelligence for Embedded Systems) – a platform-independent and continuously growing machine learning library in the C programming language. AlfES is also an important prerequisite for NeurOSmart for the development of the hardware platform for sensor and scanner control.

Fraunhofer IWU
Fraunhofer IWU’s area of expertise ranges from machine tools, forming, joining and assembly technology, precision engineering and mechatronics to digitalization in production and virtual reality in the context of mechanical engineering. As part of the project, the institute will evaluate the sensor system for applicability in an industrial environment.

Fraunhofer IAIS
Fraunhofer IAIS has research expertise in the areas of distributed learning, speech assistance systems and computer vision for autonomous driving. In this context, the institute has a powerful speech recognition system for automatic transcription and recognition of speech signals. Within the framework of NeurOSmart, the institute bundles its competencies for neural network training, among other things.

Wissenschaftliche Ansprechpartner:

Dr. Michael Mensing

Weitere Informationen:–development-of-a-smart-lidar-…

Media Contact

Claus Wacker Pressestelle
Fraunhofer-Institut für Siliziumtechnologie

All latest news from the category: Information Technology

Here you can find a summary of innovations in the fields of information and data processing and up-to-date developments on IT equipment and hardware.

This area covers topics such as IT services, IT architectures, IT management and telecommunications.

Back to home

Comments (0)

Write a comment

Newest articles

Detecting pollutants at ultra-trace levels

Material scientists develop nano-structured and reusable substrate for ultra-sensitive detection of low-concentration analytes. Surface-Enhanced Raman Scattering (or Spectroscopy), known as SERS, is an advanced analysis method that extends the range…

Stones for the climate

If climate change causes more rain, this promotes the weathering of rocks and thus the erosion of the soil. The dissolved substances reach the sea via rivers. A new model…

Nanobiosensor Developed for Detecting SARS-CoV-2

Infection and immunity status of the population are considered key parameters for handling pandemics. For this purpose, detecting antigens and antibodies is of great importance. The devices currently used for…

Partners & Sponsors