What role does ISP play in the field of intelligence
Image Signal Processor (ISP) is originally an important chip used in multimedia equipment, and plays an important role in all fields that need to capture images. With the further improvement of intelligent social life, ISP is also entering many new market application fields, and these new intelligent fields have put forward higher requirements for the performance of ISP.
In the field of autonomous driving and assisted driving, the impact of machine vision on the system is very large. Without the support of machine vision, intelligent driving technology may not be successful. The ISP is responsible for processing the corresponding RAW images for the autonomous driving system to ensure the reliability of the autonomous driving technology. However, when we drive at night, we need to ensure that the machine vision model can work normally in low light and other conditions, which requires the ISP to enhance the night vision and noise reduction processing performance of the image.
In the field of security, ISP technology is also required to be able to cope with various low-light scenarios. Make sure that the image quality of the device is not affected in low light conditions. This ensures that security monitoring can truly achieve a clear picture of full-time and full-scene monitoring.
In terms of night vision noise reduction, the current best processing is to use neural networks to complete noise reduction. Due to the different sounds in different scenarios, it is difficult to use a unified analytical formula to complete noise reduction. However, we can use neural networks to collect different situations within the same scene for real-time analysis. In this way, when the amount of collected data is large enough, the neural network can quickly and accurately complete the function of night vision noise reduction. Therefore, in a complete system using ISP, some elements of artificial intelligence need to be added to make corresponding adjustments to the ISP.
As we all know, artificial intelligence based on neural network will soon have huge network parameters, and the amount of calculation is also very large. Therefore, it is necessary to have a corresponding module (AI engine) that can efficiently process artificial intelligence calculations to meet the needs of ISPs.
At the same time, it should be noted that since the ISP has requirements for latency and power consumption, the AI engine must also consider these two factors.
The first solution is to integrate the ISP and AI engine into the same IP block. In this way, the coupling between the ISP and the AI engine is very high. Through the co-optimization of the ISP and the AI engine, higher latency and power consumption can be achieved. On the other hand, we can also utilize pipeline features based on convolutional neural networks to process pixels in batches. By deeply integrating the ISP and AI engine, the system can be co-designed for better latency.
Another solution is to build a data path between the ISP and the AI engine, and at the same time ensure that the ISP has the priority to call the AI engine. The advantage of this is that when the ISP is not activated, the AI engine can also be invoked for other applications, making the system more flexible. Of course, doing so reduces the coupling between the ISP and the AI engine, and also loses the advantage of co-optimization between the ISP and the AI engine.
The above two schemes have their own advantages and disadvantages, and it is necessary to choose the appropriate scheme according to the characteristics of the developed system. For electronic components involved in the smart field, XT-Shenzhen has a large inventory of original factories, including components produced by world-renowned semiconductor companies such as Texas Instruments, STMicroelectronics, NXP, and Xilinx.
