Intersil’s Jonpaul Jandu describes a cost efficient, easy to use processor that performs optical de-warping of a fisheye image within the camera. 
Fisheye images, spanning a field of view of up to 180 degrees, can offer fascinating and exotic qualities.They offer a density of visual information that is otherwise inaccessible to the naked eye or by the use of regular ‘pinhole’ type cameras, even when fitted with a wide-angle lens. The extreme barrel-shaped warping of the image, as if looking at a globe, is often quite acceptable – even valued – as a visually ‘interesting’ picture.
However, a different situation emerges if one considers the possible commercial or industrial applications for a fisheye image. Such applications usually depend on a regular, equidistant-matrix, rectilinear picture format. Extreme fisheye warping results in radial dislocations of the picture elements in a non-linear pattern. The pixels appear spatially shifted to the edges, away from the image’s centre of distortion – which usually, but not necessarily, is also its focal point and visual centre. One must add to that the effects of uneven luminance and colour saturation caused by the enormous directional light and shadow variations across the field of view, plus chromatic aberrations and tangential pixel dislocations introduced by the limited optical quality of the lens. Unfortunately, even though perhaps appealing at first glance, the raw data coming from a fisheye lens is not suited to computer vision applications, or the detection and recognition of objects and their movements across the picture plane. Even for visual observation in surveillance systems the extremely distorted view is almost unusable.
Basically, to be useful, fisheye images need a heavy dose of optical correction before they are suitable to use as an input for real-world applications.
The ‘fix’ is a carefully balanced image correction procedure that transforms the distorted and shifted image elements back to their ‘natural’ rectilinear matrix, if possible in real time to generate a live video stream. However the procedures required usually involve excessive post-processing – which prohibits high-end real-time applications of digital video streams as required in many live image analysis tasks. Additionally, most of these algorithms have been established and optimised to serve proprietary formats and specific applications.
In response to this, Intersil went about developing its TW2871 in Tokyo in 2009. The basic idea was to provide camera manufacturers with a versatile fisheye correction device that they could integrate easily into all kinds of real-time surveillance cameras and systems without the need for tedious adaptation and post-processing procedures. In May 2011 the company unveiled the device and is now delivering the 376-pin PBGA chip as of Q2 this year.
The device enables real-time applications through a hard-wired ASIC implementation of all the necessary corrective measures but without dependence on the usual software solutions. The processor chip integrates a high-end geometrical image correction engine. It achieves a video latency of 100msec maximum – virtually real time!
The move toward HD video
The ISP connects directly to the image sensor that feeds the highly distorted raw image to the chip. An integrated DDR2 controller manages traffic from and to the off-chip DRAM, which stores the lookup tables for geometrical pixel mapping. Any fisheye lens, and any CMOS or CCD sensor of up to 8 or 10Mp of resolution at a pixel clock up to 96MHz is accepted. This feature points the way to HD video.
Adaptation of the lens is extremely simple and is easily done by referring to its data sheet, by programming its relevant physical parameters (diameter, aperture, focal length, etc. – everything that defines its fisheye distortion pattern) into the lookup table connected to the chip. The processor then adjusts accordingly and automatically. No optical calibration or test imagery is needed. It’s a one-time, straightforward procedure.
The devices’ image signal processor (ISP) then performs a wide array of corrective actions with regards to image quality. Among these are bad-pixel, auto-exposure, auto white-balance, wide dynamic range, 3D-noise reduction and lens shading corrections, plus Bayer alignment and conversion from RGB to YCbCr colour formats.
A great feature is that it delivers up to four independent split images derived from the hemispheric fisheye view. Each of them comes in the standard definition SDTV format (CVBS), NTSC or PAL, along with 720p or 1080i HD according to SMPTE. Alternatively there are two channels of 8-bit BT.656 streams, also in NTSC or PAL.
The four corrected split images derived from the distorted fisheye view appear in a de-warped rectilinear format as if coming from four separate cameras of longer focal lengths. A novel feature is that the positions of these split images can be independently controlled, moved and zoomed by the user in real time. This is done by means of an ordinary joystick console as used in typical surveillance setups. The images are shown on a split-screen monitor, along with the original super-wide-angle fisheye view.
In other words, the optically corrected split images enable a targeted, ‘zooming-in’ on the most interesting areas of the 180-degree wide scene. This allows for closer observation of objects of interest. Positions, directions and zoom factors of the split images are freely selectable. Thus, a surveillance system fitted with this device resembles digital panning, tilting and zooming (PTZ) – a capability usually not found in today’s in-door surveillance systems. They are typically mounted to the ceilings of lobbies, hallways, and entry areas of public spaces. Usually they employ a single-view camera, which must be manually zoomed and pointed at specific areas chosen for closer examination. But with this solution, the (fisheye) camera direction is not revealed, it is hidden under a dome, appearing mirrored or black.
Another important feature is that by pointing the four split camera views generated by in the four orthogonal directions, it eliminates the nasty blind spots in the field of view as experienced with servo-mechanically controlled single-view cameras.
If motion detection is required, the the necessary algorithm are provided to do that. It even offers a masking function to avoid false alarms. Masking a certain area of a split image by simply pointing and clicking from the console excludes visual objects that show regular unsuspicious motion patterns from the scrutiny of the motion detector.
The four digital images delivered are de-warped and optically corrected up to XGA resolution, which is better than standard-definition (SD) TV quality. The final resolution is of course within the bounds of a digital zoom, which cannot go beyond the native pixel count of the original fisheye image. This limits surveillance applications mostly to indoor situations, at an object distance below 150 meters. However, future versions of the chip envisage HD resolution (1080i or 720p). In any case, optical zooming setups can now be relegated to outdoor surveillance situations that need to cover very large distances.
Most important to the designer of surveillance systems is that the new chip provides a convenient way of implementing fisheye correction without time-consuming software-driven procedures such as panorama stitching. In most cases these are too complex and too cost-intensive to be done in real time.
Besides surveillance systems, the company sees excellent opportunities for the processor in other applications including future automotive driver-assistance, video conferencing, military explosive seeking robots, medical endoscopy and home security made simple; a single-lens camera supplying various angles and levels of digital output that comply with popular digital consumer video formats and IP transfer.
Intersil
www.intersil.com
