CHAPTER 5
The Road Map
The major aim of this book has been to provide general guidelines toward the development of real-time image/video processing systems. Such guidelines have not previously appeared in one place as discussed here. In Chapter 2, several key strategies for designing simplified algorithms for real-time use were presented. In Chapter 3, an overview of currently available hardware implementation platforms including digital signal processors (DSPs), field programmable gate arrays (FPGAs), general-purpose processors (GPPs), and graphics processing units (GPUs) was given.
Finally, in Chapter 4, major software methods for obtaining real-time performance were covered.
In all these chapters, representative examples from the recent literature were carefully selected and presented to provide relevant real-world problems to illustrate the key concepts one needs to be aware of when transitioning to a real-time implementation. After having covered the algorithmic, hardware, and software aspects, a recommended road map is mentioned in this final chapter.
5.1 RECOMMENDEDROADMAP
The following steps are considered to constitute the road map when taking an image/video processing algorithm to a real-time environment running on a hardware platform:
• Step 1: Understand the algorithm
It is imperative that a deep understanding of the algorithm is first obtained beyond the high-level knowledge that is often adequate in a research environment. In this initial step, any algorithmic simplifications such as those covered in Chapter 2 should be carefully considered. This understanding also determines what hardware platform is most suitable for its real-time deployment.
• Step 2: Port the algorithm to a reference C/C++ implementation
After having verified the algorithm in the research environment, it is often necessary to create a reference C implementation, not specialized to any particular hardware. This step is needed for debugging the transition from MATLAB or LabVIEW to C and for providing a platform-independent version of the algorithm, which could be ported
over to different hardware platforms. The output of the ported algorithm should be verified using the initial version.
• Step 3: Understand the hardware
In addition to having a deep understanding of the algorithm, it is important to have a deep understanding of the underlying hardware architecture in order to maximize the available computational resources of the architecture involved. It is also useful to become acquainted with any compiler intrinsic instructions as well as any optimized
image processing libraries during this step.
• Step 4: Port the reference algorithm to the target hardware
This step involves porting the reference algorithm over to the target hardware platform.
• Step 5: Profile and identify the bottlenecks
After having ported the reference algorithm to the target hardware, next step is to profile the algorithm to identify where the bottlenecks lie. One should also make use of the integrated development environment (IDE) of the target hardware during this step.
• Step 6: Apply memory and high-level software optimizations
Once the time critical portions of the code are identified, next step includes memory optimizations and high-level software optimizations. Each modification of the code should be followed by a verification procedure to make sure that the outcome is as expected.
• Step 7: Apply low-level software optimizations if necessary
After exhausting all high-level software optimizations, if the performance is still lacking, then one should resort to low-level software optimizations such as writing a handscheduled software pipeline assembly.
• Step 8: Testing
Having achieved the desired performance, rigorous testing should be performed on the system to guarantee a smooth operation under worst-case conditions. The system should be put through a stress test to reveal any weak points to be patched up.
5.2 EPILOG
In this chapter, a recommended road map for the journey from research to reality was given.
Although this book has presented many guidelines to assist in this journey, ultimately it is up to the system designer to select an appropriate collection of the presented guidelines in a particular real-time image/video processing application of interest.
References
[1] J. Ackenhusen, Real-Time Signal Processing: Design and Implementation of Signal Processing Systems. Englewood Cliffs, NJ: Prentice-Hall, 1999.
[2] M. Akil, “Case Study of a Dynamically Reconfigurable Architecture Implementing a Watershed Segmentation,” Proceedings of SPIE-IS&T Electronic Imaging Conference on Real-Time Imaging, SPIE Vol. 4666, pp. 133–140, March 2002.
[3] M. Akil, “Dedicated Architecture for Topological Operators for Grayscale Image Processing,” Proceedings of SPIE-IS&T Electronic Imaging Conference on Real-Time Imaging, SPIE Vol. 5012, pp. 75–82, April 2003.
[4] M. Akil, “Architecture for Hardware Thinning and Crest Restoration in Graylevel Images,” Proceedings of SPIE-IS&T Electronic Imaging Conference on Real-Time Imaging, SPIE Vol. 5297, pp. 242–253, May 2004.
[5] M. Akil, “ARM-Based Embedded Processor: Real-time Implementation for Thinning and Crest Restoration in Gray-Level Images,” Proceedings of SPIE-IS&T Electronic Imaging Conference on Real-Time Imaging, SPIE Vol. 5671, pp. 102–110, February 2005.
[6] A. Amer, “Memory-Based Spatio-Temporal Real-Time Object Segmentation for Video Surveillance,” Proceedings of SPIE-IS&T Electronic Imaging Conference on Real-Time Imaging, SPIE Vol. 5012, pp. 10–21, April 2003.
[7] M. Arias-Estrada and J. Xicotencatl, “Real-Time FPGA-Based Architecture for Stereo Vision,” Proceedings of SPIE-IS&T Electronic Imaging Conference on Real-Time Imaging, SPIE Vol. 4303, pp. 59–66, April 2001.
[8] A. Atsalakis, N. Papamarkos, N. Kroupis, D. Soudris, and A. Thanailakis, “Colour Quantization Technique Based on Image Decomposition and its Embedded System Implementation,” Proceedings of IEE Vision, Image, and Signal Processing, Vol. 151, No. 6, pp. 511–524, December 2004.doi:10.1049/ip-vis:20040552
[9] N. Baba, H. Matsuo, and T. Ejima, “HeadFinder:AReal-Time Robust Head Detection and Tracking System,” Proceedings of SPIE-IS&T Electronic Imaging Conference on Real-Time Imaging, SPIE Vol. 4666, pp. 42–51, March 2002.
[10] J. Batlle, J. Marti, P. Ridao, and J. Amat, “A New FPGA/DSP-Based Parallel Architecture for Real-Time Image Processing,” Journal of Real-Time Imaging, Vol. 8, No. 5 pp. 345–356, October 2002.doi:10.1006/rtim.2001.0273
[11] A. Batur, B. Flinchbaugh, and M. Hayes III, “A DSP-Based Approach for the Implementation of Face Recognition Algorithms,” Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Vol. 2, pp. 253–256, April 2003.
[12] R. Blahut, Fast Algorithms for Digital Signal Processing. Reading, MA: Addison-Wesley, 1985.
[13] S. Borromeo and J. Aparicio, “Real-Time Implementation of a Control System for Exposure Time of CCD,” Proceedings of SPIE-IS&T Electronic Imaging Conference on Real-Time Imaging, SPIE Vol. 5671, pp. 44–51, February 2005.
[14] E. Bourennane, C. Milan, M. Paindavoine, and S. Bouchoux, “Real-Time Image Rotation Using Dynamic Reconfiguration,” Journal of Real-Time Imaging, Vol. 8, No. 4, pp. 277–289, April 2002.doi:10.1006/rtim.2002.0274
[15] A. Bovik, “Introduction to Digital Image and Video Processing,” in Handbook of Image & Video Processing, A. C. Bovik, Ed. Amsterdam: Elsevier Academic Press, 2005.
[16] M. Bramberger, J. Brunner, B. Rinner, and H. Schwabach, “Real-Time Video Analysis on an Embedded Smart Camera for Traffic Surveillance,” Proceedings of the 10th IEEE Real-Time and Embedded Technology and Applications Symposium, pp. 25–28, May 2004.
[17] M. Bramberger, Distributed Dynamic Task Allocation in Clusters of Embedded Smart Cameras, Ph.D. Dissertation, Graz University of Technology, Austria, 2005.
[18] J. Bredno, B. Martin-Leung, and K. Eck, “Software Architecture for Live Enhancement of Medical Images,” Proceedings of SPIE-IS&T Electronic Imaging Conference on Real-Time Imaging, SPIE Vol. 5297, pp. 122–133, May 2004.
[19] K. Brifault and H. Charles, “Data Cache Management on EPIC Architecture: Optimizing Memory Access for Image Processing,” ACM SIGARCH Computer Architecture News, Vol. 32, No. 3, pp. 35–42, June 2004.doi:10.1145/1024295.1024300 [20] H. Broers, W. Caarls, P. Jonker, and R. Kleihorst, “Architecture Study for Smart Cameras,” Proceedings of the European Optical Society Conference on Industrial Imaging
and Machine Vision, pp. 39–49, June 2005.
[21] C. Bruyns and B. Feldman, “Image Processing on the GPU: A Canonical Example,” Computer Architecture Course Project, Department of Computer Science, University of California Berkeley, Fall 2003.
About the Authors
NasserKehtarnavaz is a Professor of Electrical Engineering at the University ofTexas at Dallas. He haswritten four other books and numerous papers pertaining to signal and image processing, and regularly teaches undergraduate and graduate signal and image processing courses. Among his many professional activities, he is currently serving as Coeditor-in-Chief of Journal of Real-Time Image Processing, and Chair of the Dallas Chapter of the IEEE Signal Processing Society. Dr. Kehtarnavaz is a Fellow of SPIE, a Senior Member of IEEE, and a Professional Engineer.
Mark Gamadia is currently a Ph.D. Candidate and an Erik Jonsson Research Assistant Fellow in the Department of Electrical Engineering at the University of Texas at Dallas. He has written several papers involving the development and real-time implementation of image processing algorithms. Mr. Gamadia is a Student Me