HOME ::

 

 



  
 

Image-based Rendering (IBR)

 
Introduction:

Figure1. A brief summary of concentric mosaic and lightfield.
In IBR, new views of scenes are reconstructed from a collection of densely sampled images or videos.  Examples include the well-known panoramas, light fields and variants, concentric mosaics, etc .
 

Figure2. Spectrum of image-based representations.

The reconstruction problem (i.e. rendering) is treated as a multidimensional sampling problem, where new views are generated from densely sampled images and depth maps instead of building accurate 3D model of the scenes.  Depending on the functionality required, there is a spectrum of IBR as shown.  They differ from each other in the amount of geometry information of the scenes/objects being used.
At one end of the spectrum, like traditional texture mapping, we have very accurate geometric models of the scenes and objects say generated by animation techniques, but only a few images are required to generate the textures.  Given the 3D models of the objects and the lighting conditions, novel views can be rendered using conventional graphic techniques.  Moreover, interactive rendering with movable objects and light sources can be supported using advanced graphic hardware.

 

Figure3. Image formation process.
For a given lighting and geometry, a camera will capture rays from the objects and recorded them as pixels on its image plane, such as camera planes 1 and 2 in Fig. 3.  The area dA will be imaged by different cameras with different reflected radiance , which is a function of the spherical coordinate .  If  is bandlimited, then we can recover it from sufficient number of samples in .


research focus:
  • Capturing , compression, transmission and processing of dynamic image-based representations.
  • Virtual and augmented reality, cultural heritage preservation and digital museum.


Application:
  • 3-D TV and Immersive videos,
  • Digital entertainment.
  • 3-D photography, virtual tour.

Demonstration:
Click here to play

>>back to top

 

Publication:
  • S. C. Chan, H. Y. Shum, and K. T. Ng, “Image-based rendering and synthesis: technological advances and challenges,” IEEE Signal Processing Magazine: Special Issue on MVI and 3DTV, vol. 24, no. 6, pp. 22-33, Nov. 2007
  • H. Y. Shum, S. C. Chan, and S. B. Kang.  Image-based rendering. Springer, 2007.
  • S. C. Chan, K. T Ng, Z. F. Gan, K. L. Chan, and H. Y. Shum, “The plenoptic videos,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 15, no. 12, pp. 1650 -2659, Dec. 2005.
  • K. T. Ng, S. C. Chan and H. Y. Shum, “The data compression and transmission aspects of panoramic videos,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 15,  no. 1,  pp. 82 - 95, Jan. 2005.
  • H. Y. Shum, K. T. Ng and S. C. Chan, “A virtual reality system using the concentric mosaic: construction, rendering and data compression,” IEEE Trans. on Multimedia, vol. 7, no. 1,  pp. 85 - 95, Feb. 2005.
  • H. Y. Shum, S. B. Kang and S. C. Chan ,  “Survey of image-based representations and compression techniques,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 13, no. 11, pp. 1020 - 1037, Nov. 2003.


    Other Publications:
  • [1]    J. X. Chai, X. Tong, S. C. Chan, and H. Y. Shum, “Plenoptic sampling,” in Proc. of the annual conference on Computer Graphics (SIGGRAPH’00), pp. 307-318, July 2000.
  • [2]    S. C. Chan and H. Y. Shum, “A spectral analysis for light field rendering,” in Proc. of IEEE ICIP2000, vol. 2, pp. 25-28, Sept. 2000.
  • [3]    K. T. Ng, S. C. Chan, H. Y. Shum, and S. B. Kong, “On the data compression and transmission aspects of panoramic videos,” in Proc. of IEEE ICIP, vol. 3, pp. 644-647, Oct. 2001.  
  • [4]    H. Y. Shum, S. B. Kang and S. C. Chan, “Survey of Image-Based Representations and compression techniques,” IEEE Transactions Circuit and System Video Technology, vol. 13, no. 11, pp. 1020-1037, 2003.
  • [5]    S. C. Chan, K. T Ng, Z. F. Gan, K. L. Chan, and H. Y. Shum,  ”The data compression of simplified dynamic light fields,” in Proc. of IEEE Int’l Conf. on Acoustics, Speech, and Signal Processing (ICASSP’03), vol. 3, pp. 653-656, April, 2003.
  • [6]    Z. F. Gan, S. C. Chan, K. T. Ng, K. L. Chan, and H. Y. Shum, “On the rendering and post-processing of simplified dynamic light fields with depth information,” in Proc. of IEEE Int’l Conf. on Acoustics, Speech, and Signal Processing (ICASSP’04), Montreal, Canada, vol. III, pp. 321-324, May 17-21, 2004.
  • [7]    K. T. Ng, S. C. Chan, and H. Y. Shum, “The data compression and transmission aspects of panoramic videos,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 15, no. 1, pp. 82 - 95, Jan. 2005.
  • [8]    H. Y. Shum, K. T. Ng, and S. C. Chan, “A virtual reality system using the concentric mosaic: construction, rendering and data compression,” IEEE Trans. on Multimedia, vol. 7, no. 1, pp. 85-95, Feb. 2005.
  • [9]    S. C. Chan, K. T. Ng, Z. F. Gan, K. L. Chan, and H. Y. Shum, “The plenoptic videos: capturing, rendering and compression,” in Proc. of IEEE ISCAS’04, vol. 3, pp. 905-908, Vancouver, Canada, May 23-26, 2004.
  • [10]  Z. F. Gan, S. C. Chan, K. T. Ng, and H. Y. Shum, “An object-based approach to plenoptic videos,” in Proc. of IEEE Int’l Symposium on Circuits and Systems (ISCAS’05), pp. 3435-3438, May, 2005.
  • [11]  Z. F. Gan, S. C. Chan, K. T. Ng, and H. Y. Shum, “Object tracking for a class of dynamic image-based representations,” in Proc. of SPIE Visual Communications & Image Processing, pp. 1267-1274, July, 2005.
  • [12]  Z. F. Gan, S. C. Chan, and H. Y. Shum, “Object tracking and matting for a class of dynamic image-based representations,” in Proc. IEEE Advanced Video and Signal-Based Surveillance, pp. 81-86, Sep., 2005.
  • [13]  Q. Wu, K. T. Ng, S. C. Chan, and H. Y. Shum, “On object-based compression for a class of dynamic image-based representations,” in Proc. IEEE ICIP,pp. 405-408, Sept. 2005.
  • [14] Q. Wu and S. C. Chan, “An improved Macroblock level rate control algorithm for MPEG-4 video object coding,” in Proc. Int. Symp. Intelligent Signal Processing & Commun. Syst. (ISPACS’05), pp. 261-264, Hong Kong, Dec. 2005.
  • [15]  S. C. Chan, K. T Ng, Z. F. Gan, K. L. Chan, and H. Y. Shum, “The plenoptic videos,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 15, no. 12, pp. 1650-2659, Dec., 2005.
  • [16]  L. Y. Chen, S. C. Chan, and H. Y. Shum, “A joint motion-image inpainting method for error concealment in video coding,” in Proc. IEEE ICIP, pp. 2241-2244, Oct 8-11, 2006.
  • [17]  Q. Wu, S. C. Chan, and H. Y. Shum, “A convex optimization-based frame-level rate control algorithm for motion compensated hybrid DCT/DPCM video coding,” in Proc. IEEE ICIP, pp. 2469-2472, Oct. 2006. 
  • [18]  Q. Wu, S. C. Chan, and H. Y. Shum, “A convex optimization based object-level rate control algorithm for MPEG-4 video object coding,” in Proc. Asia Pacific Conf. Circuits And Syst., pp. 785-788, Dec. 2006. 
  • [19]  H. Y. Shum, S. C. Chan, and S. B. Kang.  Image-based rendering. Springer, 2007.
  • [20]  S. C. Chan, H. Y. Shum, and K. T. Ng, “Image-based rendering and synthesis: technological advances and challenges,” IEEE Signal Processing Magazine: Special Issue on MVI and 3DTV, vol. 24, no. 6, pp. 22-33, Nov. 2007.