Journal Home > Volume 9 , Issue 2

Mesh-based image warping techniques typically represent image deformation using linear functions on triangular meshes or bilinear functions on rectangular meshes. This enables simple and efficient implementation, but in turn, restricts the representation capability of the deformation, often leading to unsatisfactory warping results. We present a novel, flexible polygonal finite element (poly-FEM) method for content-aware image warping. Image deformation is represented by high-order poly-FEMs on a content-aware polygonal mesh with a cell distribution adapted to saliency information in the source image. This allows highly adaptive meshes and smoother warping with fewer degrees of freedom, thus significantly extending the flexibility and capability of the warping representation. Benefiting from the continuous formulation of image deformation, our poly-FEM warping method is able to compute the optimal image deformation by minimizing existing or even newly designed warping energies consisting of penalty terms for specific transformations. We demonstrate the versatility of the proposed poly-FEM warping method in representing different deformations and its superiority by comparing it to other existing state-of-the-art methods.


menu
Abstract
Full text
Outline
About this article

Polygonal finite element-based content-aware image warping

Show Author's information Juan Cao1,2( )Xiaoyi Zhang1,2Jiannan Huang1,2Yongjie Jessica Zhang3
School of Mathematical Sciences, Xiamen University, Xiamen361005, China
Fujian Provincial Key Laboratory of Mathematical Modeling and High-Performance Scientific Computation, Xiamen University, Xiamen 361005, China
Department of Mechanical Engineering, CarnegieMellon University, Pittsburgh, PA 15213, USA

Abstract

Mesh-based image warping techniques typically represent image deformation using linear functions on triangular meshes or bilinear functions on rectangular meshes. This enables simple and efficient implementation, but in turn, restricts the representation capability of the deformation, often leading to unsatisfactory warping results. We present a novel, flexible polygonal finite element (poly-FEM) method for content-aware image warping. Image deformation is represented by high-order poly-FEMs on a content-aware polygonal mesh with a cell distribution adapted to saliency information in the source image. This allows highly adaptive meshes and smoother warping with fewer degrees of freedom, thus significantly extending the flexibility and capability of the warping representation. Benefiting from the continuous formulation of image deformation, our poly-FEM warping method is able to compute the optimal image deformation by minimizing existing or even newly designed warping energies consisting of penalty terms for specific transformations. We demonstrate the versatility of the proposed poly-FEM warping method in representing different deformations and its superiority by comparing it to other existing state-of-the-art methods.

Keywords: finite element method (FEM), mesh generation, image warping, polygonal element

References(48)

[1]
Vaquero, D.; Turk, M.; Pulli, K.; Tico, M.; Gelfand, N. A survey of image retargeting techniques. In: Proceedings of the SPIE 7798, Applications of Digital Image Processing XXXIII, 779814, 2010.
DOI
[2]
Kiess, J.; Kopf, S.; Guthier, B.; Effelsberg, W. A survey on content-aware image and video retargeting. ACM Transactions on Multimedia Computing, Communications, and Applications Vol. 14, No. 3, Article No. 76, 2018.
[3]
Suh, B.; Ling, H.; Bederson, B. B.; Jacobs. D. W. Automatic thumbnail cropping and its effectiveness. In: Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology, 95–104, 2003.
DOI
[4]
Nishiyama, M.; Okabe, T.; Sato, Y.; Sato, I. Sensation-based photo cropping. In: Proceedings of the 17th ACM International Conference on Multimedia, 669–672, 2009.
DOI
[5]
Setlur, V.; Takagi, S.; Raskar, R.; Gleicher, M.; Gooch, B. Automatic image retargeting. In: Proceedings of the 4th International Conference on Mobile and Ubiquitous Multimedia, 59–68, 2005.
DOI
[6]
Avidan, S.; Shamir, A. Seam carving for content-aware image resizing. ACM Transactions on Graphics Vol. 26, No. 3, 10–es, 2007.
[7]
Rubinstein, M.; Shamir, A.; Avidan, S. Improved seam carving for video retargeting. ACM Transactions on Graphics Vol. 27, No. 3, 1–9, 2008.
[8]
Rubinstein, M.; Shamir, A.; Avidan, S. Multi-operator media retargeting. ACM Transactions on Graphics Vo. 28, No. 3 Article No. 23, 2009.
[9]
Barnes, C.; Shechtman, E.; Finkelstein, A.; Goldman, D. B. PatchMatch: A randomized correspondence algorithm for structural image editing. In: Proceedings of the ACM SIGGRAPH 2009 papers, Article No. 24, 2009.
DOI
[10]
Pritch, Y.; Kav-Venaki, E.; Peleg, S. Shift-map image editing. In: Proceedings of the IEEE 12th International Conference on Computer Vision, 151–158, 2009.
DOI
[11]
Guo, Y. W.; Liu, F.; Shi, J.; Zhou, Z. H.; Gleicher, M. Image retargeting using mesh parametrization. IEEE Transactions on Multimedia Vol. 11, No. 5, 856–867, 2009.
[12]
Lau, C. P.; Yung, C. P.; Lui, L. M. Image retargeting via Beltrami representation. IEEE Transactions on Image Processing Vol. 27, No. 12, 5787–5801, 2018.
[13]
Wolf, L.; Guttmann, M.; Cohen-Or, D. Non-homogeneous content-driven video-retargeting. In: Proceedings of the IEEE 11th International Conference on Computer Vision, 1–6, 2007.
DOI
[14]
Wang, Y.-S.; Tai, C.-L.; Sorkine, O.; Lee, T.-Y. Optimized scale-and-stretch for image resizing. ACM Transactions on Graphics Vol. 27, No. 5, Article No. 118, 2008.
[15]
Karni, Z.; Freedman, D.; Gotsman, C. Energy-based image deformation. Computer Graphics Forum Vol. 28, 1257–1268, 2009.
[16]
Krähenbühl, P.; Lang, M.; Hornung, A.; Gross, M. A system for retargeting of streaming video. ACM Transactions on Graphics Vol. 28, No. 5, 1–10, 2009.
[17]
Shi, M.; Yang, L.; Peng, G.; Xu, D. A content-aware image resizing method with prominent object size adjusted. In: Proceedings of the 17th ACM Symposium on Virtual Reality Software and Technology, 175–176, 2010.
DOI
[18]
Chen, R. J.; Freedman, D.; Karni, Z.; Gotsman, C.; Liu, L. G. Content-aware image resizing by quadratic programming. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, 1–8, 2010.
DOI
[19]
Cho, D.; Park, J.; Oh, T. H.; Tai, Y. W.; Kweon, I. S. Weakly- and self-supervised learning for content-aware deep image retargeting. In: Proceedings of the IEEE International Conference on Computer Vision, 4568–4577, 2017.
DOI
[20]
Tan, W. M.; Yan, B.; Lin, C. M.; Niu, X. J. Cycle-IR: Deep cyclic image retargeting. IEEE Transactions on Multimedia Vol. 22, No. 7, 1730–1743, 2020.
[21]
Zhou, Y.; Chen, Z. B.; Li, W. P. Weakly supervised reinforced multi-operator image retargeting. IEEE Transactions on Circuits and Systems for Video Technology Vol. 31, No. 1, 126–139, 2021.
[22]
Danon, D.; Arar, M.; Cohen-Or, D.; Shamir, A. Image resizing by reconstruction from deep features. Computational Visual Media Vol. 7, No. 4, 453–466, 2021.
[23]
Hu, S. M.; Li, C. F.; Zhang, H. Actual morphing: A physics-based approach to blending. In: Proceedings of the 9th ACM Symposium on Solid Modeling and Applications, 309–314, 2004.
[24]
Yan, H.-B.; Hu, S.-M.; Martin, R. Morphing based on strain field interpolation. Computer Animation and Virtual Worlds Vol. 15, Nos. 3–4, 443–452, 2004.
[25]
Bajaj, C.; Zhang, Y.; Xu, G. Physically-based surface texture synthesis using a coupled finite element system. In: Advances in Geometric Modeling and Processing. Lecture Notes in Computer Science, Vol. 4975. Chen, F.; Jüttler, B. Eds. Springer Berlin Heidelberg, 344–357, 2008.
[26]
Jacobson, A.; Tosun, E.; Sorkine, O.; Zorin, D. Mixed finite elements for variational surface modeling. Computer Graphics Forum Vol. 29, No. 5, 1565–1574, 2010.
[27]
Zhang, Y. J. Geometric Modeling and Mesh Generation from Scanned Images. New York: Chapman and Hall/CRC, 2016.
DOI
[28]
Gee, J. C.; Haynor, D. R.; Martin, R. M. D.; Bajcsy, R. K. Finite element approach to warping of brain images. In: Proceedings of the SPIE 2167, Medical Imaging 1994: Image Processing, 327–337, 1994.
[29]
Kaufmann, P.; Wang, O.; Sorkine-Hornung, A.; Sorkine-Hornung, O.; Smolic, A.; Gross, M. Finite element image warping. Computer Graphics Forum Vol. 32, No. 2pt1, 31–39, 2013.
[30]
Wachspress, E. L. A Rational Finite Element Basis. Elsevier, 1975.
[31]
Meyer, M.; Barr, A.; Lee, H.; Desbrun, M. Generalized barycentric coordinates on irregular polygons. Journal of Graphics Tools Vol. 7, No. 1, 13–22, 2002.
[32]
Hormann, K.; Floater, M. S. Mean value coordinates for arbitrary planar polygons. ACM Transactions on Graphics Vol. 25, No. 4, 1424–1441, 2006.
[33]
Rand, A.; Gillette, A.; Bajaj, C. Quadratic serendipity finite elements on polygons using generalized barycentric coordinates. Mathematics of Computation Vol. 83, 2691–2716, 2014.
[34]
Sukumar, N. Quadratic maximum-entropy serendipity shape functions for arbitrary planar polygons. Computer Methods in Applied Mechanics and Engineering Vol. 263, 27–41, 2013.
[35]
Floater, M. S.; Lai, M. J. Polygonal spline spaces and the numerical solution of the Poisson equation. SIAM Journal on Numerical Analysis Vol. 54, No. 2, 797–824, 2016.
[36]
Cao, J.; Xiao, Y. Y.; Chen, Z. G.; Wang, W. P.; Bajaj, C. Functional data approximation on bounded domains using polygonal finite elements. Computer Aided Geometric Design Vol. 63, 149–163, 2018.
[37]
Engelke, U.; Wang, J. L.; Marendy, P. Perceptual relevance based image retargeting. IEEE Signal Processing Letters Vol. 22, No. 6, 705–708, 2015.
[38]
Wang, J.; Jiang, H.; Yuan, Z.; Cheng, M.-M.; Hu, X.; Zheng, N. Salient object detection: A discriminative regional feature integration approach. International Journal of Computer Vision Vol. 123, No. 2, 251–268, 2017.
[39]
Schneider, T. Theory and applications of bijective barycentric mappings. Ph.D. Thesis. Universita della Svizzera italiana, 2017.
DOI
[40]
Sieger, D.; Alliez, P.; Botsch, M. Optimizing Voronoi Diagrams for polygonal finite element computations. In: Proceedings of the 19th International Meshing Roundtable, 335–350, 2010.
DOI
[41]
Du, Q.; Faber, V.; Gunzburger, M. Centroidal voronoi tessellations: Applications and algorithms. SIAM Review Vol. 41, No. 4, 637–676, 1999.
[42]
Gupta, S.; Mazumdar, S. G. Sobel edge detection algorithm. International Journal of Computer Science and Management Research Vol. 2, No. 2, 1578–1583, 2013.
[43]
Laffont, P.-Y.; Jun, J. Y.; Wolf, C.; Tai, Y.-W.; Idrissi, K.; Drettakis, G.; Yoon, S.-e. Interactive content-aware zooming. In: Proceedings of the Graphics Interface, 79–87, 2010.
[44]
Zhang, G.-X.; Cheng, M.-M.; Hu, S.-M.; Martin, R. R. A shape-preserving approach to image resizing. Computer Graphics Forum Vol. 28, No. 7, 1897–1906, 2009.
[45]
Tang, F.; Dong, W.; Meng, Y.; Ma, C.; Wu, F.; Li, X.; Lee, T.-Y. Image retargetability. IEEE Transactions on Multimedia Vol. 22, No. 3, 641–654, 2020.
[46]
Rubinstein, M.; Gutierrez, D.; Sorkine, O.; Shamir, A. A comparative study of image retargeting. ACM Transactions on Graphics Vol. 29. No. 6, Article No. 160, 2010.
[47]
Liu, Y. J.; Han, Y. H.; Ye, Z. P.; Lai, Y. K. Ranking-preserving cross-source learning for image retargeting quality assessment. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 42, No. 7, 1798–1805, 2020.
[48]
Li, X. Y.; Ju, T.; Hu, S. M. Cubic mean value coordinates. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 126, 2013.
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 09 February 2022
Accepted: 08 March 2022
Published: 03 January 2023
Issue date: June 2023

Copyright

© The Author(s) 2022.

Acknowledgements

We would like to thank the reviewers for their valuable comments. The research of Juan Cao was supported by the National Natural Science Foundation of China(Nos. 61872308, 61972327, and 62272402), and the XiamenYouth Innovation Funds (No. 3502Z20206029). YongjieJessica Zhang was supported in part by NSF CMMI-1953323 and a Honda grant.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.

Return