Saliency map in image visual quality assessment and processing

Vladimir Lukin, Ekaterina Bataeva, Sergey Abramov

Abstract


Images are mainly viewed and analyzed by humans. Because of this, in the characterization of image quality and effectiveness of image processing, it is necessary to take into account the peculiarities of the human vision system and cognition that are very complex. Saliency maps as well as priority and meaning maps introduced recently are the attempts to incorporate specific features of human vision into image analysis and processing fields. Many authors that consider the aforementioned maps consider them from different viewpoints. Thus, the basic subject of this paper is the factors that influence and determine these maps. Among such factors, there are low-level features as well as social and psychological ones such as emotions, age, and life values. The main goal of this paper is to give a brief survey of these factors and to consider how maps are already used in image quality assessment and processing as well as how they can be employed in the future. The tasks of the paper are to provide a definition of saliency, priority, and meaning maps, to analyze the factors that influence these maps, and to evaluate what improvement can be obtained due to taking maps into account in the assessment of image visual quality and such image processing operations as quality assessment, denoising, and lossy compression. The main result is that, by taking saliency maps into account, image quality assessment and processing efficiency can be sufficiently improved, especially for applications oriented on image viewing and analysis by observers or customers. This can be done by the simple weighting of local estimates of a given metric with further aggregation as well as by approaches based on neural networks. Using different quantitative criteria, we show what positive results can be got due to incorporating maps into quality assessment and image processing. As conclusion, we present possible directions of future research that are mainly related to an adaptation of denoising and lossy compression parameters to peculiarities of human attention.

Keywords


saliency map; quality assessment; image processing

Full Text:

PDF

References


Kussul, N., Lavreniuk, M., Shelestov, A., Skakun, S. Crop inventory at regional scale in Ukraine: Developing in season and end of season crop maps with multi-temporal optical and SAR satellite imagery. European Journal of Remote Sensing, 2018, vol. 51, no. 1, pp. 627-636. DOI: 10.1080/22797254.2018.1454265.

Zappavigna, M. Social media photography: construing subjectivity in Instagram images. Visual Communication, 2016, vol. 15, no. 3, pp. 271–292. DOI: 10.1177/1470357216643220.

Bataeva, E. V. Flanering and video mania: Modern and postmodern visual practices. Voprosy Filosofii, 2012, vol. 11, pp. 61–68. Available at: https://www.researchgate.net/publication/331037254_Flan-erstvo_i_videomania_modernye_i_postmodernye_vizualnye_praktiki (accessed 9.01.2023).

Lee, L.-K., Liew, S.-C. A Survey of Medical Image Processing Tools. 4th International Conference on Software Engineering and Computer Systems (ICSECS), 2015, pp. 171-176. DOI: 10.13140/RG.2.1.3364.4241.

Daugherty, G. (ed.) Medical Image Processing: Techniques and Applications. Springer New York, NY, 2011. 380 p. DOI: 10.1007/978-1-4419-9779-1.

Safridatul, A. et al. Visual Analysis of Satellite Landsat Images Multitemporal and GPS as a Geographic Information System for Mapping of Nugmet Plantations in Tapaktuan. 1st South Aceh International Conference on Engineering and Technology, 2019. 13 p. DOI: 10.1088/1757-899X/506/1/012037.

Treisman, A. M., Gelade, C. A feature-integration theory of attention. Cognitive Psychology, 1980, vol. 12, no 1, pp. 97-136. DOI: 10.1016/0010-0285(80)90005-5.

Koch, C., Ullman, S. Shifts in selective visual attention: towards the underlying neural circuitry. Human Neurobiology, 1985, vol. 4, pp. 219-227. DOI: 10.1007/978-94-009-3833-5_5.

Olshausen, B. A., Anderson, C. H. Van Essen, D. C. A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information. J Neuroscience, 1993, vol. 13, iss. 11, pp. 4700–4719. DOI: 10.1523/JNEUROSCI.13-11-04700.1993.

Tsotsos, J. K. et al. Modeling visual attention via selective tuning. Artificial Intelligence, 1995, vol. 78, iss. 1-2, pp. 507-545. DOI: 10.1016/0004-3702(95)00025-9.

Itti, L., Koch, C., Niebur, E. Computational modelling of visual attention. Nature Reviews Neuroscience, 2001, vol. 2, pp. 194–203. DOI: 10.1038/35058500.

Itti, L., Koch, C., Niebur, E. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998, no. 20, pp. 1254–1259. DOI: 10.1109/34.730558.

Machner, B. et al. Unbalancing the Attentional Priority Map via Gaze-Contingent Displays Induces Neglect-Like Visual Exploration. Frontiers in Human Neuroscience, 2020, vol. 14, Article no. 41. DOI: 10.3389/fnhum.2020.00041.

Li, J. Y. Visual Attention is Beyond One Single Saliency Map, 2018. Available at: https://arxiv.org/abs/1811.02650 (accessed 9.01.2023).

Cong, R. et al. Review of Visual Saliency Detection with Comprehensive Information. IEEE Transactions on Circuits and Systems for Video Technology, 2018, vol. 29, pp. 2941-2959. DOI: 10.1109/TCSVT.2018.2870832.

Fecteau, J. H., Munoz, D. P. Salience, relevance, and firing: a priority map for target selection. Trends in Cognitive Sciences, 2006, vol. 10, iss. 8, pp. 382-390. DOI: 10.1016/j.tics.2006.06.011.

Henderson, J. M., Hayes, T. R. Meaning-based guidance of attention in scenes as revealed by meaning maps. Nature Human Behaviour, 2017, no. 1, pp. 743–747. DOI: 10.1038/s41562-017-0208-0.

Rahman, S. et al. Classifying Eye-Tracking Data Using Saliency Maps. 25th International Conference on Pattern Recognition (ICPR), 2020, pp. 9288-9295. DOI: 10.1109/ICPR48806.2021.9412308.

Ramanathan, S. et al. An Eye Fixation Database for Saliency Detection in Images. Lecture Notes in Computer Science, 2010, vol. 6314, pp. 30-43. DOI: 10.1007/978-3-642-15561-1_3.

Liu, T. et al. Learning to detect a salient object. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, no. 33, iss. 2, pp. 353-367. DOI: 10.1109/TPAMI.2010.70.

Obeso, A. M. et al. Comparative study of visual saliency maps in the problem of classification of architectural images with Deep CNNs. Eighth International Conference on Image Processing Theory, Tools and Applications (IPTA), 2018, pp. 1-6. DOI: 10.1109/IPTA.2018.8608125.

Ullah, I. et al. A brief survey of visual saliency detection. Multimedia Tools and Applications, 2020, vol. 79, pp. 34605-34645. DOI: 10.1007/s11042-020-08849-y.

Liu, Z., Huang, Y., Niu, Y., Lin, L. Saliency-aware Image Quality Assessment. Proceedings of the 2016 2nd Workshop on Advanced Research and Technology in Industry Application, 2016, pp. 1352-1358. DOI: 10.2991/wartia-16.2016.280.

Layek, M. A. et al. Center-Emphasized Visual Saliency and a Contrast-Based Full Reference Image Quality Index. Symmetry, 2019, vol. 11, iss. 3, article no. 296. DOI: 10.3390/sym11030296.

Kim, C., Milanfar, P. Visual saliency in noisy images. Journal of Vision, 2013, vol. 13, pp. 1-14. DOI: 10.1167/13.4.5.

Yu, S. X., Lisin, D. A. Image Compression Based on Visual Saliency at Individual Scales. Advances in Visual Computing. Lecture Notes in Computer Science, 2009, vol. 5875, pp. 157-166. DOI: 10.1007/978-3-642-10331-5_15.

Gupta, R., Khanna, M. T., Chaudhury, S. Visual saliency guided video compression algorithm. Signal Processing: Image Communication, 2013, vol. 28, iss. 9, pp. 1006-1022. DOI: 10.1016/j.image.2013.07.003.

Zhou, B., Zhang, Z., Hao, W., Wang, Z. Image deblurring based on visual saliency. International Conference on Systems and Informatics, 2012, pp. 1919-1922. DOI: 10.1109/ICSAI.2012.6223423.

Hu, C., Jia, S., Zhang, F., Li, X. A saliency-guided street view image inpainting framework for efficient last-meters wayfinding. ISPRS Journal of Photogrammetry and Remote Sensing, 2023, vol. 195, pp. 365-379. DOI: 10.1016/j.isprsjprs.2022.11.009.

Desingh, K., Krishna, K.M., Rajan, D., Jawahar, C.V. Depth really Matters: Improving Visual Salient Region Detection with Depth. Proceedings British Machine Vision Conference, 2013, pp. 98.1-98.11. DOI: 10.5244/C.27.98.

Einhäuser, W., Rutishauser, U., Koch, C. Task-demands can immediately reverse the effects of sensory-driven saliency in complex visual stimuli. Journal of Vision, 2008, vol. 8, iss. 2, pp. 1–19. DOI: 10.1167/8.2.2.

Serences, J. T., Yantis, S. Selective visual attention and perceptual coherence, Trends in Cognitive Sciences, 2006, vol. 10, iss. 1, pp. 38-45. DOI: 10.1016/j.tics.2005.11.008.

Veale, R., Hafed, Z. M., Yoshida, M. How is visual salience computed in the brain? Insights from behavior, neurobiology and modeling. Philosophical Transactions of the Royal Society B: Biological Sciences, 2017, vol. 372, iss. 1714. DOI: 10.1098/rstb.2016.0113.

Henderson, J. M., Hayes, T. R., Peacock, C. E., Rehrig, G. Meaning and attentional guidance in scenes: A review of the meaning map approach. Vision, 2019, vol. 3, iss. 2, pp. 1-19. DOI: 10.3390/vision3020019.

Dharani, T., Aroquiaraj, I. L. A survey on content based image retrieval. International Conference on Pattern Recognition, Informatics and Mobile Engineering, 2013, pp. 485-490. DOI: 10.1109/ICPRIME.2013.6496719.

Yang, H. et al. Blind Image Quality Assessment of Natural Distorted Image Based on Generative Adversarial Networks. IEEE Access, 2019, vol. 7, pp. 179290-179303. DOI: 10.1109/ACCESS.2019.2957235.

Wang, H., Xiao, S. An Object IQA Framework Based on Saliency Map Similarity, IEEE 6th International Conference on Computer and Communications (ICCC), 2020, pp. 1985-1989. DOI: 10.1109/ICCC51575.2020.9345144.

Lu, D., Yan, L. Face Detection and Recognition Algorithm in Digital Image Based on Computer Vision Sensor. Journal of Sensors, 2021, vol. 2021, pp. 1-16. DOI: 10.1155/2021/4796768.

Humphrey, K., Underwood, G., Lambert, T. Salience of the lambs: A test of the saliency map hypothesis with pictures of emotive objects. Journal of Vision, 2012, vol. 12, iss. 1, article no. 22, pp. 1-15. DOI: 10.1167/12.1.22.

Nuthmann, A., Schütz, I., Einhäuser, W. Salience-based object prioritization during active viewing of naturalistic scenes in young and older adults. Scientific Reports, 2020, vol. 10, iss. 1, article no. 22057. DOI: 10.1038/s41598-020-78203-7.

Henderson, J. M., Hayes, T. R., Peacock, C. E., Rehrig, G. Meaning maps capture the density of local semantic features in scenes: A reply to Pedziwiatr, Kümmerer, Wallis, Bethge & Teufel. Cognition, 2021, vol. 214, article no. 104742. DOI: 10.1016/j.cognition.2021.104742.

Kulkarni, S. S., Reddy, N. P., Hariharan, S. Facial expression (mood) recognition from facial images using committee neural networks. BioMedical Engineering OnLine, 2009, vol. 8, pp. 1-16. DOI: 10.1186/1475-925X-8-16.

Liu, Z., Huang, Y., Niu, Y., Lin, L. Saliency-aware Image Quality Assessment, 2nd Workshop on Advanced Research and Technology in Industry Applications, 2016, pp. 1354-1360. DOI: 10.2991/wartia-16.2016.280.

Ponomarenko, N. et al. Image database TID2013: Peculiarities, results and perspectives. Signal Processing: Image Communication, 2015, vol. 30, pp. 57-77. DOI: 10.1016/j.image.2014.10.009.

Wang, Z., Bovik, A. C., Sheikh, H. R., Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, vol. 13, iss. 4, pp. 600-612. DOI: 10.1109/TIP.2003.819861.

Larson, E. C., Chandler, D. Most apparent distortion: Full-reference image quality assessment and the role of strategy. Journal of Electronic Imaging, 2010, vol. 19, iss. 1, article no. 011006. DOI: 10.1117/1.3267105.

Nafchi, H. Z., Shahkolaei, A., Hedjam, R., Cheriet, M. Mean Deviation Similarity Index: Efficient and Reliable Full-Reference Image Quality Evaluator. IEEE Access, 2016, vol. 4, pp. 5579-5590. DOI: 10.1109/ACCESS.2016.2604042.

Tong, Y., Konik, H, Cheikh, F.A., Tremeau, A. Full Reference Image Quality Assessment Based on Saliency Map Analysis. Journal of Imaging Science and Technology, 2010, vol. 54, iss. 3, pp. 1-21. DOI: 10.2352/J.ImagingSci.Technol.2010.54.3.030503.

Ponomarenko, N. et al. On between-coefficient contrast masking of DCT basis functions. International Conference on VPQM, USA, 2007, pp. 1-4. Available at: https://ponomarenko.info/vpqm07_p.pdf (accessed 9.01.2023).

Varga, D. Saliency-Guided Local Full-Reference Image Quality Assessment. Signals, 2022, vol. 3, iss. 3, pp. 483-496. DOI: 10.3390/signals3030028.

Zhang, L., Shen, Y., Li, H. VSI: A Visual Saliency-Induced Index for Perceptual Image Quality Assessment. IEEE Transactions on Image Processing, 2014, vol. 23, pp. 4270-4281. DOI: 10.1109/TIP.2014.2346028.

Uddin, A. F. M. S., Chung, T. C., Bae, S.-H. Visual saliency based structural contrast quality index. Electronics Letters, 2019, vol. 55, iss. 4, pp. 194-196. DOI: 10.1049/el.2018.6435.

Adrizzone, E., Bruno, A. Image Quality Assessment by saliency maps. VISAPP, 2012, vol. 1, pp. 479-483. Available at: https://www.scitepress.org/papers/2012/38677/38677.pdf (accessed 9.01.2023).

Ma, L., Li, S., Ngan, K. N. Visual Horizontal Effect for Image Quality Assessment. IEEE Signal Processing Letters, 2010, vol. 17, iss. 7, pp. 627-630. DOI: 10.1109/LSP.2010.2048726.

Ryu, J. A Visual Saliency-Based Neural Net-work Architecture for No-Reference Image Quality Assessment. Applied Sciences, 2022, vol. 12, iss. 19, article no. 9567. DOI: 10.3390/app12199567.

Charrier, C., Saadane, A., Fernandez-Maloigne, C. Blind Image Quality Assessment based on the use of Saliency Maps and a Multivariate Gaussian Distribution. 20th International Conference on Image Analysis and Processing (ICIAP), 2019, pp. 1-12. DOI: 10.1007/978-3-030-30645-8_13.

Lamichhane, K., Carli, M., Battisti, F. A CNN-based no reference image quality metric exploiting content saliency. Signal Processing: Image Communication, 2023, vol. 111, article no. 116899. DOI: 10.1016/j.image.2022.116899.

Li, F., Krivenko, S., Lukin, V. Two-step providing of desired quality in lossy image compression by SPIHT. Radioelectronic and computer systems, 2020, no. 2(94), pp. 22-32. DOI: 10.32620/reks.2020.2.02.

Oh, H., Bilgin, A., Marcellin, M. Visually lossless JPEG 2000 for remote image browsing. Information, 2016, vol. 7, no. 3, article no. 45. DOI: 10.3390/info7030045.

Bondžulić, B. et al. Efficient prediction of the first just noticeable difference point for JPEG compressed images. Acta Polytechnica Hungarica, 2021, vol. 18, iss. 8, pp. 201-220. DOI: 10.12700/APH.18.8.2021.8.11.

Yang, K., Jiang, H. Optimized-SSIM based quantization in optical remote sensing image compression. Proceedings of Sixth International Conference on Image and Graphics, 2011, pp. 117-122. DOI: 10.1109/ICIG.2011.38.

Jeong, Y. W. et al. Rate distortion optimization encoding system and method of operating the same. Patent US, no. 10,742,995 B2. 2020. Available at: https://patents.justia.com/patent/10742995 (accessed 9.01.2023).

Ortega, A., Ramchandran, K. Rate-distortion methods for image and video compression. IEEE Signal Processing Magazine, 1998, vol. 15, no. 6, pp. 23-50. DOI: 10.1109/79.733495.

Jin, L. et al. Statistical study on perceived JPEG image quality via MCL-JCI dataset construction and analysis. Proceedings of IS&T International Symposium on Electronic Imaging: Image Quality and System Performance XIII, 2016, vol. 28, pp. 1-9. DOI: 10.2352/ISSN.2470-1173.2016.13.IQSP-222.

Patel, Y., Appalaraju, S., Manmatha, R. Saliency Driven Perceptual Image Compression. IEEE Winter Conference on Applications of Computer Vision (WACV), 2021, pp. 227-236. DOI: 10.1109/WACV48630.2021.00027.

Hu, S. et al. Compressed image quality metric based on perceptually weighted distortion. IEEE Transactions on Image Processing, 2015, vol. 24, iss. 12, pp. 5594-5608. DOI: 10.1109/TIP.2015.2481319.

Golner, M., Mikhael, W., Krishnang, V. Modified JPEG Image Compression With Region-dependent Quantization. Circuits, Systems and Signal Processing, 2002, vol. 21, pp. 163–180. DOI: 10.1007/s00034-002-2004-x.

Lee, S.-H., Shin, J.-K., Lee, M. Non-uniform image compression using biologically motivated saliency map model. Proceedings of the 2004 Intelligent Sensors, Sensor Networks and Information Processing Conference, 2004, pp. 525-530. DOI: 10.1109/ISSNIP.2004.1417516.

Itti, L. Automatic foveation for video compres-sion using a neurobiological model of visual attention. IEEE Transactions on Image Processing, 2004, vol.13, pp. 1304-1318. DOI: 10.1109/TIP.2004.834657.

Hadizadeh, H., Bajić, I. V. Saliency-aware video compression. IEEE Trans. Image Process, 2014, vol. 23, iss. 1, pp. 19-33. DOI: 10.1109/TIP.2013.2282897.

Li, Z., Qin, S., Itti, L. Visual attention guided bit allocation in video compression. Image and Vision Computing, 2011, vol. 29, iss. 1, pp. 1-14. DOI: 10.1016/j.imavis.2010.07.001.

Cai, C., Chen, L., Zhang, X., Gao, Z. End-to-End Optimized ROI Image Compression. IEEE Transactions on Image Processing, 2020, vol. 29, pp. 3442-3457. DOI: 10.1109/TIP.2019.2960869.

Kaur, R., Rani, R. ROI and Non-ROI based Medical Image Compression Techniques: A Survey and Comparative Review. First International Conference on Secure Cyber Computing and Communication (ICSCCC), 2018, pp. 550-555. DOI: 10.1109/ICSCCC.2018.8703337.

Astola, J., Kuosmanen, P. Fundamentals of Nonlinear Digital Filtering. Boca Raton, CRC Press Publ., 1997. 288 p. DOI: 10.1201/9781003067832.

Lee, J.-S. Refined filtering of image noise us-ing local statistics. Computer Graphics and Image Processing, 1981, vol. 15, iss. 4, pp. 380-389. DOI: 10.1016/S0146-664X(81)80018-4.

Lee, J. S. Digital image smoothing and the sig-ma filter. Computer Vision, Graphics, and Image Processing, 1983, vol. 24, iss. 2, pp. 255-269. DOI: 10.1016/0734-189X(83)90047-6.

Lukin, V. V. et al. Digital adaptive robust algorithms for radar image filtering. Journal of Electronic Imaging, 1996, vol. 5, iss. 3, pp. 410-421. DOI: 10.1117/12.240715.

Mallat, S. A Wavelet Tour of Signal Processing, Academic Press, 2008. 832 p. Available at: https://www.sciencedirect.com/book/9780123743701/a-wavelet-tour-of-signal-processing (accessed 9.01.2023).

Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K. Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering. IEEE Transactions on Image Processing, 2007, vol. 16, iss. 8, pp. 2080-2095. DOI: 10.1109/TIP.2007.901238.

Mäkitalo, M., Foi, A., Fevralev, D., Lukin, V. Denoising of single-look SAR images based on variance stabilization and nonlocal filters. International Conference on Mathematical Methods in Electromagnetic Theory, 2010, pp. 1-4. DOI: 10.1109/MMET.2010.5611418.

Rubel, A., Rubel, O, Lukin, V. On Prediction of Image Denoising Expedience Using Neural Networks. Proceedings of PICST, 2018, pp. 629-634. DOI: 10.1109/INFOCOMMST.2018.8632050.

Zemliachenko, A. et al., Analysis of Opportunities to Improve Image Denoising Efficiency for DCT-based Filters. Radioelectronic and Computer Systems, 2018, no. 2, pp. 4-12. DOI: 10.32620/reks.2018.2.01.

Pogrebnyak, O., Lukin, V. Wiener discrete cosine transform-based image filtering. Journal of Electronic Imaging, 2012, no 4, pp. 1-14. DOI: 10.1117/1.JEI.21.4.043020.

Arun, N. et al. Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging. Radiology: Artificial Intelligence, 2021, vol. 3, pp. 1-12. DOI: 10.1148/ryai.2021200267.

Hosokawa, S. et al. Validation of a Saliency Map for Assessing Image Quality in Nuclear Medicine: Experimental Study Outcomes. Radiation, 2022, vol. 2, pp. 248-258. DOI: 10.3390/radiation2030018.

Lyudvichenko, V., Erofeev, M., Ploshkin, A., Vatolin, D. Improving Video Compression with Deep Visual-attention Models. IMIP '19: Proceedings of the 2019 International Conference on Intelligent Medicine and Image Processing, 2019, pp. 88–94. DOI: 10.1145/3332340.3332358.

Pelurson, S. et al. AI-Based Saliency-Aware Video Coding. SMPTE Motion Imaging Journal, 2022, vol. 131, iss. 4, pp. 21-29. DOI: 10.5594/JMI.2022.3160541.

Chahid, A., Serrai, H., Achten, E., Laleg-Kirati, T.-M. A New ROI-Based performance evaluation method for image denoising using the Squared Eigenfunctions of the Schrödinger Operator. 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2018, pp. 5579-5582. DOI: 10.1109/EMBC.2018.8513615.

Rubel, O., Rubel, A., Lukin, V., Egiazarian, K. Selection of Lee Filter Window Size Based on Despeckling Efficiency Prediction for Sentinel SAR Images. Remote Sensing, 2021, vol. 13, article no. 1887. DOI: 10.3390/rs13101887.




DOI: https://doi.org/10.32620/reks.2023.1.09

Refbacks

  • There are currently no refbacks.