12. Alsaleh SM, Aviles-Rivero AI, Hahn JK. Retouchimg: fusioning from-local-to-global context detection and graph data structures for fully-automatic specular reflection removal for endoscopic images. Comput Med Imag Graph (2019) 73 39–48. doi:10.1016/j.compmedimag.2019.02.002

For measurement, we assess the performance of the proposed method with visual inspection and quantitative evaluation. Specifically, true positive pixels (TPPs) and false positive pixels (FPPs) are the pixels detected as highlights within and not in the ground truth, respectively. Similarly, true negative pixels (TNPs) and false negative pixels (FNPs) are the pixels marked as nonhighlights not in and within the ground truth, respectively. In this study, we employ Precision, Recall, Accuracy, and F1-score to quantify the performance of the proposed highlight detection method, which can be written as

This work was supported by the Natural Science Foundation of Guangdong Province, Grant No. 2019A1515011940; Science and Technology Program of Guangzhou, Grant Nos. 2019050001 and 202002030353; Science and Technology Planning Project of Guangdong Province, Grant Nos. 2017B030308009, 2017KZ010101, and 2018A030313990; Special Project for Youth Top-notch Scholars of Guangdong Province, Grant No. 2016TQ03X100; and also Scientific Research Cultivation Project for Young Scholars of South China Normal University, Grant No. 19KJ16.

An inverted microscope is an upside-down standard light microscope. This type of microscope is characterized by locating the objective underneath the stage and pointing upwards to the specimen. Inverted microscopes are mainly used for live cell analysis of cell cultures growing in culture medium. Culture dishes are not only available with standard coverslip bottoms (thickness 0.17mm) but also in various material and bottom thicknesses. Therefore, for some models special objectives are available that can correct for different bottom glass thicknesses. In addition, inverse microscopes are also used for studies of thicker specimens. Inverted microscopes are also available as dark field, polarization or fluorescence microscopes.

Firstly, we evaluate the detected results of highlight on clinical data and CVC-ClinicSpec database, respectively. To be specific, we conduct experiments on Figures 2A,B,C using different criteria to evaluate the performance of each procedure of the proposed detection method. Figures 2A,B are the original images from clinical data, and Figure 2C is from CVC-ClinicSpec. Figures 2D,E,F show the results using the detection procedure in (1). Clearly, only sporadic specular pixels can be detected due to the limitation of using only local information for comparison. By using the proposed criteria in (1) and (5), highlight can be well detected, especially for the case of large region, as shown in Figures 2G,H,I. Besides, from the comparison between Figure 2I and L, the effectiveness of distinguishing highlight from dark regions using (6) is verified. We can learn from Figures 2J–L that the proposed method can detect highlight precisely and achieve visually satisfying results.

9. Chu Y, Li H, Li X, Ding Y, Yang X, Ai D, et al. Endoscopic image feature matching via motion consensus and global bilateral regression. Comput Methods Progr Biomed (2020) 190 105370. doi:10.1016/j.cmpb.2020.105370

Additionally, as shown in (3) and (4), d is calculated from the information of each patch, which means t in (5) computed by d is adaptively set for each patch.

Next, we examine the performance improvement by using the preprocess of image segmentation. As elaborated in Proposed Method, in order to improve the adaptation of the thresholds in (1) and (5) for highlight detection, we apply image segmentation to the images from CVC-ClinicSpec database (Figure 3A,B,C) and divide each image into nine equisized subgraphs with 50% overlapping with each other. In Figures 3G,H,I, it can be observed that there is misjudgment at the boundaries of each window due to the inability to effectively use the boundary information. As shown in Figure 3, the highlight detection with image segmentation provides a superior solution in comparison with that without image segmentation. The result is expected, since the proposed adaptive thresholds in (1) and (5) are sensitive to uneven lighting, which leads to errors for pixels with high intensity of highlight or dark regions, as shown in Figures 3J–L. Comparatively, the image segmentation with overlapping can effectively reduce the effect of uneven lighting and enables a superior detection performance to that with nonoverlapped windows and without segmentation.

Copyright © 2021 Yu, Chen, Zhong and Zhang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

Polarization microscopy is used for the analysis of optically anisotropic samples. The primary objective of polarization microscopy is not magnification of an object, but rather the analysis of optical properties such as refractive index or birefringence for sample analysis. The method is mainly used in mineralogy and in industry for testing plastics or mineral building materials in order to gain insights into their composition.

However, specular reflection detection based on color space mainly suffers from two challenges. On one hand, empirical thresholds are set in advance and, thus, are unable to be adjusted adaptively in different scenes. On the other hand, large specular reflection regions with high intensity cannot be well detected due to the limited difference between highlights and the neighboring areas [16].

6. Sánchez F, Bernal J, Sánchez-Montes C, Rodríguez de Miguel C, Fernández-Esparrach G. Bright spot regions segmentation and classification for specular highlights detection in colonoscopy videos. Mach Vis Appl (2017) 28:917–36. doi:10.1007/s00138-017-0864-0

The physical principle of fluorescence is used to selectively visualize and localize defined fluorescent structures, while non-fluorescent structures remain dark in order to obtain a high image contrast. For this purpose, fluorescent dyes (fluorochromes) are used with specific excitation and emission filters installed in the optical path of the microscope. A wide range of fluorescent dyes with different colors are available, which are used in molecular biological, biomedical and clinical research. For example, in immunohistochemistry, fluorescence-in-situ-hybridization and for visualization of cells or cellular components in living/fixed specimens.

A dark field microscope produces a contrast-enhanced image by indirect illumination of the specimen, thereby also unstained specimens can be displayed with high contrast. Using this technique, direct light is bypassing the objective, only light scattered by the specimen enters the objective. As a result, the background appears dark or black, only the specimen is illuminated and even small structures can be visualized with high contrast. Dark field microscopy in biology and medicine is especially used for transparent and low contrast specimens, for example, studying blood, small animals or micro-particles in material science.

Finally, to validate the robustness of the proposed method, we test 24 images from CVC-ClinicSpec and compare the proposed method with the benchmark methods from different perspective. Figure 5 illustrates the statistical comparison on Precision, Recall, Accuracy, and F1-score using boxplots. It can be observed that the proposed method outperforms the benchmarks in terms of Precision, Accuracy, and F1-score. Theoretically, Precision and Recall are negatively correlated. Although the proposed method yields a lower averaged Recall value, the efficiency of the method can be verified by the highest F1-score, which is the harmonic mean of Precision and Recall.

15. Meslouhi O, Kardouchi M, Allali H, Gadi T, Benkaddour Y. Automatic detection and inpainting of specular reflections for colposcopic images. Open Computer Science (2011) 1:341–54. doi:10.2478/s13537-011-0020-2

Endoscopic imaging systems have been widely used in disease diagnosis and minimally invasive surgery. Practically, specular reflection (a.k.a. highlight) always exists in endoscopic images and significantly affects surgeons’ observation and judgment. Motivated by the fact that the values of the red channel in nonhighlight area of endoscopic images are higher than those of the green and blue ones, this paper proposes an adaptive specular highlight detection method for endoscopic images. Specifically, for each pixel, we design a criterion for specular highlight detection based on the ratio of the red channel to both the green and blue channels. With the designed criteria, we take advantage of image segmentation and then develop an adaptive threshold with respect to the differences between the red channel and the other ones of neighboring pixels. To validate the proposed method, we conduct experiments on clinical data and CVC-ClinicSpec open database. The experimental results demonstrate that the proposed method yields an averaged Precision, Accuracy, and F1-score rate of 88.76%, 99.60% and 72.56%, respectively, and outperforms the state-of-the-art approaches based on color distribution reported for endoscopic highlight detection.

FIGURE 3. Performance of the proposed highlight detection method with different traversal methods. (A)–(C) Original images from CVC-ClinicSpec. (D)–(F) 50% overlapped windows one-fourth the size of the original images. (G)–(I) Nonoverlapped windows one-ninth the size of the original images. (J)–(L) Using global information.

A metallurgical microscope is a special version of a standard light microscope for the study of materials, such as Metals, plastics, ceramics and others. Since materials are usually not transparent solid structures, metallurgical microscopes often have an upright light unit. Moreover, this type of microscope is characterized by extensive magnification, e.g. for detailed investigations of surface structures. Application areas of metallurgical microscopy are industry, material science and research.

For natural images, most of the pioneer works are based on various color spaces. Xia et al. [1] proposed a joint hue, saturation, value (HSV) and red, green, blue (RGB) space threshold detection set using gradient magnitude in dark channel in RGB space. In [2], based on the global brightness of the images, the threshold of HSV color space is automatically set to separate specular reflections. Besides, the dichromatic reflection model is widely used for specular reflection detection of natural images [3]. Specifically, this method uses intensity ratio to extract specular and diffuse components from images [4, 5]. Owing to the different characteristics of natural images, such as uneven color distribution and lighting, oversaturated specular highlight, the above methods designed for natural images are not applicable for endoscopic images ones.

JavaScript seems to be disabled in your browser. For the best experience on our site, be sure to turn on Javascript in your browser.

dark-field microscopy

We propose an adaptive threshold for specular reflection detection. By taking advantage of the difference between the red channel and the other ones, and integrating with overlapped windowing, the proposed adaptive threshold is applicable for large highlight regions with high intensity.

To address the above issues, we, in this study, propose an adaptive detection method based on the color distribution characteristics of endoscopic images. The main contributions are summarized in the following two points.

To further improve the adaptation of the criteria of threshold for highlight detection, we apply the preprocess of segmentation by splitting an image into overlapped equal-sized windows. Therefore, the problem of uneven illumination caused by single light source can be solved through setting adaptive threshold for each patch. Specifically, in each patch, the Imean in (1) is defined as

FIGURE 4. Comparison between different methods. (A) Original image from clinical data. (B)–(C) Original images from CVC-ClinicSpec. (D)–(F) [8]. (G)–(I) [13]. (J)–(L) Proposed method.

where I(x) is the intensity of pixel x, Imean is the mean intensity of small region to which pixel x belongs, and α represents a constant parameter. The above criterion is suitable for small specular region with high intensity which is more sensitive to surrounding information. However, pixels within large specular reflection regions are hard to be recognized due to higher Imean.

7. Gao Y, Yang J, Ma S, Ai D, Lin T, Tang S, et al. Dynamic searching and classification for highlight removal on endoscopic image. Procedia Comp Sci (2017) 107:762–7. doi:10.1016/j.procs.2017.03.161

However, for darker pixels, the values of all three channels are nearly identical, like highlight pixels. To avoid the confusion between the dark and the highlight cases, we apply an empirical value of threshold in grayscale of 200 as [8], i.e.,

4. Shen HL, Zheng ZH. Real-time highlight removal using intensity ratio. Appl Optic (2013) 52 4483–93. doi:10.1364/AO.52.004483

Citation: Yu B, Chen W, Zhong Q and Zhang H (2021) Specular Highlight Detection Based on Color Distribution for Endoscopic Images. Front. Phys. 8:616930. doi: 10.3389/fphy.2020.616930

FIGURE 2. Performance of the proposed highlight detection method. (A)–(B) Original images from clinical data. (C) Original image from CVC-ClinicSpec. (D)–(F) Applying (1). (G)–(I) Applying (1) and (5). (J)–(L) Proposed method.

1. Xia W, Chen ECS, Pautler SE, Peters TM. A global optimization method for specular highlight removal from a single image. IEEE Access (2019) 7:125976–90. doi:10.1109/ACCESS.2019.2939229

BY, WC, and HZ contributed the ideas. WC performed the experiments and drafted the paper. BY, WC, QZ, and HZ all did the revision.

Using the above designed criterion, the pixels of an image satisfying (5) and (6) are detected as highlight. In summary, the overall flowchart of the proposed criteria with adaptive threshold for highlight detection is shown in Figure 1.

5. Suo J, An D, Ji X, Wang H, Dai Q. Fast and high quality highlight removal from a single image. IEEE Trans Image Process (2016) 25:5441–54. doi:10.1109/TIP.2016.2605002

To further validate the superiority of the proposed method, we compare with the benchmark specular detection methods in [8, 13], as shown in Figure 4. Figures 4A and B‐C are the original images from clinical data and CVC-ClinicSpec, respectively. As it illustrated in Figures 4D,E,F, it is seen that method of [8] tends to detect a larger area than the actual highlight because of the dilation operation. Figure 4G shows the method in [13] is unable to detect large specular highlight region due to the application of local surrounding information. From Figures 4J–L, we can observe that the proposed method can obtain accurate highlight detection results and can take both large and small highlight regions into account. To further compare the performance of different methods, we perform quantitative evaluation on images of Figure 4. As shown in Table 1, although the proposed method performs inferior to that of [8, 13] in terms of Recall due to missing specular pixels, which are marked in blue in Figures 4K,L, our method obtains the best performance in terms of Precision, Accuracy, and F1-score.

We take advantage of color distribution and propose a criterion characterized by the ratio of the red channel to both the green and blue ones, which provides a more distinguishable feature of specular reflection of endoscopic images.

14. Akbari M, Mohrekesh M, Najariani K, Karimi N, Samavi S, Soroushmehr SMR. Adaptive specular reflection detection and inpainting in colonoscopy video frames. In: 2018 25th IEEE International conference on image processing; 2018 October 7–10; Athens, Greece. (ICIP) (2018). p. 3134–8.

In the results of the experiment shown below, for the detection results generated from clinical data, the white denotes the detected highlight. As for the results from CVC-ClinicSpec, the white, black, red, and blue denote TPPs, TNPs, FPPs, and FNPs, respectively.

Bright field dark field

For endoscopic images, the highlight detection methods can be mainly classified into methods based on different color spaces and the ones with classifier. Considering the real-time detection on highlight of endoscopic images, methods based on color spaces entail lower computations in comparison with those using machine learning techniques [6, 7] and thus have advantages in practice, even though the latter can achieve higher accuracy. At present, the most commonly used color spaces are gray-level [8, 9], HSV [10, 11], and RGB [12, 13]. Specifically, preset thresholds on different color spaces are employed to determine specular highlight. To reduce the impact of halo effect of highlight, Shen et al. [8] proposed a specular detection method by adopting a morphological dilation operation to enlarge the specular reflection regions which is obtained by preset threshold on gray-level image. To address the problem that some pixels in specular reflection areas have lower intensity than nonspecular reflection areas, Oh et al. [10] defined specular reflection area into absolute bright area and relative bright area, which is determined by outlier detection. However, the detected relative bright area might include not only specular highlight, but also white tissues. Zimmerman-Moreno et al. [11] used probabilistic modeling for precise extraction of highlight from segmentation on hue and saturation components for specular regions detection inside the coarse regions. In order to detect the less intense specular highlight, Arnold et al. [13] compared the original image and median filtered image, which is modified by filling each possible specular region with surrounding information. In addition to the application of traditional color spaces, some innovative approaches are proposed. Akbari et al. [14] applied nonlinear SVM classifier trained by statistical features including the mean and standard deviation extracted from each channel of both RGB and HSV color spaces to evaluate the specular detection methods with adaptive threshold. Meslouhi et al. [15] applied the luminance and normalized chrominance of the CIE-XYZ color space to identify specular regions by thresholding.

In this paper, an adaptive specular highlight detection method for endoscopic images is proposed. Taking the color distribution characteristics of endoscopic images into account, the criteria with adaptive thresholds for highlight detection are developed. The experimental results demonstrate that the proposed method outperforms conventional schemes based on color space and can achieve averaged Precision, Accuracy, and F1-score of 88.76%, 99.60% and 72.56%, respectively.

10. Oh J, Hwang S, Lee J, Tavanapong W, Wong J, de Groen PC. Informative frame classification for endoscopy video. Med Image Anal (2007) 11 110–27. doi:10.1016/j.media.2006.10.003

16. Voronin V, Semenishchev E, Zelensky A, Agaian S. Specular reflection detection algorithm for endoscopic images. Electron Imag (2019) 2019:222–1–222–6. doi:10.2352/ISSN.2470-1173.2019.11.IPAS-222

2. Morgand A, Tamaazousti M. Generic and real-time detection of specular reflections in images. In: 2014 International conference on computer vision theory and applications (VISAPP); 2014 January 5–8; Lisbon, Portugal (2014). p. 274–82.

Endoscopic imaging systems have been widely used in disease diagnosis and minimally invasive surgery, which, compared to traditional surgery, takes shorter recovery time. Since the angles of lighting source and camera are almost identical, specular reflection always exists in endoscopic images and may conceal valuable information such as vessels and lesions, which is essential for surgeons’ observation and judgment. Therefore, specular detection and removal are significant for endoscopic techniques, where the former is the prerequisite of the latter.

To address the above issue, we propose a criterion based on the distribution of RGB color space. Basically, in nonhighlight cases, most of endoscopic images are reddish due to the presence of hemoglobin; the value of red channel could be higher than those of green and blue ones. In highlight cases, however, the values of all three channels are nearly identical and saturated, especially for large highlight regions. Therefore, we introduce the criterion for saturated specular pixels by using the ratio between red channel and the other ones as follows:

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Without loss of generality, given that the intensity of highlight pixel is higher in the surrounding area, this pixel in an image is defined as

Standard bright-field microscopes are used for daily laboratory routine in research and diagnostics for simple, standard applications that require no special equipment. Therefore simple optical systems and lenses are applied.

13. Arnold M, Ghosh A, Ameling S, Lacey G. Automatic segmentation and inpainting of specular highlights for endoscopic imaging. EURASIP Journal on Image and Video Processing (2010) 1–12. doi:10.1155/2010/814319

11. Zimmerman-Moreno G, Greenspan H. Automatic detection of specular reflections in uterine cervix images. SPIE Med Imaging (2006) 6144:2037–45. doi:10.1117/12.653089

where A represents the set of all pixels in each patch. With the defined threshold, pixels with R lower than t are marked as specular highlight pixels,

To evaluate the proposed method, we conduct experiments with clinical data provided by Guangzhou Improve Medical Instruments Co., Ltd., and open dataset, namely, CVC-ClinicSpec, which annotated specular highlight ground truth of endoscopic images [6]. The size of images from clinical data and CVC-ClinicSpec is 768 × 576 and 384 × 288, respectively. According to extensive experiments on clinical data using quadratic polynomial fitting, the parameter α in (1) is 2.4 and the parameters μ1, μ2, and μ3 in (3) are set to −2.151*10−5, 2.031*10−3, and 1.221, respectively.

where IR(x), IG(x), and IB(x) are the intensity of RGB channels of pixel x, respectively. From (2), R of nonhighlight pixels is higher than that of highlight ones. Therefore, a threshold t is introduced to discriminate highlight and nonhighlight pixels, which is defined as

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

Phase contrast microscopy is a contrast-enhancing technique to visualize structures difficult to detect with bright-field microscopy due to a lack of contrast, without the need of a staining. When penetrating a medium, light propagates with different speeds depending on the refractive index of the medium. This leads to phase differences, which are converted to differences in brightness by the microscope using phase rings. Areas of application of phase contrast microscopes are mainly the observation of living biological samples in order to resolve fine structures with high contrast.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

FIGURE 5. Comparison of Precision, Recall, Accuracy, and F1-score, where the crosses mark averages. (A) Comparison of Precision. (B) Comparison of Recall. (C) Comparison of Accuracy. (D) Comparison of F1-score.

8. Shen DF, Guo JJ, Lin GS, Lin JY. Content-aware specular reflection suppression based on adaptive image inpainting and neural network for endoscopic images. Comput Methods Progr Biomed (2020) 192 105414. doi:10.1016/j.cmpb.2020.105414