Objectification of Tongue Diagnosis in Traditional Medicine, Data Analysis, and Study Application

Summary

The present study employed U-Net and other deep learning algorithms to segment a tongue image and compared the segmentation results to investigate the objectification of tongue diagnosis.

Abstract

Tongue diagnosis is an essential technique of traditional Chinese medicine (TCM) diagnosis, and the need for objectifying tongue images through image processing technology is growing. The present study provides an overview of the progress made in tongue objectification over the past decade and compares segmentation models. Various deep learning models are constructed to verify and compare algorithms using real tongue image sets. The strengths and weaknesses of each model are analyzed. The findings indicate that the U-Net algorithm outperforms other models regarding precision accuracy (PA), recall, and mean intersection over union (MIoU) metrics. However, despite the significant progress in tongue image acquisition and processing, a uniform standard for objectifying tongue diagnosis has yet to be established. To facilitate the widespread application of tongue images captured using mobile devices in tongue diagnosis objectification, further research could address the challenges posed by tongue images captured in complex environments.

Introduction

Tongue observation is a widely utilized technique in traditional Chinese ethnic medicine (TCM). The color and shape of the tongue can reflect the physical condition and various disease properties, severities, and prognoses. For instance, in traditional Hmong medicine, the tongue's color is used to identify body temperature e.g., a red or purple tongue indicates pathological factors related to heat. In Tibetan medicine, a condition is judged by observing the tongue of a patient, paying attention to the color, shape, and moisture of the mucus. For instance, the tongues of patients with Heyi disease become red and rough or black and dry¹; patients with Xieri disease² have yellow and dry tongues; meanwhile, patients with Badakan disease³ have a white, humid, and soft tongue⁴. These observations reveal the close relationship between tongue features and physiology and pathology. Overall, the state of the tongue plays a vital role in diagnosis, disease identification, and evaluation of the treatment effect.

Simultaneously, owing to diverse living conditions and dietary practices among different ethnic groups, variations in tongue images are evident. The Lab model, established on the basis of an international standard for the determination of color, was formulated by the Commission International Eclairage (CIE) in 1931. In 1976, a color pattern was modified and named. The Lab color model is composed of three elements: L corresponds to brightness, while a and b are two color channels. a includes colors from dark green (low brightness value) to gray (medium brightness value) to bright pink (high brightness value); b goes from bright blue (low brightness value) to gray (medium brightness value) to yellow (high brightness value). By comparing the L x a x b values of the tongue color of five ethnic groups, Yang et al.⁵ found that the characteristics of tongue images of the Hmong, Hui, Zhuang, Han, and Mongolian groups were significantly distinct from each other. For example, the Mongolians have dark tongues with a yellow tongue coating, while the Hmong have light tongues with a white tongue coating, suggesting that tongue features can be used as a diagnostic indicator for assessing the health status of a population. Moreover, tongue images can function as an evaluation index for evidence-based medicine in clinical research of ethnic medicine. He et al.⁶ employed tongue images as a foundation for TCM diagnosis and systematically evaluated the safety and efficacy of Chou-Ling-Dan pellets (CLD granules-used to treat inflammatory and febrile diseases, including seasonal influenza in TCM) combined with Chinese and Western medicine. The results established the scientific validity of tongue images as an evaluation index for clinical studies. Nevertheless, traditional medical practitioners generally rely on subjectivity to observe tongue characteristics and assess patients' physiological and pathological conditions, requiring more precise indicators.

The emergence of the internet and artificial intelligence technology has paved the way for digitizing and objectifying tongue diagnosis. This process involves using mathematical models to provide a qualitative and objective description of tongue images⁷, reflecting the content of the tongue image. The process includes several steps: image acquisition, optical compensation, color correction, and geometric transformation. The pre-processed images are then fed into an algorithmic model for image positioning and segmentation, feature extraction, pattern recognition, etc. The output of this process is a highly efficient and precise diagnosis of tongue image data, thereby achieving the goal of objectification, quantification, and informatization of tongue diagnosis⁸. Thus, the purpose of high efficiency and high precision processing of tongue diagnosis data is achieved. Based on tongue diagnosis knowledge and deep learning technology, this study automatically separated the tongue body and tongue coating from tongue images using a computer algorithm, in order to extract the quantitative features of tongues for doctors, improve the reliability and consistency of diagnosis, and provide methods for subsequent tongue diagnosis objectification research⁹.

Protocol

This study has been approved by the National Natural Science Foundation of China project, Constructing Dynamic Change rules of TCM Facial image Based on Association Analysis. The ethics approval number is 2021KL-027, and the ethics committee has approved the clinical study to be carried out in accordance with the approved documents which include clinical research protocol (2021.04.12, V2.0), informed consent (2021.04.12, V2.0), subject recruitment materials (2021.04.12, V2.0), study cases and/or case reports, subject diary cards and other questionnaires (2021.04.12, V2.0), a list of participants in the clinical trial, research project approval, etc. Informed consent from the patients participating in the study was obtained. The main experimental approach of this study is to use real tongue images to validate and compare the model segmentation effects. Figure 1 presents the components of tongue diagnosis objectification.

1. Image acquisition

1. Use the self-developed hand-held lingual face diagnostic instrument to collect lingual face images of patients.
2. Fill in the patient's name, gender, age, and disease on the computer page. Images included here are from patients who came to the clinic and agreed to be photographed after being informed of the purpose and content of the study. Confirm that the patient is sitting upright, place the whole face in the image acquisition instrument, and instruct the patient to extend their tongue out of their mouth to the maximum extent.
3. Hold the image acquisition device connected to a computer and verify through the images on the computer screen that the patient is in the correct position and that the tongue and face are fully exposed.
4. Press the Shoot button on the computer screen three times to take three pictures.
   NOTE: The image acquisition instrument is currently only at the patent application stage and is not for commercial use, so it is not for sale.
5. Manually select and filter the collected tongue and face images. Filter and exclude images that have incomplete tongue and face exposure, as well as images that are too dark due to insufficient light. Figure 2 shows the image acquisition page of the software.
6. In the experimental design, collect three images from each patient at a time as alternatives and select a relatively standard, fully exposed, well-illuminated, and clear image as the sample for subsequent algorithm training and testing.
7. Collect data after the shooting, export the data for manual screening, and delete the non-standard images visible to the naked eye. Use the following filtering and exclusion criteria: incomplete tongue and face exposure, and images that are too dark as a result of insufficient light. An example of an under-lit, an incomplete, and a standard image is shown in Figure 3.
   ​NOTE: Insufficient light is generally caused by failure of the patient to place the face entirely into the instrument. Complete exposure is usually only obtained by correctly photographing the patient.

2. Tongue segmentation

1. Perform tongue image segmentation using an online annotation tool, as described below.
   1. Install Labelme, click on the Open button in the upper left corner of the label interface, select the folder where the image is located, and open the photos.
   2. Click on create polygon to start tracking points, track the tongue and lingual shapes, name them according to the selected areas (e.g., tongue and lingual surface), and save them.
   3. When all the marks are complete, click Save to save the image to the data folder. See Figure 4 for a detailed flow chart.
      NOTE: As the images may have pixel differences, the images cannot be directly used for algorithm training and testing.
2. Unify the images to the same size by edge-filling the images, with the long side of the image as the target fill length and performing white edge-filling to fill the images to a square, with the long side of the image as the edge length. The image size captured by the device is 1080 x 1920 pixels, and the size of the filled image is 1920 x 1920 pixels. See Figure 5.
3. Apply image enhancement if needed. No enhancement was applied in this study, as the images used were taken in a fixed scene and were less affected by the environment, lighting, and other factors.
4. Because three images were collected for each patient during the shooting process to account for uncontrollable factors, such as subject blinking and lens blocking, manually screen the images from each patient to retain one image per patient.
5. For the purpose of training the model, collect data from 200 people, or 600 images. After the screening, retain about 200 usable images.
6. According to the image number, randomly divide all the tongue images, placing 70% of them into the training set and 30% into the test set in a spreadsheet.

3. Tongue classification

1. Go to the official websites and download and install Anaconda, Python, and Labelme. Activate the environment and complete the installation and adjustment of the overall environment. See Figure 6 for a flow chart describing the installing and setting up of the software.
2. Build the deep learning algorithm model in the installed environment, tune the parameters, and complete the model training using the training set. Perform model selection and tuning as described in the following steps.
   1. Model selection: Choose the appropriate model based on the purpose of the research. After reviewing research on tongue image processing in the last 5 years, four algorithms, U-Net, Seg-Net, DeeplabV3, and PSPNet, were selected for validation in this study (see Supplementary Coding File 1, Supplementary Coding File 2, Supplementary Coding File 3, and Supplementary Coding File 4 for model codes).
   2. Data set construction: After completing the model selection, construct the required data set in conjunction with the research content, mainly using Labelme annotation and the uniform image size methods, as described above.
3. Perform model training as described below. Figure 7 shows details of the algorithm training operation.
   1. Input the data into the neural network for forward propagation, with each neuron first inputting a weighted accumulation of values and then inputting an activation function as the output value of that neuron to obtain the result.
   2. Input the result into the error function and compare it with the expected value to get the error and judge the degree of recognition by mistake. The smaller the loss function is, the better the model will be.
   3. Reduce the error by back propagation and determine the gradient vector. Adjust the weights by the gradient vector to the trend toward results so that the error tends to zero or shrinks.
   4. Repeat this training process until the set is completed or the error value no longer declines, at which point the model training is complete. See Figure 8 for a flow chart of the algorithm model in training and testing.
4. Test the four models using the same test data for segmentation and judge the model performance according to the segmentation effect. The four metrics of precision, recall, mean pixel accuracy (MPA), and MIoU provide a more comprehensive model performance evaluation.
5. After the results of the four models are generated, compare their values horizontally; the higher the value is, the higher the segmentation accuracy and the better the model's performance. See Figure 9, Figure 10, and Figure 11.

Results

For the comparison results, see Figure 12, Figure 13, and Table 1, where the environment constructed by this study uses the same samples to train and test the algorithm model. MIoU indicator: U-Net > Seg-Net > PSPNet > DeeplabV3; MPA indicator: U-Net > Seg-Net > PSPNet > DeeplabV3; precision indicator: U-Net > Seg-Net > DeeplabV3 > PSPNet; recall: U-Net > Seg-Net > PSPNet > DeeplabV3. The larger th...

Discussion

Based on the comparison results presented above, it is evident that the characteristics of the four algorithms under consideration are varied, and their distinct advantages and disadvantages are described below. The U-Net structure, based on the modification and expansion of a full convolution network, can obtain contextual information and precise positioning through a contracting path and a symmetrical expanding path. By classifying each pixel point, this algorithm achieves a higher segmentation accuracy and segments th...

Materials

Name	Company	Catalog Number	Comments
CPU	Intel(R) Core(TM) i7-9700K
GPU	NVIDIA GeForce RTX 3070 Ti (8192MB)
Operating systems	Microsoft Windows 10 Professional Edition (64-bit)
Programming language	Python
RAM	16G