Methods for biomedical image content extraction toward improved multimodal retrieval of biomedical articles
MetadataShow full item record
Clinicians and medical researchers routinely use online databases such as MEDLINE ® to search for bibliographic citations that are relevant to a clinical situation. Presently, they form their opinion about the relevance of a publication based on bibliographic information such as the title, abstract, MeSH ® terms, etc. However, additional information that is available in the illustrations and that may be essential, is not searchable. At present, images needed for instructional purposes or clinical decision support (CDS) appear in specialized databases or in biomedical publications and are not meaningfully retrievable using primarily text-based retrieval systems. It would greatly benefit the clinical community if the article images that are most useful for aiding CDS could be automatically found. This task can be done by automatically annotating images extracted from scientific publications with respect to their usefulness for CDS. Authors often use annotation markers (pointers, symbols, letters) to highlight specific regions of interest (ROI) that may be most relevant and important within the images, and mention them in figure captions and text discussions. Localizing those annotations could greatly assist in extracting the ROIs that can be used in indexing and retrieval of biomedical articles. As an important step toward achieving the goal, biomedical image analysis algorithms are proposed for localizing annotations to extract ROIs that can then be used to obtain meaningful image content, and associate it with the identified biomedical concepts from text captions and discussion in the article toward improved (text and image) content-based retrieval of biomedical articles. The proposed algorithms can be divided into two parts: (i) annotation recognition, and (ii) local image region analysis for indexing and retrieval of biomedical articles. In (i), annotations are segmented from background by edge- or region-based segmentation technique and their boundaries are represented by a set of line segments. The segmentation results are input to annotation classification process to be classified into one of annotation classes or noise. Several machine learning theories and techniques such as Markov Random Field (MRF), Hidden Markov Model (HMM), Dynamic Time Warping (DTW), Active Shape Model (ASM), and Neural Network (NN) are used to solve challenging problems in the annotation recognition task. In (ii), research issues such as localizing ROIs and extracting visual features from the ROIs are addressed as a medical image segmentation problem. Seeking visual features that can represent the presence of medical concepts in ROIs is the most important task in the local image analysis task. Without such visual features, contents in ROIs may not be used meaningfully for indexing and retrieval of biomedical figures and articles. A preliminary algorithm for ROI extraction is presented and its performance is evaluated through retrieval tests. Also challenges in ROI extraction and local image CBIR for retrieval and potential solutions to the challenges are discussed in depth. Various experiments are performed to evaluate the proposed algorithms and as means of exploration of potential solutions to the identified research challenges. The experiments and results show effectiveness of the proposed algorithms (especially, annotation recognition algorithms) and point to future research directions toward developing better approaches that improve image indexing quality and subsequently the indexing and retrieval of biomedical articles.