Sushil Awale

Vision-language models (VLMs) are increasingly used for image retrieval in cultural heritage collections. Yet these models are trained and evaluated on modern web-scraped data which might cause unobserved biases when applied to historical imagery. While prior work has documented gender and racial biases in VLMs using contemporary datasets, no evaluation datasets exist for historical archives. In this paper, we address this gap by introducing a new dataset for evaluating demographic bias in VLMs on photograph collections from colonial contexts. Our dataset comprises annotated photographs for demographic queries spanning gender and skin tone. Using this dataset, we examine bias in multiple VLMs through retrieval performance across queries, measuring disparities using established information retrieval metrics. Experimental results show that gender biases documented in contemporary datasets persist on historical imagery, with all tested models retrieving certain demographic groups substantially better than others. Notably, models specifically trained for debiasing fail to mitigate bias on historical images and can introduce new racial retrieval disparities. These findings underscore the need for domain-specific debiasing approaches and caution against deploying off-the-shelf VLMs in archival contexts without proper evaluation.

Most patent retrieval systems are text-based, which under utilizes the multimodal nature of patent documents. Although a few multimodal patent retrieval systems exist, they fall short in providing efficient and informative visualizations, and facilitating the interpretation of retrieval results. To address these shortcomings, this paper presents iPatent, a novel web-based multimodal patent image retrieval system. Unlike previous solutions, iPatent integrates state-of-the-art deep learning models for fine-grained unimodal, cross-modal, and multimodal patent image retrieval. Additionally, it employs both traditional machine learning techniques and modern generative methods for interactive visual exploration and insightful interpretation of retrieval results. iPatent leverages modern web technologies to provide an interactive interface that enables users to explore large patent databases efficiently and in a visually informative way.

Patent figure classification facilitates faceted search in patent retrieval systems, enabling efficient prior art search. Existing approaches have explored patent figure classification for only a single aspect and for aspects with a limited number of concepts. In recent years, large vision-language models (LVLMs) have shown tremendous performance across numerous computer vision downstream tasks, however, they remain unexplored for patent figure classification. Our work explores the efficacy of LVLMs in patent figure visual question answering (VQA) and classification, focusing on zero-shot and few-shot learning scenarios. For this purpose, we introduce new datasets, PatFigVQA and PatFigCLS, for fine-tuning and evaluation regarding multiple aspects of patent figures (i.e., type, projection, patent class, and objects). For a computational-effective handling of a large number of classes using LVLM, we propose a novel tournament-style classification strategy that leverages a series of multiple-choice questions. Experimental results and comparisons of multiple classification approaches based on LVLMs and Convolutional Neural Networks (CNNs) in few-shot settings show the feasibility of the proposed approaches.

In this work we create a question answering dataset over the DBLP scholarly knowledge graph (KG). DBLP is an on-line reference for bibliographic information on major computer science publications that indexes over 4.4 million publications published by more than 2.2 million authors. Our dataset consists of 10,000 question answer pairs with the corresponding SPARQL queries which can be executed over the DBLP KG to fetch the correct answer. DBLP-QuAD is the largest scholarly question answering dataset.