Publications

Publications are listed according to their type and in reverse chronological order. For an updated list please check on Google Scholar. * denotes equal contribution

2024

1. 

   Hierarchical Conditioning of Diffusion Models Using Tree-of-Life for Studying Species Evolution

   Mridul Khurana, Arka Daw, M. Maruf, Josef C. Uyeda, Wasila Dahdul, Caleb Charpentier, Yasin Bakis, and 8 more authors

   In ECCV, 2024

   Abs arXiv Website

   A central problem in evolutionary biology is to explore the genetic basis of evolutionary changes in the traits of organisms, such as fin structures in fish or beak colors in birds. With the growing availability of large-scale image repositories in biology and recent advances in generative modeling, there is an opportunity to study changes in evolutionary traits of species automatically from images. We introduce a novel Hierarchical Embedding (HIER-Embed) strategy to encode the evolutionary information of a species as a composition of encodings learned at every internal node in the phylogenetic tree. We use HIER-Embeddings to condition latent diffusion models to generate synthetic images of species. Further, we introduce two novel types of perturbation operations: trait masking and trait swapping, similar in spirit to gene knockout experiments, that enable us to analyze novel changes in evolutionary traits acquired at different levels of phylogeny.

2. 

   Conditioning Diffusion Models Using the Knowledge of Phylogeny for Understanding Species Evolution

   Mridul Khurana, Arka Daw, M. Maruf, Josef C. Uyeda, Wasila Dahdul, Caleb Charpentier, Yasin Bakis, and 5 more authors

   In AAAI workshop on Imageomics (Oral), 2024

   Abs PDF Poster Slides

   Understanding species evolution across the tree of life, commonly known as a phylogenetic tree is a significant area of interest for biologists. The measurement of species traits often relies on subjective assessments and is labor-intensive, leading to a notable shortage of labeled data essential for identifying traits. In this work, we explore integrating biological knowledge with generative models specifically Latent Diffusion Models. We evaluate four different conditioning techniques: Class Conditional, Scientific Name Encoding, Tree-to-Text (T2T) Encoding, and Hierarchical Level Encoding to integrate the phylogenetic information. We validate the methods with their proficiency in generating biologically meaningful images and the ability to capture phenotypical information, aiding in the discovery of novel biological traits.

3. 

   VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images

   M. Maruf, A. Daw, K. S. Mehrab, H.B. Manogaran, A. Neog, M. Sawhney, Mridul Khurana, and 15 more authors

   In NeurIPS, 2024

   Abs arXiv

   Images are increasingly becoming the currency for documenting biodiversity on the planet, providing novel opportunities for accelerating scientific discoveries in the field of organismal biology, especially with the advent of large vision-language models (VLMs). We ask if pre-trained VLMs can aid scientists in answering a range of biologically relevant questions without any additional fine-tuning. In this paper, we evaluate the effectiveness of 12 state-of-the-art (SOTA) VLMs in the field of organismal biology using a novel dataset, VLM4Bio, consisting of 469K question-answer pairs involving 30K images from three groups of organisms: fishes, birds, and butterflies, covering five biologically relevant tasks. We also explore the effects of applying prompting techniques and tests for reasoning hallucination on the performance of VLMs, shedding new light on the capabilities of current SOTA VLMs in answering biologically relevant questions using images.

4. 

   FishVista: A Multi-Purpose Dataset for Understanding and Identification of Visual Traits from Images

   K. S. Mehrab, M. Maruf, A. Daw, H.B. Manogaran, A. Neog, Mridul Khurana, Bahadir Altintas, and 10 more authors

   In review at CVPR, 2024

   arXiv

2023

1. 

   Discovering Novel Biological Traits From Images Using Phylogeny-Guided Neural Networks

   Mohannad Elhamod, Mridul Khurana, Harish Babu Manogaran, Josef C. Uyeda, Meghan A. Balk, Wasila Dahdul, Yasin Bakis, and 11 more authors

   In Proceedings of the 29th ACM SIGKDD (KDD) Conference on Knowledge Discovery and Data Mining (Oral), 2023

   Also at CVPR 2023 - CV4Animals Workshop (Oral + Poster)

   Abs PDF Poster

   Discovering evolutionary traits that are heritable across species on the tree of life (also referred to as a phylogenetic tree) is of great interest to biologists to understand how organisms diversify and evolve. However, the measurement of traits is often a subjective and labor-intensive process, making trait discovery a highly label-scarce problem. We present a novel approach for discovering evolutionary traits directly from images without relying on trait labels. Our proposed approach, Phylo-NN, encodes the image of an organism into a sequence of quantized feature vectors -or codes- where different segments of the sequence capture evolutionary signals at varying ancestry levels in the phylogeny. We demonstrate the effectiveness of our approach in producing biologically meaningful results in a number of downstream tasks including species image generation and species-to-species image translation, using fish species as a target example