Ananya Bal

I am pursuing my PhD in Robotics at the Robotics Institute, Carnegie Mellon University. I am advised by Dr. Laszlo Jeni and am currently exploring Hand-Object Interaction and Multimodal Foundational Models. Previously, during my master's at RI, I was advised by Dr. Howie Choset and Dr. John Galeotti and worked on implicit neural representations for medical ultrasound and their applications to segmentation, 3D reconstruction and modeling deformation in soft tissue. I am majorly interested in machine learning and computer vision applications for robotics.

Before joining CMU, I completed my undergrad in CS from Vellore Institute of Technology where I focused on core ML and NLP projects. I have experience with building recommendation systems, ensemble classifiers, feature-based medical image diagnosis systems, and named-entity recognition models. I briefly interned at Fidelity Investments where I worked on intelligent document layout analysis for automated information retrieval from semi-structured legal documents.

Email  /  Resume  /  Github  /  LinkedIn

Research Interests

3D Perception, Vision for manipulation, AR/VR, Physics-informed Neural Networks, Generative Models

Updates

• Mar 2024: Accepted CMU's offer for the Robotics PhD program. (I'm going to be a doctor!)
• Dec 2023: Presented my MSR Thesis Talk on Neural Implicit Representations for Medical Ultrasound Volumes and 3D Anatomy-specific Reconstructions.
• Oct 2023: Presented paper on Automated Void Detection at IROS.
• Oct 2023: Our paper on unsupervised deformable registration was presented at IROS 2023.
• May 2023: My work in Document Layout Understanding with vision and NLP during my internship at Fidelity Investments was granted a US Patent.
• May 2023: Presented paper on automated ultrasound scanning with RGBD sensing at ICRA 2023.
• May 2023: Presented a workshop paper on 3D Deformation Simulation for Vascular Tissue.
• Nov 2022: Presented a paper on comparing point cloud registration methods for large, real world disaster scenes.
• Sept 2021: Started my masters at Robotics Institute, Carnegie Mellon.

Most Robotic Ultrasound Systems (RUSs) equipped with ultrasound-interpreting algorithms rely on building 3D reconstructions of the entire scanned region or specific anatomies. These 3D reconstructions are typically created via methods that compound or stack 2D tomographic ultrasound images using known poses of the ultrasound transducer with the latter requiring 2D or 3D segmentation. While fast, this class of methods has many shortcomings. It requires interpolation-based gap-filling or extensive compounding and still yields volumes that generate implausible novel views. Additionally, storing these volumes can be memory-intensive.
These challenges can be overcome with neural implicit learning which provides interpolation in unobserved gaps through a smooth learned function as well as a lighter representation for the volume in terms of memory. In this thesis, a neural implicit representation (NIR) based on the physics of ultrasound image formation is presented. ... With this NIR, a physically-grounded version of tissue reflectivity function (TRF) is learned by regression using observed intensities in ultrasound images. Additionally, this NIR also learns a spatially-varying point spread function (PSF) of the ultrasound imaging system to improve the photorealism of rendered images. The TRF learned through this method can handle contrasting observations from different viewing-directions due to a differentiable rendering function that incorporates the angle of incidence between ultrasound rays and the tissue interfaces in the scanned volume. It is a stable representation of the tissue volume that when combined with the viewing-direction, can produce true-to-orientation ultrasound images.
Given that many diagnostic and surgical applications, robotic or otherwise, require anatomy-specific 3D reconstructions, it is not sufficient to learn entire ultrasound volumes without discerning the required anatomies. To circumvent the use of traditional 3D segmentation methods that are computationally-heavy, I demonstrate that the obtained TRF can be used to learn a neural implicit shape representation for anatomies that are largely homogeneous. This is formulated as a weakly-supervised binary voxel occupancy function that is learned in parallel with the NIR. All these contributions are substantiated on simulated, phantom-acquired and live subject-acquired ultrasound images capturing blood vessels. Finally, an application for the anatomy-specific reconstruction is discussed in the context of physical simulations for deformation modeling of soft tissue.

Incorporating viewing direction-dependence to Ultra-NeRF for rendering true-to-orientation reflectance in ultrasound images. A normal surface field is learned to supplement a modified rendering function.

Weakly supervised volumetric implicit shape representation method. Applied to 3D anatomy reconstruction from multi-view 2D ultrasound images trhough Ultra-NeRF.

Enhanced Ultra-NeRF by approximating spatially varying Point Spread Functions for the ultrasound imaging system in addition to the tissue reflectivity function of the scene. Network architecture also improved for rednering high-frequency artifacts.

A novel layering-based point cloud processing approach using NetVLAD and COLMAP for improved registration and void detection in disaster scenes.

Making RAFT training unsupervised and applying it to predict vessel deformation under forces in ultrasound images. We use this further to generate ultrasound images at multiple force values and improve segmentation by 12%

A 3D deformation simulation framework where we reduce the sim2real gap by optimizing for material properties through maximizing IoU of the vessel area from the simulation and the real-world ultrasound images.

A 3D reconstruction optimization-based method to identify a high curvature region for autonomous ultrasound scanning. A novel, comprehensive 3D reconstruction evaluation score is proposed.

Our pipeline learns deformation using 3D point clouds of the deforming object, material properties, force and its point of application and predicts a deformed version of the object. As we go from images to point clouds, our method uses 2D RGB images to learn 3D deformations.

Feel free to steal this website's source code. Do not scrape the HTML from this page itself, as it includes analytics tags that you do not want on your own website — use the github code instead. Also, consider using Leonid Keselman's Jekyll fork of this page.