My name is James Y. Huang. I’m a final-year Ph.D. student at University of Southern California (USC) advised by Prof. Muhao Chen and Prof. Fred Morstatter. I was a research intern at Microsoft Research, Adobe Research, Amazon AWS AI, and Tencent AI Lab.

My research focuses on efficient post-training methods to improve the robustness, security, and adaptablity of (multimodal) LLMs.

Before Ph.D., I received my master’s degree from University of California, Los Angeles (UCLA) where I worked with Prof. Kai-Wei Chang. I got my bachelor’s degree in Engineering Science from University of Toronto.

News

Our paper on learning compositional sentence representations won
🏆Outstanding Paper Award at EMNLP 2023!

Selected Publications

Be My Eyes: Extending Large Language Models to New Modalities Through Multi-Agent Collaboration
James Y. Huang, Sheng Zhang, Qianchu Liu, Guanghui Qin, Tinghui Zhu, Tristan Naumann, Muhao Chen, Hoifung Poon
Preprint

OmniStruct: Universal Text-to-Structure Generation across Diverse Schemas
James Y. Huang, Wenxuan Zhou, Nan Xu, Fei Wang, Qin Liu, Sheng Zhang, Hoifung Poon, Muhao Chen
Preprint

MetaScale: Test-Time Scaling with Evolving Meta-Thoughts
Qin Liu, Wenxuan Zhou, Nan Xu, James Y. Huang, Fei Wang, Sheng Zhang, Hoifung Poon, Muhao Chen
Preprint

DeAL: Decoding-time Alignment for Large Language Models
James Y. Huang, Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchoff, Dan Roth
ACL 2025

MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding
Fei Wang, Xingyu Fu, James Y. Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, Tianyi Lorena Yan, Wenjie Jacky Mo, Hsiang-Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai-Wei Chang, Dan Roth, Sheng Zhang, Hoifung Poon, Muhao Chen
ICLR 2025

Offset Unlearning for Large Language Models
James Y. Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, Muhao Chen
TMLR 2025

mDPO: Conditional Preference Optimization for Multimodal Large Language Models
Fei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu, Sheng Zhang, Hoifung Poon, Muhao Chen
EMNLP 2024

Contrastive Instruction Tuning
Tianyi Yan, Fei Wang, James Y. Huang, Wenxuan Zhou, Fan Yin, Aram Galstyan, Wenpeng Yin, Muhao Chen
ACL 2024 (Findings)

Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations
James Y. Huang, Wenlin Yao, Kaiqiang Song, Hongming Zhang, Muhao Chen, Dong Yu
EMNLP 2023 🏆Outstanding Paper Award

Robust Natural Language Understanding with Residual Attention Debiasing
Fei Wang, James Y. Huang, Tianyi Yan, Wenxuan Zhou, Muhao Chen
ACL 2023 (Findings)

Parameter-Efficient Tuning with Special Token Adaptation
Xiaocong Yang, James Y. Huang, Wenxuan Zhou, Muhao Chen
EACL 2023

Unified Semantic Typing with Meaningful Label Inference
James Y. Huang, Bangzheng Li, Jiashu Xu, Muhao Chen
NAACL 2022

Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models
James Y. Huang, Kuan-Hao Huang, Kai-Wei Chang
NAACL 2021