LLVM-AD @ ITSC 2024
About the Workshop
The 2nd Workshop on Large Language and Vision Models for Autonomous Driving (LLVM-AD) at ITSC 2024 aims to bring together professionals from academia and industry to explore the application of large language and vision models in autonomous driving. As part of this initiative, the 2nd LLVM-AD workshop launches an open dataset challenge for real-world traffic understanding.
Keynote Speakers
Workshop Recording
Organizers
Keynote Talks
Towards Scalable Autonomy (Manmohan Chandraker)
Abstract: While autonomous driving has made large strides over the past decade to be deployed in practice today, scaling it across a diversity of conditions and behaviors remains expensive. This talk explores the possibility of recent advances in large language models (LLMs) and computer vision foundational models allowing the development of more scalable driving, simulation and devops stacks for autonomous mobility. These include perception and planning that leverage language guidance, photorealistic simulation of safety-critical scenarios with neural rendering and controllable diffusion, automated devops based on vision-language understanding and a look ahead to agentic LLMs that automate complex autonomy workflows.
Bio: Manmohan Chandraker is a professor in the CSE department of the University of California, San Diego and leads computer vision research at NEC Labs. His research interests are in vision, learning and graphics, with applications to autonomous driving and augmented reality. His works have been recognized with best paper awards at CVPR, ICCV and ECCV, the NSF CAREER Award, Qualcomm and Google Research Awards. He serves on NSF panels on vision, learning and robotics and on senior program committees at CVPR, ICCV, ECCV, AAAI, NeurIPS and ICLR.
Large Language and Vision Models for Autonomous Driving (Long Chen)
Bio: Long Chen is a distinguished research lead, with a proven track record in developing disruptive AI technologies. He currently holds the position of Staff Scientist at Wayve, where he is at the forefront of building vision-language-action (VLA) models for the next wave of autonomous driving, such as Driving-with-LLMs and LINGO. Previously, he was a research engineer at Lyft Level 5, where he led the data-driven planning models from crowd-sourced data for Lyft’s self-driving cars. His extensive experience also includes applying AI technologies in various domains such as mixed reality, surgical robots, and healthcare.
Schedule
| Time | Event |
|---|---|
| 8:30-8:45 | Opening Remarks |
| 8:45-9:30 | Keynote: Large Language and Vision Models for Autonomous Driving Long Chen, Wayve https://long.ooo/ |
| 9:30-9:45 | ContextVLM: Zero-Shot and Few-Shot Context Understanding for Autonomous Driving Using Vision Language Models Shounak Sural, CMU ssural@andrew.cmu.edu |
| 9:45-10:00 | Semantic Trajectory Data Mining with LLM-Informed POI Classification Yifan Liu, UCLA https://mobility-lab.seas.ucla.edu/ |
| 10:00-10:15 | A LLM-based Multimodal Warning System for Driver Assistance Zixuan Xu, KAIST zixuan.xu@kaist.ac.kr |
| 10:15-10:30 | Real-Time Data Informed Intelligent ChatBot for Transportation Surveillance and Management Joey Cai, UW zhiyu_cai@berkeley.edu (Zoom) |
| 10:30-10:45 | Personalized Autonomous Driving with Large Language Models: Field Experiments Ziran Wang, Purdue ziran@purdue.edu |
| 10:45-11:00 | ScenarioQA: Evaluating Test Scenario Reasoning Capabilities of Large Language Models Ishaan Paranjape, UC Santa Cruz iparanja@ucsc.edu (Zoom) |
| 11:00-11:15 | Large Language Models for Human-Like Autonomous Driving: A Survey Yun LI, The University of Tokyo li-yun@g.ecc.u-tokyo.ac.jp |
| 11:15-11:30 | Evaluating and Enhancing Trustworthiness of LLMs in Perception Tasks Malsha Mahawatta, University of Gothenburg malsha.mahawatta@gu.se |
| 11:30-12:15 | Keynote: Towards Scalable Autonomy Manmohan Chandraker, NEC Labs America https://www.nec-labs.com/research/media-analytics/people/manmohan-chandraker/ |
| 12:15-12:30 | Closing Remarks |