Join the Robotics Listserv

To subscribe to event updates, send an email to with “join” in the subject line.


From Semantics to Localization in LiDAR Maps for Autonomous Vehicles

Abhinav Valada, University of Freiburg


Location: Zoom

Time: 2:40p.m.

Abstract: LiDAR-based scene interpretation and localization play a critical role in enabling autonomous vehicles to safely navigate in the environment. The last decade has witnessed unprecedented progress in these tasks by exploiting learning techniques to improve the performance and robustness. Despite these advances, the unordered spare, and irregular structure of point clouds pose several unique challenges that lead to suboptimal performance while employing standard convolutional neural networks (CNNs). In this talk, I will discuss three efforts targeted at addressing some of these challenges. First, I will present our state-of-the-art approach to LiDAR panoptic segmentation that employs a 2D CNN while explicitly leveraging the unique 3D information provided by point clouds at multiple stages in the network. I will then present our recent work that incorporates a differentiable unbalanced optimal transport algorithm to detect loop closures in LiDAR point clouds and outperforms both existing learning-based as well as hardcrafted methods. Next, to alleviate the need for expensive LiDAR sensors on every robot, I will present the first approach for monocular camera localization in LiDAR maps that effectively generalizes to new environments without any retraining and independent of the camera parameters. Finally, I will conclude the talk with a discussion on opportunities for further scaling up the learning of these tasks.

Tracking Beyond Recognition

Aljosa Osep, Technical University in Munich


Location: Zoom

Time: 2:40p.m.

Abstract: Spatio-temporal interpretation of raw sensory data is vital for intelligent agents to understand how to interact with the environment and perceive how trajectories of moving agents evolve in the 4D continuum, i.e., 3D space and time. To this end, I will first talk about our recent efforts in the semantic and temporal understanding of raw sensory data. I will first present our work on multi-object and segmentation. Then, I will discuss how to generalize these ideas towards holistic temporal scene understanding, jointly tackling object instance segmentation, tracking, and semantic understanding of monocular video sequences and LiDAR streams. Finally, I will move on to the challenging problem of scaling object instance segmentation and tracking models to the open world, in which future mobile agents will need to continuously learn without explicit human supervision. In such scenarios, intelligent agents encounter and need to react to unknown dynamic objects that were not observed during the model training.

Model-Based Visual Imitation Learning

Franziska Meier, Facebook AI Research


Location: Zoom

Time: 2:40p.m.

Abstract: How can we teach robots new skills by simply showing them what to do? In this talk I’m going to present our recent work on learning reward functions from visual demonstrations via model-based inverse reinforcement learning. Given the reward function a robot can then learn the demonstrated task autonomously. More concretely, I will show how we can frame model-based IRL, as a bi-level optimization problem, which then allows to learn reward functions by directly minimizing the distance between a demonstrated trajectory and a predicted trajectory. In order to do so from visual demonstrations, a key ingredient is a visual dynamics model, that enables the robot to predict the visual trajectory if it were to execute a policy. I will discuss, the opportunities and challenges of this research directions, and will end with an outlook for future work.

Generalized Lazy Search for Efficient Robot Motion Planning

Aditya Mandalika, University of Washington


Location: Zoom

Time: 2:40p.m.

Abstract: Robotics has become a part of the solution in various applications today: autonomous vehicles navigating busy streets, articulated robots tirelessly sorting packages in warehouses, feeding people in care homes and mobile robots assisting in rescue operations. Central to any robot that needs to navigate its environment for its application, is Motion Planning: the task of computing a collision-free motion for a (robotic) system between given start and goal states in an environment cluttered with obstacles. As tasks become more complex, there is a need to develop more sophisticated motion planning algorithms that can compute high quality solutions for the robot quickly.

In this talk, we will specifically investigate the computational bottlenecks in search for the shortest path on a graph: search effort and collision evaluations. Lazy search algorithms can efficiently solve shortest path problems where evaluating edges for collision is expensive, as is the case in robotics. We show that the existing algorithms can provably minimize the number of collision evaluations, but at the cost of increased graph operations. This can be prohibitively expensive in cluttered environments that necessitate large graphs. In this talk, we discuss a framework of lazy search algorithms that seamlessly interleave lazy search with edge evaluations to prevent wasted computational effort and to minimize the total planning time. I will close the talk with a brief discussion on the efficacy of the framework, the potential extensions and (exciting) future work.

Improving Model Predictive Control in Model-based Reinforcement Learning

Nathan Lambert, University of California, Berkeley


Location: Zoom

Time: 2:40p.m.

Abstract: Model-based reinforcement learning is developing into a useful candidate in data-efficient control synthesis for complex robotic tasks. Using simple one-step dynamics models learned from few data has proven useful in a wide variety of simulated and experimental tasks. Frequently, the one step-models are unrolled to form longer trajectory predictions for optimization in model-predictive control. In this talk, we detail how the dual optimizations of accurate one-step predictions and then a trajectory control mechanism can result in an objective mismatch. We then detail work that can begin to address this mismatch and improve the peak performance and computational efficiency of model-based reinforcement learning.

Open-source legged robotics, from hardware to control software

Majid Khadiv, Max-Planck Institute for Intelligent Systems


Location: Zoom

Time: 10:00a.m.

Abstract: Legged robots (especially humanoids) are the most suitable robot platform that could be deployed in our daily lives in the future. However, the complexity in the mechanical structure of these robots as well as the need for an advanced control software hindered progress in this field. On the hardware side, there is no standard hardware such that researchers can use to benchmark and compare their algorithms. Furthermore, legged robots are expensive and not every lab can afford to buy them for research. On the control side, the dynamics of these robots are highly complex which makes their control extremely challenging. This complexity has several aspects: 1) These robots are under-actuated and could easily fall down if not controlled properly, 2) locomotion can only be realized through establishing and breaking contact which enforces a hybrid dynamics, 3) The system is very high dimensional (up to 100 states and 50 control inputs) and the dynamic model is highly nonlinear, 4) the system is extremely constrained due to the limited amount of contact forces between the robot and the environment, etc. In this talk, I will first briefly present our recent efforts in the Open Dynamic Robot Initiative (ODRI) to provide the community with low-cost, but high-performance legged platforms that are fully open-source and can be replicated quickly using 3D-printing technology. I will also extensively talk about my recent efforts to find tractable ways at the intersection of optimal control and reinforcement learning to safely control legged robots in the presence of different uncertainties and disturbances.

Trait-based Coordination of Heterogenous Multi-Agent Teams

Harish Ravichandar, Georgia Institute of Technology


Location: Zoom

Time: 2:40p.m.

Abstract:Heterogeneous multi-agent teams have the potential to carry out complex multi-task operations that are intractable for their homogeneous counterparts. Indeed, heterogeneous teams can impact a wide variety of domains, such as disaster relief, warehouse automation, autonomous driving, defense, and environmental monitoring. However, effective coordination of such teams requires the careful consideration of the teams’ diverse, but finite, resources, as well as its ability to satisfy complex requirements associated with concurrent tasks. In this talk, I will introduce a family of application-agnostic approaches that can coordinate heterogenous multi-agent teams by effectively leveraging the relative strengths of agents when satisfying the requirements of different tasks. A unifying theme across these approaches is that both agents and tasks are modeled in terms of capabilities (i.e., traits). As such, these approaches are readily generalizable to new teams and agents without much additional modeling or computational effort. In particular, I will discuss techniques and challenges associated with i) forming effective coalitions that can satisfy known requirements of heterogenous tasks, and ii) learning how to coordinate heterogenous teams from human experts when exact requirements are unavailable.