Marin Vlastelica

Zero-shot RL methods like forward-backward representations (FB) have so far been decoupled from the exploration problem. We design exploration policies that arise naturally from the FB representation by minimizing its posterior variance, hence its epistemic uncertainty, and show that this considerably improves sample complexity over other exploration methods.

We propose a principled offline algorithm for unsupervised skill discovery that maximizes diversity while ensuring each learned skill imitates state-only expert demonstrations to a certain degree. Our main analytical contribution connects Fenchel duality, reinforcement learning, and unsupervised skill discovery to maximize a mutual information objective subject to KL-divergence state occupancy constraints. Policies trained on a custom offline quadruped dataset transfer well to the real 12-DoF robot. An earlier version appeared at EWRL 2023 as “Diverse Offline Imitation via Fenchel Duality”.

We take a constrained optimization viewpoint on the quality-diversity trade-off and obtain diverse policies by imposing constraints on value functions defined through distinct rewards, with a Van der Waals attract-repel term controlling the diversity level. The learned skills transfer to the real 12-DoF quadruped Solo12 and exhibit diverse agile obstacle-traversal behaviors.

Instead of discarding “spurious” features whose relationship with the label changes across domains, we show they can be safely exploited in the test domain without labels: pseudo-labels from stable features provide sufficient guidance when stable and unstable features are conditionally independent given the label. Our Stable Feature Boosting algorithm learns an asymptotically-optimal predictor without test-domain labels.

We propose an improved sampling procedure for a diffusion generative model for approximate sampling from a target distribution that is defined by an energy (cost) function of designs in a complex simulation environment.

As a continuation of the blackbox-backprop line of work, we introduce a simple modification to the algorith, namely treating the solver as an identity mapping in the computation graph on the backward pass. This, coupled with projections that avoid degenerate cases, works comparably well as blackbox-backprop.

Latent action reinforcement learning for task-oriented dialogue has seen success at benchmarks such as MultiWOZ. Categorical latents have been argued to be the best choice. We show that with continuous latents and reformulation of the ELBO objective and the reinforcmenet learning stage, we can achieve state-of-the-art performance on MultiWOZ.

In this work we proposed a method for imitating rough demonstrations by a robot quadruped based on Wasserstein adversarial learning of an imitation reward. We trained a policy via domain randomization and successfully transferred it to the real system, where to robot was able to reproduce agile motions.

Adversarial learning to facilitate unsupervised skill discovery resulting in a controllable skill set by a latent variable with skills that are useful for solving the downstream task.

In this work we address the problem of efficient uncertainty estimation in zero-order trajectory optimization via the Cross Entropy Method (CEM). Moreover, we show that it is essential that we can distinguish between epistemic and aleatoric uncertainty in order to avoid so-called “risky” behaviour.

As a continuation of the blackbox-differentiation line of work, we propose to use time-dependent shortest-path solvers in order to enhance generalization capabilities of neural network policies. With imposing a prior on the underlying goal-conditioned MDP structure, we are able to extract well-performing policies through imitation learning that utilize blackbox solvers for receding horizon planning at execution time. Again, this comes with absolutely no sacrifices to the optimality of the solver used.

Problems that are inherently combinatorial still remain a hinderance for classical deep learning methods. Traditional methods that try to do gradient propagation through combinatorial solvers rely on sample-based estimates or solver relaxations. We show that for a specific class of solver, we are able to efficiently compute gradients of an implicit piecewise-linear interpolation of the objective. This allows us to achieve unprecedented generalization performance on representation learning tasks with combinatorial flavor.

As another continuation of the blackbox differentiation line of work, we show that we can cast the ranking problem as a blackbox solver that satisfies the conditions for efficient gradient calculation, therefore enabling us to optimize rank-based metrics by simply using efficient implementations of sorting algorithms instead of learning a differentiable sort operation. We apply this insight to optimizing mean average precision and recall in object detection and retrieval tasks, where we achieve comparable results to state-of-the-art at the time.

In this work we propose a hierarchical reinforcement learning algorithm that is able to construct planning graphs while managing to use computational resources efficiently, guided by intrinsic motivation in form of prediction error and measure of task improvement.

In this work we propose a liquid state machine approach to reinforcement learning of continuous motor control. The liquid state machine approach with smart spectral radius initialization has shown to extract useful features for motor control which can be used with classical RL algorithms.

In this work we looked at what kind of features we can use for prediction of video valence, violence and sentiment.