publications

2024

1. AAMAS 2024

   Informativeness of Reward Functions in Reinforcement Learning Rati Devidze, Parameswaran Kamalaruban, and Adish Singla In International Conference on Autonomous Agents & Multiagent Systems, AAMAS'24 [Abs] [HTML]

   Reward functions are central in specifying the task we want a reinforcement learning agent to perform. Given a task and desired optimal behavior, we study the problem of designing informative reward functions so that the designed rewards speed up the agent’s convergence. In particular, we consider expert-driven reward design settings where an expert or teacher seeks to provide informative and interpretable rewards to a learning agent. Existing works have considered several different reward design formulations; however, the key challenge is formulating a reward informativeness criterion that adapts w.r.t. the agent’s current policy and can be optimized un- der specified structural constraints to obtain interpretable rewards. In this paper, we propose a novel reward informativeness crite- rion, a quantitative measure that captures how the agent’s current policy will improve if it receives rewards from a specific reward function. We theoretically showcase the utility of the proposed informativeness criterion for adaptively designing rewards for an agent. Experimental results on two navigation tasks demonstrate the effectiveness of our adaptive reward informativeness criterion.

2. AISTATS 2024

   Ethics in Action: Training Reinforcement Learning Agent for Moral Decision-making In Text-based Adventure Games Weichen Li, Rati Devidze, Waleed Mustafa, and Sophie Fellenz In International Conference on Artificial Intelligence and Statistics, AISTATS'24 [Abs] [HTML]

   Reinforcement Learning (RL) has demonstrated its potential in solving goal-oriented sequential tasks. However, with the increasing capabilities of RL agents, ensuring morally responsible agent behavior is becoming a pressing concern. Previous approaches have included moral considerations by statically assigning a moral score to each action at runtime. However, these methods do not account for the potential moral value of future states when evaluating immoral actions. This limits the ability to find trade-offs between different aspects of moral behavior and the utility of the action. In this paper, we aim to factor in moral scores by adding a constraint to the RL objective that is incorporated during training, thereby dynamically adapting the policy function. By combining Lagrangian optimization and meta-gradient learning, we develop an RL method that is able to find a trade-off between immoral behavior and performance in the decision-making process.

3. 2023

   1. ECML PKDD 2023

      Learning to Play Text-based Adventure Games with Maximum Entropy Reinforcement Learning Weichen Li, Rati Devidze, Sophie Fellenz In Joint European Conference on Machine Learning and Knowledge Discovery in Databases [Abs] [HTML]

      Text-based adventure games are a popular testbed for language based reinforcement learning (RL). In previous work, deep Qlearning is most often used as the learning agent. Q-learning algorithms are difficult to apply to complex real-world domains, for example due to their instability in training. Therefore, we adapt the Soft-Actor-Critic (SAC) algorithm to the domain of text-based adventure games in this paper. To deal with sparse extrinsic rewards from the environment, we combine the SAC with a potential-based reward shaping technique to provide more informative (dense) reward signals to the RL agent. The SAC method achieves higher scores than the Q-learning methods on many games with only half the number of training steps. Additionally, the reward shaping technique helps the agent to learn the policy faster and improve the score for some games. Overall, our findings show that the SAC algorithm is a well-suited approach for text-based games.

   2. 2022

      1. NeurIPS 2022

         Exploration-Guided Reward Shaping for Reinforcement Learning under Sparse Rewards Rati Devidze, Parameswaran Kamalaruban, and Adish Singla In Advances in Neural Information Processing Systems, NeurIPS'22 [Abs] [HTML]

         We study the problem of reward shaping to accelerate the training process of a reinforcement learning agent. Existing works have considered a number of different reward shaping formulations; however, they either require external domain knowledge or fail in environments with extremely sparse rewards. In this paper, we propose a novel framework, Exploration-Guided Reward Shaping (EXPLORS), that operates in a fully self-supervised manner and can accelerate an agent’s learning even in sparse-reward environments. The key idea of EXPLORS is to learn an intrinsic reward function in combination with exploration-based bonuses to maximize the agent’s utility w.r.t. extrinsic rewards. We theoretically showcase the usefulness of our reward shaping framework in a special family of MDPs. Experimental results on several environments with sparse/noisy reward signals demonstrate the effectiveness of EXPLORS.

      2. 2021

         1. NeurIPS 2021

            Explicable Reward Design for Reinforcement Learning Agents Rati Devidze, Goran Radanovic, Parameswaran Kamalaruban, and Adish Singla In Advances in Neural Information Processing Systems, NeurIPS'21 [Abs] [HTML]

            We study the design of explicable reward functions for a reinforcement learning agent while guaranteeing that an optimal policy induced by the function belongs to a set of target policies. By being explicable, we seek to capture two properties: (a) informativeness so that the rewards speed up the agent’s convergence, and (b) sparseness as a proxy for ease of interpretability of the rewards. The key challenge is that higher informativeness typically requires dense rewards for many learning tasks, and existing techniques do not allow one to balance these two properties appropriately. In this paper, we investigate the problem from the perspective of discrete optimization and introduce a novel framework, EXPRD, to design explicable reward functions. EXPRD builds upon an informativeness criterion that captures the (sub-)optimality of target policies at different time horizons in terms of actions taken from any given starting state. We provide a mathematical analysis of EXPRD, and show its connections to existing reward design techniques, including potential-based reward shaping. Experimental results on two navigation tasks demonstrate the effectiveness of EXPRD in designing explicable reward functions.

         2. JMLR 2021

            Policy Teaching in Reinforcement Learning via Environment Poisoning Attacks Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla. In Journal of Machine Learning Research, JMLR'21 [Abs] [HTML]

            We study a security threat to reinforcement learning where an attacker poisons the learning environment to force the agent into executing a target policy chosen by the attacker. As a victim, we consider RL agents whose objective is to find a policy that maximizes reward in infinite-horizon problem settings. The attacker can manipulate the rewards and the transition dynamics in the learning environment at training-time, and is interested in doing so in a stealthy manner. We propose an optimization framework for finding an optimal stealthy attack for different measures of attack cost. We provide lower/upper bounds on the attack cost, and instantiate our attacks in two settings: (i) an offline setting where the agent is doing planning in the poisoned environment, and (ii) an online setting where the agent is learning a policy with poisoned feedback. Our results show that the attacker can easily succeed in teaching any target policy to the victim under mild conditions and highlight a significant security threat to reinforcement learning agents in practice.

         3. NeurIPS 2021

            Curriculum Design for Teaching via Demonstrations: Theory and Applications Gaurav Yengera, Rati Devidze, Parameswaran Kamalaruban, and Adish Singla In Advances in Neural Information Processing Systems, NeurIPS'21 [Abs] [HTML]

            We consider the problem of teaching via demonstrations in sequential decisionmaking settings. In particular, we study how to design a personalized curriculum over demonstrations to speed up the learner’s convergence. We provide a unified curriculum strategy for two popular learner models: Maximum Causal Entropy Inverse Reinforcement Learning (MaxEnt-IRL) and Cross-Entropy Behavioral Cloning (CrossEnt-BC). Our unified strategy induces a ranking over demonstrations based on a notion of difficulty scores computed w.r.t. the teacher’s optimal policy and the learner’s current policy. Compared to the state of the art, our strategy doesn’t require access to the learner’s internal dynamics and still enjoys similar convergence guarantees under mild technical conditions. Furthermore, we adapt our curriculum strategy to the setting where no teacher agent is present using task-specific difficulty scores. Experiments on a synthetic car driving environment and navigation-based environments demonstrate the effectiveness of our curriculum strategy

         4. 2020

            1. ICML 2020

               Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla. In International Conference on Machine Learning, ICML'20 [Abs] [HTML]

               We study a security threat to reinforcement learning where an attacker poisons the learning environment to force the agent into executing a target policy chosen by the attacker. As a victim, we consider RL agents whose objective is to find a policy that maximizes average reward in undiscounted infinite-horizon problem settings. The attacker can manipulate the rewards or the transition dynamics in the learning environment at training-time and is interested in doing so in a stealthy manner. We propose an optimization framework for finding an \emph{optimal stealthy attack} for different measures of attack cost. We provide sufficient technical conditions under which the attack is feasible and provide lower/upper bounds on the attack cost. We instantiate our attacks in two settings: (i) an \emph{offline} setting where the agent is doing planning in the poisoned environment, and (ii) an \emph{online} setting where the agent is learning a policy using a regret-minimization framework with poisoned feedback. Our results show that the attacker can easily succeed in teaching any target policy to the victim under mild conditions and highlight a significant security threat to reinforcement learning agents in practice.

            2. IJCAI 2020

               Understanding the Power and Limitations of Teaching with Imperfect Knowledge Rati Devidze, Farnam Mansouri, Luis Haug, Yuxin Chen, and Adish Singla. In International Joint Conference on Artificial Intelligence, IJCAI'20 [Abs] [HTML]

               Machine teaching studies the interaction between a teacher and a student/learner where the teacher selects training examples for the learner to learn a specific task. The typical assumption is that the teacher has perfect knowledge of the task—this knowledge comprises knowing the desired learning target, having the exact task representation used by the learner, and knowing the parameters capturing the learning dynamics of the learner. Inspired by real-world applications of machine teaching in education, we consider the setting where teacher’s knowledge is limited and noisy, and the key research question we study is the following: When does a teacher succeed or fail in effectively teaching a learner using its imperfect knowledge? We answer this question by showing connections to how imperfect knowledge affects the teacher’s solution of the corresponding machine teaching problem when constructing optimal teaching sets. Our results have important implications for designing robust teaching algorithms for real-world applications.

            3. 2019

               1. IJCAI 2019

                  Interactive Teaching Algorithms for Inverse Reinforcement Learning Parameswaran Kamalaruban, Rati Devidze, Volkan Cevher, and Adish Singla. In International Joint Conference on Artificial Intelligence , IJCAI'19 [Abs] [HTML]

                  We study the problem of inverse reinforcement learning (IRL) with the added twist that the learner is assisted by a helpful teacher. More formally, we tackle the following algorithmic question: How could a teacher provide an informative sequence of demonstrations to an IRL learner to speed up the learning process? We present an interactive teaching framework where a teacher adaptively chooses the next demonstration based on learner’s current policy. In particular, we design teaching algorithms for two concrete settings: an omniscient setting where a teacher has full knowledge about the learner’s dynamics and a blackbox setting where the teacher has minimal knowledge. Then, we study a sequential variant of the popular MCE-IRL learner and prove convergence guarantees of our teaching algorithm in the omniscient setting. Extensive experiments with a car driving simulator environment show that the learning progress can be speeded up drastically as compared to an uninformative teacher.

               2. ICML 2019

                  Learning to Collaborate in Markov Decision Processes Goran Radanovic, Rati Devidze, David Parkes, and Adish Singla. In International Conference on Machine Learning, ICML'19 [Abs] [HTML]