Explainable Reinforcement Learning (XRL) has emerged as a promising approach for improving the transparency of Reinforcement Learning (RL) agents. However, a gap remains between complex RL policies and domain experts, driven by the limited comprehensibility of XRL outputs and the fragmented coverage of current XRL methods. This leaves users uncertain about selecting the most suitable tools for their needs. To address these challenges, we introduce TalkToAgent, a multi-agent Large Language Models (LLM) framework that provides interactive, natural language explanations for RL policies. With five specialized agents–Coordinator, Explainer, Coder, Evaluator, and Debugger–TalkToAgent automatically maps user queries to appropriate XRL tools and interprets agent actions through either key state variables, expected outcomes, or contrastive explanations. Notably, our approach extends conventional contrastive explanations by deriving alternative scenarios from qualitative descriptions about control behavior, or generating entirely new rule-based policies. We validated TalkToAgent on three process control benchmarks with diverse objectives. Results demonstrate that TalkToAgent accurately translates user queries into XRL tasks, while the Coder-Evaluator-Debugger loop minimizes failures in policy generation. Furthermore, qualitative evaluation confirms that TalkToAgent effectively contextualized agent behaviors within their specific problem domains.