In this study, we propose a two-stage optimal control framework for a fed-batch bioreactor. The high-level controller aims to obtain the optimal feed trajectory that maximizes the final time productivity and yield using a nominal model. By contrast, the low-level controller maintains the high-level performance in the presence of the model-plant mismatch and real-time disturbances. This two-stage decomposition can perform the closed-loop operation with less online recomputation. To solve the high-level optimization, differential dynamic programming (DDP), a model-based reinforcement learning that employs the derivatives of the model is applied. Three types of low-level controllers are proposed: DDP controller, a model predictive control (MPC) that tracks the high-level trajectory, and an economic MPC. We first validate that DDP yields as good result as the direct method. Second, we compare the three low-level controllers and verify the necessity of the two-stage decomposition through the studies on a bioreactor.