A reinforcement learning based adaptive PID controller was presented for the attitude stabilization of a kind of bionic underwater robot with two bionic undulating fins. The scheme of the reinforcement learning based adaptive PID controller was given concretely including the control law and the parameter adaptive method based on reinforcement learning. Simulation experiments of yaw angle stabilization based on actual model parameters were carried out. The results indicate that the stabilization performance of yaw angle is improved distinctly after several iterations of learning control and the controller can overcome ordinary disturbances in short time, exhibiting its preferable adaptability.