Abstract: Aiming at the flexibility issue in the collaborative handling of the heterogeneous three-robot system, a reinforcement learning control strategy based on Proximal Policy Optimization (PPO) is proposed. A simulation environment for the collaborative handling of the heterogeneous three robots is established in the CoppeliaSim robot simulator, and the simulations of force control and reinforcement learning control are performed respectively. The comparison of simulation results show that under the reinforcement learning control, the trajectory error of the mass center of the handled object is the smallest in the Z direction, which is only 4.7% of that under the force control. The end effector velocity change of robot 2 and the angular velocity change of its typical joint are much smoother. Through the sim2real solution, both control methods are deployed in the three-robot collaborative handling experiment. The experimental results indicate that under the reinforcement learning control, the object trajectory tracking error in the Z direction is also the smallest, only 5.4% of that under the force control. The velocity change of robot in the X direction is only 20.7% of that under the force control, and its typical joint 2 reveals better flexibility, with the angular velocity change being only 35.2% of that under the force control. The simulation and experimental results demonstrate that the reinforcement learning method is provided with better performance, and it also has the feasibility of being transferred from simulation to reality.