Abstract: A formation matrix,which is presented to identify the positions of the mobile robots and state space with lower dimension,is constructed. By analyzing the merit and demerit of various formations,state criterion and reinforcement function are provided. The effect on the learning result owing to uncertain partition of sensed area is weakened. Simulation experiment shows that the presented approach is feasible. Finally,some deficiencies are analyzed.