Abstract: To detect the occurrence of abnormal human behaviors under video surveillance in real time accurately, a recognition algorithm of abnormal human behavior based on pose estimation was proposed in this paper. First, the human pose estimation algorithm based on deep learning was used to extract the coordinates of the key points of the human skeleton to form a spatial temporal graph model containing spatial information and time series information. Each node in the model corresponded to a joint of the human and contained two types of edges at the same time. One was the space edge that conformed to the natural connectivity of the joints, and the other was the temporal edge across continuous time. After that, the spatial temporal graph was subjected to multi-stage spatial temporal graph convolution operations to extract advanced features. Finally the Softmax classifier was used for behavior classification. the behavior result was obtained and whether it is abnormal behavior was judged. Compared with the current advanced methods, the experimental results of KTH single-person dataset and HMDB51 multi-person interaction dataset show that the accuracy is better. The real-time video was tested, and the frame rate of real-time detection and recognition reached 25 frames per second to realize real-time monitoring video processing.