Abstract: Researches of human behavior recognition in depth video focused on depth video’s action feature representation was conducted to obtain a discriminative feature representation. Firstly a LBP operator based on the surface normal in depth video as a lower feature was proposed. Then the features were used to train a dictionary to get sparse representation. Lastly the original depth video was divided into some sub depth video by an adaptive spatio-temporal pyramid and a pooling method was adopted to normalize the lower features and the sparse coefficient to get a higher representation. The high representation realizes an accurate recognition of human behavior. The experiments on the action recognition dataset MSR Action3D and gesture recognition dataset MSR Gesture3D prove the author’s improved encoding algorithm’s feasibility and superiority.