Abstract: In achieve a high compression ratio,the authors proposed order to a DCT-based energy function,which is used to estimate the textural complexity of residual blocks of intra prediction.Then no more than 4 modes instead of 9 intra prediction modes for 4×4 block can be chosen to perform RDO according to the textural complexity of residual block.Consequently,the complexity decreases significantly.Experiments show that the proposed algorithm can save about 48% of the computation time stably while maintaining the PSNR-Bitrate performance of H.264/AVC.