Abstract: To optimize the application of H.264 video encoder in DSP,on the basis of the research on the block size prediction of intra & inter frame,grey histogram and autocorrelation are adopted to improve the intra frame prediction.Combining the sum of absolute difference,which is provided as the half stop rule,and on the basis of the analysis of texture difference,a novel fast inter frame prediction algorithm is proposed.The two algorithms have achieved significant improvement as compared to the original encoder.Finally,the optimization techniques of H.264 on DM642 DSP platform,which can effectively enhance the video coding speed,are proposed.The experimental results show that without the variety of image,basically the encoding speed is 15 frame/s on average,some of video sequences may achieve 18 frame/s.