Abstract: Original online weighted sample learning tracking assumes that each sample is more independent and its contribution to the package is all the same, and the same weight is given for all the positive samples. Therefore, it does not agree with the fact that the contribution degree of the target is not all the same with the distance between the target position and the sample of package. Additionally, the original algorithm cannot accurately and comprehensively represent the sample of the target package because of single feature, thus, it affects the robustness of the algorithm. Regarding the problem of original algorithm, this paper put forward a type of visual tracking method based on weighted sample learning. The method fused HOG features and Haar features at the same time, trained a classifier under the framework of learning, and gave different weights according to the similarity of the sample features. The experiments were conducted on the image sequence of different scenarios, and compared with a variety of current mainstream algorithms. Results show that the created appearance model has higher ability to distinguish between foreground and background, the algorithm has higher accuracy and stronger adaptability and can effectively overcome the traditional sample classifier degradation problems in learning.