Abstract: To solve the problem such as interference of stray light and uneven gray distribution when extracting defects from rail surface, an adaptive segmentation algorithm for rail surface defect image was proposed based on background subtraction. Firstly, through the statistics for pixel gray feature of each row in rail image, rail surface area was quickly located by combining the distribution curve of gray mean and standard deviation for each row. Secondly, the mean window was adaptively selected based on feature of region and edge. Finally, a background image model was set up based on the mean fuzzy principle, and the image subtraction operation was made, in which the segmentation of rail surface defects was achieved. Results show that the extraction method for rail surface area proposed in this paper is fast and effective, and the adaptive segmentation algorithm for rail surface defects can highlight the defects in the image and effectively reduce the effect of illumination change and uneven reflections. The recall and accuracy of the proposed method are 95.4% and 81.3% respectively.