Abstract: To improve synthesis realistic of sign language videos, a method to describe sign language video semantics is proposed, and the sign language video database based on semantic description for sign language synthesis is constructed.Chinese sign language videos in specific research field are captured, then sign language video units and multi-dimensional transition units are cut from the captured sign language videos.By describing the semantic information of every frame in sign language videos, which include locations, hand shapes and rhythm information, their multi-dimensional semantic models are constructed.During sign language video synthesis, useful information can be used in real-time by parsing multi-dimensional semantic models.This method provides real-time and coherent semantic information for sign language video synthesis, and in the process of joining two sign language videos, different rhythm information can be parsed out from their semantic models, then interpolated locations of transition frames can be moderately adjusted to make the rhythm in transition frames gradually change.