Online Handwritten Mathematical Expression Recognition Based on Dual-mode Encoder-decoder Framework

To make full use of the handwriting order feature and global two-dimensional structure feature of online handwritten mathematical expression, a dual-mode recognition model based on an encoder-decoder framework was designed by combining online mode and offline mode. The model can accept the handwritten mathematical expression data in the form of one-dimension coordinate point sequence and two-dimensional static image. The model can extract the handwriting order feature information from the input coordinate point sequence through the online encoder, and extract the two-dimensional structure feature information from the static image through the offline encoder, so as to fully retain the handwriting order feature and global two-dimensional structure feature. In the encoder stage, sinusoidal coding was proposed for the online mode to encode the input coordinate point sequence and supplement the stroke level information, which can effectively avoid the loss of stroke information caused by fuzzy stroke interval. For the offline mode, the smooth attention mechanism was proposed. By adopting the smooth window, adaptive adjustment of the receptive field of each pixel feature in the feature map was realized, which had solved the problem that the ordinary attention mechanism cannot filter the effective feature information of handwritten symbols with large size differences at the same time. It had effectively improved the ability of the attention mechanism to capture the effective handwritten area. The experimental results show that the accuracy rate of mathematical expression recognition of the dual-mode model can reach 58.76%, and compared with other recognition models in the same field, the dual-mode model can improve the accuracy rate of mathematical expression recognition by 1.56%-4.71%, reaching a higher level.