Handwritten Formula Recognition Based on Segmentation of Adhesive Symbols and Multi-feature Fusion

FU Pengbin; LI Jianjun; YANG Huirong

doi:10.11936/bjutxb2020120030

Journal of Beijing University of Technology > 2021 > 47(8): 842-853. > DOI: 10.11936/bjutxb2020120030

FU Pengbin, LI Jianjun, YANG Huirong. Handwritten Formula Recognition Based on Segmentation of Adhesive Symbols and Multi-feature Fusion[J]. Journal of Beijing University of Technology, 2021, 47(8): 842-853. DOI: 10.11936/bjutxb2020120030

Citation:

PDF (1890 KB)

Handwritten Formula Recognition Based on Segmentation of Adhesive Symbols and Multi-feature Fusion

Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China

More Information

Received Date: December 29, 2020
Available Online: August 03, 2022
Published Date: August 09, 2021

Graphical Abstract

Abstract

Abstract

To solve the problem that character adhesion affects the automatic recognition of offline handwritten mathematical formulas, a segmentation method of single-point adhesive characters was proposed in this paper based on character contour features. The segmentation point and directions were obtained by using the information of contour direction codes on the upper and lower sides of characters. Then, the multi-feature fusion was realized by combining the geometric characteristics such as width, height, number of corners and projection contour. Finally, combined with the spatial position relationship characteristics of the special symbols and the surrounding characters, such as up, down, left, right, overlap and half encirclement, the structure analysis was conducted, and the common characters recognized by convolution neural network were substituted by the structure analysis sequence to elicit the overall recognition of the formula. The experimental results show that this method can effectively deal with the adhesion and special symbol recognition in mathematical formulas. The segmentation accuracy of adhesive characters has reached 87.25%. At the same time, the overall recognition rate of handwritten mathematical formula is further improved.

FullText(HTML)

References (20)

References

[1]	STRATHY N W, SUEN C Y, KRZYZAK A. Segmentation of handwritten digits using contour features[C]//International Conference on Document Analysis & Recognition. Piscataway: IEEE, 1993: 577-580.
[2]	VELLASQUES E, OLIVEIRA L S, BRITTO A S, et al. Filtering segmentation cuts for digit string recognition[J]. Pattern Recognition, 2008, 41(10): 3044-3053. doi: 10.1016/j.patcog.2008.03.019
[3]	TAMHANKAR P A, MASALKAR K D, KOLHE S R. A novel approach for character segmentation of offline handwritten marathi documents written in MODI script[J]. Procedia Computer Science, 2020, 171(3): 179-187. http://www.sciencedirect.com/science/article/pii/S1877050920309844
[4]	CONGEDO G, DIMAURO G, IMPEDOVO S, et al. Segmentation of numeric strings[C]//International Conference on Document Analysis & Recognition. Los Alamitos, CA: IEEE Computer Society, 1995, 2(1): 1038-1041.
[5]	PAL U, BELAIED A, CHOISY C. Touching numeral segmentation using water reservoir concept[J]. Pattern Recognition Letters, 2003, 24(1/3): 261-272. http://www.sciencedirect.com/science/article/pii/S0167865502002404
[6]	WANG J, DU J, ZHANG J, et al. Multi-modal attention network for handwritten mathematical expression recognition[C]//2019 International Conference on Document Analysis and Recognition (ICDAR). Piscataway: IEEE, 2020: 1181-1186.
[7]	LIU X, ZHANG T, YU X. An end-to-end trainable system for offline handwritten chemical formulae recognition[C]//2019 International Conference on Document Analysis and Recognition (ICDAR). Piscataway: IEEE 2019: 577-582.
[8]	SHAN G, WANG H, LIANG W, et al. Robust encoder-decoder learning framework for offline handwritten mathematical expression recognition based on a multi-scale deep neural network[J]. Science China Information Sciences, 2021, 64(3): 1-3. http://qikan.cqvip.com/Qikan/Article/Detail?id=7104293108
[9]	ZHANG J, DU J, DAI L. Track, attend, and parse (TAP): an end-to-end framework for online handwritten mathematical expression recognition[J]. IEEE Transactions on Multimedia, 2019, 21(1): 221-233. doi: 10.1109/TMM.2018.2844689
[10]	KHARE V, SHIVAKUMARA P, NAVYA B J, et al. Weighted-gradient features for handwritten line segmentation[C]//2018 24th International Conference on Pattern Recognition (ICPR). Piscataway: IEEE, 2018: 3651-3656.
[11]	LI Z H, ZHANG L, WANG J M, et al. Research on image acquisition and preprocessing method for visual measurement of pipeline inner surface corrosion[C]//2017 13th IEEE International Conference on Electronic Measurement & Instruments (ICEMI). Piscataway: IEEE, 2017: 364-368.
[12]	田学东, 张艳, 王亮. 基于轮廓特征的印刷公式粘连符号的切分[J]. 计算机工程与应用, 2008, 44(4): 176-178. https://www.cnki.com.cn/Article/CJFDTOTAL-JSGG200804057.htm TIAN X D, ZHANG Y, WANG L. Segmentation of touching symbols in typeset mathematical expressions based on profile characteristic[J]. Computer Engineering and Applications, 2008, 44(4): 176-178. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JSGG200804057.htm
[13]	FUJISAWA H, NAKANO Y, KURINO K. Segmentation methods for character recognition: from segmentation to document structure analysis[J]. Proceedings of the IEEE, 1992, 80(7): 1079-1092. http://ieeexplore.ieee.org/document/156471
[14]	付鹏斌, 彭荆旋, 杨惠荣, 等. 基于多重几何特征和CNN的脱机手写算式识别[J]. 计算机系统应用, 2020, 29(8): 271-279. https://www.cnki.com.cn/Article/CJFDTOTAL-XTYY202008040.htm FU P B, PENG J X, YANG H R, et al. Off-line handwritten equation recognition based on multiple geometric features and CNN[J]. Computer Systems & Applications, 2020, 29(8): 271-279. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-XTYY202008040.htm
[15]	王改, 郑启龙, 邓文齐, 等. 基于BWDSP众核的CNN计算任务划分优化[J]. 计算机系统应用, 2019, 28(9): 88-94. https://www.cnki.com.cn/Article/CJFDTOTAL-XTYY201909013.htm WANG G, ZHENG Q L, DENG W Q, et al. Optimization of CNN computing task partition based on many-core BWDSP[J]. Computer Systems & Applications, 2019, 28(9): 88-94. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-XTYY201909013.htm
[16]	MAHDAVI M, ZANIBBI R, MOUCHERE H, et al. ICDAR 2019 CROHME+TFD: competition on recognition of handwritten mathematical expressions and typeset formula detection[C]//2019 International Conference on Document Analysis and Recognition (ICDAR). Piscataway: IEEE, 2019: 1533-1538.
[17]	MOUCHERE H, VIARD-GAUDIN C, ZANIBBI R, et al. ICFHR 2016 CROHME: competition on recognition of online handwritten mathematical expressions[C]//International Conference on Frontiers in Handwriting Recognition. Piscataway: IEEE, 2017: 607-612.
[18]	李兴国, 高炜. 基于滴水算法的验证码中粘连字符分割方法[J]. 计算机工程与应用, 2014, 50(1): 163-166. https://www.cnki.com.cn/Article/CJFDTOTAL-JSGG201401037.htm LI X G, GAO W. Segmentation method for merged characters in CAPTCHA based on drop fall algorithm[J]. Computer Engineering and Applications, 2014, 50(1): 163-166. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JSGG201401037.htm
[19]	OKAMOTO M, IMAI H, TAKAGI K. Performance evaluation of a robust method for mathematical expression recognition[C]//International Conference on Document Analysis & Recognition. Los Alamitos, CA: IEEE Computer Society, 2001: 121-128.
[20]	杨庆海, 卢波, 颜子夜, 等. 基于马尔科夫随机场的粘连字符串切分算法[J]. 计算机工程, 2013, 39(4): 258-262. doi: 10.3969/j.issn.1000-3428.2013.04.059 YANG Q H, LU B, YAN Z Y, et al. Touched string segmentation algorithm based on Markov random field[J]. Computer Engineering, 2013, 39(4): 258-262. (in Chinese) doi: 10.3969/j.issn.1000-3428.2013.04.059

[1]	RUAN Xiaogang, LI Ang, HUANG Jing. Pose Estimation Method Based on Self-supervised Recurrent Convolutional Neural Networks[J]. Journal of Beijing University of Technology, 2021, 47(12): 1311-1320. DOI: 10.11936/bjutxb2020040001
[2]	JIA Xibin, LI Jia. Lightweight Pyramid Prototype Alignment Network for Few-shot Semantic Segmentation[J]. Journal of Beijing University of Technology, 2021, 47(5): 455-462, 519. DOI: 10.11936/bjutxb2020110010
[3]	LIN Lan, WU Yuchao, WANG Jingxuan, WU Shuicai. Research Progress of Semantic Segmentation Technique Based on Convolutional Neural Network and Its Application in Brain Neuroimage[J]. Journal of Beijing University of Technology, 2021, 47(1): 85-92. DOI: 10.11936/bjutxb2019070002
[4]	JIA Xibin, LU Chen, Siluyele Ntazana, Mazimba Windi. Multi-scale Feature Fusion Network for Person Re-identication[J]. Journal of Beijing University of Technology, 2020, 46(7): 788-794. DOI: 10.11936/bjutxb2019090005
[5]	JI Junzhong, ZHANG Qi. Influence Function-based Convolutional Neural Network for Move Prediction in Go[J]. Journal of Beijing University of Technology, 2019, 45(1): 54-60. DOI: 10.11936/bjutxb2017060053
[6]	LI Yujian, SHEN Chengkai, YANG Hongli, HU Haihe. PCA Shuffling Initialization of Convolutional Neural Networks[J]. Journal of Beijing University of Technology, 2017, 43(1): 22-27. DOI: 10.11936/bjutxb2016060070
[7]	SUN Yan-feng, QI Guang-lei, HU Yong-li, ZHAO Lu. Deep Convolution Neural Network Recognition Algorithm Based on Improved Fisher Criterion[J]. Journal of Beijing University of Technology, 2015, 41(6): 835-841. DOI: 10.11936/bjutxb2014110040
[8]	SUN Guang-min, ZHANG Cheng, WANG Peng, DENG Chao. Handwritten Character Recognition Based on the Nonlinear PCA Neural Network[J]. Journal of Beijing University of Technology, 2007, 33(9): 915-919. DOI: 10.3969/j.issn.0254-0037.2007.09.004
[9]	Hu Jian, Wang Qingbao, Tu Chengyu. The Application of the Feedforword Neural Networks with Multi-layer in the Recognition of Handwritten Numerals[J]. Journal of Beijing University of Technology, 1996, 22(4): 127-133.
[10]	Hao Zhentao, Wang Qingbao. On Handprinted Arabic Numeral Recognition Using Artificial Neural Networks[J]. Journal of Beijing University of Technology, 1993, 19(3): 44-48.

Cited By

Cited by

Periodical cited type(4)

1.	付鹏斌，徐宇，杨惠荣. 基于多尺度特征融合的互学习脱机手写数学公式识别. 华南理工大学学报(自然科学版). 2024(02): 23-31 .
2.	周伟. 一种改进的线性变换与多尺度位置编码方法. 厦门理工学院学报. 2023(03): 58-66 .
3.	方洪波，万广，陈忠辉，黄以卫，张文勇，谢本亮. 基于改进YOLOv5s的离线手写数学符号识别. 图学学报. 2022(03): 387-395 .
4.	贾华颖，颜佩珊，仲昭婷，梁银. 手写口算题字符分割方法研究. 福建电脑. 2022(12): 30-33 .

Other cited types(1)

Get Citation

PDF

XML

Article views (209) PDF downloads (73) Cited by(5)

Turn off MathJax

Article Contents

Abstract

References

Handwritten Formula Recognition Based on Segmentation of Adhesive Symbols and Multi-feature Fusion