Abstract: There are many characteristics of tongue that need to be analyzed in traditional Chinese medicine (TCM). Different characteristics are often analyzed by individual methods, which significantly increases the overall implementation complexity of the analysis system. Therefore, this paper proposes a collaborative classification method of tongue color and coating color in TCM based on continual learning. This method takes tongue color classification as an old task and coating color classification as a new task, which makes full use of the similarity and relevance of the two tasks to realize the accurate classification of tongue color and coating color simultaneously under a single network framework. First, a dual branch network structure with global local hybrid attention (GLHA) mechanism was designed, which aggregates high-level semantic features with low-level features to improve the representative capability of features. Second, a continual learning strategy based on the combination of regularization and rehearsal was proposed, which made the network effectively prevent forgetting the knowledge learned from old task while learning new task. The experimental results on two self-established TCM tongue datasets show that, the proposed collaborative classification method can achieve a comparable classification performance with a single task, and simultaneously, reduce the overall complexity of the two classification tasks by almost half. Among them, the accuracy of tongue color classification reaches 93.92% and 92.97%, the precision reaches 93.69% and 92.87%, the recall reaches 93.96% and 93.16%, respectively. While that of the coating color classification reaches 90.17% and 90.26%, the precision reaches 90.05% and 90.17%, the recall reaches 90.24% and 90.29%, respectively.