Chinese segmentation problem

Abstract

The correct segmentation of a sentence into words is essential in the computerized analysis of natural languages, and the generation of a sentence involves the proper composition of individually separate words. Unlike most languages written in phonetuc alphabets, the Chinese texts do not indicate word boundaries as spacing remains constant between ideographic characters. We demonstrate the problem of segmenting a written Chinese sentence into words in the context of machine translation and present some of the previous partial solutions: pre-editing, `maximal matching`, frequence priority, and other heuristic strategies. Furthermore, we propose the use of domain-specific frequency and a `no-window` principle in the implementation of the maximal matching strategy in combination with other heuristic rules as a more through scheme for Chinese segmentation.