Tools Used in Natural Language Processing

Morphological Analysis

MeCab

• https://taku910.github.io/mecab/
• Designed with the basic principle of being a general-purpose tool independent of language, dictionary, and corpus
• Operates at very high speed
• Outputs the analysis result that minimizes cost based on word occurrence likelihood (generation cost) and part-of-speech connectivity (connection cost)

JUMAN

• http://nlp.ist.i.kyoto-u.ac.jp/index.php?JUMAN
• Designed so that users can easily modify grammar definitions and word connectivity rules
• JUMAN Demo

JUMAN++

• http://nlp.ist.i.kyoto-u.ac.jp/index.php?JUMAN++
• Uses RNNLM to perform analysis that considers semantic naturalness
• High accuracy but slow processing speed
• Currently in a development version
• JUMAN++ Demo

Sudachi

• https://github.com/WorksApplications/Sudachi
• Features include the ability to use multiple segmentation units simultaneously, and extensible character normalization and unknown word handling

Janome

• https://mocobeta.github.io/janome/
• A morphological analysis library written in Pure Python with a built-in dictionary
• Has few dependencies but is still under development

Dependency Parsing

CaboCha

• https://taku910.github.io/cabocha/
• Uses MeCab for the morphological analysis component

Libraries

GiNZA

• https://megagonlabs.github.io/ginza/
• Released on 2019/4/2
• Provides morphological analysis, dependency parsing, and word dependency structure analysis
• Uses spaCy as its framework and incorporates SudachiPy internally