6.2 KiB
pycorrector
中文错别字纠正工具。音似、形似错字（或变体字）纠正，可用于中文拼音、笔画输入法的错误纠正。python开发。
pycorrector依据语言模型检测错别字位置，通过拼音音似特征、笔画五笔编辑距离特征及语言模型困惑度特征纠正错别字。
特征
语言模型
- Kenlm（统计语言模型工具）
- RNNLM（TensorFlow、PaddlePaddle均有实现栈式双向LSTM的语言模型）
错误检测
- 字粒度：语言模型困惑度（ppl）检测某字的似然概率值低于句子文本平均值，则判定该字是疑似错别字的概率大。
- 词粒度：切词后不在词典中的词是疑似错词的概率大。
错误纠正
- 通过错误检测定位所有疑似错误后，取所有疑似错字的音似、形似候选词，
- 使用候选词替换，基于语言模型得到类似翻译模型的候选排序结果，得到最优纠正词。
思考
- 现在的处理手段，在词粒度的错误召回还不错，但错误纠正的准确率还有待提高，更多优质的纠错集及纠错词库会有提升，我更希望算法上有更大的突破。
- 另外，现在的文本错误不再局限于字词粒度上的拼写错误，需要提高中文语法错误检测（CGED, Chinese Grammar Error Diagnosis）及纠正能力，列在TODO中，后续调研。
使用说明
安装
- 全自动安装：pip install pycorrector 或者 pip3 install pycorrector
- 半自动安装：下载 https://github.com/shibing624/pycorrector, 解压缩并运行 python setup.py install
纠错
使用示例:
import pycorrector
corrected_sent, detail = pycorrector.correct('少先队员因该为老人让坐')
print(corrected_sent, detail)
输出:
少先队员应该为老人让座 [[('因该', '应该', 4, 6)], [('坐', '座', 10, 11)]]
自定义语言模型
语言模型对于纠错步骤至关重要，目前我能收集到的语料数据有人民日报数据。大家可以用中文维基（繁体转简体）等更大的语料数据训练效果更好的语言模型， 对于纠错效果会有比较好的提升。
- kenlm语言模型训练工具的使用，请见博客：http://blog.csdn.net/mingzai624/article/details/79560063
- 附上训练语料<人民日报2014版熟语料>，包括： 1）标准人工切词及词性数据people2014.tar.gz， 2）未切词文本数据people2014_words.txt， 3）kenlm训练字粒度语言模型文件及其二进制文件people2014corpus_chars.arps/klm， 4）kenlm词粒度语言模型文件及其二进制文件people2014corpus_words.arps/klm。
网盘链接:https://pan.baidu.com/s/1971a5XLQsIpL0zL0zxuK2A 密码:uc11。尊重版权，传播请注明出处。
贡献及优化点（TODO）
- 使用RNN语言模型来提高纠错准确率。
- 优化形似字字典，提高形似字纠错准确率。
- 整理中文纠错集，使用seq2seq做深度中文纠错模型。
- 添加中文语法错误检测及纠正能力。
参考
- Norvig’s spelling corrector
- 《Chinese Spelling Error Detection and Correction Based on Language Model, Pronunciation, and Shape》[Yu, 2013]
- 《Chinese Spelling Checker Based on Statistical Machine Translation》[Chiu, 2013]
- 《Chinese Word Spelling Correction Based on Rule Induction》[yeh, 2014]
- 《Neural Language Correction with Character-Based Attention》[zhang, 2016]
pycorrector
Chinese text error correction tool.
pycorrector Use the language model to detect errors, pinyin feature and shape feature to correct chinese text error, it can be used for Chinese Pinyin and stroke input method.
Features
language model
- Kenlm
- RNNLM
Usage
install
- pip install pycorrector / pip3 install pycorrector
- Or download https://github.com/shibing624/pycorrector, Unzip and run: python setup.py install
correct
input:
import pycorrector
corrected_sent, detail = pycorrector.correct('少先队员因该为老人让坐')
print(corrected_sent, detail)
output:
少先队员应该为老人让座 [[('因该', '应该', 4, 6)], [('坐', '座', 10, 11)]]
Future work
-
P(c), the language model. We could create a better language model by collecting more data, and perhaps by using a little English morphology (such as adding "ility" or "able" to the end of a word).
-
P(w|c), the error model. So far, the error model has been trivial: the smaller the edit distance, the smaller the error. Clearly we could use a better model of the cost of edits. get a corpus of spelling errors, and count how likely it is to make each insertion, deletion, or alteration, given the surrounding characters.
-
It turns out that in many cases it is difficult to make a decision based only on a single word. This is most obvious when there is a word that appears in the dictionary, but the test set says it should be corrected to another word anyway: correction('where') => 'where' (123); expected 'were' (452) We can't possibly know that correction('where') should be 'were' in at least one case, but should remain 'where' in other cases. But if the query had been correction('They where going') then it seems likely that "where" should be corrected to "were".
-
Finally, we could improve the implementation by making it much faster, without changing the results. We could re-implement in a compiled language rather than an interpreted one. We could cache the results of computations so that we don't have to repeat them multiple times. One word of advice: before attempting any speed optimizations, profile carefully to see where the time is actually going.