在进行文本训练和处理之前难免要进行下预处理，过滤掉没有用的符号等，简单用python 的正则表达式过滤一下。
#!/usr/bin/python# encoding: UTF-8import re# make English text cleandef clean_en_text(text):# keep English, digital and spacecomp = pile('[^A-Z^a-z^0-9^ ]')return comp.sub('', text)# make Chinese text cleandef clean_zh_text(text):# keep English, digital and Chinesecomp = pile('[^A-Z^a-z^0-9^\u4e00-\u9fa5]')return comp.sub('', text)if __name__ == '__main__':text_en = '$How old are you? Could you give me your pen?'text_zh = '$你好！我是个程序猿，标注码农￥'print(clean_en_text(text_en))print(clean_zh_text(text_zh))
测试
(base) ➜ src python test.py
How old are you Could you give me your pen
你好我是个程序猿标注码农
