保持字符串转换为ASCII前后的一致性

问题描述

我有许多 unicode 格式的字符串，例如 carbonpolymers—III\n12-Géotechnique\n 以及更多具有许多不同 unicode 字符的字符串，位于名为 txtWords.

I have many strings in unicode format such as carbon copolymers—III\n12- Géotechnique\n and many more having many different unicode characters, in a string variable named txtWords.

我的目标是删除所有非 ASCII 字符同时保持字符串的一致性.例如，我想将第一句变成 carbonpolymers III 或 carbonpolymers iii(这里不区分大小写)，第二句变成 geotechnique\n 等等...

My goal is to remove all non-ASCII characters while preserving the consistency of the strings. For instance I want to first sentence turn into carbon copolymers III or carbon copolymers iii (no case-sensitivity here) and the second one to geotechnique\n and so on ...

目前我正在使用以下代码，但它并没有帮助我实现我的期望.当前代码将碳共聚物III更改为碳共聚物iii，这绝对不是它应该的样子:

Currently I am using the following code but it doesn't help me achieve what I expect. The current code changes carbon copolymers III to carbon copolymersiii which is definitely not what it should be:

import unicodedata, re
txtWords = unicodedata.normalize('NFKD', txtWords.lower()).encode('ascii','ignore')
txtWords = re.sub(r'[^a-z^\n]',r' ',txtWords)

如果我先使用正则表达式代码，那么我会得到更糟的东西(就我的期望而言):

If I use the regex code first then I get something worse (in terms of what I expect):

    import unicodedata, re
    txtWords = re.sub(r'[^a-z^\n]',r' ',txtWords)
    txtWords = unicodedata.normalize('NFKD', txtWords.lower()).encode('ascii','ignore')

这样，对于字符串 Géotechnique\n 我得到 otechnique！

This way, for the string Géotechnique\n I get otechnique!

我该如何解决这个问题?

How can I resolve this issue?

推荐答案

使用 \w 正则表达式在分解技巧前去除非字母数字:

Use the \w regular expression to strip non-alphanumerics before the decomposing trick:

#coding:utf8
from __future__ import unicode_literals,print_function
import unicodedata as ud
import re
txtWords = 'carbon copolymers—III\n12- Géotechnique\n'
txtWords = re.sub(r'[^\w\n]',r' ',txtWords.lower(),flags=re.U)
txtWords = ud.normalize('NFKD',txtWords).encode('ascii','ignore').decode()
print(txtWords)

输出(Python 2 和 3):

Output (Python 2 and 3):

carbon copolymers iii
12  geotechnique

这篇关于保持字符串转换为ASCII前后的一致性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！