问题描述
:) 我尝试使用 w = Word(printables),但它不起作用.我应该如何为此提供规范.'w' 用于处理印地语字符 (UTF-8)
:) I tried using w = Word(printables), but it isn't working. How should I give the spec for this. 'w' is meant to process Hindi characters (UTF-8)
代码指定了语法并进行了相应的解析.
The code specifies the grammar and parses accordingly.
671.assess :: अहसास ::2
x=number + "." + src + "::" + w + "::" + number + "." + number
如果只有英文字符,它可以工作,所以代码对于 ascii 格式是正确的,但代码对于 unicode 格式不起作用.
If there is only english characters it is working so the code is correct for the ascii format but the code is not working for the unicode format.
我的意思是,当我们有某种形式的东西时,代码就可以工作了671.assess::ahsaas::2
I mean that the code works when we have something of the form671.assess :: ahsaas ::2
即它以英文格式解析单词,但我不确定如何解析然后以 unicode 格式打印字符.我需要这个用于英语印地语单词对齐.
i.e. it parses words in the english format, but I am not sure how to parse and then print characters in the unicode format. I need this for English Hindi word alignment for purpose.
python 代码如下所示:
The python code looks like this:
# -*- coding: utf-8 -*-
from pyparsing import Literal, Word, Optional, nums, alphas, ZeroOrMore, printables , Group , alphas8bit ,
# grammar
src = Word(printables)
trans = Word(printables)
number = Word(nums)
x=number + "." + src + "::" + trans + "::" + number + "." + number
#parsing for eng-dict
efiledata = open('b1aop_or_not_word.txt').read()
eresults = x.parseString(efiledata)
edict1 = {}
edict2 = {}
counter=0
xx=list()
for result in eresults:
trans=""#translation string
ew=""#english word
xx=result[0]
ew=xx[2]
trans=xx[4]
edict1 = { ew:trans }
edict2.update(edict1)
print len(edict2) #no of entries in the english dictionary
print "edict2 has been created"
print "english dictionary" , edict2
#parsing for hin-dict
hfiledata = open('b1aop_or_not_word.txt').read()
hresults = x.scanString(hfiledata)
hdict1 = {}
hdict2 = {}
counter=0
for result in hresults:
trans=""#translation string
hw=""#hin word
xx=result[0]
hw=xx[2]
trans=xx[4]
#print trans
hdict1 = { trans:hw }
hdict2.update(hdict1)
print len(hdict2) #no of entries in the hindi dictionary
print"hdict2 has been created"
print "hindi dictionary" , hdict2
'''
#######################################################################################################################
def translate(d, ow, hinlist):
if ow in d.keys():#ow=old word d=dict
print ow , "exists in the dictionary keys"
transes = d[ow]
transes = transes.split()
print "possible transes for" , ow , " = ", transes
for word in transes:
if word in hinlist:
print "trans for" , ow , " = ", word
return word
return None
else:
print ow , "absent"
return None
f = open('bidir','w')
#lines = ["'
#5# 10 # and better performance in business in turn benefits consumers . # 0 0 0 0 0 0 0 0 0 0
#5# 11 # vHyaapaar mEmn bEhtr kaam upbhOkHtaaomn kE lIe laabhpHrdd hOtaa hAI . # 0 0 0 0 0 0 0 0 0 0 0
#'"]
data=open('bi_full_2','rb').read()
lines = data.split('!@#$%')
loc=0
for line in lines:
eng, hin = [subline.split(' # ')
for subline in line.strip('
').split('
')]
for transdict, source, dest in [(edict2, eng, hin),
(hdict2, hin, eng)]:
sourcethings = source[2].split()
for word in source[1].split():
tl = dest[1].split()
otherword = translate(transdict, word, tl)
loc = source[1].split().index(word)
if otherword is not None:
otherword = otherword.strip()
print word, ' <-> ', otherword, 'meaning=good'
if otherword in dest[1].split():
print word, ' <-> ', otherword, 'trans=good'
sourcethings[loc] = str(
dest[1].split().index(otherword) + 1)
source[2] = ' '.join(sourcethings)
eng = ' # '.join(eng)
hin = ' # '.join(hin)
f.write(eng+'
'+hin+'
')
f.close()
'''
如果源文件的示例输入语句是:
if an example input sentence for the source file is:
1# 5 # modern markets : confident consumers # 0 0 0 0 0
1# 6 # AddhUnIk baajaar : AshHvsHt upbhOkHtaa . # 0 0 0 0 0 0
!@#$%
输出看起来像这样:-
1# 5 # modern markets : confident consumers # 1 2 3 4 5
1# 6 # AddhUnIk baajaar : AshHvsHt upbhOkHtaa . # 1 2 3 4 5 0
!@#$%
输出说明:-这实现了双向对齐.这意味着英语现代"的第一个词映射到印地语AddhUnIk"的第一个词,反之亦然.这里甚至字符也被当作单词,因为它们也是双向映射的一个组成部分.因此,如果您观察印地语单词 '.'有一个空对齐,它相对于英语句子没有映射,因为它没有句号.当我们处理一些您试图实现双向映射的句子时,输出中的第 3 行基本上代表一个分隔符.
Output Explanation:-This achieves bidirectional alignment.It means the first word of english 'modern' maps to the first word of hindi 'AddhUnIk' and vice versa. Here even characters are take as words as they also are an integral part of bidirectional mapping. Thus if you observe the hindi WORD '.' has a null alignment and it maps to nothing with respect to the English sentence as it doesn't have a full stop.The 3rd line int the output basically represents a delimiter when we are working for a number of sentences for which your trying to achieve bidirectional mapping.
如果我有 Unicode(UTF-8) 格式的印地语句子,我应该做哪些修改才能使其工作.
What modification should i make for it to work if the I have the hindi sentences in Unicode(UTF-8) format.
推荐答案
作为一般规则,不要不要处理编码的字节串:将它们变成正确的 unicode 字符串(通过调用它们的 .decode
方法),请始终在 unicode 字符串上进行所有处理,然后,如果您必须出于 I/O 目的,.encode
将它们重新转换为您需要的任何字节串编码.
As a general rule, do not process encoded bytestrings: make them into proper unicode strings (by calling their .decode
method) as soon as possible, do all of your processing always on unicode strings, then, if you have to for I/O purposes, .encode
them back into whatever bytestring encoding you require.
如果您在谈论文字,就像您在代码中一样,尽快"是立即:使用 u'...' 来表达你的文字.在更一般的情况下,当您被迫以编码形式进行 I/O 时,它立即在输入之后进行(就像如果您需要以特定编码形式执行输出一样,它就在输出之前).
If you're talking about literals, as it seems you are in your code, the "as soon as possible" is at once: use
u'...'
to express your literals. In a more general case, where you're forced to do I/O in encoded form, it's immediately after input (just as it's immediately before output if you need to perform output in a specific encoded form).
这篇关于Python - pyparsing unicode 字符的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!