通过Amazon SQS将压缩文本从PHP发送到NodeJS

问题描述

我似乎停留在通过Amazon SQS将压缩消息从PHP发送到NodeJS上.

I seem to be stuck at sending the compressed messages from PHP to NodeJS over Amazon SQS.

在PHP方面，我有:

$SQS->sendMessage(Array(
    'QueueUrl'    => $queueUrl,
    'MessageBody' => 'article',
    'MessageAttributes' => Array(
        'json' => Array(
            'BinaryValue' => bzcompress(json_encode(Array('type'=>'article','data'=>$vijest))),
            'DataType' => 'Binary'
        )
    )
));

注1:我也尝试过将压缩数据直接放入消息中，但是该库给了我一些无效字节数据的错误

NOTE 1: I also tried putting compressed data directly in the message, but the library gave me an error with some invalid byte data

在节点"一侧，我有:

body = decodeBzip(message.MessageAttributes.json.BinaryValue);

消息来自sqs.receiveMessage()调用，该部分有效，因为它适用于原始(未压缩的消息)

Where message is from sqs.receiveMessage() call and that part works since it worked for raw (uncompressed messages)

我得到的是 TypeError:格式不正确

我也尝试使用:

gzcompress()-zlib.inflateraw()

gzcompress() - zlib.inflateraw()

gzdeflate()-zlib.inflate()

gzdeflate() - zlib.inflate()

gzencode()-zlib.gunzip()

gzencode() - zlib.gunzip()

这些对中的每对都给了我相同错误的版本(本质上，输入数据是错误的)

And each of those pairs gave me their version of the same error (essentially, input data is wrong)

考虑到我开始怀疑消息传输中的某处错误

Given all that I started to suspect that an error is somewhere in message transmission

我在做什么错了?

编辑1 :似乎错误发生在传输中，因为php中的bin2hex()和Node中的.toString('hex')返回完全不同的值.似乎PHP中的Amazon SQS API使用base64传输BinaryAttribute，但是Node无法对其进行解码.我设法通过关闭Amazon aws配置文件中的自动转换，然后在node中手动解码base64来部分解码它，但是它仍然无法解码.

EDIT 1: It seems that the error is somewhere in transmission, since bin2hex() in php and .toString('hex') in Node return totally different values. It seems that Amazon SQS API in PHP transfers BinaryAttribute using base64 but Node fails to decode it. I managed to partially decode it by turning off automatic conversion in amazon aws config file and then manually decoding base64 in node but it still was not able to decode it.

编辑2 :通过在php端使用base64_encode()，并将base64作为messageBody发送(不使用MessageAttributes)，我设法完成了同样的事情.在节点一侧，我使用了新的Buffer(messageBody，'base64')，然后在其上使用了解码Bzip.一切正常，但是我仍然想知道为什么MessageAttribute无法正常工作.当前的base64增加了开销，我希望按预期使用服务，而不是通过变通办法.

EDIT 2: I managed to accomplish the same thing by using base64_encode() on the php side, and sending the base64 as a messageBody (not using MessageAttributes). On the node side I used new Buffer(messageBody,'base64') and then decodeBzip on that. It all works but I would still like to know why MessageAttribute is not working as it should. Current base64 adds overhead and I like to use the services as they are intended, not by work arounds.

推荐答案

这是所有SQS库在后台执行的操作.您可以获取SQS库的php源代码并亲自查看.二进制数据将始终一直以base64编码(无论是否使用 MessageAttributes ，都没有关系)，以此满足具有表单URL编码消息的API要求.

This is what all the SQS libraries do under the hood. You can get the php source code of the SQS library and see for yourself. Binary data will always be base64 encoded (when using MessageAttributes or not, does not matter) as a way to satisfy the API requirement of having form-url-encoded messages.

我不知道$ vijest中的数据有多长时间，但我敢打赌，在压缩后再进行base64编码后，它会比以前更大.

I do not know how long the data in your $vijest is, but I am willing to bet that after zipping and then base64 encoding it will be bigger than before.

所以我对您的回答分为两部分(如果您确实很顽固，则再加上三分之一):

So my answer to you would be two parts (plus a third if you are really stubborn):

• 在查看基础原始API时，绝对清楚不使用MessageAttributes不会增加base64的额外开销.相反，由于SQS php库强制执行的数据结构，因此使用MessageAttributes会增加一些额外的开销.因此，不使用MessageAttributes显然不是一种解决方法，如果您想自己压缩数据并以这种方式工作，则应该这样做.
• 由于http POST请求的性质，在应用程序内部压缩数据是一个非常糟糕的主意. Base64开销可能会使压缩优势无效，并且最好发送纯文本.
• 如果您完全不相信我或API规范或HTTP规范并且想要继续，那么我建议在BinaryValue参数中发送一个简单的短字符串"teststring"，并将发送的内容与获得的内容进行比较.这将使您很容易理解SQS库对BinaryValue参数所做的转换.

这篇关于通过Amazon SQS将压缩文本从PHP发送到NodeJS的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！