如何清理 JdbcMetadataStore?

问题描述

最初我们与 google Pub/Sub 的交流流程是这样的:

Initially our flow of cimmunicating with google Pub/Sub was so:

1. 应用程序接受消息
2. 检查它在idempotencyStore中不存在
3. 3.1 如果不存在 - 将其放入幂等存储(key 是唯一标头的值，value 是当前时间戳)
   3.2 如果存在 - 忽略此消息
4. 处理完成后 - 发送确认
5. 在确认成功回调中 - 从元数据存储中删除此消息

第 5 点是错误的，因为理论上即使在消息处理后我们也可以得到重复的消息.此外，我们发现有时即使调用成功回调，消息也可能不会被删除(确认后一次又一次地从 Google Pub/Sub 订阅收到消息[Heisenbug]) 所以我们决定在消息处理后更新值并将时间戳替换为FiNISHED" 字符串

The point 5 is wrong because theoretically we can get duplicated message even after message has processed. Moreover we found out that sometimes message might not be removed even although successful callback was invoked( Message is received from Google Pub/Sub subscription again and again after acknowledge[Heisenbug]) So we decided to update value after message is proccessed and replace timestamp with "FiNISHED" string

但迟早我们会遇到这张桌子会人满为患.所以我们必须清理 MetaDataStore 中的消息.我们可以删除已处理且处理时间超过 1 天的消息.

But sooner or later we will encounter that this table will be overcrowded. So we have to cleanup messages in the MetaDataStore. We can remove messages which are processed and they were processed more 1 day.

正如在 https://stackoverflow.com/a/51845202/2674303 的评论中提到的，我可以添加metadataStore 表中的附加列，我可以在其中标记消息是否已处理.这根本不是问题.但是我怎样才能在我的清洁工中使用这个标志呢?MetadataStore 只有键和值

As was mentioned in the comments of https://stackoverflow.com/a/51845202/2674303 I can add additional column in the metadataStore table where I could mark if message is processed. It is not a problem at all. But how can I use this flag in the my cleaner? MetadataStore has only key and value

推荐答案

我完全看不出这一步的原因.

I don't see a reason in this step at all.

因为您说您在 value 中存储了一个时间戳，这意味着您可以不时分析此表以删除绝对旧的条目.

Since you say that you store in the value a timestamp that means that you can analyze this table from time to time to remove definitely old entries.

在我的一些项目中，我们在数据库中有一项日常工作，以归档一张表以提高主进程的性能.对，只是因为我们不再需要旧数据了.出于这个原因，我们肯定会检查原始文件中的一些时间戳，以确定它是否应该进入存档.我不会因为有可能从外部系统重新传送而在处理后立即删除数据.

In some my project we have a daily job in DB to archive a table for better main process performance. Right, just because we don't need old data any more. For this reason we definitely check some timestamp in the raw to determine if that should go into archive or not. I wouldn't remove data immediately after process just because there is a chance for redelivery from external system.

另一方面，为了获得更好的性能，我会在元数据表中添加具有 timestamp 类型的额外索引列，并在每次更新或插入时通过触发器填充一个值.好吧，MetadataStore 只需从 MetadataStoreSelector 插入一个条目:

On the other hand for better performance I would add extra indexed column with timestamp type into that metadata table and would populate a value via trigger on each update or instert. Well, MetadataStore just insert an entry from the MetadataStoreSelector:

return this.metadataStore.putIfAbsent(key, value) == null;

因此，您需要一个 on_insert 触发器来填充该日期列.这样您最终会知道是否需要删除条目.

So, you need an on_insert trigger to populate that date column. This way you will know in the end of day if you need to remove an entry or not.

这篇关于如何清理 JdbcMetadataStore?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！