AppEngine MapReduce NDB，DeadlineExceededError

问题描述

我们正在努力在我们的项目中大量使用 MapReduce 。
现在我们遇到了这个问题，在日志中有很多' DeadlineExceededError '错误...

we're trying to heavily use MapReduce in our project.Now we have this problem, there is a lots of 'DeadlineExceededError' errors in the log...

它（追溯每次都有所不同）：

One example of it ( traceback differs each time a bit ) :

Traceback (most recent call last):
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 207, in Handle
    result = handler(dict(self._environ), self._StartResponse)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
    return handler.dispatch()
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/s~sba/1.362471299468574812/mapreduce/base_handler.py", line 65, in post
    self.handle()
  File "/base/data/home/apps/s~sba/1.362471299468574812/mapreduce/handlers.py", line 208, in handle
    ctx.flush()
  File "/base/data/home/apps/s~sba/1.362471299468574812/mapreduce/context.py", line 333, in flush
    pool.flush()
  File "/base/data/home/apps/s~sba/1.362471299468574812/mapreduce/context.py", line 221, in flush
    self.__flush_ndb_puts()
  File "/base/data/home/apps/s~sba/1.362471299468574812/mapreduce/context.py", line 239, in __flush_ndb_puts
    ndb.put_multi(self.ndb_puts.items, config=self.__create_config())
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3625, in put_multi
    for future in put_multi_async(entities, **ctx_options)]
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 323, in get_result
    self.check_success()
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 318, in check_success
    self.wait()
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 302, in wait
    if not ev.run1():
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/ndb/eventloop.py", line 219, in run1
    delay = self.run0()
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/ndb/eventloop.py", line 181, in run0
    callback(*args, **kwds)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 365, in _help_tasklet_along
    value = gen.send(val)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/ndb/context.py", line 274, in _put_tasklet
    keys = yield self._conn.async_put(options, datastore_entities)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1560, in async_put
    for pbs, indexes in pbsgen:
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1350, in __generate_pb_lists
    incr_size = pb.lengthString(pb.ByteSize()) + 1
DeadlineExceededError

我的问题是：

• 我们如何避免此错误？


• 工作会发生什么，它会重试吗（如果是的话，我们如何控制它）？或不？


• 最终会导致数据不一致吗？

推荐答案

很明显，你正在执行太多的put操作，而不能插入到一个数据存储区调用中。您有多种选择：

Apparently you are doing too many puts than it is possible to insert in one datastore call. You have multiple options here:

1. 如果这是一个比较罕见的事件 - 忽略它。 Mapreduce会重试切片并降低放置池大小。确保您的地图是幂等的。


2. 查看 http://code.google.com/p/appengine-mapreduce/source/browse/trunk/python/src/mapreduce/context.py - 在你的main.py中，你可以降低 DATASTORE_DEADLINE ， MAX_ENTITY_COUNT 或 MAX_POOL_SIZE 来减小整个mapreduce池的大小。
   

1. If this is a relatively rare event - ignore it. Mapreduce will retry the slice and will lower put pool size. Make sure that your map is idempotent.
2. Take a look at http://code.google.com/p/appengine-mapreduce/source/browse/trunk/python/src/mapreduce/context.py - in your main.py you can lower DATASTORE_DEADLINE, MAX_ENTITY_COUNT or MAX_POOL_SIZE to lower the size of the pool for the whole mapreduce.

这篇关于AppEngine MapReduce NDB，DeadlineExceededError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！