使用过多的内存从{raster}包中提取

问题描述

我一直在使用raster包中的extract函数，使用shapefile定义的区域从栅格文件中提取数据.但是，此过程现在需要的内存量存在问题.我确实有很多shapefile(〜1000个).光栅文件很大(〜1.6gb)

I have been using the extract function from the raster package to extract data from raster files using an area defined by shapefiles. However, I am having problems with the amount of memory that this process is now requiring. I do have a large number of shapefiles (~1000). The raster files are large (~1.6gb)

我的过程是:

shp <- mclapply(list.files(pattern="*.shp",full.names=TRUE), readShapePoly,mc.cores=6)
ndvi <- raster("NDVI.dat")
mc<- function(y) {
temp <- gUnionCascaded(y)
extract <- extract(ndvi,temp)
mean <- range(extract, na.rm=T )[1:2]
leng <- length(output)
}
output <- lapply(shp, mc)

我可以做些更改以减少内存负载吗?我尝试加载较少的shapefile，该文件在内存再次达到峰值之前可以工作约5分钟.其四核计算机2.4GHz与8GB内存

Are there any changes I can make to reduce the memory load? I tried loading fewer shapefiles which worked for about 5 min before the memory spiked again. Its a quad core computer 2.4ghz with 8gb ram

推荐答案

我会这样做(未经测试):

I would do this (untested):

## Clearly we need these packages, and their dependencies
library(raster)
library(rgeos)
shpfiles <- list.files(pattern="*.shp",full.names=TRUE)
ndvi <- raster("NDVI.dat")
## initialize an object to store the results for each shpfile
res <- vector("list", length(shpfiles))
names(res) <- shpfiles
## loop over files
for (i in seq_along(shpfiles)) {
  ## do the union
  temp <- gUnionCascaded(shpfiles[i])
  ## extract for this shape data (and don't call it "extract")
  extracted <- extract(ndvi,temp)
  ## further processing, save result
  mean <- range(extracted, na.rm = TRUE )[1:2]
  res[[i]] <- mean  ## plus whatever else you need
}

目前尚不清楚上面mc()的返回值是什么意思，因此我将其忽略.与您最初尝试的相比，这将大大提高内存效率和速度.我怀疑在这里完全可以使用并行的东西.

It's not at all clear what the return value of mc() above is meant to be, so I ignore it. This will be far more memory efficient and fast than what you tried originally. I doubt it's worth using parallel stuff at all here.

这篇关于使用过多的内存从{raster}包中提取的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！