本文介绍了如何从gcs中获取文件列表?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
遵循google的,我使用以下代码获取列表在远程目录中的所有文件
class GCSFileStorage {
String bucket =bucket_name;
String remoteDirectoryPath =remote / path;
int fetchBlockSize = 1024 * 1024;
GcsService gcsService =
GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());
列表< String> list(){
List< String> filenames = new List();
ListResult listResult = gcsService.list(bucket,ListOptions.DEFAULT);
while(listResult.hasNext()){
ListItem listItem = listResult.next();
文件名+ = listItem.getName();
}
返回文件名;
}
}
GCSFileStorage gcs = new GCSFileStorage();
gcs.list();
但是这段代码会失败并产生一个异常:
java.io.IOException:com.google.appengine.tools.cloudstorage.RetriesExhaustedException:
...
导致:java.io.IOException:java .lang.NullPointerException
...
导致:java.lang.NullPointerException
在com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService $ BlobStorageAdapter。< init>(LocalRawGcsService .java:123)
at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService $ BlobStorageAdapter.getInstance(LocalRawGcsService.java:184)
我怀疑我应该以gcs授权,这可能是失败的原因。然而,我还没有找到正确的方法来启动gcs工作所需的一切。
我使用的https://cloud.google.com/appengine/docs/java/googlecloudstorageclient/ =nofollow>客户端是特定于App Engine的。它是通过依赖关系添加的。 $ b
com.google.appengine.tools:appengine-gcs-client:0.5
code>
改为应该被使用。它的依赖关系是
com.google.apis:google-api-services-storage:v1-rev44-1.20.0
然后,获取文件列表的代码可能如下所示:
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.jackson2.JacksonFactory;
导入com.google.api.services.storage.Storage;
导入com.google.api.services.storage.StorageScopes;
导入com.google.api.services.storage.model.Objects;
import com.google.api.services.storage.model.StorageObject;
导入com.google.common.collect.Lists;
import java.io.File;
import java.io.IOException;
import java.security.GeneralSecurityException;
import java.util.LinkedList;
import java.util.List;
class GCSFileStorage {
String bucket =bucket_name;
String remoteDirectoryPath =remote / path;
存储空间
public GCSFileStorage()抛出GeneralSecurityException,IOException {
storage = setupStorage();
}
列表< String> list()抛出IOException {
List< String> allItems = new LinkedList< String>();
对象响应= storage.objects().list(bucket)。
setPrefix(remoteDirectoryPath).execute();
for(StorageObject obj:response.getItems()){
allItems.add(obj.getName());
while(response.getNextPageToken()!= null){
String pageToken = response.getNextPageToken();
response = storage.objects()。list(bucket)。
setPrefix(remoteDirectoryPath).setPageToken(pageToken).execute();
for(StorageObject obj:response.getItems()){
allItems.add(obj.getName());
}
}
返回allItems;
}
存储setupStorage()抛出GeneralSecurityException,IOException {
GoogleCredential凭证=新的GoogleCredential.Builder()。
setTransport(新的NetHttpTransport())。
setJsonFactory(new JacksonFactory())。
setServiceAccountId(your_account_id)。
setServiceAccountScopes(
Lists.newArrayList(StorageScopes.DEVSTORAGE_FULL_CONTROL))。
setServiceAccountPrivateKeyFromP12File(
new File(/ local / path / to / private / key.p12))。
build();
返回新的存储。
Builder(新的NetHttpTransport(),
新的JacksonFactory(),凭据)。
setApplicationName(foo)。build();
}
}
Following google's Getting Started I use following code to get the list of all files in a remote directory
class GCSFileStorage {
String bucket = "bucket_name";
String remoteDirectoryPath = "remote/path";
int fetchBlockSize = 1024 * 1024;
GcsService gcsService =
GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());
List<String> list() {
List<String> filenames = new List();
ListResult listResult = gcsService.list(bucket, ListOptions.DEFAULT);
while (listResult.hasNext()) {
ListItem listItem = listResult.next();
filenames += listItem.getName();
}
return filenames;
}
}
GCSFileStorage gcs = new GCSFileStorage();
gcs.list();
But this code fails with an exception:
java.io.IOException: com.google.appengine.tools.cloudstorage.RetriesExhaustedException:
...
Caused by: java.io.IOException: java.lang.NullPointerException
...
Caused by: java.lang.NullPointerException
at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService$BlobStorageAdapter.<init>(LocalRawGcsService.java:123)
at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService$BlobStorageAdapter.getInstance(LocalRawGcsService.java:184)
I suspect that I somehow should authorize in gcs and this may be the reason of failure. However I haven't found proper way to init everything that gcs needs for work.
解决方案
As @ozarov mentioned the client I was using is specific for App Engine. It was added through dependency
com.google.appengine.tools:appengine-gcs-client:0.5
Instead REST API client should be used. Its dependency is
com.google.apis:google-api-services-storage:v1-rev44-1.20.0
Then the code to fetch files list may look as follows
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.storage.Storage;
import com.google.api.services.storage.StorageScopes;
import com.google.api.services.storage.model.Objects;
import com.google.api.services.storage.model.StorageObject;
import com.google.common.collect.Lists;
import java.io.File;
import java.io.IOException;
import java.security.GeneralSecurityException;
import java.util.LinkedList;
import java.util.List;
class GCSFileStorage {
String bucket = "bucket_name";
String remoteDirectoryPath = "remote/path";
Storage storage
public GCSFileStorage() throws GeneralSecurityException, IOException {
storage = setupStorage();
}
List<String> list() throws IOException {
List<String> allItems = new LinkedList<String>();
Objects response = storage.objects().list(bucket).
setPrefix(remoteDirectoryPath).execute();
for (StorageObject obj: response.getItems()) {
allItems.add(obj.getName());
}
while (response.getNextPageToken() != null) {
String pageToken = response.getNextPageToken();
response = storage.objects().list(bucket).
setPrefix(remoteDirectoryPath).setPageToken(pageToken).execute();
for (StorageObject obj: response.getItems()) {
allItems.add(obj.getName());
}
}
return allItems;
}
Storage setupStorage() throws GeneralSecurityException, IOException {
GoogleCredential credential = new GoogleCredential.Builder().
setTransport(new NetHttpTransport()).
setJsonFactory(new JacksonFactory()).
setServiceAccountId("your_account_id").
setServiceAccountScopes(
Lists.newArrayList(StorageScopes.DEVSTORAGE_FULL_CONTROL)).
setServiceAccountPrivateKeyFromP12File(
new File("/local/path/to/private/key.p12")).
build();
return new Storage.
Builder(new NetHttpTransport(),
new JacksonFactory(), credential).
setApplicationName("foo").build();
}
}
这篇关于如何从gcs中获取文件列表?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!