批量上传Azure Blob Storage文件的最快方法是什么?我尝试了两种方法,sync
和async
上传,async
显然是最快的,但是我想知道是否有更好的方法?是否有对批量上传的内置支持?我在文档中找不到任何内容,但可能会错过它。
这是我运行的测试:
static void Main(string[] args)
{
int totalFiles = 10; //10, 50, 100
byte[] randomData = new byte[2097152]; //2mb
for (int i = 0; i < randomData.Length; i++)
{
randomData[i] = 255;
}
CloudStorageAccount cloudStorageAccount = CloudStorageAccount.Parse(ConfigurationManager.AppSettings["StorageConnectionString"]);
var blobClient = cloudStorageAccount.CreateCloudBlobClient();
var container = blobClient.GetContainerReference("something");
container.CreateIfNotExists();
TimeSpan tsSync = Test1(totalFiles, randomData, container);
TimeSpan tsAsync = Test2(totalFiles, randomData, container);
Console.WriteLine($"Sync: {tsSync}");
Console.WriteLine($"Async: {tsAsync}");
Console.ReadLine();
}
public static TimeSpan Test2(int total, byte[] data, CloudBlobContainer container)
{
Stopwatch sw = new Stopwatch();
sw.Start();
Task[] tasks = new Task[total];
for (int i = 0; i < total; i++)
{
CloudBlockBlob blob = container.GetBlockBlobReference(Guid.NewGuid().ToString());
tasks[i] = blob.UploadFromByteArrayAsync(data, 0, data.Length);
}
Task.WaitAll(tasks);
sw.Stop();
return sw.Elapsed;
}
public static TimeSpan Test1(int total, byte[] data, CloudBlobContainer container)
{
Stopwatch sw = new Stopwatch();
sw.Start();
for (int i = 0; i < total; i++)
{
CloudBlockBlob blob = container.GetBlockBlobReference(Guid.NewGuid().ToString());
blob.UploadFromByteArray(data, 0, data.Length);
}
sw.Stop();
return sw.Elapsed;
}
输出是:
10个文件
Sync: 00:00:08.7251781
Async: 00:00:04.7553491
DMLib: 00:00:05.1961654
Sync: 00:00:08.1169861
Async: 00:00:05.2384105
DMLib: 00:00:05.4955403
Sync: 00:00:07.6122464
Async: 00:00:05.0495365
DMLib: 00:00:06.4714047
50个文件
Sync: 00:00:39.1595797
Async: 00:00:22.5757347
DMLib: 00:00:25.2897623
Sync: 00:00:40.4932800
Async: 00:00:22.3296490
DMLib: 00:00:26.0631829
Sync: 00:00:39.2879245
Async: 00:00:24.0746697
DMLib: 00:00:26.9243116
我希望这对SO是一个有效的问题。
谢谢
编辑:
我已经使用“DMLib”测试更新了结果,以响应到目前为止给出的答案。 DMLib是一项测试,没有配置更改(请参见上文),没有性能提升
根据文档推荐,我对
ServicePointManager.DefaultConnectionLimit = Environment.ProcessorCount * 8;
进行了更多测试,这使上传速度大大提高,但同时也提高了异步方法的上传速度。到目前为止,DMlib并没有给我带来任何值得提高的性能。我在下面的配置更改中添加了第二组测试结果。我还设置了
ServicePointManager.Expect100Continue = false;
,但这对速度没有影响。使用
ServicePointManager.DefaultConnectionLimit = Environment.ProcessorCount * 8;
的测试结果10个文件
Sync: 00:00:07.6199307
Async: 00:00:02.9615565
DMLib: 00:00:02.6629716
Sync: 00:00:08.7721797
Async: 00:00:02.8246599
DMLib: 00:00:02.7281091
Sync: 00:00:07.8437682
Async: 00:00:03.0171246
DMLib: 00:00:03.0190045
50个文件
Sync: 00:00:40.2395863
Async: 00:00:10.3157544
DMLib: 00:00:10.5107740
Sync: 00:00:40.2473358
Async: 00:00:10.8190161
DMLib: 00:00:10.2585441
Sync: 00:00:41.2646137
Async: 00:00:13.7188085
DMLib: 00:00:10.8686173
我是否在错误使用该库,因为它似乎没有比我自己的方法提供更好的性能。
最佳答案
请使用Azure Storage Data Movement Library,它是AzCopy的核心。该库正是解决您问题的工具。 :)
关于c# - 将文件批量上传到Azure(C#)的最快方法,我们在Stack Overflow上找到一个类似的问题:https://stackoverflow.com/questions/37386073/