问题描述
我想从FTP目录中递归下载多个下载文件,为此,我正在使用FluentFTP库,而我的代码是这样的:
I would like to download multiple download files recursively from a FTP Directory, to do this I'm using FluentFTP library and my code is this one:
private async Task downloadRecursively(string src, string dest, FtpClient ftp)
{
foreach(var item in ftp.GetListing(src))
{
if (item.Type == FtpFileSystemObjectType.Directory)
{
if (item.Size != 0)
{
System.IO.Directory.CreateDirectory(Path.Combine(dest, item.Name));
downloadRecursively(Path.Combine(src, item.Name), Path.Combine(dest, item.Name), ftp);
}
}
else if (item.Type == FtpFileSystemObjectType.File)
{
await ftp.DownloadFileAsync(Path.Combine(dest, item.Name), Path.Combine(src, item.Name));
}
}
}
我知道您每次下载都需要一个FtpClient,但是我该如何最大程度地使用一定数量的连接,我想这个想法是针对我找到的每个文件创建,连接,下载和关闭.同时具有X个下载文件.另外,我不确定是否应该使用异步,线程和我最大的问题(如何实现所有这些功能)创建Task.
I know you need one FtpClient per download you want, but how can I make to use a certain number of connections as maximum, I guess that the idea is to create, connect, download and close per every file I find but just having a X number of downloading files at the same time. Also I'm not sure if I should create Task with async, Threads and my biggest problem, how to implement all of this.
来自 @Bradley 的答案似乎不错,但问题是确实读取了必须从外部下载的每个文件文件,并且没有最大的并发下载值,因此我不确定如何应用这两个要求.
Answer from @Bradley here seems pretty good, but the question does read every file thas has to download from an external file and it doesn't have a maximum concurrent download value so I'm not sure how to apply these both requirements.
推荐答案
使用:
-
ConcurrentBag
类以实现连接池; -
Parallel
类使操作并行化; -
ParallelOptions.MaxDegreeOfParallelism
限制并发线程数.
var clients = new ConcurrentBag<FtpClient>();
var opts = new ParallelOptions { MaxDegreeOfParallelism = maxConnections };
Parallel.ForEach(files, opts, file =>
{
file = Path.GetFileName(file);
string thread = $"Thread {Thread.CurrentThread.ManagedThreadId}";
if (!clients.TryTake(out var client))
{
Console.WriteLine($"{thread} Opening connection...");
client = new FtpClient(host, user, pass);
client.Connect();
Console.WriteLine($"{thread} Opened connection {client.GetHashCode()}.");
}
string remotePath = sourcePath + "/" + file;
string localPath = Path.Combine(destPath, file);
string desc =
$"{thread}, Connection {client.GetHashCode()}, " +
$"File {remotePath} => {localPath}";
Console.WriteLine($"{desc} - Starting...");
client.DownloadFile(localPath, remotePath);
Console.WriteLine($"{desc} - Done.");
clients.Add(client);
});
Console.WriteLine($"Closing {clients.Count} connections");
foreach (var client in clients)
{
Console.WriteLine($"Closing connection {client.GetHashCode()}");
client.Dispose();
}
另一种方法是启动固定数量的线程,每个线程具有一个连接,并让它们从队列中选择文件.
Another approach is to start a fixed number of threads with one connection for each and have them pick files from a queue.
有关实现的示例,请参阅我关于WinSCP .NET程序集的文章:
通过SFTP/FTP协议自动并行连接中的传输
For an example of an implementation, see my article for WinSCP .NET assembly:
Automating transfers in parallel connections over SFTP/FTP protocol
关于SFTP的类似问题:
使用C#Parallel.ForEach循环处理SFTP文件而不处理下载
这篇关于使用FluentFTP从FTP同时下载多个文件,最大值的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!