parallel-processing - BLAS v.Julia SharedArray对象的并行更新

我对将Julia SharedArray用于科学计算项目感兴趣。我当前的实现对所有矩阵矢量运算都吸引BLAS，但是我认为SharedArray可能会在多核计算机上提供一些加速。我的想法是简单地按索引更新输出向量，将索引更新填充到工作进程中。

先前有关SharedArray的here和有关共享内存对象的here的讨论并未对此问题提供明确的指导。从直观上看似乎很简单，但是经过测试，我对这种方法为何如此差的效果感到困惑(请参见下面的代码)。对于初学者来说，@parallel for似乎分配了很多内存。如果我在循环前添加@sync前缀(如果以后需要整个输出向量，这似乎是一件很明智的事情)，那么并行循环会慢得多(尽管没有@sync，该循环非常快)。

我是否错误地解释了SharedArray对象的正确用法？还是我没有有效地分配计算？

### test for speed gain w/ SharedArray vs. Array ###

# problem dimensions
n = 10000; p = 25000

# set BLAS threads; 64 seems reasonable in testing
blas_set_num_threads(64)

# make normal Arrays
x = randn(n,p)
y = ones(p)
z = zeros(n)

# make SharedArrays
X = convert(SharedArray{Float64,2}, x)
Y = convert(SharedArray{Float64,1}, y)
Z = convert(SharedArray{Float64,1}, z)

# run BLAS.gemv! on Arrays twice, time second case
BLAS.gemv!('N', 1.0, x, y, 0.0, z)
@time BLAS.gemv!('N', 1.0, x, y, 0.0, z)

# does BLAS work equally well for SharedArrays?
# check timing result and ensure same answer
BLAS.gemv!('N', 1.0, X, Y, 0.0, Z)
@time BLAS.gemv!('N', 1.0, X, Y, 0.0, Z)
println("$(isequal(z,Z))")  # should be true

# SharedArrays can be updated in parallel
# code a loop to farm updates to worker nodes
# use transposed X to place rows of X in columnar format
# should (hopefully) help with performance issues from stride
Xt = X'
@parallel for i = 1:n
    Z[i] = dot(Y, Xt[:,i])
end

# now time the synchronized copy of this
@time @sync @parallel for i = 1:n
    Z[i] = dot(Y, Xt[:,i])
end

# still get same result?
println("$(isequal(z,Z))")  # should be true

具有4个工作器+ 1个主节点的测试输出:

elapsed time: 0.109010169 seconds (80 bytes allocated)
elapsed time: 0.110858551 seconds (80 bytes allocated)
true
elapsed time: 1.726231048 seconds (119936 bytes allocated)
true

最佳答案

您遇到了几个问题，其中最重要的是Xt[:,i]创建了一个新数组(分配内存)。这是一个使您更接近所需内容的演示:

n = 10000; p = 25000

# make normal Arrays
x = randn(n,p)
y = ones(p)
z = zeros(n)

# make SharedArrays
X = convert(SharedArray, x)
Y = convert(SharedArray, y)
Z = convert(SharedArray, z)

Xt = X'

@everywhere function dotcol(a, B, j)
    length(a) == size(B,1) || throw(DimensionMismatch("a and B must have the same number of rows"))
    s = 0.0
    @inbounds @simd for i = 1:length(a)
        s += a[i]*B[i,j]
    end
    s
end

function run1!(Z, Y, Xt)
    for j = 1:size(Xt, 2)
        Z[j] = dotcol(Y, Xt, j)
    end
    Z
end

function runp!(Z, Y, Xt)
    @sync @parallel for j = 1:size(Xt, 2)
        Z[j] = dotcol(Y, Xt, j)
    end
    Z
end

run1!(Z, Y, Xt)
runp!(Z, Y, Xt)
@time run1!(Z, Y, Xt)
zc = copy(sdata(Z))
fill!(Z, -1)
@time runp!(Z, Y, Xt)

@show sdata(Z) == zc

结果(启动julia -p 8时):

julia> include("/tmp/paralleldot.jl")
elapsed time: 0.465755791 seconds (80 bytes allocated)
elapsed time: 0.076751406 seconds (282 kB allocated)
sdata(Z) == zc = true

为了进行比较，在同一台计算机上运行时:

julia> blas_set_num_threads(8)

julia> @time A_mul_B!(Z, X, Y);
elapsed time: 0.067611858 seconds (80 bytes allocated)

因此，原始的Julia实现至少可以与BLAS竞争。

关于parallel-processing - BLAS v.Julia SharedArray对象的并行更新，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/28360850/

sharedArray

parallel-processing - BLAS v.Julia SharedArray对象的并行更新