问题描述
我正在运行一个声明性管道,其中一个步骤将运行(很长)集成测试.我试图将测试分为几个较小的测试,并在多个节点上并行运行它们.我有8个较小的测试,并且有8个节点(在标签下),所以我希望每个测试都在单独的节点上运行.不幸的是,当两个测试在同一节点上运行时,它们会相互干扰,因此都失败了.
I am running a Declarative Pipeline where one of the steps runs a (very long) integration test. I'm trying to split my test into several smaller ones and run them in parallel over several nodes. I have 8 of these smaller tests and I have 8 nodes (under a label), so I'd like to have each test run on a separate node. Unfortunately, two tests — when run on the same node — interfere with each other, and so both fail.
我需要能够首先获得可用节点的列表,然后并行运行每个节点中的一个较小的测试;如果没有足够的节点,则较小的测试之一需要等到该节点完成.
I need to be able to first get the list of available nodes, and then run the smaller tests in parallel, one of each node; if there are not enough nodes, one of the smaller tests need to wait until the node is finished.
但是,发生的事情是,当按标签要求一个节点时,两个较小的测试通常会得到相同的节点,因此都失败了.节点配置为最多可运行3个执行器,否则整个系统将停止运行,因此我无法更改它.
However, what happens is that when asking for a node by label, two of the smaller tests usually get the same node, and so both fail. Nodes are configured to run up to 3 executors, otherwise the whole system halts, so I can't change that.
我目前用于较小测试的配置是:
My current configuration for the smaller test is:
stage('Integration Tests') {
when {
expression {params.TESTS_INTEGRATION}
}
parallel {
stage('Test1') {
agent {node {label 'my_builder'}}
steps {
script {
def shell_script = getShellScript("Test1")
sh "${shell_script}"
}
}
}
我可以从这样的标签中获取可用的从站列表:
I am able to get the list of available slaves from a label like this:
pipeline {
stages {
// ... other stages here ...
stage('NodeList'){
steps {
script {
def nodes = getNodeNames('my_builder')
free_nodes = []
for (def element = 0; element < nodes.size(); element++) {
usenode = nodes[element]
try {
// Give it 5 seconds to run the nodetest function
timeout(time: 5, unit: 'SECONDS') {
node(usenode) {
nodetest()
free_nodes += usenode
}
}
} catch(err) {
}
}
println free_nodes
}
}
}
哪里
def getNodeNames (String label) {
def lgroup = Jenkins.instance.getLabel(label)
def nodes = lgroup.getNodes()
def result = []
if (nodes.size() > 0) {
for (def element = 0; element < nodes.size(); element++) {
result += nodes[element].getNodeName()
}
}
return result
}
def nodetest() {
sh('echo alive on \$(hostname)')
}
如何从free_nodes
数组中以编程方式获取节点名称,并指示舞台使用该名称?
How can I get the node name programmatically out of the free_nodes
array and direct the stage to use that?
推荐答案
我已经弄清楚了,所以对于未来的人们来说:
I've figured it out, so for the people from the future:
事实证明,您可以在声明性管道中运行脚本化管道,如下所示:
It turns out you can run a Scripted Pipeline inside a Declarative Pipeline, like this:
pipeline {
stage('SomeStage') {
steps {
script {
// ... your scripted pipeline here
}
}
}
该脚本可以执行任何操作,其中包括...运行管道!
The script can do anything, and that includes... running a pipeline!
这是脚本:
script {
def builders = [:]
def nodes = getNodeNames('my_label')
// let's find the free nodes
String[] free_nodes = []
for (def element = 0; element < nodes.size(); element++) {
usenode = nodes[element]
try {
// Give it 5 seconds to run the nodetest function
timeout(time: 5, unit: 'SECONDS') {
node(usenode) {
nodetest()
free_nodes += usenode
}
}
} catch(err) {
// do nothing
}
}
println free_nodes
def tests = params.TESTS_LIST.split(',')
for(int i = 0; i < tests.length; i++) {
// select the test to run
def the_test = tests[i]
// select on which node to run it
def the_node = free_nodes[i % free_nodes.length]
// here comes the scripted pipeline: prepare steps
builders[the_test] = {
// run on the selected node
node(the_node) {
// lock the resource with the name of the node so two tests can't run there at the same time
lock(the_node) {
// name the stage
stage(the_test) {
println "Running on ${NODE_NAME}"
def shell_script = getShellScript("${the_test}")
sh "${shell_script}"
}
}
}
}
}
// run the steps in parallel
parallel builders
}
这篇关于Jenkins声明式管道作业-如何在从站之间分配并行步骤?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!