如何在Bash中的多个变量中存储curl输出

问题描述

在我的脚本中，我有两个http请求.我想重用连接，所以例如，我要做的是:

In my script, I have two http requests. I would like reuse the connection, so for example what I do is:

curl -v 'http://example.com?id=1&key1=value1' 'http://example.com?id=1&key2=value2'

是否可以将每个http请求的输出存储在两个不同的变量中?我一直在搜寻.我还没有找到任何解决方案.

Is there any way to store the output of each http request in two different variables? I have been searching. I haven't found any solution yet.

我了解我可以执行以下操作以将输出存储在两个不同的文件中.

I understand I can do the following to store output in two different files.

curl -v 'http://example.com?id=1&key1=value1' -o output1 'http://example.com?id=1&key2=value2' -o output2

这是我的用例

我有一个cronjob，它每隔几分钟在下面运行一次并行(GNU并行)命令.而"get_data.sh"将运行2000次，因为input.csv中有2000行.我想避免使用tmp文件来获得最佳性能.

I have a cronjob that runs the parallel (GNU parallel) command below every few minutes. And 'get_data.sh' will be run 2000 times, because there are 2000 rows in input.csv. I would like to avoid using tmp file to get the best performance.

parallel \
  -a input.csv \
  --jobs 0 \
  --timeout $parallel_timeout \
  "get_data.sh {}"

在get_data.sh中:

In get_data.sh:

id=$1
curl -v "http://example.com?id=${id}&key1=value1" -o output1 \
"http://example.com?id=${id}&key2=value2" -o output2

stat1=$(cat output1 | sed '' | cut ..)
stat2=$(cat output2 | awk '')

推荐答案

您正在寻找 parset .它是 env_parallel 的一部分，它是GNU Parallel软件包的一部分( https://www.gnu.org/software/parallel/parset.html ):

You are looking for parset. It is part of env_parallel which is part of the GNU Parallel package (https://www.gnu.org/software/parallel/parset.html):

parset myarr \
  -a input.csv \
  --jobs 0 \
  --timeout $parallel_timeout \
  get_data.sh {}

echo "${myarr[3]}"

您可以让 parset 运行所有组合-就像使用GNU Parallel:

You can have parset run all combinations - just like you would with GNU Parallel:

echo www.google.com > input.txt
echo www.bing.com >> input.txt

# Search for both foo and bar on all sites
parset output curl https://{1}/?search={2} :::: input.txt ::: foo bar

echo "${output[1]}"
echo "${output[2]}"

如果您要对 foo 和 bar 做不同的处理，则可以创建函数并运行这些函数:

If you are doing different processing for foo and bar you can make functions and run those:

# make all new functions, arrays, variables, and aliases defined after this
# available to env_parset
env_parallel --session

foofunc() {
  id="$1"
  curl -v "http://example.com?id=${id}&key1=value1" | sed '' | cut -f10
}

barfunc() {
  id="$1"
  curl -v "http://example.com?id=${id}&key2=value2" | awk '{print}'
}

# Run both foofunc and barfunc on all sites
env_parset output {1} {2} ::: foofunc barfunc :::: input.txt

echo "${output[1]}"
echo "${output[2]}"
env_parallel --end-session

如果您不想 export -f 您要使用的函数和变量，则需要

-(end-)session 和 env_parset 在功能中使用.

--(end-)session and env_parset are needed if you do not want to export -f the functions and variables that you use in the functions.

GNU Parallel使用临时文件.如果您的命令运行速度很快，则这些临时文件在删除之前永远不会接触磁盘.相反，它们保留在RAM的磁盘缓存中.您甚至可以通过将-tmpdir 指向虚拟磁盘来强制它们保留在RAM中:

GNU Parallel uses tempfiles. If your command runs fast then these tempfiles never touch the disk before they are deleted. Instead they stay in the disk cache in RAM. You can even force them to stay in RAM by pointing --tmpdir to a ramdisk:

mkdir /dev/shm/mydir
parset output --tmpdir /dev/shm/mydir ...

这篇关于如何在Bash中的多个变量中存储curl输出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！