高铭杰

高铭杰

关注
发信
关注(28)粉丝(399)

Postgresql源码（127）投影ExecProject的表达式执行分析

1 投影函数

用例

create table t1(i int primary key, j int, k int);
insert into t1 select i, i % 10, i % 100 from generate_series(1,10000000) t(i);
explain analyze select abs(k),abs(k),abs(k),abs(k),abs(k),exp(k),exp(k),exp(k),exp(k),exp(k) from t1;

对于这样这一条查询来说，每扫描一行，都会调用投影函数ExecProject，完成最终结果的构造。

投影函数：

static inline TupleTableSlot *
ExecProject(ProjectionInfo *projInfo)
{
	ExprContext *econtext = projInfo->pi_exprContext;
	ExprState  *state = &projInfo->pi_state;
	TupleTableSlot *slot = state->resultslot;
	bool		isnull;

	/*
	 * Clear any former contents of the result slot.  This makes it safe for
	 * us to use the slot's Datum/isnull arrays as workspace.
	 */
	ExecClearTuple(slot);

	/* Run the expression, discarding scalar result from the last column. */
	(void) ExecEvalExprSwitchContext(state, econtext, &isnull);

	/*
	 * Successfully formed a result row.  Mark the result slot as containing a
	 * valid virtual tuple (inlined version of ExecStoreVirtualTuple()).
	 */
	slot->tts_flags &= ~TTS_FLAG_EMPTY;
	slot->tts_nvalid = slot->tts_tupleDescriptor->natts;

	return slot;
}

总结：

  • projInfo->pi_exprContext记录了需要执行表达式的上下文信息
    • 具体存放：t1表扫出来的一行
    • 具体存放：表达式执行的内存上下文
  • projInfo->pi_state记录了表达式执行状态
    • 具体存放：表达式执行的每一个step，每一个step放到ExecInterpExpr中按顺序执行得到最终结果tuple。

2.1 表达式执行上下文结构ExprContext

其中ecxt_scantuple用来保存待处理输入tuple，通常指向一个 tuple table slot。

ExprContext *econtext = {
  type = T_ExprContext,
  ecxt_scantuple = 0x1520918,    <<<<<<<< 待处理tuple
  ecxt_innertuple = 0x0,
  ecxt_outertuple = 0x0,
  ecxt_per_query_memory = 0x15203e0,
  ecxt_per_tuple_memory = 0x151e3d0,
  ecxt_param_exec_vals = 0x0,
  ecxt_param_list_info = 0x0,
  ecxt_aggvalues = 0x0,
  ecxt_aggnulls = 0x0,
  caseValue_datum = 0,
  caseValue_isNull = true,
  domainValue_datum = 0,
  domainValue_isNull = true,
  ecxt_estate = 0x15204e0,
  ecxt_callbacks = 0x0
}

2.2 表达式执行状态ExprState

ExprState  *state = {
  type = T_ExprState,
  flags = 6 '\006',
  resnull = false,
  resvalue = 0,
  resultslot = 0x151f278,
  steps = 0x1525c38,
  evalfunc = 0x74d749 <ExecInterpExprStillValid>,
  expr = 0x24c9618,
  evalfunc_private = 0x74af83 <ExecInterpExpr>,
  steps_len = 37,
  steps_alloc = 64,
  parent = 0x151e718,
  ext_params = 0x0,
  innermost_caseval = 0x0,
  innermost_casenull = 0x0,
  innermost_domainval = 0x0,
  innermost_domainnull = 0x0,
  escontext = 0x0
}

注意实际执行使用ExecInterpExpr函数完成表达式计算：

2.3 执行表达式

执行时，通过state->evalfunc函数完成具体的表达式计算动作。具体怎么计算流程放在steps中：

(gdb) p/x state->steps[0]->opcode
$15 = 0x74b0ad
(gdb) p/x state->steps[1]->opcode
$16 = 0x74b1ec
(gdb) p/x state->steps[2]->opcode
$17 = 0x74b784
(gdb) p/x state->steps[3]->opcode
$18 = 0x74b591
(gdb) p/x state->steps[4]->opcode
$19 = 0x74b1ec
(gdb) p/x state->steps[5]->opcode
$20 = 0x74b784
(gdb) p/x state->steps[6]->opcode
$21 = 0x74b591
(gdb) p/x state->steps[7]->opcode
$22 = 0x74b1ec
...
...

opcode需要再reverse_dispatch_table中确定当前计算走哪个ExecInterpExpr中的分支：

（这里不太方便调试因为这里把一个超大的switch case改成了goto，为了性能！）

p reverse_dispatch_table
$13 = {{
    opcode = 0x74b01a <ExecInterpExpr+151>,
    op = EEOP_DONE
  }, {
    opcode = 0x74b041 <ExecInterpExpr+190>,
    op = EEOP_INNER_FETCHSOME
  }, {
    opcode = 0x74b077 <ExecInterpExpr+244>,
    op = EEOP_OUTER_FETCHSOME
  }, {
    opcode = 0x74b0ad <ExecInterpExpr+298>,
    op = EEOP_SCAN_FETCHSOME
  }, {
    opcode = 0x74b0e6 <ExecInterpExpr+355>,
    op = EEOP_INNER_VAR
  }, {
    opcode = 0x74b169 <ExecInterpExpr+486>,
    op = EEOP_OUTER_VAR
  }, {
    opcode = 0x74b1ec <ExecInterpExpr+617>,
    op = EEOP_SCAN_VAR
  }, {
    opcode = 0x74b26f <ExecInterpExpr+748>,
    op = EEOP_INNER_SYSVAR
  }, {
    opcode = 0x74b29e <ExecInterpExpr+795>,
    op = EEOP_OUTER_SYSVAR
  }, {
    opcode = 0x74b2cd <ExecInterpExpr+842>,
    op = EEOP_SCAN_SYSVAR
  }, {
    opcode = 0x74b2fc <ExecInterpExpr+889>,
    op = EEOP_WHOLEROW
  }, {
    opcode = 0x74b32a <ExecInterpExpr+935>,
    op = EEOP_ASSIGN_INNER_VAR
  }, {
    opcode = 0x74b3f7 <ExecInterpExpr+1140>,
    op = EEOP_ASSIGN_OUTER_VAR
  }, {
    opcode = 0x74b4c4 <ExecInterpExpr+1345>,
    op = EEOP_ASSIGN_SCAN_VAR
  }, {
    opcode = 0x74b591 <ExecInterpExpr+1550>,
    op = EEOP_ASSIGN_TMP
  }, {
    opcode = 0x74b615 <ExecInterpExpr+1682>,
    op = EEOP_ASSIGN_TMP_MAKE_RO
  }, {
    opcode = 0x74b6e1 <ExecInterpExpr+1886>,
    op = EEOP_CONST
  }, {
    opcode = 0x74b717 <ExecInterpExpr+1940>,
    op = EEOP_FUNCEXPR
  }, {
    opcode = 0x74b784 <ExecInterpExpr+2049>,
    op = EEOP_FUNCEXPR_STRICT
  }, {
    opcode = 0x74b853 <ExecInterpExpr+2256>,
    op = EEOP_FUNCEXPR_FUSAGE
  }, {
    opcode = 0x74b881 <ExecInterpExpr+2302>,
    op = EEOP_FUNCEXPR_STRICT_FUSAGE
  }, {
    opcode = 0x74b8af <ExecInterpExpr+2348>,
    op = EEOP_BOOL_AND_STEP_FIRST
  }, {
    opcode = 0x74b8ba <ExecInterpExpr+2359>,
    op = EEOP_BOOL_AND_STEP
  }, {
    opcode = 0x74b92c <ExecInterpExpr+2473>,
    op = EEOP_BOOL_AND_STEP_LAST
  }, {
    opcode = 0x74b98f <ExecInterpExpr+2572>,
    op = EEOP_BOOL_OR_STEP_FIRST
  }, {
    opcode = 0x74b99a <ExecInterpExpr+2583>,
    op = EEOP_BOOL_OR_STEP
  }, {
    opcode = 0x74ba09 <ExecInterpExpr+2694>,
    op = EEOP_BOOL_OR_STEP_LAST
  }, {
    opcode = 0x74ba69 <ExecInterpExpr+2790>,
    op = EEOP_BOOL_NOT_STEP
  }, {
    opcode = 0x74bab3 <ExecInterpExpr+2864>,
    op = EEOP_QUAL
  }, {
    opcode = 0x74bb38 <ExecInterpExpr+2997>,
    op = EEOP_JUMP
  }, {
    opcode = 0x74bb63 <ExecInterpExpr+3040>,
    op = EEOP_JUMP_IF_NULL
  }, {
    opcode = 0x74bbae <ExecInterpExpr+3115>,
    op = EEOP_JUMP_IF_NOT_NULL
  }, {
    opcode = 0x74bbfc <ExecInterpExpr+3193>,
    op = EEOP_JUMP_IF_NOT_TRUE
  }, {
    opcode = 0x74bc61 <ExecInterpExpr+3294>,
    op = EEOP_NULLTEST_ISNULL
  }, {
    opcode = 0x74bc9d <ExecInterpExpr+3354>,
    op = EEOP_NULLTEST_ISNOTNULL
  }, {
    opcode = 0x74bcea <ExecInterpExpr+3431>,
    op = EEOP_NULLTEST_ROWISNULL
  }, {
    opcode = 0x74bd18 <ExecInterpExpr+3477>,
    op = EEOP_NULLTEST_ROWISNOTNULL
  }, {
    opcode = 0x74bd46 <ExecInterpExpr+3523>,
    op = EEOP_BOOLTEST_IS_TRUE
  }, {
    opcode = 0x74bd86 <ExecInterpExpr+3587>,
    op = EEOP_BOOLTEST_IS_NOT_TRUE
  }, {
    opcode = 0x74be01 <ExecInterpExpr+3710>,
    op = EEOP_BOOLTEST_IS_FALSE
  }, {
    opcode = 0x74be7c <ExecInterpExpr+3833>,
    op = EEOP_BOOLTEST_IS_NOT_FALSE
  }, {
    opcode = 0x74bebc <ExecInterpExpr+3897>,
    op = EEOP_PARAM_EXEC
  }, {
    opcode = 0x74beea <ExecInterpExpr+3943>,
    op = EEOP_PARAM_EXTERN
  }, {
    opcode = 0x74bf18 <ExecInterpExpr+3989>,
    op = EEOP_PARAM_CALLBACK
  }, {
    opcode = 0x74bf48 <ExecInterpExpr+4037>,
    op = EEOP_CASE_TESTVAL
  }, {
    opcode = 0x74bfbe <ExecInterpExpr+4155>,
    op = EEOP_DOMAIN_TESTVAL
  }, {
    opcode = 0x74c034 <ExecInterpExpr+4273>,
    op = EEOP_MAKE_READONLY
  }, {
    opcode = 0x74c08a <ExecInterpExpr+4359>,
    op = EEOP_IOCOERCE
  }, {
    opcode = 0x74c26c <ExecInterpExpr+4841>,
    op = EEOP_IOCOERCE_SAFE
  }, {
    opcode = 0x74c293 <ExecInterpExpr+4880>,
    op = EEOP_DISTINCT
  }, {
    opcode = 0x74c3a6 <ExecInterpExpr+5155>,
    op = EEOP_NOT_DISTINCT
  }, {
    opcode = 0x74c496 <ExecInterpExpr+5395>,
    op = EEOP_NULLIF
  }, {
    opcode = 0x74c57f <ExecInterpExpr+5628>,
    op = EEOP_SQLVALUEFUNCTION
  }, {
    opcode = 0x74c5a6 <ExecInterpExpr+5667>,
    op = EEOP_CURRENTOFEXPR
  }, {
    opcode = 0x74c5cd <ExecInterpExpr+5706>,
    op = EEOP_NEXTVALUEEXPR
  }, {
    opcode = 0x74c5f4 <ExecInterpExpr+5745>,
    op = EEOP_ARRAYEXPR
  }, {
    opcode = 0x74c61b <ExecInterpExpr+5784>,
    op = EEOP_ARRAYCOERCE
  }, {
    opcode = 0x74c649 <ExecInterpExpr+5830>,
    op = EEOP_ROW
  }, {
    opcode = 0x74c670 <ExecInterpExpr+5869>,
    op = EEOP_ROWCOMPARE_STEP
  }, {
    opcode = 0x74c7be <ExecInterpExpr+6203>,
    op = EEOP_ROWCOMPARE_FINAL
  }, {
    opcode = 0x74c8cf <ExecInterpExpr+6476>,
    op = EEOP_MINMAX
  }, {
    opcode = 0x74c8f6 <ExecInterpExpr+6515>,
    op = EEOP_FIELDSELECT
  }, {
    opcode = 0x74c924 <ExecInterpExpr+6561>,
    op = EEOP_FIELDSTORE_DEFORM
  }, {
    opcode = 0x74c952 <ExecInterpExpr+6607>,
    op = EEOP_FIELDSTORE_FORM
  }, {
    opcode = 0x74c980 <ExecInterpExpr+6653>,
    op = EEOP_SBSREF_SUBSCRIPTS
  }, {
    opcode = 0x74c9df <ExecInterpExpr+6748>,
    op = EEOP_SBSREF_OLD
  }, {
    opcode = 0x74c9e1 <ExecInterpExpr+6750>,
    op = EEOP_SBSREF_ASSIGN
  }, {
    opcode = 0x74c9e1 <ExecInterpExpr+6750>,
    op = EEOP_SBSREF_FETCH
  }, {
    opcode = 0x74ca11 <ExecInterpExpr+6798>,
    op = EEOP_CONVERT_ROWTYPE
  }, {
    opcode = 0x74ca3f <ExecInterpExpr+6844>,
    op = EEOP_SCALARARRAYOP
  }, {
    opcode = 0x74ca66 <ExecInterpExpr+6883>,
    op = EEOP_HASHED_SCALARARRAYOP
  }, {
    opcode = 0x74ca94 <ExecInterpExpr+6929>,
    op = EEOP_DOMAIN_NOTNULL
  }, {
    opcode = 0x74cabb <ExecInterpExpr+6968>,
    op = EEOP_DOMAIN_CHECK
  }, {
    opcode = 0x74cae2 <ExecInterpExpr+7007>,
    op = EEOP_XMLEXPR
  }, {
    opcode = 0x74cb09 <ExecInterpExpr+7046>,
    op = EEOP_JSON_CONSTRUCTOR
  }, {
    opcode = 0x74cb37 <ExecInterpExpr+7092>,
    op = EEOP_IS_JSON
  }, {
    opcode = 0x74cb5e <ExecInterpExpr+7131>,
    op = EEOP_JSONEXPR_PATH
  }, {
    opcode = 0x74cb9f <ExecInterpExpr+7196>,
    op = EEOP_JSONEXPR_COERCION
  }, {
    opcode = 0x74cbcd <ExecInterpExpr+7242>,
    op = EEOP_JSONEXPR_COERCION_FINISH
  }, {
    opcode = 0x74cbf4 <ExecInterpExpr+7281>,
    op = EEOP_AGGREF
  }, {
    opcode = 0x74cc82 <ExecInterpExpr+7423>,
    op = EEOP_GROUPING_FUNC
  }, {
    opcode = 0x74cca9 <ExecInterpExpr+7462>,
    op = EEOP_WINDOW_FUNC
  }, {
    opcode = 0x74cd40 <ExecInterpExpr+7613>,
    op = EEOP_MERGE_SUPPORT_FUNC
  }, {
    opcode = 0x74cd6e <ExecInterpExpr+7659>,
    op = EEOP_SUBPLAN
  }, {
    opcode = 0x74cd9c <ExecInterpExpr+7705>,
    op = EEOP_AGG_STRICT_DESERIALIZE
  }, {
    opcode = 0x74cdd7 <ExecInterpExpr+7764>,
    op = EEOP_AGG_DESERIALIZE
  }, {
    opcode = 0x74ce8a <ExecInterpExpr+7943>,
    op = EEOP_AGG_STRICT_INPUT_CHECK_ARGS
  }, {
    opcode = 0x74cf18 <ExecInterpExpr+8085>,
    op = EEOP_AGG_STRICT_INPUT_CHECK_NULLS
  }, {
    opcode = 0x74cf9f <ExecInterpExpr+8220>,
    op = EEOP_AGG_PLAIN_PERGROUP_NULLCHECK
  }, {
    opcode = 0x74d02c <ExecInterpExpr+8361>,
    op = EEOP_AGG_PLAIN_TRANS_INIT_STRICT_BYVAL
  }, {
    opcode = 0x74d147 <ExecInterpExpr+8644>,
    op = EEOP_AGG_PLAIN_TRANS_STRICT_BYVAL
  }, {
    opcode = 0x74d22c <ExecInterpExpr+8873>,
    op = EEOP_AGG_PLAIN_TRANS_BYVAL
  }, {
    opcode = 0x74d2fb <ExecInterpExpr+9080>,
    op = EEOP_AGG_PLAIN_TRANS_INIT_STRICT_BYREF
  }, {
    opcode = 0x74d413 <ExecInterpExpr+9360>,
    op = EEOP_AGG_PLAIN_TRANS_STRICT_BYREF
  }, {
    opcode = 0x74d4f5 <ExecInterpExpr+9586>,
    op = EEOP_AGG_PLAIN_TRANS_BYREF
  }, {
    opcode = 0x74d5c1 <ExecInterpExpr+9790>,
    op = EEOP_AGG_PRESORTED_DISTINCT_SINGLE
  }, {
    opcode = 0x74d648 <ExecInterpExpr+9925>,
    op = EEOP_AGG_PRESORTED_DISTINCT_MULTI
  }, {
    opcode = 0x74d6cf <ExecInterpExpr+10060>,
    op = EEOP_AGG_ORDERED_TRANS_DATUM
  }, {
    opcode = 0x74d6fd <ExecInterpExpr+10106>,
    op = EEOP_AGG_ORDERED_TRANS_TUPLE
  }}

翻译后

(gdb) p/x state->steps[0]->opcode
$15 = 0x74b0ad                       EEOP_SCAN_FETCHSOME
(gdb) p/x state->steps[1]->opcode
$16 = 0x74b1ec                       EEOP_SCAN_VAR
(gdb) p/x state->steps[2]->opcode
$17 = 0x74b784                       EEOP_FUNCEXPR_STRICT
(gdb) p/x state->steps[3]->opcode
$18 = 0x74b591                       EEOP_ASSIGN_TMP
(gdb) p/x state->steps[4]->opcode
$19 = 0x74b1ec                       EEOP_SCAN_VAR
(gdb) p/x state->steps[5]->opcode
$20 = 0x74b784                       EEOP_FUNCEXPR_STRICT
(gdb) p/x state->steps[6]->opcode
$21 = 0x74b591                       EEOP_ASSIGN_TMP
(gdb) p/x state->steps[7]->opcode
$22 = 0x74b1ec                       EEOP_SCAN_VAR
...
...
(gdb) p/x state->steps[34]->opcode
$27 = 0x74b784                       EEOP_FUNCEXPR_STRICT
(gdb) p/x state->steps[35]->opcode
$28 = 0x74b591                       EEOP_ASSIGN_TMP
(gdb) p/x state->steps[36]->opcode
$29 = 0x74b01a                       EEOP_DONE

可以看到表达式计算的流程：

第一步：EEOP_SCAN_FETCHSOME
  1. 从econtext->ecxt_scantuple读取到scanslot（当前要处理的一行数据）
  2. slot_getsomeattrs函数确保这一行数据中，至少有op->d.fetch.last_var个列是可以直接访问的（这里是3，因为t1表就三列，后面的处理可能需要访问这三列的任意一列）。为什么说有时不能直接访问，因为列有可能指向toast表。
		EEO_CASE(EEOP_SCAN_FETCHSOME)
		{
			CheckOpSlotCompatibility(op, scanslot);

			slot_getsomeattrs(scanslot, op->d.fetch.last_var);

			EEO_NEXT();
		}
第二步：EEOP_SCAN_VAR

输入

p state->steps[1].d.fetch
$46 = {
  last_var = 2,
  fixed = 23,
  known_desc = 0x0,
  kind = 0x0
}

执行，从行中拿到第2列的值（0列、1列、2列）

		EEO_CASE(EEOP_SCAN_VAR)
		{
			int			attnum = op->d.var.attnum;   // attnum = 2

			/* See EEOP_INNER_VAR comments */

			Assert(attnum >= 0 && attnum < scanslot->tts_nvalid);
			*op->resvalue = scanslot->tts_values[attnum];
			*op->resnull = scanslot->tts_isnull[attnum];

			EEO_NEXT();
		}

结果

(gdb) p state->steps[1].resvalue
$41 = (Datum *) 0x151f918
(gdb) p *state->steps[1].resvalue
$39 = 1

(gdb) p state->steps[1].resnull
$42 = (_Bool *) 0x151f920
(gdb) p *state->steps[1].resnull
$40 = false
第三步：EEOP_FUNCEXPR_STRICT

输入

(gdb) p state->steps[2].d.func
$47 = {
  finfo = 0x151f8a8,
  fcinfo_data = 0x151f8f8,
  fn_addr = 0xa8ea89 <int4abs>,
  nargs = 1
}

执行

		EEO_CASE(EEOP_FUNCEXPR_STRICT)
		{
			FunctionCallInfo fcinfo = op->d.func.fcinfo_data;
			NullableDatum *args = fcinfo->args;
			int			nargs = op->d.func.nargs;
			Datum		d;

			/* strict function, so check for NULL args */
			for (int argno = 0; argno < nargs; argno++)
			{
				if (args[argno].isnull)
				{
					*op->resnull = true;
					goto strictfail;
				}
			}
			fcinfo->isnull = false;
			d = op->d.func.fn_addr(fcinfo);
			*op->resvalue = d;
			*op->resnull = fcinfo->isnull;

	strictfail:
			EEO_NEXT();
		}

注意这里有一个地方很有意思，按理说EEOP_SCAN_VAR执行完才把值拿到，但从EEOP_FUNCEXPR_STRICT的执行来看，并没有发现把前一步的结果，放到函数args的步骤。

但是从GDB来看，函数的入参的地址和上一步取值后存放结果的地址是相同的，也就是上一步取值就是为了拿入参的args：

(gdb) p state->steps[4].resvalue
$59 = (Datum *) 0x151f9b8

(gdb) p state->steps[5].d.func->fcinfo_data->args
$58 = 0x151f9b8

具体是怎么做到的呢？在构造steps时：
Postgresql源码（127）投影ExecProject的表达式执行分析-LMLPHP
对于函数入参value会调用ExecInitExprRec去取值，在这个过程中，把参数的value指向新step的resvalue：
Postgresql源码（127）投影ExecProject的表达式执行分析-LMLPHP
而这个新的step就是EEOP_SCAN_VAR：
Postgresql源码（127）投影ExecProject的表达式执行分析-LMLPHP

第四步：暂存结果集

暂存结果集到resultslot中：

		EEO_CASE(EEOP_ASSIGN_TMP)
		{
			int			resultnum = op->d.assign_tmp.resultnum;

			Assert(resultnum >= 0 && resultnum < resultslot->tts_tupleDescriptor->natts);
			resultslot->tts_values[resultnum] = state->resvalue;
			resultslot->tts_isnull[resultnum] = state->resnull;

			EEO_NEXT();
		}
第五步：继续上述流程直到执行完成
(gdb) p/x state->steps[0]->opcode
$15 = 0x74b0ad                       EEOP_SCAN_FETCHSOME   第一步
(gdb) p/x state->steps[1]->opcode
$16 = 0x74b1ec                       EEOP_SCAN_VAR         第二步
(gdb) p/x state->steps[2]->opcode
$17 = 0x74b784                       EEOP_FUNCEXPR_STRICT  第三步
(gdb) p/x state->steps[3]->opcode
$18 = 0x74b591                       EEOP_ASSIGN_TMP       第四步
(gdb) p/x state->steps[4]->opcode
$19 = 0x74b1ec                       EEOP_SCAN_VAR         第五步 和上述流程相同，每个函数的执行流程都是相似的
(gdb) p/x state->steps[5]->opcode
$20 = 0x74b784                       EEOP_FUNCEXPR_STRICT
(gdb) p/x state->steps[6]->opcode
$21 = 0x74b591                       EEOP_ASSIGN_TMP
(gdb) p/x state->steps[7]->opcode
$22 = 0x74b1ec                       EEOP_SCAN_VAR
...
...
(gdb) p/x state->steps[34]->opcode
$27 = 0x74b784                       EEOP_FUNCEXPR_STRICT
(gdb) p/x state->steps[35]->opcode
$28 = 0x74b591                       EEOP_ASSIGN_TMP
(gdb) p/x state->steps[36]->opcode
$29 = 0x74b01a                       EEOP_DONE
05-01 07:28
Powered by LMLPHP ©2024 高铭杰 0.002321