如何在不向每个规则添加 EOF 的情况下测试 ANTLR 翻译

问题描述

我正在重新编写我的翻译器，这次我对测试更加自律，因为这个版本可能会存在几个星期以上.

I am in the middle of re-writing my translator and I am being much more disciplined about tests this time, since this version is likely to live for more than a few weeks.

因为您可以从任何节点开始运行访问者，所以您几乎可以像这样编写漂亮的小测试......

Because you can run a visitor starting at any node, you can almost write beautiful small tests like this ...

expect(parse("some test code", "startGrammarRule")).toEqual(new ASTForGrammarRule())

然后为每个访问者函数编写一个(或其中几个)

and then write one ( or a few of these ) for each visitor function

除非您调用的规则是子规则，因此没有EOF"；在里面，所以如果我的语法有什么地方

EXCEPT that the rule you are invoking is a sub rule, and so does not have "EOF" in it, so if my grammar has somewhere in it

numberList: NUMBER ( ',' NUMBER )* ;

... 然后 parse("1,2,3", "numberList") 只解析1"；(因为它只是一个EOF"，它会使解析器饿到足以消耗所有字符串).

... then parse("1,2,3", "numberList") only parses "1" (because it is only an "EOF" which would make the parser hungry enough to consume all the string).

编辑规则以添加 EOF 是一个非启动项.我可以为每个我编写测试的规则添加一个规则的测试版本......

Editing the rule to add EOF is a non starter. I could, for every rule I write a test for, add a test version of the rule ...

numberList: NUMBER ( ',' NUMBER )* ;
numberList_TEST: numberList EOF ;

...但这会使语法变得混乱，并担心必须始终严格维护 _TEST 规则...

... but that is going to make the grammar cluttered and introduce worry that the _TEST rules have to always be maintained scrupulously ...

当我创建一个解析器时，我想要一个标志，它动态地构造那个虚假的 TEST 规则，然后从那里解析，或者类似的东西......

I want a flag when I create a parser which constructs that faux TEST rule dynamically and then parses from there, or something like that ...

是否有更好的方法来为我的解析器编写测试，但我还没有想出?

Is there a better way to write tests for my parser that I haven't figured out yet?

推荐答案

在 Java 项目中，我使用自定义匹配器来检查解析的令牌是否为 100% 的令牌流，如果不是，将失败.

In a Java project, I'm using a custom matcher to check if the parsed tokens are 100% of the tokenstream, and if not, will fail.

您似乎使用了 TypeScript 目标，因此在 TypeScript 中可能如下所示:

You seem to use the TypeScript target, so in TypeScript that could look like this:

grammar T;

parse      : numberList EOF;
numberList : NUMBER ( ',' NUMBER )*;

NUMBER : [0-9]+;
ID     : [a-zA-Z]+;
WS     : [ \t\r\n]+ -> channel(HIDDEN);

parserMatchers.ts

import { TLexer } from '../src/parser/TLexer';
import { BailErrorStrategy, CharStreams, CommonTokenStream } from 'antlr4ts';
import { TParser } from '../src/parser/TParser';
import { Lexer } from 'antlr4ts/Lexer';

expect.extend({
  toBeCompletelyParsedBy: (source: string, ruleName: string) => {
    const lexer = new TLexer(CharStreams.fromString(source));
    lexer.removeErrorListeners();
    const tokenStream = new CommonTokenStream(lexer);
    const parser = new TParser(tokenStream);
    parser.removeErrorListeners();
    parser.errorHandler = new BailErrorStrategy();
    const context = parser[ruleName]();

    // Collect the real tokens: non-HIDDEN and non-EOF tokens
    const realTokens = tokenStream.getTokens().filter((t) =>
      t.channel === Lexer.DEFAULT_TOKEN_CHANNEL && t.type !== Lexer.EOF);

    let indexOfStop = realTokens.indexOf(context.stop);
    let pass = realTokens.length === (indexOfStop + 1);

    let message = () => {

      if (pass) {
        return `Expected '${source}' not to be completely parsed by rule '${ruleName}', but it did.`;
      }

      let offending = realTokens[indexOfStop + 1];

      return `Expected '${source}' to be completely parsed by rule '${ruleName}', but '${offending.text}' ` +
        `(${offending.line}:${offending.charPositionInLine}) was not included!`;
    };

    return { pass, message };
  }
});

declare global {
  namespace jest {
    interface Matchers<R> {
      toBeCompletelyParsedBy(ruleName: string): R
    }
  }
}

export {};

在你的单元测试中，你现在可以这样做:

And in you unit tests, you can now do this:

import './parserMatchers';

test('the numberList parser rule', () => {
  expect('3, 4, 5').toBeCompletelyParsedBy('numberList');
  expect('3, 4, 5 FOO').not.toBeCompletelyParsedBy('numberList');
});

这篇关于如何在不向每个规则添加 EOF 的情况下测试 ANTLR 翻译的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！