我不能得到令牌的字符串值

问题描述

我试图实现一个词法分析器与升压精神有点编程语言。

I try to implement a Lexer for a little programming language with Boost Spirit.

我要得到令牌的价值，我得到一个bad_get异常：

I have to get the value of a token and I get a bad_get exception :

抛'的boost :: bad_get结果一个实例后终止叫
  什么（）的boost :: bad_get：失败值获得使用boost ::得到中止

我在做的时候得到这个异​​常：

I obtain this exception when doing :

std::string contents = "void";

base_iterator_type first = contents.begin();
base_iterator_type last = contents.end();

SimpleLexer<lexer_type> lexer;

iter = lexer.begin(first, last);
end = lexer.end();

std::cout << "Value = " << boost::get<std::string>(iter->value()) << std::endl;

我的词法分析器定义这样的：

My lexer is defined like that :

typedef std::string::iterator base_iterator_type;
typedef boost::spirit::lex::lexertl::token<base_iterator_type, boost::mpl::vector<unsigned int, std::string>> Tok;
typedef lex::lexertl::actor_lexer<Tok> lexer_type;

template<typename L>
class SimpleLexer : public lex::lexer<L> {
    private:

    public:
        SimpleLexer() {
            keyword_for = "for";
            keyword_while = "while";
            keyword_if = "if";
            keyword_else = "else";
            keyword_false = "false";
            keyword_true = "true";
            keyword_from = "from";
            keyword_to = "to";
            keyword_foreach = "foreach";

            word = "[a-zA-Z]+";
            integer = "[0-9]+";
            litteral = "...";

            left_parenth = '(';
            right_parenth = ')';
            left_brace = '{';
            right_brace = '}';

            stop = ';';
            comma = ',';

            swap = "<>";
            assign = '=';
            addition = '+';
            subtraction = '-';
            multiplication = '*';
            division = '/';
            modulo = '%';

            equals = "==";
            not_equals = "!=";
            greater = '>';
            less = '<';
            greater_equals = ">=";
            less_equals = "<=";

            whitespaces = "[ \\t\\n]+";
            comments = "\\/\\*[^*]*\\*+([^/*][^*]*\\*+)*\\/";

            //Add keywords
            this->self += keyword_for | keyword_while | keyword_true | keyword_false | keyword_if | keyword_else | keyword_from | keyword_to | keyword_foreach;
            this->self += integer | litteral | word;

            this->self += equals | not_equals | greater_equals | less_equals | greater | less ;
            this->self += left_parenth | right_parenth | left_brace | right_brace;
            this->self += comma | stop;
            this->self += assign | swap | addition | subtraction | multiplication | division | modulo;

            //Ignore whitespaces and comments
            this->self += whitespaces [lex::_pass = lex::pass_flags::pass_ignore];
            this->self += comments [lex::_pass = lex::pass_flags::pass_ignore];
        }

        lex::token_def<std::string> word, litteral, integer;

        lex::token_def<lex::omit> left_parenth, right_parenth, left_brace, right_brace;

        lex::token_def<lex::omit> stop, comma;

        lex::token_def<lex::omit> assign, swap, addition, subtraction, multiplication, division, modulo;
        lex::token_def<lex::omit> equals, not_equals, greater, less, greater_equals, less_equals;

        //Keywords
        lex::token_def<lex::omit> keyword_if, keyword_else, keyword_for, keyword_while, keyword_from, keyword_to, keyword_foreach;
        lex::token_def<lex::omit> keyword_true, keyword_false;

        //Ignored tokens
        lex::token_def<lex::omit> whitespaces;
        lex::token_def<lex::omit> comments;
};

是否有其他方式来获得令牌的价值？

Is there an other way to get the value of a Token ?

推荐答案

您可以随时使用默认标记的数据（这是iterator_range的源迭代器类型）。

You can always use the 'default' token data (which is iterator_range of the source iterator type).

std::string tokenvalue(iter->value().begin(), iter->value().end());

在升压库研究测试用例后，我发现了一些东西：

After studying the test cases in the boost repository, I found out a number of things:

• 这是由设计


• 还有一个更简单的方法


• 中的更容易的赐予Lex中语义动作（例如，使用_1）并使用齐词法分析器令牌时自动化;分配将自动转换为齐属性类型


• 这有（确实）得到了'懒，一次性的，评价在文档中提到的语义

• this is by design
• there is an easier way
• the easier way comes automated in Lex semantic actions (e.g. using _1) and when using the lexer token in Qi; the assignment will automatically convert to the Qi attribute type
• this has (indeed) got the 'lazy, one-time, evaluation' semantics mentioned in the docs

的系带是该令牌数据是变体，其中开出作为原料输入迭代范围。只有经过'的在的'强制分配，转换后的属性在变型缓存。您可以亲眼目睹过渡：

The cinch is that the token data is variant, which starts out as the raw input iterator range. Only after 'a' forced assignment, the converted attribute is cached in the variant. You can witness the transition:

lexer_type::iterator_type iter = lexer.begin(first, last);
lexer_type::iterator_type end = lexer.end();

assert(0 == iter->value().which());
std::cout << "Value = " << boost::get<boost::iterator_range<base_iterator_type> >(iter->value()) << std::endl;

std::string s;
boost::spirit::traits::assign_to(*iter, s);
assert(1 == iter->value().which());
std::cout << "Value = " << s << std::endl;

正如你所看到的，属性分配在这里勉强，直接使用 assign_to 实施特质

全部工作示范：

#include <boost/spirit/include/lex_lexertl.hpp>

#include <iostream>
#include <string>

namespace lex = boost::spirit::lex;

typedef std::string::iterator base_iterator_type;
typedef boost::spirit::lex::lexertl::token<base_iterator_type, boost::mpl::vector<int, std::string>> Tok;
typedef lex::lexertl::actor_lexer<Tok> lexer_type;

template<typename L>
class SimpleLexer : public lex::lexer<L> {
    private:

    public:
        SimpleLexer() {
            word = "[a-zA-Z]+";
            integer = "[0-9]+";
            literal = "...";

            this->self += integer | literal | word;
        }

        lex::token_def<std::string> word, literal;
        lex::token_def<int> integer;
};

int main(int argc, const char* argv[]) {
    SimpleLexer<lexer_type> lexer;

    std::string contents = "void";

    base_iterator_type first = contents.begin();
    base_iterator_type last = contents.end();

    lexer_type::iterator_type iter = lexer.begin(first, last);
    lexer_type::iterator_type end = lexer.end();

    assert(0 == iter->value().which());
    std::cout << "Value = " << boost::get<boost::iterator_range<base_iterator_type> >(iter->value()) << std::endl;

    std::string s;
    boost::spirit::traits::assign_to(*iter, s);
    assert(2 == iter->value().which());
    std::cout << "Value = " << s << std::endl;

    return 0;
}

这篇关于我不能得到令牌的字符串值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！