json - 为简单的json创建Avro模式

我正在尝试为以下json构建avro模式:

{
  "id":1234,
  "my_name_field": "my_name",
  "extra_data": {
      "my_long_value": 1234567890,
      "my_message_string": "Hello World!",
      "my_int_value":  777,
      "some_new_field": 1
  }
}

“id”和“my_name_field”的值是已知的，但“extra_data”中的字段会动态更改，并且是未知的。

我想到的avro模式是:

{
    "name":"my_record",
    "type":"record",
    "fields":[
        {"name":"id", "type":"int", "default":0},
        {"name":"my_name_field", "type":"string", "default":"NoName"},
        { "name":"extra_data", "type":{"type":"map", "values":["null","int","long","string"]}     }
    ]
}

我的第一个想法是使用 map 将'extra_data'记录下来，但这是行不通的:

{ "name":"extra_data", "type":{"type":"map", "values":["null","int","long","string"]} }

我得到:

AvroTypeException: Expected start-union. Got VALUE_NUMBER_INT

apache在https://cwiki.apache.org/confluence/display/Hive/AvroSerDe中给出了一些不错的示例，但似乎没有一个能胜任。

这是我要检查的单元测试:

公共(public)类(class)AvroTest {

@Test
public void readRecord() throws IOException {

    String event="{\"id\":1234,\"my_name_field\":\"my_name\",\"extra_data\":{\"my_long_value\":1234567890,\"my_message_string\":\"Hello World!\",\"my_int_value\":777,\"some_new_field\":1}}";

    SchemaRegistry<Schema> registry = new com.linkedin.camus.schema.MySchemaRegistry();
    DecoderFactory decoderFactory = DecoderFactory.get();

    ObjectMapper mapper = new ObjectMapper();
    GenericDatumReader<GenericData.Record> reader = new GenericDatumReader<GenericData.Record>();
    Schema schema = registry.getLatestSchemaByTopic("record_topic").getSchema();
    reader.setSchema(schema);

    HashMap hashMap = mapper.readValue(event, HashMap.class);
    long now = Long.valueOf(hashMap.get("now").toString())*1000;
    GenericData.Record read = reader.read(null, decoderFactory.jsonDecoder(schema, event));
}

希望能对此有所帮助，
谢谢。

最佳答案

如果额外数据字段的列表确实未知，则使用多个可选值字段可能会有所帮助，如下所示:

{
    "name":"my_record",
    "type":"record",
    "fields":[
        {"name":"id", "type":"int", "default":0},
        {"name":"my_name_field", "type":"string", "default":"NoName"},
        {"name":"extra_data", "type": "array", "items": {
            {"name": "extra_data_entry", "type":"record", "fields": [
                {"name":"extra_data_field_name", "type": "string"},
                {"name":"extra_data_field_type", "type": "string"},
                {"name":"extra_data_field_value_string", "type": ["null", "string"]},
                {"name":"extra_data_field_value_int", "type": ["null", "int"]},
                {"name":"extra_data_field_value_long", "type": ["null", "long"]}
            ]}
        }}
    ]
}

然后，您可以基于该字段的extra_data_field_value_*选择extra_data_field_type值。