如何在USQL UDO中记录某些内容?

问题描述

我有自定义提取器，我正尝试从中提取一些消息.

I have custom extractor, and I'm trying to log some messages from it.

我尝试了类似Console.WriteLine的明显操作，但是找不到输出在哪里.但是，我在adl://<my_DLS>.azuredatalakestore.net/system/jobservice/jobs/Usql/.../<my_job_id>/中找到了一些系统日志.

I've tried obvious things like Console.WriteLine, but cannot find where output is. However, I found some system logs in adl://<my_DLS>.azuredatalakestore.net/system/jobservice/jobs/Usql/.../<my_job_id>/.

如何记录某些内容?是否可以在Data Lake Store或Blob Storage Account上的某处指定日志文件?

How can I log something? Is it possible to specify log file somewhere on Data Lake Store or Blob Storage Account?

推荐答案

U-SQL的最新版本已为UDO添加了诊断日志记录.请参阅发行说明此处.

A recent release of U-SQL has added diagnostic logging for UDOs. See the release notes here.

// Enable the diagnostics preview feature
SET @@FeaturePreviews = "DIAGNOSTICS:ON";


// Extract as one column
@input =
    EXTRACT col string
    FROM "/input/input42.txt"
    USING new Utilities.MyExtractor();


@output =
    SELECT *
    FROM @input;


// Output the file
OUTPUT @output
TO "/output/output.txt"
USING Outputters.Tsv(quoting : false);

这是我来自UDO的诊断行:

This was my diagnostic line from the UDO:

Microsoft.Analytics.Diagnostics.DiagnosticStream.WriteLine(System.String.Format("Concatenations done: {0}", i));

这是整个UDO:

using System.Collections.Generic;
using System.IO;
using System.Text;
using Microsoft.Analytics.Interfaces;

namespace Utilities
{
    [SqlUserDefinedExtractor(AtomicFileProcessing = true)]
    public class MyExtractor : IExtractor
    {
        //Contains the row
        private readonly Encoding _encoding;
        private readonly byte[] _row_delim;
        private readonly char _col_delim;

        public MyExtractor()
        {
            _encoding = Encoding.UTF8;
            _row_delim = _encoding.GetBytes("\n\n");
            _col_delim = '|';
        }

        public override IEnumerable<IRow> Extract(IUnstructuredReader input, IUpdatableRow output)
        {
            string s = string.Empty;
            string x = string.Empty;
            int i = 0;

            foreach (var current in input.Split(_row_delim))
            {
                using (System.IO.StreamReader streamReader = new StreamReader(current, this._encoding))
                {
                    while ((s = streamReader.ReadLine()) != null)
                    {
                        //Strip any line feeds
                        //s = s.Replace("/n", "");

                        // Concatenate the lines
                        x += s;
                        i += 1;

                    }

                    Microsoft.Analytics.Diagnostics.DiagnosticStream.WriteLine(System.String.Format("Concatenations done: {0}", i));

                    //Create the output
                    output.Set<string>(0, x);
                    yield return output.AsReadOnly();

                    // Reset
                    x = string.Empty;

                }
            }
        }
    }
}

这些是我在以下目录中找到的结果:

And these were my results found in the following directory:

/system/jobservice/jobs/Usql/2017/10/20.../diagnosticstreams

这篇关于如何在USQL UDO中记录某些内容?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！