问题描述
我正在编写一个应用程序，可以通过摄像头输入对视频进行编码，并通过解码-编辑-编码步骤处理视频.对于相机，我使用 Camera 类而不是 Intent 来配置相机的详细信息设置.然后我将相机帧提供给编码器(API 16 中的 MediaCodec)和多路复用器(我使用 ffmpeg 多路复用器，因为我想在 4.1 设备上工作).
I'm writing an APP that can encode video by camera input and process video by decode-edit-encode steps. For the camera, I use the Camera class rather than Intent to configure the details settings of the camera. Then I feed the camera frames to the encoder (MediaCodec in API 16) and the muxer (I use ffmpeg muxer since I want to work on 4.1 Devices).
我通过系统纳米时间测量相机帧的时间码，并选择帧子集以适合所需的 FPS(当前为 15).时间值中有一些小的噪音"，例如(以毫秒为单位):0, 60718, 135246, 201049, ... 而不是 0, 66000, 133000, 200000, ... .
I measure the time code of camera frames by system nano time, and select a subset of frames to fit a desired FPS (currently 15). There are some small "noises" in the time values, for example (in ms): 0, 60718, 135246, 201049, ... rather than 0, 66000, 133000, 200000, ... .
在尝试正确配置多路复用器后(如 这个问题)，我可以制作一个视频(使用 AVC 编解码器)，可以由设备上的视频播放器播放.播放速度正确，所以我认为视频应该有正确的帧时间信息.
After some trying of configuring the muxer correctly (As this question), I can produce a video (with AVC codec) that can be playback by the video player on devices. The playback speed is correct so I think the video should have correct time information of the frames.
但是，当我尝试解码视频以执行视频编辑过程时遇到了问题.我使用标准视频提取/解码步骤作为这些示例，如下所示:
However, I got a problem when I try to decode the video to perform the video editting process. I use the standard video extract/decode steps as these samples, like this:
int decode_input_index = decoder.dequeueInputBuffer(TIMEOUT_USEC);
if (decode_input_index >= 0)
{
ByteBuffer decoder_input_buffer = decode_input_buffers[decode_input_index];
int sample_size = extractor.readSampleData(decoder_input_buffer, 0);
if (sample_size < 0)
{
decoder.queueInputBuffer(decode_input_index, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
is_decode_input_done = true;
}
else
{
long sample_time = extractor.getSampleTime();
decoder.queueInputBuffer(decode_input_index, 0, sample_size, sample_time, 0);
extractor.advance();
}
}
else
{
Log.v(TAG, "Decoder dequeueInputBuffer timed out! Try again later");
}
当我对视频进行编码时，来自 getSampleTime() 的采样时间具有正确的值.(例如，在我们中，它们恰好是 0、60718、135246、201049……).也是decoder.queueInputBuffer()的输入中的呈现时间.当解码器继续解码此帧时，我通过以下方式获得帧时间:
The sample time from getSampleTime() has correct value as I encode the video. (e.g., they are exactly 0, 60718, 135246, 201049, ... in us). It is also the presentation time in the input of decoder.queueInputBuffer(). When the decoder proceeds to decode this frame, I get the frame time by:
int decode_output_index = decoder.dequeueOutputBuffer(decode_buffer_info, TIMEOUT_USEC);
switch (decode_output_index)
{
....
(some negative-value flags in MediaCodec)
....
default:
{
ByteBuffer decode_output_buffer = decode_output_buffers[decode_output_index];
long ptime_us = decode_buffer_info.presentationTimeUs;
boolean is_decode_EOS = ((decode_buffer_info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0);
....
}
}
我希望设置与解码器输入相同的时间序列，但我从解码器输出的 BufferInfo 中得到很多 0.解码后的帧内容似乎是正确的，但大部分呈现时间值为0.只有最后几帧的呈现时间正确.
I expect to set the same time sequence as the one in decoder input, but I get a lot of 0's from the BufferInfo at decoder output. The decoded frame contents seems to be correct, but most of the presentation time values are 0. Only the last few frames has correct presentation time.
我在装有 Android 4.3 的设备上测试了整个相同的过程(即使使用相同的 ffmpeg muxer 而不是 API 18 中的 MediaMuxer)，一切看起来都很好.在 4.1/4.2 设备上，如果我通过设备上的内置摄像头 APP 捕获视频然后解码视频，那么呈现时间也是正确的，尽管时间值也会因摄像头延迟而产生噪音.
I test the whole same process on a device with Android 4.3 (even with the same ffmpeg muxer rather than MediaMuxer in API 18), and everything looks fine. On 4.1/4.2 devices, if I capture the video by the built-in camera APP on the device and then decode the video, then the presentation time is also correct although the time values also have noises due to camera delay.
视频或解码过程有什么问题，视频可以正常播放和解码，但采样时间正确，呈现时间错误?我可能不得不使用一种解决方法来通过采样时间来衡量演示时间(使用队列很容易)，但我想弄清楚我的工作中是否有任何遗漏的部分.
What's wrong with the video or the decode process, when the video can be playbacked and decoded normally, but with correct sample time and bad presentation time? I may have to use a workaround to measure the presentation time by the sample time (It's easy by using a queue), but I want to figure out if there is any missing part in my work.
推荐答案
不能保证 MediaCodec 在 Android 4.3 之前正确处理演示时间戳.这是因为确认 PTS 行为的 CTS 测试直到那时才被添加.
There is no guarantee that MediaCodec handles presentation time stamps correctly before Android 4.3. This is because the CTS tests that confirm the PTS behavior were not added until then.
我确实记得某些供应商的 AVC 编解码器中的时间戳处理存在问题.我不记得细节了，但是如果你从 EncodeDecodeTest 在各种 4.1/4.2 设备上，您会出现一些故障.(当然，您需要去掉表面对表面的测试.)
I do recall that there were problems with the timestamp handling in the AVC codecs from certain vendors. I don't remember the details offhand, but if you run the buffer-to-buffer and buffer-to-surface tests from EncodeDecodeTest on a variety of 4.1/4.2 devices you'll turn up some failures. (You'd need to strip out the surface-to-surface tests, of course.)
您的时间戳处理代码看起来不错.时间戳不是 H.264 流的一部分，因此它实际上只是作为元数据通过编解码器转发，您似乎在所有正确的位置拾取并转发它.最重要的是，如果您传入有效的 PTS 值并获得良好的视频，但 PTS 值是垃圾，则编解码器中的某些内容处理不当.
Your timestamp handling code looks fine. The timestamp isn't part of the H.264 stream, so it's really just being forwarded through the codec as metadata, and you seem to be picking it up and forwarding it on in all the right places. The bottom line is, if you're passing valid PTS values in and getting good video but garbage PTS values, something in the codec is mishandling them.
您需要通过单独传递值来解决这个问题，或者——如果输入帧速率总是规则的——简单地计算它.理论上，编码器可以对帧重新排序，因此您将时间戳传递给编码器的顺序可能与它们出现的顺序不同……但是由于您知道制作电影时时间戳是递增的，因此您应该如果这在实践中存在问题，可以对它们进行排序.
You'll need to work around it by passing the values separately, or -- if the input frame rate is always regular -- trivially computing it. In theory the encoder can reorder frames, so the order in which you pass time stamps into the encoder might not be the same order in which they come out... but since you know the timestamps were ascending when you made the movie, you should be able to just sort them if this were a problem in practice.
另一方面，如果您在帧到达应用程序时抓取 System.nanoTime()，系统中的延迟将导致您在时间戳值中看到的摆动".您可以在带有 Surface 输入的 Android 4.3 中做得更好，因为 SurfaceTexture 保存的时间戳设置得更接近捕获帧的时间.(我知道这对你目前的努力没有用，但想给未来带来一些希望.)
On a different note, delays in the system will cause the "wobble" you're seeing in the timestamp values if you grab System.nanoTime() when the frame arrives in the app. You can do a bit better in Android 4.3 with Surface input because the SurfaceTexture holds a timestamp that is set much closer to when the frame was captured. (I know that's not useful for your current efforts, but wanted to give some hope for the future.)
这篇关于视频解码期间的采样时间/演示时间不一致的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！