IOSurfaces - 视频中的伪影，无法抓取视频表面

问题描述

这是一个两部分的问题.我有以下代码工作，它抓取当前显示表面并从表​​面创建视频(一切都在后台发生).

This is a 2-part Question. I have the following code working which grabs the current display surface and creates a video out of the surfaces (everything happens in the background).

for(int i=0;i<100;i++){
        IOMobileFramebufferConnection connect;
        kern_return_t result;
        IOSurfaceRef screenSurface = NULL;

        io_service_t framebufferService = IOServiceGetMatchingService(kIOMasterPortDefault, IOServiceMatching("AppleH1CLCD"));
        if(!framebufferService)
            framebufferService = IOServiceGetMatchingService(kIOMasterPortDefault, IOServiceMatching("AppleM2CLCD"));
        if(!framebufferService)
            framebufferService = IOServiceGetMatchingService(kIOMasterPortDefault, IOServiceMatching("AppleCLCD"));

        result = IOMobileFramebufferOpen(framebufferService, mach_task_self(), 0, &connect);

        result = IOMobileFramebufferGetLayerDefaultSurface(connect, 0, &screenSurface);

        uint32_t aseed;
        IOSurfaceLock(screenSurface, kIOSurfaceLockReadOnly, &aseed);
        uint32_t width = IOSurfaceGetWidth(screenSurface);
        uint32_t height = IOSurfaceGetHeight(screenSurface);
        m_width = width;
        m_height = height;
        CFMutableDictionaryRef dict;
        int pitch = width*4, size = width*height*4;
        int bPE=4;
        char pixelFormat[4] = {'A','R','G','B'};
        dict = CFDictionaryCreateMutable(kCFAllocatorDefault, 0, &kCFTypeDictionaryKeyCallBacks, &kCFTypeDictionaryValueCallBacks);
        CFDictionarySetValue(dict, kIOSurfaceIsGlobal, kCFBooleanTrue);
        CFDictionarySetValue(dict, kIOSurfaceBytesPerRow, CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &pitch));
        CFDictionarySetValue(dict, kIOSurfaceBytesPerElement, CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &bPE));
        CFDictionarySetValue(dict, kIOSurfaceWidth, CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &width));
        CFDictionarySetValue(dict, kIOSurfaceHeight, CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &height));
        CFDictionarySetValue(dict, kIOSurfacePixelFormat, CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, pixelFormat));
        CFDictionarySetValue(dict, kIOSurfaceAllocSize, CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &size));

        IOSurfaceRef destSurf = IOSurfaceCreate(dict);

        IOSurfaceAcceleratorRef outAcc;
        IOSurfaceAcceleratorCreate(NULL, 0, &outAcc);

        IOSurfaceAcceleratorTransferSurface(outAcc, screenSurface, destSurf, dict, NULL);

        IOSurfaceUnlock(screenSurface, kIOSurfaceLockReadOnly, &aseed);
        CFRelease(outAcc);

        // MOST RELEVANT PART OF CODE

        CVPixelBufferCreateWithBytes(NULL, width, height, kCVPixelFormatType_32BGRA, IOSurfaceGetBaseAddress(destSurf), IOSurfaceGetBytesPerRow(destSurf), NULL, NULL, NULL, &sampleBuffer);

        CMTime frameTime = CMTimeMake(frameCount, (int32_t)5);

        [adaptor appendPixelBuffer:sampleBuffer withPresentationTime:frameTime];

        CFRelease(sampleBuffer);
        CFRelease(destSurf);
        frameCount++;
    }

P.S:最后 4-5 行代码是最相关的(如果需要过滤).

P.S: The last 4-5 lines of code are the most relevant(if you need to filter).

1) 制作的视频有伪像.我以前做过视频，以前也遇到过这样的问题.我想这可能有两个原因:
一世.传递给适配器的 PixelBuffer 在处理(编码 + 写入)完成之前被修改或释放.这可能是由于异步调用.但我不确定这本身是否是问题以及如何解决它.
ii.传递的时间戳不准确(例如，具有相同时间戳的 2 帧或时间戳低于前一帧的帧).我注销了时间戳值，这似乎不是问题.

1) The video that is produced has artefacts. I have worked on videos previously and have encountered such an issue before as well. I suppose there can be 2 reasons for this:
i. The PixelBuffer that is passed to the adaptor is getting modified or released before the processing (encoding + writing) is complete. This can be due to asynchronous calls. But I am not sure if this itself is the problem and how to resolve it.
ii. The timestamps that are passed are inaccurate (e.g. 2 frames having the same timestamp or a frame having a lower timestamp than the previous frame). I logged out the timestamp values and this doesn't seem to be the problem.

2) 上面的代码在播放视频或玩游戏时无法抓取表面.我得到的只是输出中的一个空白屏幕.这可能是由于在这种情况下发生的硬件加速解码.

2) The code above is not able to grab surfaces when a video is played or when we play games. All I get is a blank screen in the output. This might be due to hardware accelerated decoding that happens in such cases.

对这两个问题中的任何一个部分的任何输入都会非常有帮助.另外，如果您有任何关于 IOSurfaces 的一般阅读链接，请在此处发布.

Any inputs on either of the 2 parts of the questions will be really helpful. Also, if you have any good links to read on IOSurfaces in general, please do post them here.

推荐答案

我做了一些实验，得出的结论是，即使在内容传输完成之前，从中复制内容的屏幕表面也在发生变化(调用 IOSurfaceAcceleratorTransferSurface()).我正在使用一个锁(尝试了异步和只读)，但它被 iOS 覆盖了.我将锁定/解锁部分之间的代码更改为以下最小值:

I did a bit of experimentation and concluded that the screen surface from which the content is copied is changing even before the transfer of contents is complete (call to IOSurfaceAcceleratorTransferSurface() ). I am using a lock (tried both asynchronous and read-only) but it is being overridden by the iOS. I changed the code between the lock/unlock part to the following minimal:

IOSurfaceLock(screenSurface, kIOSurfaceLockReadOnly, &aseed);
aseed1 = IOSurfaceGetSeed(screenSurface);
IOSurfaceAcceleratorTransferSurface(outAcc, screenSurface, destSurf, dict, NULL);
aseed2 = IOSurfaceGetSeed(screenSurface);
IOSurfaceUnlock(screenSurface, kIOSurfaceLockReadOnly, &aseed);

GetSeed 函数告诉表面的内容是否已经改变.而且，我记录了一个计数，指示种子更改的帧数.计数非零.所以，下面的代码解决了这个问题:

The GetSeed function tells if the contents of the surface have changed. And, I logged a count indicating the number of frames for which the seed changes. The count was non-zero. So, the following code resolved the problem:

if(aseed1 != aseed2){
//Release the created surface
continue; //Do not use this surface/frame since it has artefacts
}

但这确实会影响性能，因为许多框架/表面由于伪影而被拒绝.对此的任何补充/更正都会有所帮助.

This however does affect performance since many frames/surfaces are rejected due to artefacts.Any additions/corrections to this will be helpful.

这篇关于IOSurfaces - 视频中的伪影，无法抓取视频表面的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！