MediaCodec和TextureView的Z顺序问题

问题描述

在我的Android应用中，我需要使用以下Z顺序渲染三个视图:

In my Android app I have the need to render three views with the following Z-order:

1. 在底部，MediaCodec解码器的输出表面覆盖整个屏幕.我要求必须转换MediaCodec产生的图像(例如，缩放图像)
2. 在中间，是GLSurfaceView(或我定义的其他运行表面/视图的GL着色器)，覆盖了整个屏幕.显然，此层中的某些像素将是透明的，以便查看下面的MediaCodec输出.
3. 在顶部，还有其他视图-说ImageView.不确定我是否需要为这些最上面的视图提供透明性，也许完全不透明的矩形视图也可以-它们只是不会覆盖整个屏幕，而是会四处移动.

1. At bottom, the output surface of a MediaCodec decoder covering the whole screen. I have the requirement that I have to transform the image produced by MediaCodec (e.g. scale it)
2. In the middle, a GLSurfaceView (or other surface/view running GL shaders I define), covering the whole screen. Obviously some of the pixels in this layer will be transparent, in order to see the MediaCodec output beneath.
3. On top, any other view - say an ImageView. Not sure if I will require transparency for these topmost views, maybe fully opaque rectangular Views are OK - they just won't cover the whole screen and will move around.

看起来这是不可能的，但是也许我错过了一些东西，或者有一种方法可以在较低的水平上做更多的工作(例如，EGL上下文或类似的东西……我目前尚不了解)

It's looking like this is impossible but maybe I'm missing something or there is a way to do it with more effort at a lower level (e.g. EGL contexts or something like that...which I currently don't understand).

我无法使它正常工作并担心这是不可能的原因是:

The reason I haven't been able to get this to work and am worrying it is impossible is:

• 对于底部的MediaCodec输出层(1)，我必须能够转换图像.因此，我给MediaCodec渲染的表面必须来自TextureView
• 为了能够透视中间GLSurfaceView(2)的透明像素，我必须调用GLSurfaceView.setZOrderOnTop(true).否则，GLSurfaceView是不透明的.
• 但是调用GLSurfaceView.setZOrderOnTop(true)意味着GLSurfaceView之上没有其他视图(3)呈现.例如. ImageView始终会出现在GLSurfaceView的不透明像素后面.

• For the bottom MediaCodec output layer (1), I have to be able to transform the image. So, the surface I give MediaCodec to render to must be from a TextureView
• In order to be able to see through the transparent pixels of the middle GLSurfaceView (2), I have to call GLSurfaceView.setZOrderOnTop(true). Otherwise the GLSurfaceView is opaque.
• But calling GLSurfaceView.setZOrderOnTop(true) means that no other views (3) are rendered on top of the GLSurfaceView. E.g. an ImageView will always appear behind the opaque pixels of the GLSurfaceView.

似乎调用GLSurfaceView.setZOrderMediaOverlay(true)而不是GLSurfaceView.setZOrderOnTop(true)是为了解决此问题并简化此类Z排序.如果最底下的MediaCodec输出层是SurfaceView，它也会这样做.但是我需要将其作为TextureView，以便可以对其进行转换.并且GLSurfaceView.setZOrderMediaOverlay(true)在下面有TextureView时似乎不起作用:TextureView被中间的GLSurfaceView层完全遮盖，而不是通过透明像素显示.

It looks like calling GLSurfaceView.setZOrderMediaOverlay(true) instead of GLSurfaceView.setZOrderOnTop(true) is intended to address this and facilitate this type of Z-ordering. And it does if the bottommost MediaCodec output layer is a SurfaceView. But I need it to be a TextureView so I can transform it. And GLSurfaceView.setZOrderMediaOverlay(true) doesn't seem to work when there is a TextureView beneath it: the TextureView is completely obscured by the middle GLSurfaceView layer rather than showing through the transparent pixels.

这种Z排序不可能吗是正确的吗?还是可以通过搞乱EGL和上下文等来实现?

Is it correct that this Z-ordering is impossible? Or can it be accomplished by messing around with EGL and contexts, etc?

推荐答案

EGL上下文在这里并不是真正相关的.您的战斗就是使用SurfaceFlinger和视图系统.

EGL contexts aren't really relevant here. Your fight is with SurfaceFlinger and the view system.

如果运行adb shell dumpsys SurfaceFlinger，则可以看到系统合成器知道的所有层的完整列表.如果您要在SurfaceView中播放320x240的视频，则看起来像这样(为简洁起见，删除了几列和许多其他内容):

If you run adb shell dumpsys SurfaceFlinger you can see something a complete list of all the layers that the system compositor knows about. If you're playing a 320x240 video in a SurfaceView, it looks something like this (several columns and lots of other stuff removed for brevity):

    type    |          source crop              |           frame           name
------------+-----------------------------------+--------------------------------
        HWC | [    0.0,    0.0,  320.0,  240.0] | [   48,  411, 1032, 1149] SurfaceView
        HWC | [    0.0,   75.0, 1080.0, 1776.0] | [    0,   75, 1080, 1776] com.android.grafika/com.android.grafika.PlayMovieSurfaceActivity
        HWC | [    0.0,    0.0, 1080.0,   75.0] | [    0,    0, 1080,   75] StatusBar
        HWC | [    0.0,    0.0, 1080.0,  144.0] | [    0, 1776, 1080, 1920] NavigationBar
  FB TARGET | [    0.0,    0.0, 1080.0, 1920.0] | [    0,    0, 1080, 1920] HWC_FRAMEBUFFER_TARGET

图层从后到前按Z顺序排列. SurfaceView的表面层位于背面，应用程序UI层位于其顶层，系统状态+导航栏位于所有内容之上.

The layers are in Z-order, from back to front. The SurfaceView's surface layer is at the back, the app UI layer is on top of that, and the system status + nav bars are above everything.

应用程序的视图"层次结构中的所有内容都呈现在单个图层上.其中包括TextureView.您无法控制其相对于其他硬件组合层的Z顺序.

Everything in the app's View hierarchy is rendered on a single layer. That includes TextureView. You can't control its Z-ordering relative to other hardware-composited layers.

SurfaceView之所以如此，是因为View部分只是一个透明的占位符，而实际动作发生在该单独的层上，您可以控制其Z顺序(一点点).您可以将其放在三个不同的级别上:

SurfaceViewis fancy in that the View portion is just a transparent placeholder, while the real action happens on that separate layer, whose Z-ordering you can control (a little). You can put it on three different levels:

• 媒体"(默认)
• 媒体叠加层"
• (应用程序用户界面在此处)
• 面板"(ZOrderOnTop)

因此，您要做的是将MediaCodec输出放置在默认层，将GLES输出放置在媒体覆盖"层.您都需要使用SurfaceView来完成这两项.

So what you want to do is put your MediaCodec output at the default layer, and your GLES output at the "media overlay" layer. You need to do both of these with a SurfaceView.

很难从这里提供更好的建议，因为您描述的是您尝试解决方案所遇到的问题，而不是您要解决的问题(即您要构建的是什么?)，但是我可以提供一些建议建议.

It's hard to provide better advice from here because you described the problems you're having with your attempted solution, rather than the problem you're trying to solve (i.e. what are you building?), but I can offer a couple of suggestions.

首先，您可以缩放SurfaceView.如果您查看上面的dumpsys输出，您会注意到"SurfaceView"行的源裁剪矩形为320x240(视频大小)，而目标矩形为984x738.这来自 Grafika的播放视频(SurfaceView)"，SurfaceView的大小可保留4:3视频的宽高比. SurfaceFlinger负责缩放内容以匹配视图.

First, you can scale a SurfaceView. If you look at the dumpsys output above, you'll notice that the "SurfaceView" line has a source crop rect of 320x240 (the size of the video), and a destination rect that's 984x738. This is coming from Grafika's "Play video (SurfaceView)", which sized the SurfaceView to preserve the 4:3 aspect ratio of the video. SurfaceFlinger takes care of scaling the content to match the view.

第二，如果您不显示受DRM保护的视频内容，则可以将其发送到SurfaceTexture并在渲染其他所有内容时仅使用GLES进行渲染. (这正是TextureView的工作，这就是为什么它需要硬件加速的原因.) Grafika中的连续捕获".

Second, if you're not showing DRM-protected video content, you can send it to a SurfaceTexture and just render it with GLES when you're rendering everything else. (This is exactly what TextureView does, which is why it requires hardware acceleration.) See e.g. "Continuous capture" in Grafika.

更新:可以在 Android系统级图形文档.

这篇关于MediaCodec和TextureView的Z顺序问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！