在Unity SRP中实现TAA效果

TAA的原理

首先是要了解画面上的锯齿是如何产生的。锯齿发生在光栅化的阶段，光栅化的时候会丢失掉小于一个像素宽的细节，也就导致了锯齿的产生。

从字面上来看，TAA (Temporal Anti-Aliasing)的抗锯齿效果来源于Temporal一词，是一种时间上的抗锯齿。TAA会结合当前渲染的画面和之前渲染的画面，通过这两个画面之间的融合，达成抗锯齿的效果。基本思想是在光栅化的时候对画面进行抖动，让亚像素的细节在不同帧渲染到不同的像素上，最后再对这些像素按时间的权重来混合，就能达到抗锯齿的效果。

Temporal Reprojection Anti-Aliasing

Temporal Reprojection Anti-Aliasing是由PlayDead在他们的游戏Inside中使用的一种TAA的方法，他们在GDC2016的演示中分享了这个方法。相较于普通的TAA来说，Temporal Reprojection Anti-Aliasing中使用了Velocity Buffer中的屏幕像素的速度信息和Depth Buffer中对应的屏幕像素的世界坐标信息，这样当物体移动或者相机移动的时候，在做到抗锯齿的同时也减少了TAA带来的拖影效果，同时也把TAA和运动模糊相结合达到更理想的抗锯齿的效果。

PlayDead提供了对应的源代码。本博客中TAA在SRP中的实现也参考了sienaiwun的TAA代码。

在Unity SRP中实现TAA的操作

我们通过RendererFeature的方式在渲染管线中加入TAA。在ForwardRendererData中加入RendererFeature后，往Global Volume中添加Temporal Anti-Aliasing以在场景中启用TAA效果。启用TAA效果后，会现在渲染不透明物体之前调用一个Jitter Pass对相机的栅格化阶段进行抖动；在渲染TAA Pass时（在Bloom等跟物体渲染相关的后处理效果之后，在Chromatic Aberration等跟屏幕空间位置相关的后处理效果之前）根据抖动值还原出正常的不抖动的画面，并和AccumTexture进行混合，获得最终的渲染画面。因此我们需要TAARendererFeature、TAAJitterPass、TAARenderPass这三个脚本来处理渲染管线，TemporalAntiAliasing这个脚本来处理Volume，TAAShader这个Shader文件来进行TAA的混合操作。
对栅格化阶段进行抖动，也就相当于是修改了相机的透视变换矩阵的第一第二行的第三位的值，抖动值最好和TexelSize相结合，这样在TAA反向抖动还原正常值的时候，在shader中会比较好写。抖动值和TAA的反向抖动是正比关系，因此可以不需要特别纠结于计算，在shader中传入一个debug值再和抖动值相乘用作反向抖动，观察最后的画面是否存在抖动，就能很好的判断出这两个值的比例了。抖动的方式有很多，纯随机的抖动也可以选择，不过稍不如使用均匀分布的随机抖动的效果好，这里使用Inside中的方式即利用Halton数列进行抖动。
为了让相机移动时也能有较好的抗锯齿效果且削弱拖影现象，Temporal Reprojection Anti-Aliasing需要采样当前的深度贴图，还原出物体的世界空间的坐标，再计算出这个世界空间在AccumTexture中的UV值(Reprojection)，使用这个UV值采样AccumTexture再和当前渲染画面进行融合。
因为Velocity Buffer比较麻烦，这里暂且忽略掉物体移动对TAA带来的影响。
在ScriptableRenderPass中使用cmd.GetTemporaryRT()获得的Render Texture，在当帧过后就会被回收，因此AccumTexture需要使用RenderTexture.GetTemporary()来获取。这里我把AccumTexture放在TemporalAntiAliasing.cs中，方便使用。

TemporalAntiAliasing.cs

除了普通的Volume的设置之外，还需要提供Render Texture的接口。lastFrame的x值和y值分别对应最后渲染画面中对AccumTexture进行线性插值的最小和最大系数。

using System;

namespace UnityEngine.Rendering.Universal
{
    [Serializable, VolumeComponentMenu("Post-processing/Temporal Anti-Aliasing")]
    public class TemporalAntiAliasing : VolumeComponent, IPostProcessComponent
    {
        public BoolParameter isEnabled = new BoolParameter(false);

        public NoInterpFloatRangeParameter lastFrame = new NoInterpFloatRangeParameter(new Vector2(0.2f, 0.8f), 0f, 1f);

        public Vector2Parameter jitterIntensity = new Vector2Parameter(Vector2.one);

        private RenderTexture[] accumTextures;


        public bool IsActive()
        {
            return isEnabled.value;
        }

        public bool IsTileCompatible()
        {
            return false;
        }

        void EnsureArray<T>(ref T[] array, int size, T initialValue = default(T))
        {
            if (array == null || array.Length != size)
            {
                array = new T[size];
                for (int i = 0; i != size; i++)
                    array[i] = initialValue;
            }
        }

        bool EnsureRenderTarget(ref RenderTexture rt, int width, int height, RenderTextureFormat format, FilterMode filterMode, string name, int depthBits = 0, int antiAliasing = 1)
        {
            if (rt != null && (rt.width != width || rt.height != height || rt.format != format || rt.filterMode != filterMode || rt.antiAliasing != antiAliasing))
            {
                RenderTexture.ReleaseTemporary(rt);
                rt = null;
            }
            if (rt == null)
            {
                rt = RenderTexture.GetTemporary(width, height, depthBits, format, RenderTextureReadWrite.Default, antiAliasing);
                rt.name = name;
                rt.filterMode = filterMode;
                rt.wrapMode = TextureWrapMode.Clamp;
                return true;// new target
            }
            return false;// same target
        }

        public void EnsureRT(RenderTextureDescriptor descriptor)
        {
            EnsureArray(ref accumTextures, 2);
            EnsureRenderTarget(ref accumTextures[0], descriptor.width, descriptor.height, descriptor.colorFormat, FilterMode.Bilinear, "TAA_Accum_One");
            EnsureRenderTarget(ref accumTextures[1], descriptor.width, descriptor.height, descriptor.colorFormat, FilterMode.Bilinear, "TAA_Accum_Two");
        }

        public RenderTexture GetRT(int index)
        {
            return accumTextures[index];
        }
    }
}

TAARendererFeature.cs

在TAARendererFeature中生成相机抖动的值，通过TAAJitterPass对相机的投影矩阵进行抖动，通过TAARenderPass反向抖动还原正常的画面。Halton序列的生成方式可以进行优化，这里暂且略过。这里也暂且忽略了DX11和OpenGL的平台差异化处理（无非就是UV的Y轴翻转的问题）。

namespace UnityEngine.Rendering.Universal
{
    public class TAARendererFeature : ScriptableRendererFeature
    {
        [System.Serializable]
        public class TAASettings
        {
            public bool isEnabled = true;
            //最好是AfterRenderingPostProcessing，不过会有CameraTarget的问题，需要更多的设置
            public RenderPassEvent renderPassEvent = RenderPassEvent.BeforeRenderingPostProcessing;
        }

        public TAASettings settings = new TAASettings();
        TAARenderPass taaRenderPass;
        TAAJitterPass taaJitterPass;
        private TAAData taaData;
        private int haltonIndex = 0;
        private Vector2 lastOffset;
        private Matrix4x4 lastProj = Matrix4x4.identity;
        private Matrix4x4 lastView = Matrix4x4.identity;

        [SerializeField, HideInInspector]
        private Shader taaShader;

        public override void Create()
        {
            taaShader = Shader.Find("Hidden/Universal Render Pipeline/TAAShader");
            if (taaShader == null)
            {
                Debug.LogWarning("Shader was not found. Please ensure it compiles correctly");
                return;
            }

            taaData = new TAAData();
            taaData.Initialize();
            taaJitterPass = new TAAJitterPass();
            taaRenderPass = new TAARenderPass(settings);
        }

        public struct TAAData
        {
            public Vector2 offset;
            public Vector2 lastOffset;
            public Matrix4x4 lastProj;
            public Matrix4x4 lastView;
            public Matrix4x4 jitteredProj;
            public Matrix4x4 currentView;

            public void Initialize()
            {
                offset = Vector2.zero;
                lastOffset = Vector2.zero;
                lastProj = Matrix4x4.identity;
                lastView = Matrix4x4.identity;
                jitteredProj = Matrix4x4.identity;
                currentView = Matrix4x4.identity;
            }
        }

        private float HaltonSeq(int prime, int index = 1/* NOT! zero-based */)
        {
            float r = 0.0f;
            float f = 1.0f;
            int i = index;
            while (i > 0)
            {
                f /= prime;
                r += f * (i % prime);
                i = (int)Mathf.Floor(i / (float)prime);
            }
            return r;
        }

        //dx11 only?
        private Matrix4x4 GetJitteredProjectionMatrix(Camera camera, Vector2 offset, Vector2 jitterIntensity)
        {
            Matrix4x4 originalProjMatrix = camera.nonJitteredProjectionMatrix;

            float near = camera.nearClipPlane;
            float far = camera.farClipPlane;

            Vector2 matrixOffset = offset * new Vector2(1f / camera.pixelWidth, 1f / camera.pixelHeight) * jitterIntensity;
            //[row, column]
            originalProjMatrix[0, 2] = matrixOffset.x;
            originalProjMatrix[1, 2] = matrixOffset.y;
            return originalProjMatrix;
        }

        public override void AddRenderPasses(ScriptableRenderer renderer, ref RenderingData renderingData)
        {
            Camera camera = renderingData.cameraData.camera;

            TemporalAntiAliasing taaComponent = VolumeManager.instance.stack.GetComponent<TemporalAntiAliasing>();
            if(renderingData.cameraData.cameraType == CameraType.Game && taaComponent.IsActive())
            {
                // 获取Offset值
                haltonIndex = (haltonIndex + 1) & 1023;
                Vector2 offset = new Vector2(
                    HaltonSeq(2, haltonIndex + 1) - 0.5f,
                    HaltonSeq(3, haltonIndex + 1) - 0.5f);

                // 获取jittered projection matrix，并记录之前的matrix
                // jittered projection的ab应该对应0.5 * texel_size.xy
                lastOffset = taaData.offset;
                taaData.lastOffset = lastOffset;
                taaData.offset = new Vector2(offset.x / camera.pixelWidth, offset.y / camera.pixelHeight) * taaComponent.jitterIntensity.value;
                taaData.jitteredProj = GetJitteredProjectionMatrix(camera, offset, taaComponent.jitterIntensity.value);
                taaData.lastProj = lastProj;
                taaData.lastView = lastView;
                lastProj = camera.projectionMatrix;
                lastView = camera.worldToCameraMatrix;
                taaData.currentView = lastView;

                // 第一个Pass对相机使用jittered projection matrix
                taaJitterPass.Setup(taaData);
                renderer.EnqueuePass(taaJitterPass);

                //第二个Pass执行真正的TAA
                //暂时不考虑motion blur/运动物体的TAA
                taaRenderPass.Setup(taaShader, taaData, taaComponent);
                renderer.EnqueuePass(taaRenderPass);
            }
        }
    }
}

TAAJitterPass.cs

这个Pass仅用于改变相机的透视变换矩阵，在某些情况下渲染透明物体时会重置透视变换矩阵，这种情况下需要在BeforeRenderingTransparent的时候再额外执行一遍TAAJitterPass。

namespace UnityEngine.Rendering.Universal
{
    public class TAAJitterPass : ScriptableRenderPass
    {
        private const string profilerTag = "TAA Jitter Pass";
        private ProfilingSampler taaSampler = new ProfilingSampler("TAA Jitter Pass");

        private TAARendererFeature.TAAData taaData;

        public TAAJitterPass()
        {
            profilingSampler = taaSampler;
            renderPassEvent = RenderPassEvent.BeforeRenderingOpaques;
        }

        public void Setup(TAARendererFeature.TAAData taaData)
        {
            this.taaData = taaData;
        }

        public override void Execute(ScriptableRenderContext context, ref RenderingData renderingData)
        {
            CommandBuffer cmd = CommandBufferPool.Get(profilerTag);
            context.ExecuteCommandBuffer(cmd);
            cmd.Clear();

            using (new ProfilingScope(cmd, profilingSampler))
            {
                CameraData cameraData = renderingData.cameraData;
                cmd.SetViewProjectionMatrices(cameraData.camera.worldToCameraMatrix, taaData.jitteredProj);
            }
            context.ExecuteCommandBuffer(cmd);
            CommandBufferPool.Release(cmd);
        }
    }
}

TAARenderPass.cs

由于_CameraColorTexture没有开启随机读写，使用Compute Shader进行TAA的计算会需要额外的blit，这里就使用普通的shader来进行TAA的操作了。

using System;

namespace UnityEngine.Rendering.Universal
{
    public class TAARenderPass : ScriptableRenderPass
    {
        private const string profilerTag = "My TAA Pass";

        private ProfilingSampler taaSampler = new ProfilingSampler("TAA Pass");

        RenderTargetHandle cameraColorHandle;
        RenderTargetIdentifier cameraColorIden;
        RenderTargetHandle cameraDepthHandle;
        RenderTargetIdentifier cameraDepthIden;

        private Vector2Int screenSize;
        private int accumIndex;

        private TAARendererFeature.TAASettings settings;
        private TAARendererFeature.TAAData taaData;
        private Material taaMaterial;
        private TemporalAntiAliasing taaComponent;

        private Vector2 lastFrame;

        public TAARenderPass(TAARendererFeature.TAASettings settings)
        {
            profilingSampler = new ProfilingSampler(profilerTag);
            cameraColorHandle.Init("_CameraColorTexture");
            cameraColorIden = cameraColorHandle.Identifier();
            cameraDepthHandle.Init("_CameraDepthTexture");
            cameraDepthIden = cameraDepthHandle.Identifier();

            this.settings = settings;
        }

        public void Setup(Shader taaShader, TAARendererFeature.TAAData taaData, TemporalAntiAliasing taaComponent)
        {
            this.taaData = taaData;
            renderPassEvent = settings.renderPassEvent;
            this.lastFrame = taaComponent.lastFrame.value;
            taaMaterial = new Material(taaShader);
            this.taaComponent = taaComponent;
        }

        public override void Configure(CommandBuffer cmd, RenderTextureDescriptor cameraTextureDescriptor)
        {
            RenderTextureDescriptor descriptor = cameraTextureDescriptor;
            screenSize = new Vector2Int(descriptor.width, descriptor.height);
            taaComponent.EnsureRT(descriptor);
        }

        private void DoTAA(CommandBuffer cmd, TemporalAntiAliasing taa, RenderTargetIdentifier colorid, int index)
        {
            RenderTexture accumRead = taa.GetRT(index);
            int tempIndex = 1 - index;
            RenderTexture accumWrite = taa.GetRT(tempIndex);

            taaMaterial.SetVector("_TAAOffsets", new Vector4(taaData.offset.x, taaData.offset.y, taaData.lastOffset.x, taaData.lastOffset.y));
            taaMaterial.SetVector("_TAALastFrame", taaComponent.lastFrame.value);
            taaMaterial.SetVector("_TextureSize", new Vector4(screenSize.x, screenSize.y, 1f / screenSize.x, 1f / screenSize.y));
            Matrix4x4 lastViewProj = taaData.lastProj * taaData.lastView;
            taaMaterial.SetMatrix("_LastViewProj", lastViewProj);
            taaMaterial.SetTexture("_AccumTexture", accumRead);

            cmd.Blit(colorid, accumWrite, taaMaterial);
            cmd.Blit(accumWrite, colorid);
        }

        public override void Execute(ScriptableRenderContext context, ref RenderingData renderingData)
        {
            CommandBuffer cmd = CommandBufferPool.Get(profilerTag);
            context.ExecuteCommandBuffer(cmd);
            cmd.Clear();

            using (new ProfilingScope(cmd, taaSampler))
            {
                DoTAA(cmd, taaComponent, cameraColorIden, accumIndex);
                accumIndex = 1 - accumIndex;
            }
            context.ExecuteCommandBuffer(cmd);
            cmd.Clear();
            CommandBufferPool.Release(cmd);
        }

        public override void FrameCleanup(CommandBuffer cmd)
        {
            if (cmd == null) throw new ArgumentNullException("cmd");
        }
    }
}

TAAShader.shader

多次采样能够明显的削弱拖影效果，tempMain = min(mainTexture, color_avg * 1.25);这一行能够在一定程度上减少高光的闪烁，clip_aabb和k_feedback似乎用处不是很大，不过我还是写进去了。

Shader "Hidden/Universal Render Pipeline/TAAShader"
{
    Properties
    {
        _MainTex("Main Texture", 2D) = "white"{}
    }

    HLSLINCLUDE
#include "Packages/com.unity.render-pipelines.core/ShaderLibrary/Common.hlsl"
#include "Packages/com.unity.render-pipelines.universal/ShaderLibrary/core.hlsl"
#include "Packages/com.unity.render-pipelines.core/ShaderLibrary/Color.hlsl"

    texture2D _MainTex;
    texture2D _CameraDepthTexture;
    texture2D _AccumTexture;

    SamplerState sampler_LinearClamp;
    SamplerState sampler_PointClamp;

    float4 _TAAOffsets;
    float2 _TAALastFrame;
    float4 _TextureSize;
    float4x4 _LastViewProj;

    struct Attributes
    {
        float4 positionOS   : POSITION;
        float2 texcoord     : TEXCOORD0;
    };

    struct Varyings
    {
        float4 positionCS   : SV_POSITION;
        float2 texcoord     : TEXCOORD0;
    };

    float3 clip_aabb(float3 aabb_min, float3 aabb_max, float3 avg, float3 input_texel)
    {
        float3 p_clip = 0.5 * (aabb_max + aabb_min);
        float3 e_clip = 0.5 * (aabb_max - aabb_min) + FLT_EPS;
        float3 v_clip = input_texel - p_clip;
        float3 v_unit = v_clip / e_clip;
        float3 a_unit = abs(v_unit);
        float ma_unit = max(a_unit.x, max(a_unit.y, a_unit.z));

        if (ma_unit > 1.0)
            return p_clip + v_clip / ma_unit;
        else
            return input_texel;
    }

    Varyings TAAVert(Attributes input)
    {
        Varyings output = (Varyings)0;
        VertexPositionInputs vertexPositionInputs = GetVertexPositionInputs(input.positionOS.xyz);
        output.positionCS = vertexPositionInputs.positionCS;
        output.texcoord = input.texcoord;
        return output;
    }

    float4 TAAFrag(Varyings input) : SV_TARGET
    {
        float2 sampleUV = input.texcoord;
        float2 currentOffset = _TAAOffsets.xy;
        float2 lastOffset = _TAAOffsets.zw;

        float2 unJitteredUV = sampleUV - 0.5 * currentOffset;

        float3 mainTexture = _MainTex.SampleLevel(sampler_LinearClamp, unJitteredUV, 0).rgb;

        float2 du = float2(_TextureSize.z, 0);
        float2 dv = float2(0, _TextureSize.w);

        float3 ctl = _MainTex.Sample(sampler_LinearClamp, unJitteredUV - dv - du).rgb;
        float3 ctc = _MainTex.Sample(sampler_LinearClamp, unJitteredUV - dv).rgb;
        float3 ctr = _MainTex.Sample(sampler_LinearClamp, unJitteredUV - dv + du).rgb;
        float3 cml = _MainTex.Sample(sampler_LinearClamp, unJitteredUV - du).rgb;
        float3 cmc = _MainTex.Sample(sampler_LinearClamp, unJitteredUV).rgb;
        float3 cmr = _MainTex.Sample(sampler_LinearClamp, unJitteredUV + du).rgb;
        float3 cbl = _MainTex.Sample(sampler_LinearClamp, unJitteredUV + dv - du).rgb;
        float3 cbc = _MainTex.Sample(sampler_LinearClamp, unJitteredUV + dv).rgb;
        float3 cbr = _MainTex.Sample(sampler_LinearClamp, unJitteredUV + dv + du).rgb;

        float3 color_min = min(ctl, min(ctc, min(ctr, min(cml, min(cmc, min(cmr, min(cbl, min(cbc, cbr))))))));
        float3 color_max = max(ctl, max(ctc, max(ctr, max(cml, max(cmc, max(cmr, max(cbl, max(cbc, cbr))))))));
        float3 color_avg = (ctl + ctc + ctr + cml + cmc + cmr + cbl + cbc + cbr) / 9.0;

        float depthTexture = _CameraDepthTexture.SampleLevel(sampler_PointClamp, unJitteredUV, 0).r;

        float4 positionNDC = float4(sampleUV * 2 - 1, depthTexture, 1);
#if UNITY_UV_STARTS_AT_TOP
        positionNDC.y = -positionNDC.y;
#endif

        float4 worldPos = mul(UNITY_MATRIX_I_VP, positionNDC);
        worldPos /= worldPos.w;
        float4 lastPositionCS = mul(_LastViewProj, worldPos);
        float2 lastUV = lastPositionCS.xy / lastPositionCS.w;
        lastUV = lastUV * 0.5 + 0.5;

        float3 accumTexture = _AccumTexture.SampleLevel(sampler_LinearClamp, lastUV, 0).rgb;
        float3 tempMain = 0;
        accumTexture = clip_aabb(color_min, color_max, color_avg, accumTexture);
        tempMain = min(mainTexture, color_avg * 1.25);

        float lum0 = Luminance(mainTexture);
        float lum1 = Luminance(accumTexture);

        float unbiased_diff = abs(lum0 - lum1) / max(lum0, max(lum1, 0.2));
        float unbiased_weight = 1.0 - unbiased_diff;
        float unbiased_weight_sqr = unbiased_weight * unbiased_weight;
        float k_feedback = lerp(_TAALastFrame.x, _TAALastFrame.y, unbiased_weight_sqr);

        float3 returnColor = lerp(tempMain, accumTexture, k_feedback);
        if (unJitteredUV.x >= 0.5)
        {
            returnColor = mainTexture;
        }
        return float4(returnColor, 1);
    }

    ENDHLSL

    SubShader
    {
        ZTest Always Cull Back ZWrite Off
    
        pass
        {
            Name "TAA Pass"
            HLSLPROGRAM
            #pragma vertex TAAVert
            #pragma fragment TAAFrag
            ENDHLSL
        }
    }
}

一些思考

效果总体来说还是不错的，但是TAAShader中lerp当前渲染画面和AccumTexture的算法应该有待提高，目前特别细长的亚像素特征（如呈线状的高光）会有锯齿爬行的感觉，画面不够稳定，特别小的细节会有一闪一闪的感觉，应该还能再优化优化。PlayDead的源代码有点看不下去。。。就僵在这里了，大概有个80分吧。

我也想多放图来着，但是实在没啥好放的。。。导致博客越来越枯燥了。

TAA的原理#

Temporal Reprojection Anti-Aliasing#

在Unity SRP中实现TAA的操作#

TemporalAntiAliasing.cs#

TAARendererFeature.cs#

TAAJitterPass.cs#

TAARenderPass.cs#

TAAShader.shader#

一些思考#