There’s Vulkan 1.4 based GPU support available to test in the vukvuk2 branch, just install the most recent artifact. It’s feature and API complete and can be used for testing. Changes to the API and how it works will only be made if someone has high quality feedback.
All internal core filters now also have GPU support and apart from a few of the most esoteric resize transfer options there is complete feature parity in every filter with one exception, GPU filters don’t accept variable format/size input. Also filters that accept multiple nodes require all nodes to be either local or GPU, no mixing. If you don’t use any GPU filters these builds will work more or less exactly like the latest R79 release.
General structure for users
Basically the available types have been extended with GPU video nodes and GPU video frames. These are more or less identical to the good old video frames except that they’re stored on the GPU. GPU nodes are simply nodes that output GPU frames.
To transfer between local and GPU type for nodes you simply do:
clip.std.GPUUpload()
clip.std.GPUDownload()Uploading or downloading a clip already in the desired location does nothing. Note that implicit transfers are inserted when needed, for example if you do something like:
clip = core.bs.VideoSource("filename", gpu=False)
clip = clip.nnedi3vk.NNEDI3()
clip = clip.vivtc.VFM(order=1)This will work but due to the implicit transfers before nnedi3vk and VFM you’ll get two warnings. It’s also possible to set call set_output() on GPU nodes, they’ll simply get a GPUDownload() inserted implicitly and work like you expect.
By default the best GPU is selected. Best means that discrete devices are picked first and if multiple are found the amount of VRAM is the tie breaker. If you don’t like this you can enumerate all available using core.vulkan_devices() and specify which one to use with core.set_vulkan_device(device index) BEFORE running any GPU filters.
Resource management and performance quirks
In addition to the normal max_cache_size() there is now a max_vram_cache_size() that works much in the same way. There are however a few important quirks to mention, since there’s no swap for GPU memory you literally crash when it runs out. VapourSynth internally retries allocations a few times with delays to reduce the chance of it happening but you will need more conservative numbers here. The default is ~2/3 of discrete GPU VRAM.
Note how I said discrete GPU VRAM. Integrated GPUs with unified memory use much more conservative numbers since the two pools effectively compete for the same RAM. It’s also important to mention that most internal filters run faster on the CPU than on an integrated GPU due to this, the CPU algorithms have much better memory access patterns in general and you otherwise end up memory bandwidth limited very fast.
The internal frame structure
The model chosen is somewhat inspired by FFmpeg’s Vulkan GPU filter framework but has many notable differences. Frames are stored in vkBuffer objects with exactly the same memory layout as a normal frame including the stride and other padding. One vkBuffer per plane, individually reference counted and possible to pass through. Everything works and looks like a normal VSFrame with one exception, getWritePtr() doesn’t have the rarely used copy-on-write semantics, instead you have to explicitly make a copy yourself in a new frame before modifying frame data due to how the GPU pipeline works. For those of you wondering FFmpeg instead uses vkImage as the underlying frame type.
Writing filters and Vulkan usage
Generally it can be said that 3 levels of abstraction are provided. Plain Vulkan (NOT RECOMMENDED), the VSGPUExecPool and other machinery provided in VSVULKANAPI and finally the gpudriver.h header.
Many filters can use gpudriver.h, for example all core filters except resize use this so you can simply look up how the most similar filter was implemented in the core and base your code off that. You can find the invert sample here. As you can see it’s mostly about declaring inputs, output and the shaders. To see more clearly which helper macros (STORE, SRCN and so on) are predefined look up how vsgpu::SimpleFilter::prelude is set. This header will probably be promoted to public at some point with minor changes but for now everyone’s allowed to freely copy and relicense it.
The second also recommended way is to use the VSGPUExecPool object and its related functions. An example of this can be found in the normal GPU invert sample here. Basically it’s normal Vulkan with some additional helpers so the core can keep track of resources. Note that the GPUExecPool is an important part of the memory management machinery. You can still write mostly raw Vulkan API plugins however then the core can’t account for its resource usage and hence is why it’s not recommended even if technically possible.
CUDA users and other feature/extension quirks
There are many versions of the invert sample, including one for CUDA. Basically you import all the input and output frame memory from Vulkan using the export handles. It looks quite simple and I have no idea how CUDA works at all so if you need help go ask AI or something.
We also have another case where the core’s vkDevice isn’t created with the features and extensions required by a plugin. Or for codebases that really prefer to keep their own vkDevice internally. One example of both is BestSource when using FFmpeg’s Vulkan decoding. When using hardware decoding each available decoder needs its own extensions enabled in addition to several general hardware decoding ones. Due to how Vulkan works you can only specify the enabled features/extensions when a vkDevice is created. You can however have multiple vkDevices on the same physical GPU to allow fast memory copies/sharing between them. They’re also not extensions that are generally useful to video processing filters like atomic floats and such so in these cases you just have to have your own vkDevice.
Over time I plan to enable more generally useful extensions when available and do an API version bump to indicate it.