Just a correction to what I actually tested.
The reason for my decision to test NVENC vs x264 was because I noticed a part in my 1440p vod that broke down drastically. I checked and the same cases seemed to break down similarly in quality and it looked unlike anything I had seen with h264 and x264 slow. It must have been an edge case that the encoder just can't handle properly.
I just re-encoded the same raw video with 1440p HEVC at 9k bitrate and it actually looked better than x264 slow with tweaked parameters.
My understanding is that hardware encoders typically need to cheat to produce decent results and I must have encountered one of the rare cases where the efficiency fell apart.
With this in mind, I don't see a reason to use x264 at all when HEVC is available. TY to
@Game_Dev_Carto and
@SDGolfGuy for sharing their experiences causing me to look into this a bit further.
More testing would be required across different games to really say it's better in all cases but I think as long as it's 50%+ of cases, there's no reason to consider x264.
TLDR; Use twitch enhanced broadcast for best quality.
I came to a similar conclusion although I do think 1440p HEVC will look better than 1080p 8k NVENC. HEVC is a bit more bitrate efficient so you can think of it as being equivalent to 1440p at 12k bitrate using H264.
It's still not enough for fast-paced gameplay with non-cartoon and/or non-simple graphics.
I tested apex legends and it still struggles with drastic lighting changes, lots of reds, cam quality deteriorates signficantly as well in high action.
x264 slow even with no tweaked parameters blows any NVENC h264 or HEVC out of the water at the possible twitch and kick streaming bitrates. You can get it even better with tweaked up x264 settings.
I did some extensive side-by-side subjective testing with apex legends and it just doesn't compare.
That being said, if you can't reliably do x264 slow or are multistreaming, server-side transcodes + 1440p HEVC at 9k is a no brainer. and you can reuse that encode for youtube as well since youtube supports HEVC and AV1 and doesn't require the viewer to have a device that supports it since they do server-side transcoding for every quality setting -- including the highest "source" quality.