If you were a professional, with a VERY large budget, and appropriate IT support, then sure, go for lossless Recording, otherwise... hold that thought
As for 'sweet spot' sure... but it depends [sorry, couldn't help myself] but seriously, depends on content, workflow, and quite a few variables ... many of which will be unique to your circumstance (and video editor) ...
My recommendation - start with which video editor you are going to use... as that will be primary determinant of desired video format, bitrate, color depth, etc. As a rough rule of thumb (and me not knowing better... this isn't my area of expertise), I start with your end video output requirement/expectation, and be Recording at 50% to 100% higher bitrate (maybe even higher if your system can handle it?). Also, make sure you look into your GPU and the impact of Lookahead and PsychoVisual settings, as well as B-Frames
- Then, to preserve video quality, make sure you understand your video editor, especially that you edit, re-encode and check, but your next edit will be against the original video file, NOT the one you just re-encoded/output [the edits are like a recipe applied to base video, you can just keep tweaking. But starting a fresh edit against your output will create the generational quality loss. If you weren't already aware... be absolutely sure you understand how this works, its impact, etc in your editor of choice]
Once you have a handle on the above, the settings in OBS Studio become more obvious. An item I found not obvious, is that for best video quality, don't re-scale (especially not using your CPU) ... set OBS Studio Base Canvas and output to same resolution (ideally same as monitor)... so if you want 4K output, best to have 4K monitor. [in part, per qhobbes comment below]
Here are some other notes I captured years ago that might be a good starting point for consideration
https://obsproject.com/forum/threads/best-settings.140188/#post-514693 @FerretBomb comment #2
1) NEVER RECORD TO MP4 DIRECTLY, FOR ANY REASON. It is not a recording-safe format; if anything goes wrong during the recording, even for a split second, the ENTIRE recording will be corrupted and absolutely not recoverable by any means. Record to MKV, and remux to MP4 after the recording is complete from OBS' File menu, Remux Recordings. [Lawrence 2025 note ... this applies to hybrid MP4 as well... just don't]
2) Record using CQP or CRF, not CBR. CBR is only used for streaming, where the back-end infrastructure requires it. CQP/CRF are quality-target based encodes, and will use as much or as little bitrate as is needed to maintain a constant image quality. No wasting bitrate on simple/slow scenes, no choking on fast-moving or complex scenes. 22 is a good starting point. 16 will result in much larger files, but near-perfect video. 12 should only be used if you plan to edit and re-encode later, and will be VERY large. Anything lower than 12 shouldn't be used unless you know exactly why you need it, and what problems it can cause.
3) Use the Quality preset, not Max Quality. Likewise, turn off Psychovisual Tuning. Both of these options use CUDA cores, and tend to cause significant problems like encoding overload when it should otherwise not be happening.
Related to # 2 above ["FerretBomb, post: 529433, member: 4349"]
Don't record with CBR or VBR, use CQP instead. CQP is a quality-based encoding target that uses as much or as little bitrate as is needed to maintain a given image quality level.
22 is the normal 'good' point, 16 for 'visually lossless', and 12 is generally the lowest you'll want to go even if you plan to edit the video later (to cut down on re-encoding artifacts). The lower the number, the closer to 'lossless' video it gets. But below 16 the filesizes get ridiculously large very fast.
For easy setting, use simple output mode instead of advanced and choose "High Quality, Medium File Size" as quality. Or if you insist on advanced output mode, increase the cq value to get smaller files. Increasing the cq value by 3-4, you're halving the file size.
…. recording, use a quality based rate control like CQP (if you use nvenc on a Nvidia GPU) or CRF (if you use x264) or ICQ (if you use Quicksync on a Intel iGPU). CBR/VBR is for streaming only.
Jun 9 2021 NVENC. Tools > Auto-Configuration Wizard
Look-ahead allows the encoder to dynamically select the number of B-Frames, between 0 and the number of B-Frames you specify. B-frames are great because they increase image quality, but they consume a lot of your available bitrate (you should use CQP for recording so bitrate is not an issue), so they reduce quality on high motion content. Look-ahead enables the best of both worlds. This feature is CUDA accelerated; toggle this off if your GPU utilization is high to ensure a smooth recording.
Psycho Visual Tuning enables the Rate Distortion Optimization in the encoder, which greatly optimizes the way you use bitrate (not sure exactly how this works with CQP, possibly same quality at lower bitrate), improving image quality on movement. This feature is also CUDA accelerated.
If you're using something that uses the GPU, such as games, don't enable those. It depends on your source.
rescaling in the encoder in settings->output->streaming->rescale output. Disable this. This runs in CPU space and is a bottleneck. Instead just set your output resolution in settings->video->output resolution. This rescaling takes place on the GPU and uses next to none resources, since the GPU is specialized on that kind of operation.