Right, that’s as intended. If you set all currency values to three digits precision, 33.333 x 3 will equal 99.999.
If you want items to have three digits precision but everything else two, you need to do it via doctype customization, not system settings.