Configuring caching affects not the speed of the first site load for a new visitor, but the speed of the site for all subsequent visits and page transitions.
The files are saved in the user's browser and when they are needed again, they are no longer downloaded from your server, but immediately displayed, since they were downloaded earlier.
Many files on the site are very rare and can be cached for a long time, and some files should not be cached at all.
You can check on the popular Google PageSpeed Insights - https://developers.google.com/speed/pagespeed/insights/
If there is a phrase «Leverage browser caching», it does not mean that you have problems with it. You need to expand the description by clicking on the link «Show how to fix», and if there are files located on your domain among the files, then safely order the kwork.
If the PSI resource only shows files on third-party resources (google-analytics, yandex metrika, widgets, pixels, etc. ), then they do not need to be cached either. And we can not influence the cache of other resources!