...
If you only need to see the worklogs of your team, but not of all users in your instance, then you should only grant yourself the "View Worklogs" permission for the teams you need to view the time logged.
...
However, if you need to see the worklogs of all users on issues of a specific Jira project, you will need the "View all worklogs" permission for each of the JIRA projects you are viewing.
Please note that in all cases, the viewer needs to have access to the issues other users have logged time on (Browse Projects permission for the Jira project)
...