Contacts are in the .pst file which is the message store for Outlook.
Outlook does not prompt to separately access the contacts in its message
store. It may prompt to open the message store itself if you have
passworded it.
You installed add-ins or macros that are attempting to access your contacts
list. The security model in Outlook will prompt when something wants access
to your contacts (which means the add-ins were not properly coded for the
security model or are unsigned macros that you didn't grant permission to
have access).