You can use selected-calendars endpoint in Cal.com's API suite to keep track of which calendars should be checked for conflicts.
Find all selected calendars
GET
/selected-calendars
Create a new selected calendar
POST
/selected-calendars
Find a selected calendar
GET
/selected-calendars/{userId}_{integration}_{externalId}
Edit an existing selected calendar
PATCH
/selected-calendars/{userId}_{integration}_{externalId}
Remove an existing selected calendar
DELETE
/selected-calendars/{userId}_{integration}_{externalId}