I think your problem is related to the access token scopes. I am working on a similar app but i am using the scopes https://outlook.office365.com/IMAP.AccessAsUser.All https://outlook.office365.com/SMTP.Send to obtain access tokens and it seems not to work if I try to use SMTP.Send and IMAP.AccessAsUserAll.

My access token looks like this: {"token_type":"Bearer", "scope":"https://outlook.office365.com/IMAP.AccessAsUser.All https://outlook.office365.com/SMTP.Send", "expires_in":3599, "ext_expires_in":3599, "access_token":"eyJ0eXAiOi..."}

In the azure app config, I just have the Microsoft Graph permissions for IMAP and SMTP.