The plugin can check once a day, by signing in to your mail service and straight out again, with nothing sent. If the service refuses twice in a row, ten minutes apart, a notice appears on every wp-admin screen and you get an email, usually before any customer's message has failed. You can run the same check any time with Send a test → Check the sign-in now, or wp email-delivery check from the command line.

The daily check is on for new installs. If you had Email Delivery before it existed, it stays off until you switch it on: the Dashboard offers it under Worth doing next, or use Settings → When it does not arrive.

If the notice says This site could not connect to its mail server instead, the password is not the problem: the site never got as far as signing in. The mail service may be down, or your web host may be blocking the connection (see "The test says connection timed out" below).

If it says The mail server did not offer this site a sign-in, the site connected but was never asked for a password. Set Encryption to TLS (or SSL on port 465) in Settings, because Gmail and Microsoft 365 offer a sign-in only once the connection is encrypted. If it is already set, your web host may be sending mail through a server of its own; ask them.

It checks the sign-in and nothing else. A service can accept the sign-in and still refuse a message, and that is reported when a message fails.

How do I know my mail password still works?