This can be perfectly normal.

There is an 'inactivity timer' setting on the token that normally defaults to "0". In the installation instructions, we recommend changing that value to "2", though certain organizations may have slightly different requirements, so you may want to verify with your local System Administrators.

The retrieval process can ask you to enter the token passphrase up to 6 times while the token is going through the certificate retrieval process. If your 'inactivity timer' is set to "0", then you will be prompted every time the token needs to be accessed.

Note: If you are entering the passphrase incorrectly, you will see a message that specifically indicates the passphrase entered was not correct.