Firstly: 

If a person forgets and says when bowing “Subhaana rabbiy al-a‘la” by mistake, or he says when prostrating, “Subhaana rabbiy al-‘azeem”, it is one of two scenarios: 

(i)                He remembers that he did not say the dhikr in the right place, in which case he should say “Subhaan rabbiy al-‘azeem” before rising from bowing, and he should say when prostrating “Subhaana rabbiy al-a‘la” before he sits up. 

In this case he does not have to do the prostration of forgetfulness, because he did not omit something that is obligatory; rather it is mustahabb for him to do the prostration of forgetfulness, because he did not say the dhikr in the right place. 

(ii)              He does not remember that he did not say the dhikr in the right place until after rising from bowing or prostration. 

In this case he has to do the prostration of forgetfulness because he omitted something obligatory. 

In this case the prostration should come before the salaam. 

We have previously discussed this issue in the answer to question no. 7743 

Shaykh Ibn ‘Uthaymeen (may Allah have mercy on him) said: If a person says a prescribed phrase in the wrong place, it is prescribed for him to do the prostration of forgetfulness, such as if he says “Subhaana rabbiy al-a‘la” whilst bowing, then he remembers and says “Subhaana rabbiy al-‘azeem” In this case, he said a phrase that is prescribed, namely “Subhaana rabbiy al-a‘la”, but it is prescribed to say “Subhaana rabbiy al-a‘la” when prostrating. So if he says it when bowing, we say: You said something that is prescribed in the wrong place, so doing the prostration (of forgetfulness) is Sunnah in your case.

End quote from ash-Sharh al-Mumti‘, 3/359 

Shaykh Ibn Jibreen (may Allah have mercy on him) said: If he says a prescribed phrase in the wrong place by mistake, it does not invalidate the prayer… so if he recites Qur’aan when he is sitting or recites the Tashahhud when he is standing, or he says “Subhaana rabbiy al-a‘la” whilst bowing or “Subhaana rabbiy al-‘azeem” whilst prostrating, i.e., he does something that is Sunnah in the wrong place even though it is prescribed, then it is Sunnah for him to do the prostration of forgetfulness, but it is not obligatory, because these phrases all come under the heading of adhkaar al-salaat (dhikrs recited during the prayer), so the prayer is not invalidated by doing so.  End quote from Sharh Akhdar al-Mukhtasaraat 

Secondly: 

If the worshipper was praying behind an imam and did any of the things mentioned above, then he should do the prostration of forgetfulness at the end of his prayer, if he joined the prayer late. If he offered the entire prayer from the beginning with the imam, then he should say the salaam with the imam and he does not have to do the prostration of forgetfulness. 

Shaykh Ibn ‘Uthaymeen (may Allah have mercy on him) said: If one who is praying behind an imam makes a mistake in his prayer, and he did not join the prayer late, i.e., he did all the rak‘ahs with the imam, such as if he forgot to say “Subhaana rabbiy al-‘azeem” when bowing, he does not have to do the prostration of forgetfulness, because the imam has borne it for him. But if we assume that the one who prayed behind the imam made a mistake that caused one of the rak‘ahs to be invalidated, such as if he forgot to recite al-Faatihah, then in this case, when the imam says the salaam, he has to stand up and do the rak‘ah that was invalidated because of his mistake, then he should recite the tashahhud, say the salaam and do the prostration of forgetfulness after the salaam. 

But if the one who prays behind the imam makes a mistake in his prayer and he joined the prayer late, then he should do the prostration of forgetfulness, whether his mistake came when he was praying with the imam or after he stood up to complete what he had missed, because if he prostrates he will not be going against his imam, because his imam will have already finished his prayer. End quote. 

See: Risaalah fi Ahkaam Sujood as-Sahw, by Shaykh Ibn ‘Uthaymeen (may Allah have mercy on him). See also the answer to question no. 35909. 

And Allah knows best.