The 45-year-old actress will portray Queen Elizabeth in the fourth and fifth series of the regal saga and she admitted it's tough following in the footsteps of Claire Foy, whose turn as the monarch won her several awards.
Discussing replacing Claire, Olivia said: "It's horrendous.
"Everyone loves Claire Foy, so I have got the worst job in the world at the moment. You're saying all the worst things, thanks!
""It's the same as any classical play you do -- everyone will have already played that part before."
And Olivia admitted she couldn't help but do "Claire impressions" during her first days on set - but doesn't think that was necessarily a bad thing.
She admitted in an interview with Entertainment Weekly: "The first week, I did feel myself trying to do Claire impressions. 'What would she have done?' "
But as the series picks up right where series two ended, Olivia jokingly added: "Maybe it was a stroke of genius that I was doing that. That's what it was! Yes, that was preplanned!"
The 'Favourite' star was thrilled to be offered the lead role in the show and didn't even try to play it cool when producers approached her about the part.
She admitted: "I was incredibly uncool about it.
"The producers went, 'So...' [I said,] 'Yes! Yes! I'm really excited! Thank you very much!' I loved the first two seasons."
Showrunner Peter Morgan admitted he only ever had the Oscar-winning star in mind to take over as the queen.
He said: "Olivia Colman was a list of one. I think I wanted to know [she would play the part] even before negotiations were done for seasons three and four."
However, he confessed it wasn't ideal that her performance as Queen Anne in 'The Favourite' would come out first.
He added: "Obviously I'd have preferred her not to be playing another queen before.
"But it's so different -- such a different tone."