The music mogul was raced to hospital and had to the first live shows of the current series on Saturday and Sunday.
Now the Sun’s Bizarre column can reveal he is feeling well enough to start work again on the ITV programme.
A source close to Simon, 58, said: “He’s feeling much better now.
“He didn’t want to race back and hinder his recovery – especially given doctor’s orders.
“He’s already started working with his acts and has met up with them all to prepare for the weekend’s performances.
“Everyone knows how much of a workaholic Simon is so he’s obviously gagging to get back into the swing of things.”
In the early hours of last Friday the X Factor boss fell down stairs in his London home after fainting as he went to fetch some milk.
He was alone with son Eric, three, whilst partner Lauren Silverman was in the US with her children.
Simon was taken to hospital where tests were done on him and he was released later that day.
He was advised not to fulfil work commitments that weekend and former guest judge Alesha Dixon filled in for him on Sunday’s show with just Louis Walsh, Sharon Osbourne and Nicole Scherzinger on the Saturday’s programme.
Speaking to the Sun after the fall, Simon revealed he was told by doctors the fall was down to low blood pressure and insisted he felt lucky to still be alive.
He said: “Sometimes we get a reminder that we’re not invincible and this was certainly mine. It was a huge shock.
“They think I fainted because I had low blood pressure and so I have got to really take good care of myself to sort that out.
“After all I am a dad and have more responsibility than ever.”
Recalling the accident in detail for the first time, he said: “I’d gone to get some hot milk because I felt ropey. On the way back upstairs, I just remember feeling really dizzy.
“Next thing I know someone was putting a neck brace on me and I had a terrible headache, which must have been from me hitting the stairs. I was worried at first that I’d done some real damage.
“But I’m on the mend now. I know I was very lucky I didn’t hurt myself seriously.
“It could have been a lot worse. I must say, everyone at the hospital were incredible. I’m truly grateful.”