Speaking at the BBC Radio Solent fans forum, Pardew insisted: "I would say it is going to be tough.
"But am I not saying it is impossible, I would never say that."
Meanhwile, Pardew has also revealed that he would have no objection in principle to the appointment of a director of football at St Mary's.
"Somewhere down the road a director of football here wouldn't be a bad thing because you do need to continue a vision for a club that goes past managers," he said.
"Managers will come and want to change the vision of the club and if you want a constant vision then sometimes I can see the logic of a director of football."