Swede Robin Soderling also found the going tough before staging a remarkable fight-back to beat Spaniard Fernando Verdasco 2-6, 7-5, 6-4.
Fourth seed Murray, knocked out in the third round of the Madrid Masters last week by unseeded Thomaz Bellucci, struggled for rhythm on his serve and was made to sweat by his aggressive Belgian opponent before winning in just under two hours.
"It was a good win but it could have been better," Murray said.
"I need to keep improving. It's about getting into the right frame of mind not just for every game but for every point."
After saving two break points in the third game of the first set, Murray broke Malisse's serve in the next game and another two times to take the first set.
"I started well but then began rushing through my service games," Murray said.
"On clay I feel you always have time to get back into a game. Even when I was losing games I felt I was hitting the ball well."
But with the Rome crowd rooting for underdog Malisse, Murray dropped the first two games of the second set and was then broken twice more as the Belgian pounced on the Scot's short second serve and poorly executed drop-shots to square the match.
In the darkening skies in the Italian capital, Murray cut an anxious figure, slamming his racket on the ground, gesticulating and admonishing himself without a coach to look up to.
But the Briton, who has never lost to Malisse, took the first two games of the decider and then held his composure to improve his first serve average to 50 per cent.
After breaking Malisse, ranked 41st in the world, yet again, Murray finished with an ace to set up a meeting with either Victor Troicki of Serbia or Italy's Potito Starace.
Earlier in the heat of the day, seventh seed Tomas Berdych made the third round by beating Argentine Juan Monaco 6-2, 6-2.
The Czech was joined in the last 16 by 16th seeded Frenchman Richard Gasquet, who crushed Igor Andreev 6-1, 6-2 to set up a tie against third seed Roger Federer or Jo-Wilfried Tsonga.
Paolo Lorenzi stirred the crowds with a convincing 7-6 6-3 upset win over Brazilian Thomaz Bellucci, who last week reached the Madrid semi-finals.
The gutsy Italian qualifier's reward is a meeting with top seed and champion Rafa Nadal for a place in the last 16.
-ReutersTags: sport, tennis, italy First posted May 11, 2011 14:38:00