Madonna's lawyer appeared in court in the African nation's capital of Lilongwe (03.04.09) expecting to finalise the adoption of four-year-old orphan Mercy James.
But Judge Esme Chombo rejected Madonna's case because the 50-year-old star has not lived in Malawi for 18 months, as is required by law.
The residency rule was waived when Madonna adopted David, now three, from the country in 2006, and it is not clear why Judge Chombo ruled differently.
A Malawian official previously hinted the judge had already told the star she would be granted custody.
Madonna is expected to appeal the ruling to Malawi's Supreme Court.
In papers lodged at the court last Monday, Madonna promised to love Mercy - whose real name is Chifundo - like she was her own daughter.
She said: "I am able and willing to securely provide for Chifundo James and make her a permanent and established member of my family.
Application endorsed
"To deny Chifundo James the opportunity to be adopted by me could expose her to hardship and emotional trauma which is otherwise avoidable."
Malawi's child welfare minister also publicly endorsed Madonna's adoption application.
She said: "We have close to two million orphans in Malawi who need help. We can't look after all of them as a country. If people like Madonna adopt even one such orphan, it's one mouth less we have to feed."
Madonna arrived in Malawi on Sunday, March 29, with her children Lourdes, 12, Rocco, eight, and David.
She had been expecting to fly her family - including her new daughter - to her home in London, where staff has been preparing for the new arrival for months.