The Japan Meteorological Agency said that up to 13 inches of additional rain are expected through Thursday in parts of the southern main island of Kyushu.
The country's Fire and Disaster Management Agency issued a directive for more than 1 million residents to evacuate to designated shelters in three southern prefectures. Directives are stronger than advisories, but there are no penalties if people do not comply.
"The sound of the rain is so strong that it worries me," one evacuee told Kyodo News.
All 590,000 residents of the city of Kagoshima were told to evacuate and were seen filling evacuation shelters.
Heavy rain has continued in southern Japan since Friday, killing an elderly woman in a mudslide in Kagoshima. The rain also has flooded dozens of homes.
Some parts of southern Kyushu have seen nearly 40 inches of rainfall since Friday, according to Japanese broadcaster NHK.
Deputy Chief Cabinet Secretary Kotaro Nogami said 14,000 ground troops were standing by for emergency rescue and search operations if needed.
He also urged residents to use caution and try to evacuate early "to protect your lives" rather than waiting until the last minute.
A stalled front has left extremely humid air sitting over the region, bringing more rain to already saturated ground, according to NHK. Officials about the possibilities of "deep-seated landslides" that can cover large areas and be over 300 feet deep.
Officials echoed the warnings issued last July when heavy rain led to mudslides and flooding left more than 200 people dead.
The Associated Press contributed to this report.