Japanese prime minister's ruling coalition has failed to secure a majority in upper house vote, Japan's NHK TV says.