Remote play lag absolutely isn't doing you any favors. While the input latency might not be as impactful for other genres, for something like a rhythm game where the timing windows are usually open for a fraction of a second, that lag is likely enough to break your prospects, even if your sense of rhythm is otherwise accurate. Bear in mind, the image you're seeing (and the audio you're hearing) on your remote play device is slightly behind what the actual PS4 is outputting because of delays in video encoding and transmission, so you hitting in time to what your remote play device is displaying is consistently going to be too late according to the game's logic because the actual console is already ahead of what you're seeing. Even if you try to compensate for this, it might not always work because of fluctuating network and encoding conditions, resulting in lost controller inputs.
Basically, I'd try playing on an actual TV and compare the difference if I were you. You might not suddenly get perfect scores, but you'll almost certainly notice a night and day in difference in terms of what you get, even if you find some stretches of songs tricky still.