If the developer has a big budget, they probably have the resources to bring in some random people to play the game, and to get feedback on whether they think the difficulty is too hard, too easy.
But I think most developers look at whatever the QA testers can do (in this case, time trials), and just lower expectation a little bit, because most people won't be able to compete with testers who have played the same game day in, day out for like 6 months.
I remember listening to the in-game commentary for Star Wars: Rogue Squadron II, and there was a part where the lead designer briefly talked about the first Rogue Squadron, and he basically acknowledged that they lost sight of what was reasonable and started listening to feedback from the testers way too much, to the point that some of the missions in the last act of the game are of completely ridiculous difficulty, not to mention nearly impossible to get a gold medal on.
Log in to comment