So I played this game a lot when it first went Early Access, then nothing until final release. Anecdotally, everyone I've ever talked to about the game thinks you should keep your light at 75-100 pretty much always; you get better chances of surprising enemies and of scouting the area ahead, and most importantly you incur almost no stress from simply walking around. The trade off is that you get slightly better/more loot when your light is lower (for some arbitrary reason) but you also get surprised more by enemies, and more stress penalties from the darkness and maybe something I'm forgetting, and it just doesn't seem like there is any kind of significant reason to not keep your torch constantly topped up. The cost of torches when you deploy is really quite low, and they don't take much inventory space.

I feel like I really don't get what the devs were going for with the torch mechanic, because I just keep it constantly at 75-100.

I think the bonus loot is well worth the raised difficulty. Especially when training lower leveled characters in small beginner dungeons I usually go with 0 torchlight to get the most out of these runs.

Given how quickly the torch level can go down I get the feeling they're essential for long runs. I've tried medium runs and have had to think harder about their management. For short runs they are kinda easy to spam.

Low light and no light runs are totally a thing. You get a lot more money and loot when there is no light. When I was playing a couple weeks before release, I was frequently running out of inventory space not even halfway into a mission because of the increased stuff you find. That never happened when I was still playing the "keep the torch maxed at all times" game. And its a great way to train and weed out new recruits. If they die it doesn't matter, so you really get the most bang for your buck. I'm not going to claim that I'm some master of the strategies in the games, but no light runs felt invaluable on my first attempts at the game. How best to balance that is something you'd have to put some thought into though. In my opinion if you are keeping your torch up high every run, you're not going to progress as quickly as you could otherwise, and you may miss out on some cool equip-able items that you would have to pay a small fortune for in town. You also find a lot more of the materials that you use to upgrade places in town, so that's quite helpful as well.

Eventually once you get some guys who have so many negatives that you just don't want to spend the money on them, assign them to a suicide squad and put them through tougher dark light runs. If they die, they'll get you some good stuff on the way and you don't have to deal with the cost of removing their penalties. If they don't die, great! Send them out to suicide again for more stuff.

I don't know why you would want to keep torches maxed anyway unless you're afraid of losing people. You get no real benefit aside from easier combat which only helps you in the short term of that particular mission. The only time my torch is maxed anymore is when I'm going on a boss run, or if I'm doing poorly on the mission because of rng.

I've found that there is definitely a tangible increase in loot by keeping the torch lower, so I'm not quite as quick to top it off unless I run into an encounter where the extra physical and sanity damage from monsters would make a difference.

I guess it makes me think of XCOM. If you take your sweet time upgrading in that game (maybe you retreat from missions and don't recover resources or capture anything) you fall behind: Sectoids become Floaters become Mutons, harder and harder to kill, so you need to get your ass in gear and get some laser weapons and better armour pretty soon. If you take your sweet time in Darkest Dungeon (lower rate of heirlooms and gold due to high light), kinda nothing happens to punish you on the meta layer, because you can generally keep doing low tier and mid tier missions indefinitely. You'll upgrade the town a little slower, but higher light lets you avoid unnecessary risks when it comes to your party getting surprised or getting stressed, at really no penalty other than time.

You get much better loot (both trinkets and gold) and all crits do a lot more damage. if you tool up a high damage group with high crit chance, low light quirks and good low light trinkets then you deal massive damage and get big returns. Like everything in this game it's all risk reward.