For me regardless of the system I've found four players to be my magic number. This doesn't mean I HAVE to run games with exactly four players, but instead that's my sweet spot. Growing the number to five or dropping down to three doesn't change much so me so four leaves that wiggle room. Any more in either direction starts to really show the pros and cons of having a crowed or barren gaming tables. Yes I did in fact say there were pros to bother having too many people and even the possability of having only a single player.
Going Above Five
I'm pretty sure I;m in the lucky few that tend to have this problem come up somewhat frequently, but thanks to online games allowing for more access to more players I'm sure this is becoming a more and more common problem. I don't like telling people they can't play in my games, especially if they are my friends outside of gaming. Recently this ending up in a game with seven players plus me.
The great thing about this many people playing is it's basically a tiny party when we all get to the table. There's also no really worry of the group not having the tools to handle a challenge I throw at them (unless you have Snow GM and the Seven Murder Hobos at your table). The thing is knowing the party always has every base covered means challenge take a big hit. IF I want the group out numbered I need to throw a small army. When it comes to social, puzzle, or trap type challenges there's always going to be one person built to just walk past 90% of what I throw out there and more often than not a second player that can be back up in case the dice go wrong.
When it comes to combat seven people all taking a turn can very quickly turn into the waiting game even if everyone is quick and only take 5 minutes max on a turn (we all know players that can take MUCH longer) that's still over a half hour till your next turn. That's not to mention my turn as like I said if I want to outnumber the party it means massive amounts of enemies. Narratively this causes a problem too. A seven player game means sever stars which leads to either planning constant spot light moments for everyone or making the adventure the spotlight .
Going Below Three
I actually really enjoy single and two man player groups, but I'm not blind to the problems this creates. Namely that unless the characters are well built and well rounded then what could just be a spotlight moment or a minor challenge for a group of four could grind a story to a halt or worse leave the characters dead in a ditch. Combat is usually a last resort (not something our combat monster players like to hear). On the plus side with only one or two heroes they can shine as actual stars of a story tailored around them. Narrative can take center stage with only a couple players and creating challenge usually isn't too hard (hell like I just said it might be a bit TOO easy).
The biggest issue with smaller groups come in who you play with. I would suggest only going the small group route with players you have gaming history with. In a larger group if you have a flake or a problem player there's room to cut the fat, but in a small group cutting the fat might mean cutting the game.
The Wiggle Room
I stand by that four really is the best number with three and five being good, but not perfect. I didn't throw three and five player groups under the bus though because like I said they are just fine and more importantly a four player party can become either of these in a heart beat. The real magic on four is that if a friend shows up and someone gets sick you can keep trucking along. Really you usually don't even have to change what you have prepared, just as long as everyone understands things might be a little easier or tougher because of the change. Sure if you planned a session to require a specific player to be able to do something and they don't show up that's a big wrench, but a good GM should have at least two alternate options in case the party doesn't want to do what you expect anyways.
The short and simple of it when it comes to group size in my opinion is built into how much you want your tabletop game to feel like a war game. Bigger groups lose narrative, but their combat ability is buffed meaning you can throw epic battles their way lesser groups wouldn't be able to handle. Smaller groups turn combat deadly, but let the story shine and focus of personal development. For me I like a balance and I find that balance at four, the magic number.