I agree with all of this. A player in a Forgotten Realms game already knows, in broad strokes, how the pantheon of gods interact with the world, the basic cosmology of the planes, and has an understanding of the major bio-forms and how they act/interact.
Playing in Skippy’s Realm means the players need to spend time and energy learning these things, and that can really break the narrative for players and GMs: “Wait, are aqua-dogs friendly or aggressive? Do we all believe in the gods, and do they interact with us? Why do the Mousefolk hate the Lemurfolk so much?” etc.
For me, as player, having a wholly new world is hard to role play, because I don’t know what my character would already know, having grown up there. That’s not to say it can’t be fun, and that exploration and discovery isn’t rewarding. It’s just a different game.
As a GM, if I build a whole world in my head, it can be frustrating some times trying to present its rich nuances to players. They may take a very different vibe than what I had in my head, which can cause some unintended friction.