What is downtime? Technically, downtime is the time between each player's two turns. Say you're playing Chess - when it's your turn to move, you're actively engaged in thinking about the game. Then you make your move, and you look at your opponent - it's your time to wait. Sure, you may be thinking of your next move and consider possible board states, but you can't really make a decision until you know for sure what your opponent chose to do, and you probably already consider some of their possible moves when you made yours. In other words, you are waiting. This is downtime.
Naturally, gamers and designers are always after games with low or no downtime. Though this is quite a consensus, once you start thinking you see that actually downtown can have different meanings and it bothers people to different degrees. Some people think players actually need some downtime, to think about their turns in strategic games or take a little rest in intense games (like real time games). We can now see clue that the issue with downtime is not merely about time but also about timing: how much of a game feels like downtime depends on whether a break in the action is welcomed or not. Downtime is therefore not just the time between a player's turns but more precisely, the time in which a player isn't actively engaged. Games with similar turn length could have different amount of downtime, depending on how much you engaged a player is during another player's turn. A player may be engaged during another player's turn because they're preparing for their own turn, enjoying the spectacle of their friend's turn, anxiously following to see its impact on their own position or taking a needed break after an intense turn.
Short turns is the way Scythe tries to work around the issue of downtime (as the formal game description tells us), as the action-selection system only lets you choose one action during your turn. But making turns short is not always a solution and I think Scythe is a good example of that's point: turns can actually be too short, and if they are too short they don't feel quite satisfying. The upshot is that downtime is not just a function of how long it takes for other people to take their turns - it's also a function of how fulfilling each turn is. In Star Realms, for example, players build their engine throughout the game and then, at some time, you get an explosive turn where everything comes together and you play half your deck, blowing through anything. After such an epic turn, players will have more patience to wait for their next turn, and will be happy to take a breather. But with Scythe, even when my engine gets churning - I can still do only a little thing every turn and you never really get that epic turn (unless you attack in multiple places). That makes the difference between four and five players much greater than it would have otherwise been.
There is another factor that affects the experience of downtime and that is interaction. Since downtime is not just a matter of time but of engagement, if you're really engaged during other players' turns, it won't feel like downtime. In Magic: The Gathering and other collectible card games you can actually interrupt another player during their turn to play a card or cancel their effect. In war games you will typically be sitting on the edge of your seat during other players turns, waiting to see where or whether they choose to attack you. But there's more and more games that choose to create meaningful interaction in non-confrontational ways, borrowing from the rising tide of Eurogames. Scythe is one of these games, where fighting occurs rather infrequently and interaction with other players comes partly from positioning on the board but mostly through a variety of intricate indirect ways, such as the way some actions your neighbors take may give you resources. Yet the most important way other players affect your play in Scythe is the fact the game ends when someone unlocks all their achievements. In practice, the game ends when someone decides to end it - which means you have to make sure your game plan aligns with the game's time horizon as it might end before you are ready.
The bottom line is - if you want to avoid downtime in your game, you have to think not just about the time but also about other dimensions of the experience of downtime - timing of breaks, fulfillment during players' turns, and engagement during other people's turns.