If I remember correctly, people experienced it with 2-3 participants too in the past. But due to the nature of how things work, it can depend on a lot of things, mostly your "natural" lag and the plant updates sent within a given time.
That's also the reason why not all people experienced it. If your natural lag was low enough, your client was able to cope with larger amounts of incoming data too. But if it was relatively high, the large amounts of plant data basically clogged up your client, it could not keep up with realtime updates, which delayed all the rest of the network data, and it all went downhill from there. (At least that's how I understood it as a non-network expert.)