Somehow limiting the number of people who can target a single unit would be one idea. Rationale behind it could be something along the lines of "There is too much interference for your targetting system to lock on" or something. However, this would be easy to exploit by having X number of friendlies lock onto the said target, and that would be a big issue. If there were some way around this, it might encourage smaller groups. Or, it would just make people learn to call targets better.
I like the idea of limiting squad size. Perhaps make the size relational to some skill located under the.. (sorry at work, cant remember the name) teamwork/leadership extensions. Everyone starts with a 5 person squad ability, then each level adds 2 more members, for a total of a 25 member squad. This wouldn't really prevent zergs though, since you could just run multiple squads without too much trouble.
Or, along the same lines as the targeting thing, maybe the more people in your immediate vicinity the longer it takes to lock onto targets (due to some sort of interference) but, this could also be easily exploited by a huge group vs a small group, as everybodies lock times would increase and not just the larger group.
I don't think we will ever see a "roving band of 100 people" walking around beta island solely looking for PVP targets. I think this will only be an issue during schedule corp combat.
I've only participated once (in the battle that crashed all of M2S).. but, having multiple objectives that must be completed and defended simultaneously, fairly spread apart, would not stop a corp from amassing 150 people.. but it might at least split that 150 up into 5 smaller groups. Having say, 7 objectives, and the first to either defend/capture 4 of the 7 is declared the winner.. vs 1 defense as it is now. Or maybe 3 defenses to win and 4 captures to win. Shrug.
Anything that somehow limits a "blob" will be exploitable in some fashion I think, unless that blob can be split up.
Just my thoughts.