The group I ride with varies in size, depending on availability. Like who's working or laying in hospital, broken.
Some of us ride like complete loons and get there before the others, some of us ride in the slower group and meet up where we're going. There's always two groups of us, it works fine, the plodders know they don't have to ride like the hooligans so they don't even try.
I don't mind the size of the group just as long as they don't mind if some go fast or some go slow, just ride at your own pace.