As b3mwe says, don't lose heart. Most "meets" you'll see on the forums are fairly small anyway. Don't worry if you're not acheiving the McRae Gathering... the rest of us are human too (well... mostly!)
Keep it small, keep it regular, set a date and venue and stick with it. Don't wait for folk to suggest dates and times... they won't generally. Set it up and let folk come to you. If you get a lot of interest, but people are suggesting another date, THEN consider changing it.
The Tayside meet has a core of about 6 or 7 very regular attendees with a few others who dip in and out. But it started with just a couple of us... so keep at it!