Last years Vic swap meet was rather disappointing, I thought. There were a few dozen tables- mostly old codgers who thought their Nikon F3s were worth $5000. Kudos to the organizers, but it was hardly an earth-shattering event.
Unless you're from Victoria- and it doesn't sound like you are- it's not worth the 60 dollar (each way, with car) ferry ride from Vancouver out to the island.
Edit: I will take this chance to plug
Camera Traders, though. Totally a bang-up outfit, and if you decide to go to Vic anyway, make sure to check them out.