I am not sure, but I don't think people are allowed to wander into American federal buildings and start shooting photos. Maybe if you are on a guided tour, but not self-guided tours. I could be wrong. See how little I know about my own country?
Anywho, when in Victoria, it's well worth a quick stop inside Parliament to check out the area that's open to tourists. It's never crowded. There are hourly tours, but if you are pressed for time, you can just go inside and wander around and take photos. The stained glass is amazing, but the windows have been placed where they are in both bright light and dark shadow from the tiny courtyards and light spaces behind the windows. It's a massive building to walk around from the outside too.