Pollen may cause allergies and aggravate your skin to flush, but I doubt it causes breakouts. If it did, everyone would have breakouts. I think a good product to use to make sure you don't have any gunk left behind even after a cleanser is a good toner. If you've tried drugstore toners with no success, try to find a natural one or a one without alcohol. I cannot exactly recommend a specific one unless I know your skin type, but I think it's good to explore starting from the top sellers on the Sephora site. I hope this helps!
I think it's a good idea to make sure your pores are kept clean and unclogged especially when allergens are really bad. Bliss Daily Detoxifying Facial Toner is a gentle protecting and purifying formula that smoothes, moisturizes and provides potent antioxidant protection against smoke, smog, and other environmental toxins (such as allergens). You can find this toner here: http://www.sephora.com/browse/product.jhtml;jsessi
This toner will also help to keep your pores clear of dirt, bacteria, makeup, etc., so it should help to prevent breakouts.
I hope this helps you.