Goodwill might be an obvious choice if you have older family who used to take you to flea markets on the weekend when you were a child.
I choose/observe the type of person selling me the records and the location. I've come across some very nice people but also some bitter or strange people when buying records. I try to get a sense of the place using those factors before I get comfortable shopping.
I like stopping at used bookstores lately or the few specialized stores left like Big City, I used to go to conventions more in the 90's, I hope to do that more this year. The last one I went to was WFMU here in Manhattan where I saw Chairman Mao, Cut Chemist and Kon of Kon & Amir.
I hope to increase my purchases using mail order because I see stores overseas that have gems, but I always like traveling and getting my fingers dusty.
I like Wax Poetics magazine for inspiration on how other people dig
Check out this blog which features photography of the homes of record collectors: http://www.dustandgrooves.com