I love brown in quilts, clothes, etc. Maybe at some point those people who don't like brown quilts were frightened by a really bad one. But if they will just look at yours, they would fall in love with them.
My husband loved brown, blue, and peach combined in a quilt!