for hand quilting i use regular old pencils, sharpened to a fine point.
i've never had it not wash out, though some people say that happens.
handquilting for me means the quilt...is more precious. i just recently used a frixion pen and they work beautifully, but some part of my gut just does not trust them. the chemical stays in the fabric, as evidenced by its return when the quilt gets cold. i have used them for some projects though. the water soluble blue pens work really well too, but they run out quickly and the price can really add up. (and with both of those methods i always feel a bit guilty about the landfill waste i am creating...but i am a little weird about that stuff.)
i have also used ivory soap when handquilting on dark fabrics. i mark as i go with that, so the pattern has to be carefully thought out. it works like a charm, but it's a bit hard on your hands after a while.
aileen