Well, Sarah did actual say "Is she Raven's mother?" rather than "She is Raven's mother!" so yeah she's guessing based on what she knows, she only knows about Adrian being an elf, there may be other elves in the world, but Adrian's in Moperville, Box is in Moperville, and Box is responsible for the ambient energy and magic marks in Moperville. That would be enough to make anyone wonder if they're related. Edward knows Adrian as well and he might have been introduced to Pandora as Adrian's mother, but he might not realize that the child that appeared in the woods and on TV is her. I would suspect that if Edward did know that it was Adrian's mother causing problems in Moperville, he'd be on Adrian's case to get her to stop.
This just occurred to me, If Grace has told Sarah about her experience with Adrian and the boar, being under an illusion that made her look a lot like Susan, and being told that Elves can't have children. Sarah is in a position to guess what Box was talking about, the lie she passed on, the "of course they were his!" line. Though Sarah might jump to the conclusion that Adrian is Susan's dad which might not be too far from the truth (more likely grandfather or maybe great grandfather) but Sarah doesn't know enough in this case to be as accurate.