Or maybe the ghosts don't see you because you're small. Of course, ghosts generally walk through people by accident or if they're particularly mean. Were they referring to possession when they told you that?

((Heeey, it's our little, somewhat-creepy buddy!))