I think the COTF were much more numerous and powerful in the times when the drawings beneath Dragonstone were made. Their powers in those days may have allowed them to reach Dragonstone. Perhaps they flew dragons, or other creatures that no longer exist.
The “Old Gods”, of which the COTF are a part, have been on the wane for quite sometime. They just don’t have the mojo they once had. In fact, I think the presumption is that we the Ice Walker assault on their one remaining enclave they are all dead now.