The Skull Dungeon resides in a cave in the top-left hand corner of the Calico Desert, accessed by the bus.