You probably won't want to linger in Karavostassi, perhaps the least enticing port in the Cyclades. Best to hop on a local bus and chug the 4km (2 1/2 miles) up to Hora.
Cliff-side Hora is centered around five closely connected squares, connected by meandering streets, lined with houses, restaurants, and shops. Even from the bus-stop square, the sheer drop of the cliff offers an awesome sight. On the right in the next square, you'll find the Kastro: two narrow pedestrian streets connected by tunnel-like walkways, squeezed between the town and the sea cliffs, with remnants of the medieval castle. One majestic church, Kimisis Theotokou, dominates the skyline of Hora. It's particularly beautiful at night, when it's illuminated. Built at the highest point in town -- and with fine views over the island -- it stands on the foundations of the ancient Greek town. Townspeople parade through the village with the church's icon of the Virgin with great ceremony each Easter Sunday. The other church to see here is the deserted Monastery of the Panagia, north of town, also with lovely views.
As you rush from island to island, checking in and out of hotels, it's not always easy to feel the rhythm of island life. One great way to do that is to visit Folegandros's small Folk Museum, in Ano Meria. If you want to see some countryside, head west from Hora to the village of Ano Meria. This tiny hamlet of scattered farms is the island's second-largest village. Some of the tools and household items on view in the museum have been used for generations, and you can see some still in use today. The museum (no phone; free admission, but donation appropriate) is open 5 to 8pm weeknights in July and August, and the local bus can drop you a pleasant stroll away. If the museum turns out to be closed, console yourself that this, too, is an insight into the rhythms of island life!