You can circumnavigate the island in 3 to 4 hours, depending on your pace. The best hike is along the coastal path heading north of town, where you'll see the Municipio, or island headquarters. Head left here, taking the trail along the north coast that leads past an old cemetery. This hike hugs the steep cliffs on the northern side of the island, part of the marine reserve, and the views are stunning.
Eventually you'll come to Punta di Megna, on the western coast, on the exact opposite side of the island from Ustica village. The offshore rock so appreciated by scuba divers, Scoglio del Medico, can be seen from here.
The road continues along the southwestern coast as far as the battered ruins of the old tower, Punta Spalmatore, where you can go swimming. (There is no beach here, however.)
Below this point, at Punta Cavazzi, along the southern rim of the island, is Piscina Naturale, a sheltered seawater pool and the best place on Ustica for swimming. If there are a lot of tourists on the island at the time of your visit, this "hole" is likely to be crowded with bathers in the briefest of swimwear.
At this point, the route no longer follows the coast and cuts inland all the way northeast to Ustica village once again.