Interesting ceramics are for sale at Hephaestus, on the waterfront at 21 May; Alexander's, on the corner of Plateia Museio (the square 1 block back from the waterfront); and The Mistral, 6 Vironis, up the hill opposite the post office, offering the work of the potter/owner.
For a taste of the local cuisine, consider Kefalonia's prized Golden Honey, tart quince preserve, or almond pralines. Another possibility is a bottle of one of Kefalonia's highly praised wines. You can visit Calliga Vineyard (selling white Robola and red Calliga Cava) or Gentilini Vineyard (with more expensive wines), both near Argostoli; or Metaxas Wine Estate, south of Argostoli. The tourist office on the waterfront will tell you how to arrange a tour.