Only a half-hour's drive from Kampot is the place that was once considered the Cambodian Riviera in the time of the French, continuing under the rule of Sihanouk. Indeed, Sihanouk's old seaside villa still stands, though he never actually stayed in it since the area became unstable before the curtains were chosen and the place was furnished. Now it is crumbling and pocked with bullet holes, but one can easily imagine how grand it once could have been. Kep (and indeed much of southern Cambodia) was a fierce battleground over the years of war and the derelict feel still remains, although developers are moving in fast and things are changing. There is very little beach at Kep and what there is remains pretty scruffy and stony. There are some secluded resorts that appeal to some, but at present facilities are limited. Kep is better done as a short, relaxed day trip from Kampot, unless it provides the particular kind of seclusion you crave. It is a pretty drive from one to the other.
Where Kep is a world beater is when it comes to seafood. They fish some of the best crab in the world off Kep, and Kampot is renowned for producing the best pepper on the globe. The killer recipe doesn't take much working out and Kep crab in Kampot pepper lives up to anyone's expectations. There is a strip of restaurants right on the beach road coming from Kampot (it is signposted) where the crab is unloaded. This place is called, not surprisingly, the crab market.