Clem's Café (tel. 802/775-3337) is a casually hip cafe housed in a former bank, right downtown at 100 Merchants Row. Grab a seat at a sidewalk table or move inside and choose a room. (The bank vault is tiny and nicely painted, and you can have a lively conversation there -- with an echo.) A good selection of coffees is available, plus baked goods such as banana-nut tarts, croissants, and cheese Danishes. It's open Monday to Friday until 2pm, until 1pm on Saturday and Sunday.
And, for families, there's an outlet of the popular northern New England fish-house chain Weathervane (tel. 802/773-0382), which I happen to like (and find that kids seem to enjoy, too). It's located at 124 Woodstock Ave., which is Route 4 a few blocks uphill (going toward Killington) from the central crossroads of Routes 4 and 7. How they get fresh fish all the way from the Maine coast over here to Rutland every morning is beyond me, but it's worth checking out.
Note: This information was accurate when it was published, but can change without notice. Please be sure to confirm all rates and details directly with the companies in question before planning your trip.