I think it can be done with public transportation, but it will be complicated, since there is no train through the Dolomiti area itself, and I don't believe there is one bus that will get you from the east to Ortisei (or some other western town). You can go by train from Venice to Belluno, but after that you'll need a series of bus connections. For details of how to do that I would recommend that if no one here can help that you ask on the TripAdvisor message forums for Cortina d'Ampezzo and Ortisei. You might also try the Thorntree forum on Lonely Planet, because its participants tend to skew toward people on a tight budget, and they very often know how to ace all the bus schedules.
You might also look into the possibility of taking a train from Venice to Belluno, then a public bus to Cortina d'Ampezzo, and after that taking an all-day commercial coach excursion that travels the Grand Dolomites Road. Such an excursion might get you all the way to Ortisei or Bolzano, with a lunch stop. It will obviously not be anywhere near as cheap as the public bus, but might be worth the comfort. There is a train station in Bolzano, so from there you can keep going by public transportation to Lago di Como (it will be a full day of travel). But again, I am not sure what is available, if anything, in the way of long coach excursions in that area. Never done it myself. (Another forum to try if you are interested: Rick Steves forum)
But unless you are really very averse to driving, you might reconsider the idea of getting a car. You can rent one in Venice or Belluno and if you stay on the Grand Dolomites Road from Cortina d'Ampezzo to Ortisei, it is a beautifully engineered road that is not difficult or exhausting to drive. You can do the entire road in less than a day, with stops for lunch and photo taking. After a night or two in Ortisei, you can drop offf the car in Bolzano and take the train onwards, or continue with your car to take a scenic route to Lago di Como (but I have never done this, and don't know if the drive through those mountains is difficult).
Two other things I'll add: Late September is already outside the optimal time period for dry, clear days in the Dolomiti. The risks for rain and overcast in the Alpine north mountains begin very early in September. The spectacular unique peaks of these mountains -- which is what you are there to see -- can be completely invisible if there is a low cloud cover or rain. There is literally nothing to see. Of course in some years the late September weather is picture-perfect, dry and clear. You just can't know. If at all possible, book hotels with a great cancellation policy, and decide when you are in Venice if it makes sense to head to the Dolomiti given the immediate weather forecast. Of course, if it is raining up there and you decide not to go, then you also need to have booked a Plan B itinerary to do instead. Fortunately the train line between Venice and Milano is overloaded with beautiful historic towns, several with fascinating historic gardens -- the hardest part is chooosing.
Final thing is if you are using the train from Bolzano or Trento to head to Bellagio in Lago di Como, you can get leave the train in Bergamo and switch to a commuter train that will take you to Lecco on the eastern shore of Lago di Como. From Lecco you can take a ferry to Bellagio. to work out the details for making connections, try the Tripadvisor forum for Bergamo.
Hope you have a great trip!