CiviCRM Community Forums (archive)

 * The dedicated server and VPS sound pretty comparable spec-wise and appear to be in the same price range (~$120-~$140/mo).
 * The time/labor cost of switching from your shared server to VPS or dedicated is probably the same.
 * If you were more concerned about consistent throughput/performance, then the dedicated hardware would be a safer bet.
 * The ISP has probably planned more effectively for managing hardware failures on the VPS than on dedicated hardware (given the same price-point). For example, they list redundant hard disks (RAID-10) as a standard feature of the VPS; on dedicated hardware, it's an add-on ($30-$80/mo). More generally, if the VPS host goes down, it will affect multiple customers, which can be a good motivator.
 * The hardware sounds OK for most of your stats. I'm a little nervous about the 100k contacts, though, because I don't have personal experience at that DB size. I can't say if you'll notice the difference between 3GB and 4GB. Perhaps someone else can comment on whether 3-4GB is OK for 100k contacts.

Personally, I'd probably go for the VPS -- but (a) keep notes while you setup/transfer the hosting and (b) only make a 1 month commitment. If it works out in the first month, then extend to a yearly commitment. If there are still performance issues (eg b/c the ISP has crammed too many customers on its hardware), then you could switch to dedicated. Making the switch a second time would be easier because you'd have your experience+notes.