The *big* difference is the focus breathing which is really in favour of the f4 at closer distances. Its much lighter (shouldn't make a difference for portraits), has better VR (shouldn't make much of a difference unless you are shooting available light without tripod somewhere dim = use the 2.8 anyway), sharpness is equal or slightly better than f2.8 (even with 2.8 stopped to f4, but again shouldn't make a difference as they are both very sharp lenses). If you aren't shooting at f2.8 then the f4 is the better option. Since I don't do either of those types of shooting, I'm moving to the f/4 and adding the 85/1.8 with the savings for some additional creative options. On the other hand, there isn't much savings and the f/2.8 is clearly superior for sports and events overall (where you need all the light you can get and you want shallower DoF at longer focus distances). If you're only worried about portraits, I feel that the f/4 version is just as good. It is more convenient for working in close quarters and doing closeups.
One of the main reasons that I am selling my 2.8 is because of the shorter minimum focus distance and higher magnification of the f/4 (while still being just as sharp wide open). Simply put, because of the f/2.8's focal length breathing properties and the f/4's reduced amount of breathing, if you have the space to back off a bit and use a little longer focal length with the f/4, you will actually get the same or even less DoF at f/4 compared to the 2.8 f/2.8. The best very accurate description is here: It is a complicated subject, but the short answer is that yes, as a portrait lens the 70-200/4 is actually pretty much just as good as the 70-200 f/2.8 VR II due to differences in focal length breathing.