This happened to me, last month, when I was flying home from the East Coast.
My United flight was completely full. Although I had a seat in one of the last rows, I ended up in the last boarding group since I had an aisle seat and United boards window seats first and then moves inboard.
When I got to my row, the window seat was still not occupied and there was a guy seated in the middle seat. I took my aisle seat and then, a few minutes later, a couple with two small children showed up. Mom and the kids were in the row in front of me and dad was assigned the window seat in my row.
The father asked me if I'd mind switching my aisle seat for his window seat so he could help out with the kids. What could I say? Of course I obliged. But, in the blink of an eye, the guy in the middle seat said, "Let me just move over a seat," as he took the window seat.
Things had just gone from a so-so window seat to an unpleasant middle seat faster than I could respond and, with each passing second, I realized that I couldn't say anything polite without sounding like a curmudgeon. Now what?
I could tell, when the father and I made eye contact, that he felt bad about what just happened. Although he mentioned that this was, in no way, his intention, I didn't say anything. Mr. former-middle-seat looked like he was in the military and I imagined that, had I said anything, his response could have been something on the order of, "I'm just returning from Afghanistan and I was hoping that I could have the window seat so that I could look at the country that I've sworn to defend."
But, playing the quiet victim did end up having its advantages. About ten minutes after takeoff, the father, in the aisle seat, moved up into the row in front of me and sat there, with his daughter on his lap, for the rest of the flight. So, I ended up next to the only open seat on the plane. Meanwhile, Mr. former-middle-seat took out a military technical manual for an aircraft radar system and quietly read it.