Castiel had not reacted to being grabbed. Except to look at Dean as if he might be a bit addled. (Which, given where he has been, is a distinct possibility.)

When Dean abruptly lets him go and falls back with a distressed sound, Castiel leans back only slightly. He is busy examining Dean for some sign, some indication of what tack he might take to reach him.

Castiel has gotten better at assessing and responding to human feelings. But it is still not what one would call one of his strengths.

"I will not take you back. Can not," he says.

"You are needed elsewhere."

"Do you understand me, Dean?"