That statement alone is enough to prove to Castiel that she's no longer one of them, memories or no.

It makes it easier in a way.

He could refute her, but what would be the point? Even if he had the luxury of time to belabor the point, Castiel would have no desire to do so before an audience.

"Still," he says, "we have a history."

Not always a good one, but a history.

She could at least grant him that. That this is not an easy task God set before him.