I think you nailed it with the ‘knowing about Jesus versus being known by him’ distinction.
The people he turns away have impressive spiritual resumes, but the key phrase is ‘I never knew you.’ They defend themselves by pointing to their own works (‘did we not prophesy, cast out demons’). Their confidence was in what they did for God rather than in belonging to him.
As for how anyone can know where they stand, the fact you’re asking that question puts you in a different category than the people in the passage. They show up on judgment day confident, with no self-examination at all.
I think the ‘will of the Father’ he mentions is the ordinary, unglamorous stuff he’d spent the whole Sermon on the Mount describing: forgiving people, honesty, mercy, loving enemies.