What are some of the basic concepts we'd expect programmer askers to be familiar with?
That's simple; expect no knowledge whatsoever on this front. Remember that users are perfectly entitled to ask entry level crypto questions, as long as the questions are according to the rules we've established. Often, if you ask a question, you have no idea about the complexity that an answer may entail.
For particular function programmers want to achieve, what algorithms/primitives/protocols do we expect them to know?
None. We can try and answer to the best of our abilities. If you expect it is not possible, we either expect no answers or short answers that the asker doesn't understand. That's a shame, but that cannot be helped.
Suggestions for identifying and remedying XY questions?
I don't think that there is a generic method for detecting XY questions.
It is often useful to still answer the original question, if just for future readers that are in a different situation. Pointing them in the right direction and indicating that they are probably asking the wrong question in a comment or answer is fine.
Please beware that rewriting the question in such a way that the original question is invalidated is not allowed on StackExchange. So please make sure that you don't steer users in that direction. Asking a new question is of course always allowed.
Adding or commenting security warnings and implementation hints is very much appreciated but strictly optional. The user is responsible for the end security. Please don't send the user deliberately in the wrong direction; we will definitely have to act against that kind of misinformation.
It's nice if the user gets an answer they understand, and preferably answers should take the expected level of understanding by the user in mind. However, we have no obligation to educate a user beyond the question.
If the user starts to argue, feel free to disengage before it gets nasty. If the users ask additional questions through the comments then feel free to flag them if you don't want to answer them. Answers should not be held hostage to such followups.
We're not a programming site and if the question is just about implementation or debugging then the question should be closed and / or migrated to StackOverflow. Feel free to flag those so the moderators can take a look. Note that we may still not migrate if we feel that the question is not welcome on the other site; migrations cannot be undone nor migrated once more, so they are quite definite.