My opinion is that it should return a 5xx as a way to tell the client they did not do anything wrong. If the client is a user-operated web browser there is not much to do about it, but if the client is a library it tells the programmer that the error is not their fault (i.e. packets are not malformed nor data missing and so).
Good question, anyhow.