I have never taught middle school, or been in a middle school, so I cannot comment on what is more difficult.
Don’t make your decision based on the opinions of others. You need to make the decision for what is best for you, and where you’d rather be. Take time volunteering in schools to see what you’d prefer. Good luck.
Teaching In Real Life 