I agree, but the reason is not that "the Desktop folder has enough to do."
There are two reasons why I think it's a bad idea:
1. It clutters up the desktop. The more icons you have there, the harder it
is to find what you want.
2. Normally, almost everything on the desktop is a shortcut, and knowing
that, you tend to rely on its being a shortcut. that makes most people think
they can delete anything on the desktop with impunity. Put a mixture of
actual files and shortcuts on the desktop and you run the risk that you will
someday delete a real file you need thinking you are just deleting a
shortcut to it.
That goes double if you are one of those people who also remove the shortcut
arrows (which I also don't recommend), but the risk is there even if you
don't.