.....
As you say, it's ridiculous to claim that land-owners must grant permission for anyone to fly a drone in their airspace. If that was so, no-one would be able to fly a drone anywhere, ever. All land is owned by someone. If droners follow the guidelines there shouldn't be any problem. Idiots will ignore the signs anyway, so I don't see what they would achieve.
Different hobby, same issue with land ownership, so a similar conversation.
I can use my air rifle in my garden as it's my land, but I must ensure that the pellets do not go further than my boundary.
I can use my air rifle on another person's land, with their permission, again ensuring that the pellets do not go further than the boundary.
Also, at all times, I must ensure I do not cause concern to any member of the public - e.g. if I was to sit on the driveway plinking at targets I could expect a visit from people in authority.
As you say though, idiots ignore the rules, and give the sport a bad press amongst the public who perceive that all shooters are just there to stop the neighbour's cat pooping in the garden.
I have no issues with the planes that fly over my garden, they come and go and peace returns (and most are so high I don't hear them). I have no issue with drone users in their own/permitted land. I do however have an issue with the persistent noise (I assume it's why they are called drones) as a stranger flies a daily grid pattern over my garden and all others in the neighbourhood, or flies slowly along the beach we're all sitting on and I'm wondering what they are trying to see on their cameras*.
This is not an attack on legitimate drone users, in the same way I don't expect anybody to attack my hobbies (as motorists always seem to when I swap the car for a bicycle).
*I have nothing to hide, but want to know why I am being filmed. In a public space a few years ago somebody started photographing myself and my children. He was within his rights as it was a public space, but I felt uncomfortable being obviously photographed. I got up and challenged him, he locked his phone and put it away saying "I was photographing my grand daughter" whilst pointing at a little girl a few feet away. I knew the phone was pointed at myself and not her... but it was a really crowded space and I just decided to walk away rather than grab the phone and cause a scene.