Yep, you can plaster straight onto stone or brick.
It needs to be clean though and you need a fatty lime, so it sticks.
The thing with building, and I fall into the trap all the time, is it's easy to read too much about it and overthink it. I've found it's best to be pragmatic and just do it and see if it works.