Hurt? Yes. Kill? Unclear.
Both the (presumably) fully grown demogorgon in the first season and the "demo-dogs" in the second season appeared to be physically harmed by things other than Eleven's telekinetic powers (what the question calls "magic").
In season one, Nancy and Jonathan seem to be able to hurt the Demogorgon using fire and other weapons, forcing it to retreat to the Upside Down. In season two, the demo-dogs are shown to be affected by gunshots, especially at the end of the last episode when Eleven is attempting to close the gate.
However, in all cases they appear to be highly resilient (a few gun shots aren't sufficient to do serious damage), and may even have rapid regeneration abilities (in or outside of the Upside Down). We also don't know if it's possible to fatally wound any of them as that's never been explicitly shown.