The weird acting and line delivery is a stylistic choice that Yorgos Lanthimos seems to prefer. You can see it in his other films as well, for instance in The Lobster.
In the interviews on The Killing of a Sacred Deer DVD, Lanthimos and Colin Farrell talk about this style. One thing that was mentioned in the interview (I forget now which interviewee brought it up), is that with a more normal acting style, the actor is doing all the feeling and emoting. The audience doesn't actually have to figure out what to feel or do much of the feeling themselves, aside from mirroring the actor. With this style, the audience does more emotional work: they have to figure out what to feel and then feel it, rather than just watching someone else emoting.
Personally, I also think that feeling alienated within a society that has incomprehensible rules is a major theme of both these movies, so a very flat line delivery style helps to make the audience feel that alienation and confusion.
The DVD interview doesn't appear to be available online, so I can only suggest going to your local library and checking out the DVD for yourself. There is an LA Times interview with the cast that includes these quotes, though:
"The dialogue is pretty heavy and dark, so the less emotion you attach to it, the bigger the impact," Keoghan added.
"the characters onscreen are acting like it's normal. And that shakes us in a way. If the characters were resisting and shaking their fists to the high heavens, they're doing the work for us. But the characters in Yorgos' films don't necessarily process things the way we are used to seeing people process things, or the way we process them ourselves, and I think that becomes unnerving," [said Farrell]