Aranofsky is nothing if not an enigmatic director, so barring him commenting, it can only be speculation. You point about computer bugs is quite insightful, and certainly a legitimate answer!
The bugs were specifically ants. There is some discussion of the symbolism of ants. They are certainly industrious. This may relate to the rigor required for the type of mathematics Max is dealing in, particularly number theory. Proverbs 6:6 may be translated as "Go to the ant, O sluggard; consider her ways, and be wise." The movie definitely has a biblical theme.
They are also almost certainly a reference to Salvador Dalí, as the film can be taken as a surrealist work. Daliparis.com explains them as a "symbol of decay and decomposition. Dalí met ants the first time as a child, watching the decomposed remains of small animals eaten by them. He watched with fascination and repulsion, and continued to use them in his work, as a symbol of decadence and ephemeral."
I'll have to think more about the dichotomy, but paradox is staple of early mathematics.
With all that said, I suspect there are even more meanings. Aranofsky is just that kind of artist.