Unknown, currently.
This isn't an issue the Academy has to address yet, and they've not published any guidance on it. As so few people identify as transgender/gender-queer, and more importantly, almost no major films have cast transgender/gender-queer people in high profile roles.
The statistics all show a tendency for white men to get the most Oscar worthy roles. There's currently a huge debate about how to cast more women and non-white actors/actresses in these high profile roles. Until that imbalance is addressed, I think it is unlikely transgender and gender-queer people will become prominent in them - so I suspect this is an issue the Academy will not need to address for quite some time.
There have been two transgender composers nominated. But as that entire category is gender neutral, there was no controversy there.
I would suspect that transgender people would be nominated based on the gender they identified as. Whether that is the gender they identified as at the time of the film or the gender they identify as now is unknown - but I'd suspect the more controversial the candidate, the less likely an elderly, white Academy selection committee is to nominate them.
For gender-queer, we have absolutely no idea. They may just choose a category and if the person rejects the category, ignore them. Of course, this is all just speculation - we just don't know.
A further point of interest is that there is a number of articles suggesting making the Oscars gender neutral. However, whilst many feminists are in favour of this, many others are not, as the evidence suggests the representation of women and non-white actors at the Oscars would simply diminish. Consider the gender neutral Best Director award. In the 85-year history of the Academy Awards, only 4 females have been nominated with just 1 winner - Kathryn Bigelow for The Hurt Locker.
So, whilst creating gender neutral Awards could help the Academy recognise a transgender or gender-queer person in future, it would also likely strongly increase the amount of white men winning the awards - so along with the positives, there are some very strong negatives to any move the Academy could potentially make here