It is absolutely 100% realistic.
Hand signals (which also include head movements) and other visual signals are a crucial form of plane-to-plane communication used by the military. Air crews practice these hand signals frequently to maintain a level of fluency that allows them to send and receive them as easily as communicating verbally.
Visual signals can communicate complex thoughts and detailed instructions. For instance, numbers are given with one hand by fingers held sideways (1-5) or up-and-down (6-0/10), so "Come to a heading of 217 degrees" can be flashed by hand almost as fast as it can be spoken.
There are multiple reasons why military pilots use hand and other visual signals.
Military air crews use hand signals between their aircraft in a variety of situations:
Loss of communications from radio damage/malfunction, injury to the pilot, or any other reason is an obvious case where hand signals are critical. Hand signals can be life saving in these instances. A pilot who still has comms can assist one who has lost them to find a carrier or airfield and land by relaying directions from the carrier/tower to the pilot who has lost comms.
Avoiding triangulation of location is another. When beyond radar range, aircraft can be detected by their radio transmissions and located by triangulating on them. This can be especially critical if the carrier is running radio/radar silent to avoid detection in a time of war. In such cases, hand signals are crucial to minimize radio transmissions and avoid detection.
Operational security is a third. The bad guys are always listening and verbal communications can give away a lot of information about an aircraft or a group of aircraft. For example, it could reveal the number of planes in a given grouping, their speed and heading, their objective, etc.
Practice. Air crews will use hand signals even when they don't need to--sometimes simulating situations like radio loss--so that they can build and retain fluency for when they do need them.
Using hand signals poses little difficulty when aircraft are not in combat.
Military aircraft typically fly in close formation during patrols and other missions and one reason is to facilitate visual signals. Maintaining close distance with other aircraft and matching speed and heading are basic skills that pilots master to make formation flying second nature. They maintain visual contact with each other to ensure nothing goes wrong with the other planes or their crews, so they are in position to exchange hand signals much of the time.
Contrary to what another answer suggests, pilots do not have to keep their eyes glued to their instruments most of the time. They constantly scan their surroundings and can easily devote attention to hand and other visual signals. At night, air crews use flashlight signals instead of hand signals, and as the materials I post below demonstrate, pilots can also communicate with the way they maneuver their aircraft.
Obviously hand signals are no use in combat, but Top Gun: Maverick never suggests that they are.
Another answer mentions that signals would be impossible during combat. This is true because in combat planes do not fly in close formation (usually) and the air crews' attention can not be easily diverted to such communications. But nowhere in the movie were hand signals used during actual combat. They were used during close formation flying.
The movie was mostly realistic in its depiction of the use of hand signals.
In the movie, the enemy nation is unable to raise the F-14 on the radio, so the enemy pilot came into close formation with the plane to establish visual communication and find out what was going on with the aircraft--this is very realistic, it's just what an air force would do with an apparently friendly aircraft not responding on the radio. The signal used by Maverick for "I've lost comms" (when trying to fool the enemy pilot) was pointing to his ear and giving thumbs down. This is the actual US Navy hand signal for "I've lost comms."
It is also probably realistic that foreign air forces have their own signals for a lot of things and Maverick and Rooster might not know the ones the other pilot was using. Different cultures have different conventions for non-verbal communications. For instance, a non-Western culture might not have context for the thumbs-up/thumbs-down sign. Given that many signals are so intuitive, there'd probably be some overlap, but it's unlikely all their hand signals would be the same.
In both Top Gun and Top Gun: Maverick's earlier training scenes, the student pilots communicate almost exclusively by radio when flying in formation during exercises. This was actually a little unrealistic, but it was done so the audience could easily follow along. In reality, the student pilots would have used hand signals during formation flying to prevent the aggressor pilots (Viper and Jester in Top Gun and Maverick in Top Gun: Maverick) from listening in to what they were saying and what they were planning.
Here are some US Navy cheat sheets that show some other standard hand signals for plane-to-plane communication. In the last one you can see that even maneuvering one's plane in a certain way can be used to communicate a command.
Other uses for hand signals
There is a whole other sign language used between pilots and ground crews. Ground crews must communicate with and direct aircraft on the ground, but they do not have radios and jet noise and lowered canopies make verbal communications impossible. So pilots use hand signals extensively even when they are still on the ground.
Military pilots definitely use hand and other visual signals for plane-to-plane communications. Top Gun: Maverick is highly realistic in its depiction of hand signals. The film's only lapse in realism with regard to hand signals is when the pilots in training don't use them during formation flying to hide their communications from the aggressor pilot; however, this was probably a good directing choice because it allows the audience to follow along more easily.