my o vibrates his tail like in a territorial way and flares his gills getting all puffed up does this mean he is male or do females behave this way too? usually its when my dog walks near his tank

It sounds like your Oscar is showing aggression towards your dog, I would love to know what they actually make of a dog. Flaring their gills and opening their mouths wide is what I would imagine is them making themselves look bigger and more intimidating. The vibrating tail is I believe a form of communication. I also believe this can be aggression and non-aggression as well

both male and female do the same flaring of gills and mouth wide
open either during aggression or mating..... and they do react to
any uncommon sight like bright color of dresses or anything that
moves in front of the tank

my oscar does that and theres not even another oscar in the tank

my red once went crazy like that , just because i had placed some cardboard boxes close to the tank.
To me that was proof that the Oscar thinks of the surroundings of his tank, as being "his" territory as well.
He stopped the flaring right after i moved the boxes away into the hallway...
My dog still get that special attention from my Other oscar from time to time.