If I have center location "loc", I want to get all mobs within a radius of X in a cylinder area (ignoring Y axis) How can I do this? I am useless with maths

Get the entities of the world the loc resides in and then filter out those where the distance (Location#distance(Location)) is greater than the radius of the cylinder. To optimize, calculate the squared location and filter all entities out where that is greater than the desired distance squared.

This is not a cylinder but a sphere. You'd get all the entities in the world. And then, for a point of with x=Xm, z=Zm and radius R, you'd check if an entity's X and Z satisfy (X - Xm)^2 + (Z - Zm)^2 <= R^2.