Please, can anybody tell how to solve this problem, which algorithm is used, how could the answer of sample test be 4.828427125?

maybe a maximum of (2xy+x*sqrt(a*a-x*x)+y*sqrt(b*b-y*y))/2 should do