I've checked my formula with both kinds of formulas given in this article:

http://www.jstor.org/discover/10.2307/2691523?uid=3739232&uid=2129&uid=2&uid=70&uid=4&sid=21101178420913 (if someone is interested in full version of article - mailto Oracle@acm.lviv.ua)

I use BigInteger in Java - should be no precision issues.

Results are the same for all rectangles with 1 <= sides <= 100 and for about a billion random test cases with 1 <= side <= 1000000.

And still WA#4((( Is it definitely correct?

Now I wonder if the problem is exactly in checking if one rectangle fits into others? Or I've missed something?