At the moment my way is to look through all the squares of length LEN (using binary search) and to check, whether a hash of a square was met before. It is of O(N*M*log(min{N, M})) complexity.
But I get WA#10 again and again. Please, if someone had a problem with this test, share your impressions, 'cause I'm going slightly mad.

You have a collisions with hash. (I've got this problem also because second number x was to small and close to first one).

Thank you, it really had to do with collisions. Actually, my self-made hashset implementation was fantastically awful. :-)

Edited by author 06.07.2012 15:48

Why you have used binary search? How we can say that the sq. matrix 2x2 has less value than 3x3. and 3x3 has less than 4x4. how can we justify?

We run a binary search on the length of the square's side. For a fixed length of the square's side, we calculate the hash of all squares with that side length, and look for a pair of squares with the same hash.