ENG  RUSTimus Online Judge
Online Judge
Задачи
Авторы
Соревнования
О системе
Часто задаваемые вопросы
Новости сайта
Форум
Ссылки
Архив задач
Отправить на проверку
Состояние проверки
Руководство
Регистрация
Исправить данные
Рейтинг авторов
Текущее соревнование
Расписание
Прошедшие соревнования
Правила
вернуться в форум

Обсуждение задачи 1394. Корабли. Версия 2

Orient SSP solver [40] // Задача 1394. Корабли. Версия 2 11 июн 2020 01:19
Didn't know will it be useful for someone and is it on the right path to solve the problem, but I wish to publish this code for SSP solver. To invent good heuristics for MSSP is utmost harder.

using I = short;

template<typename T = I, typename S = I>
struct DDP // dynamic dynamic programming solver for subset sum problem O(n^3 * 2^sqrt(n)) (n is number of input bits)
{
    T terms[2][10001];
    S sums[2][10001];

    template<typename Term, typename Sum>
    std::pair<Term, Sum> solve(const Term termsBeg, const Term termsEnd, const Sum sumsBeg, const Sum sumsEnd)
    {
        static_assert(std::is_same<typename std::iterator_traits<Term>::value_type, T>::value, "!");
        static_assert(std::is_same<typename std::iterator_traits<Sum>::value_type, S>::value, "!");

        auto sumMax = std::max_element(sumsBeg, sumsEnd);

        auto termsSrc = terms[0];
        auto termsDst = terms[1];

        auto sumsSrc = sums[0];
        auto sumsDst = sums[1];

        *termsSrc = {};
        *sumsSrc = {};

        auto sumsSrcEnd = std::next(sumsSrc);
        for (auto t = termsBeg; t != termsEnd; ++t) {
            auto termsSrcBeg = termsSrc;
            auto termsDstBeg = termsDst;

            auto sumsSrcBeg = sumsSrc;
            auto sumsShiftedBeg = sumsSrc;
            auto sumsDstBeg = sumsDst;

            // merge unique
            assert(std::size_t(std::distance(sumsSrcBeg, sumsSrcEnd)) <= std::extent<std::remove_reference_t<decltype(sums[0])>>::value);
            while (sumsSrcBeg != sumsSrcEnd) {
                if (sumsShiftedBeg == sumsSrcEnd) {
                    break;
                }
                I shiftedSum = *sumsShiftedBeg + *t;
                if (shiftedSum < *sumsSrcBeg) {
                    *sumsDstBeg++ = shiftedSum;
                    ++sumsShiftedBeg;
                    *termsDstBeg++ = *t;
                } else {
                    if (shiftedSum == *sumsSrcBeg) {
                        ++sumsShiftedBeg;
                        assert(*sumsShiftedBeg + *t != *sumsSrcBeg);
                    }
                    *sumsDstBeg++ = *sumsSrcBeg++;
                    *termsDstBeg++ = *termsSrcBeg++;
                }
                assert(std::size_t(std::distance(sumsDst, sumsDstBeg)) <= std::extent<std::remove_reference_t<decltype(sums[0])>>::value);
            }
            if (sumsShiftedBeg == sumsSrcEnd) {
                sumsSrcEnd = std::copy(sumsSrcBeg, sumsSrcEnd, sumsDstBeg);
                termsDstBeg = std::copy_n(termsSrcBeg, std::distance(sumsDstBeg, sumsSrcEnd), termsDstBeg);
            } else {
                assert(sumsSrcBeg == sumsSrcEnd);
                sumsSrcEnd = std::transform(sumsShiftedBeg, sumsSrcEnd, sumsDstBeg, [t] (S s) { return s + *t; });
                termsDstBeg = std::fill_n(termsDstBeg, std::distance(sumsDstBeg, sumsSrcEnd), *t);
            }

            std::swap(termsSrc, termsDst);
            std::swap(sumsSrc, sumsDst);

            for (auto sum = sumsBeg; sum != sumsEnd; ++sum) {
                sumsSrcBeg = std::lower_bound(sumsSrc, sumsSrcEnd, *sum);
                if ((sumsSrcBeg != sumsSrcEnd) && (*sumsSrcBeg == *sum)) {
                    sumsSrcEnd = sumsSrcBeg;
                    assert((*std::next(termsSrc, std::distance(sumsSrc, sumsSrcEnd)) == *t));
                    auto termsDstEnd = termsDst;
                    do {
                        auto term = *std::next(termsSrc, std::distance(sumsSrc, sumsSrcEnd));
                        *termsDstEnd++ = term;
                        sumsSrcEnd = std::lower_bound(sumsSrc, sumsSrcEnd, *sumsSrcEnd - term);
                    } while (sumsSrcEnd != sumsSrc);
                    assert((std::accumulate(termsDst, termsDstEnd, S{}) == *sum));
                    termsDstBeg = termsDst;
                    auto termsSrcEnd = t;
                    assert(*termsSrcEnd == *termsDstBeg);
                    auto termsTail = termsSrcEnd;
                    while (++termsDstBeg != termsDstEnd) {
                        while (*--termsSrcEnd != *termsDstBeg) {
                            *termsTail-- = *termsSrcEnd;
                        }
                    }
                    while (termsBeg != termsSrcEnd) {
                        *termsTail-- = *--termsSrcEnd;
                    }
                    assert(std::distance(termsBeg, termsTail) + 1 == std::distance(termsDst, termsDstEnd));
                    auto term = termsTail;
                    termsDstBeg = termsDst;
                    assert(termsDstBeg != termsDstEnd);
                    do {
                        *termsTail-- = *termsDstBeg++;
                    } while (termsDstBeg != termsDstEnd);
                    assert(termsTail == std::prev(termsBeg));
                    return {term, sum}; // not past the end element for found range but exactly last element
                }
            }
            sumsSrcEnd = std::lower_bound(sumsSrc, sumsSrcEnd, *sumMax); // crop
        }
        return {termsEnd, sumsEnd};
    }
};

auto x = solve(std::begin(ships), std::end(ships), std::cbegin(rows), std::cend(rows)) if succeeded, it gives rearranged ships range such, that range [std::begin(ships), x.first] is a solution and sum of all ships is *x.second. [std::begin(ships), std::end(ships)) still consists of exactly the same ships. If not succeeded then it return a pair of std::end(ships) and std::cend(rows)

Edited by author 11.06.2020 12:53
Manciu Ion Re: SSP solver [36] // Задача 1394. Корабли. Версия 2 14 июн 2020 11:18
What algorithm do you use to pass 68 tests ?

Edited by author 15.06.2020 21:48
Manciu Ion Re: SSP solver [2] // Задача 1394. Корабли. Версия 2 14 июн 2020 12:29
I use bactracking and dynamic programming, it runs 30 seconds for the test 67th, the hardest one for my solution.


Edited by author 17.06.2020 15:02
Manciu Ion Re: SSP solver // Задача 1394. Корабли. Версия 2 14 июн 2020 22:08
Manciu Ion Re: SSP solver // Задача 1394. Корабли. Версия 2 17 июн 2020 15:00


Edited by author 17.06.2020 15:37
Manciu Ion Re: SSP solver // Задача 1394. Корабли. Версия 2 15 июн 2020 21:44
it's too hard to figure out what's going on here!
Orient Re: SSP solver [31] // Задача 1394. Корабли. Версия 2 17 июн 2020 03:25
Not specific kind of algorithm but just a good seed. I have no backtracking. No dp or dfs over that primitive above. Free search by combining of ships from one of found rows with pool of ships that cannot form some of the rows. Here used stability of the algorithm above to rearrange ships for better mixing of them. 67 is also hardest test for mine. 8 solutions per minute on E5-2667 v2 when running 16 threads (I use -fopenmp to parallelize). 70 gives about 220 solutions for that configuration.

Edited by author 17.06.2020 03:36
Manciu Ion Re: SSP solver [30] // Задача 1394. Корабли. Версия 2 20 окт 2020 21:15
What algorithm do you use to pass 82 tests?
Manciu Ion Re: SSP solver [29] // Задача 1394. Корабли. Версия 2 21 окт 2020 14:42
is it the same one you described above?
Orient Re: SSP solver [28] // Задача 1394. Корабли. Версия 2 23 окт 2020 00:56
Manciu Ion писал(a) 21 октября 2020 14:42
is it the same one you described above?
The same, but with small improvements, which detect a situation when heuristics goes to infinite loop then reshuffles a part of partial solution.
Strictly speaking above algorithm is only a part of the whole algorithm. Anyways that algorithm is just a heuristics. Although it is the only "beautiful" heuristics I can imagine, it is still a heuristics.

(!!!) Just now I found, that full brute force for tests like 67, 70 and 80, which enumerates all the possible solutions (literally ALL (+ simple b&b of course)), finishes in 3 seconds max. Even if I don't exit when first solution is found. It is incredible after all that research in heuristics field. I think I'll publish the algorithm. Not too soon. After solving the problem.
Manciu Ion Re: SSP solver [25] // Задача 1394. Корабли. Версия 2 25 окт 2020 00:32
Congratulations!
Orient Re: SSP solver [24] // Задача 1394. Корабли. Версия 2 25 окт 2020 01:14
Thx. It seems it is mistake of OJ, because I cannot reproduce.
rip&pvs Re: SSP solver [23] // Задача 1394. Корабли. Версия 2 25 окт 2020 05:55
jury must add a test in which several inputs are verified with single verdict: passed or failed to protect them from collecting information :)


Edited by author 25.10.2020 06:23
Orient Re: SSP solver [22] // Задача 1394. Корабли. Версия 2 25 окт 2020 07:24
I bet you have a number of 70+ tests))
rip&pvs Re: SSP solver [21] // Задача 1394. Корабли. Версия 2 25 окт 2020 09:28
only one, and realized that it would be faster to solve it by changing the heuristic parameter (every 3-5 works) than to spend 100+ submissions to determine it.
Orient Re: SSP solver // Задача 1394. Корабли. Версия 2 25 окт 2020 14:11
You heuristics is fast. Mine is not. Can you give us a hint?
Orient Re: SSP solver [18] // Задача 1394. Корабли. Версия 2 26 окт 2020 23:44
 > solve it by changing the heuristic parameter (every 3-5 works)

Your heuristics is highly effective, if I understand correctly. Is it true, that you can select a parameter ("seed") for the heuristics by just 3-5 tries? For each hard test case. Then you write `for (int seed : {seed_test70, seed_test67, seed_test80, seed_test69, seed_test77, seed_test79}) heuristics(seed);` and even for test79 (e.g., here "last" one) all iterations run in less then 1 second? I.e. less then a small number of milliseconds per every seed value, you collect?

Edited by author 26.10.2020 23:46
rip&pvs Re: SSP solver [17] // Задача 1394. Корабли. Версия 2 27 окт 2020 23:14
nothing like this, my solution is common for all test and direct randomization is minimal. Actually I have kind of deterministic one (stuck on test#67 and can't pass #82 too, but not critical) and ideas for improving it, but don't have motivation and time to do this.
I'm not going to share an idea or give hints.
Orient Re: SSP solver // Задача 1394. Корабли. Версия 2 28 окт 2020 00:18
Спасибо и на этом.
Manciu Ion Re: SSP solver [15] // Задача 1394. Корабли. Версия 2 28 окт 2020 14:22
Do you use one algorithm or you mix them ?

Edited by author 29.10.2020 11:56
Orient Re: SSP solver [14] // Задача 1394. Корабли. Версия 2 29 окт 2020 16:29
I use two algorithms. Heuristics and branch and bounded brute force. They called in the next order: 1 try of heuristics (I think 90% of tests are covered by this first one) (it take about 1-2ms), then highly clipped b&b brute force (I think it get about a couple of hundreds of milliseconds in worst case - on test 77 or 79 it fails and made complete (unsuccessful) enumeration through all clipped DoD), then infinite number of tries of the same heuristics (it get about first tens or mb one hundred of milliseconds average, but for some good seed (in fact I enumerate first 20 of them - two was good enough) it get mb less than ten milliseconds or so).

A vast majority of test are covered by heuristics. It works well on those tests, that have big actual number of unique solutions. For these brute force produces too big number of solutions, that results in MLE. Heuristics is simple and have regular structure. Actually it can find solutions for any test, but for some of them (test67) the probability per try is too low to reliable fit in 1 seconds of time limit.
On the contrary brute force is highly effective on those tests, which have one or two unique solutions (or maybe thousands -- don't even know, because I have only test67 and test70, which have such a property). Branch and bounding works highly effective on these.

The only "cheating" to get solution for me was to find right seed for test79 (I don't know its structure and I want to collect it maybe). Don't even know how rare it -- I find it from the first try. Another cheating -- is to optimize. I made holes in DoD of brute force. So enumeration was reduced a lot for minor number of tests. Optimization is not valuable at all from algorithmic point of view. It is a bullshit full of magic numbers.
Manciu Ion Re: SSP solver [13] // Задача 1394. Корабли. Версия 2 29 окт 2020 18:03
My backtracking algorithm runs in 4.0 sec on Java and 8.0 sec on C++ for test 78, 0.178 sec on Java and
0.165 sec on C++ for test 67.
Orient Re: SSP solver [10] // Задача 1394. Корабли. Версия 2 29 окт 2020 18:15
I can conclude, that your backtracking algorithm is (exactly) as optimal as mine. I bet you use something like `std::vector<std::bitset<99>>` to store solutions per every row and something like `std::unordered_map<std::bitset<9>, std::unordered_set<std::bitset<99>>>` for branch and bounding, isn't it?
Manciu Ion Re: SSP solver [9] // Задача 1394. Корабли. Версия 2 29 окт 2020 18:28
Yes, you are right, but I use my bit set implementation. I have found another constraint but I don't know for now how to implement optimal!
Orient Re: SSP solver [2] // Задача 1394. Корабли. Версия 2 29 окт 2020 18:29
What is the constraint? Is it a secret for now?
Manciu Ion Re: SSP solver [1] // Задача 1394. Корабли. Версия 2 29 окт 2020 18:35
It's no secret, but it won't be fair to post it here
Orient Re: SSP solver // Задача 1394. Корабли. Версия 2 29 окт 2020 18:38
OK, I think I'll come up with one too. Knowing that there is something besides `std::unordered_map<std::bitset<9>, std::unordered_set<std::bitset<99>>>` is enough.

Edited by author 29.10.2020 18:39
Orient Re: SSP solver [5] // Задача 1394. Корабли. Версия 2 31 окт 2020 01:36
 > I use my bit set implementation

Do you use _bit_scan_forward/_bit_scan_reverse when recover solution? Can it be an improvement over std::bitset?
Manciu Ion Re: SSP solver // Задача 1394. Корабли. Версия 2 31 окт 2020 10:52
Yes, It can, but I don't think it will improve more than 0.1 second!

Edited by author 31.10.2020 12:56
Manciu Ion Re: SSP solver [3] // Задача 1394. Корабли. Версия 2 31 окт 2020 13:32
It is not critic recovering and saving solution!
Anatoliy V Tomilov Re: SSP solver [2] // Задача 1394. Корабли. Версия 2 31 окт 2020 18:29
Anyways it is interesting to implement next-bit-set-iterator.
Manciu Ion Re: SSP solver [1] // Задача 1394. Корабли. Версия 2 5 ноя 2020 14:05
Do you use parallel programing ?
Anatoliy V Tomilov Re: SSP solver // Задача 1394. Корабли. Версия 2 5 ноя 2020 23:25
No. Because it have no sense. User time of all the threads are summed up.
Manciu Ion Re: SSP solver [1] // Задача 1394. Корабли. Версия 2 6 ноя 2020 14:33


Edited by author 20.11.2020 22:00
Anatoliy V Tomilov Re: SSP solver // Задача 1394. Корабли. Версия 2 8 ноя 2020 02:58
Good to know. Do you want to trade our solutions when you solve?)
Orient Re: SSP solver // Задача 1394. Корабли. Версия 2 26 окт 2020 23:44


Edited by author 26.10.2020 23:45
Manciu Ion Re: SSP solver [1] // Задача 1394. Корабли. Версия 2 26 окт 2020 16:55
test 80 has 11881161 solutions! Your algorithm generate all of them in just 3 seconds!?
Orient Re: SSP solver // Задача 1394. Корабли. Версия 2 26 окт 2020 21:18
 > test 80 has 11881161 solutions! Your algorithm generate all of them in just 3 seconds!?

I was wrong. The algorithm (backtracking) generates all the (15741649, "all the" contains a large amount of convention, because of branch and bounding) solutions for test80 in ~72 seconds. First solution found in 3.223853s for standard random seed and 0.563442s for search in lexicographical order. But my heuristics finds solution for test80 in 0.000041s and it goes before the backtracking, so test80 is not the problem at all.

test67 (first solution): 0.290 (lexicographical order)
test67 (complete enumeration, 1 solution): 0.373 (lexicographical order)
test70 (first solution): 0.179 (lexicographical order)
test70 (complete enumeration, 2 solutions): 0.187 (lexicographical order)

Number of solutions for each separate line for test80 (lines sorted):
109
115
209
3981
4442
10091
38691
0 // this one is not even calculated, because not needed

Hardware: Intel Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz (3.90GHz Trubo Boost per single core, vs 2.40GHz (same arch) on OJ system). Specially bought for this problem).

BTW test5 is a tough nut to crack for the algorithm.

Edited by author 26.10.2020 22:38
David Sun Re: SSP solver [2] // Задача 1394. Корабли. Версия 2 4 июл 2020 22:52
Mind elaborating? Cannot see the linkage between this code and an O(n^3 * 2^sqrt(n)) algorithm.

My email sunzheng.david[at]gmail.com.
Orient Re: SSP solver [1] // Задача 1394. Корабли. Версия 2 6 июл 2020 01:34
This is implementaton of third algorithm from "An Empirical Study of Algorithms for the Subset Sum Problem" O'Neil. It is not the code of solution of the whole problem (which is "multiple subset sum problem"), but a trivial part of it ("subset sum problem").
Obmen Re: SSP solver // Задача 1394. Корабли. Версия 2 6 июл 2020 12:26
Hello, Orient, could you tell us more about your idea by email facttacf@gmail.com?