| *** ocurieles has joined #idempiere | 04:32 | |
| *** ocurieles has quit IRC | 04:37 | |
| *** norbertbede has joined #idempiere | 04:54 | |
| *** norbertbede has quit IRC | 05:15 | |
| *** tsvikruha has joined #idempiere | 05:30 | |
| *** norbertbede has joined #idempiere | 05:48 | |
| *** a42niem has joined #idempiere | 06:08 | |
| *** nmicoud has joined #idempiere | 06:28 | |
| *** norbertbede has quit IRC | 06:33 | |
| *** norbertbede has joined #idempiere | 06:58 | |
| *** norbertbede has quit IRC | 07:29 | |
| *** norbertbede has joined #idempiere | 07:32 | |
| *** KermitTheFragger has joined #idempiere | 07:47 | |
| *** ocurieles has joined #idempiere | 08:09 | |
| *** ocurieles has quit IRC | 08:14 | |
| *** is-mw has joined #idempiere | 08:52 | |
| *** norbertbede1 has joined #idempiere | 09:03 | |
| *** norbertbede1 has joined #idempiere | 09:03 | |
| *** norbertbede has quit IRC | 09:05 | |
| *** norbertbede1 has quit IRC | 09:08 | |
| *** norbertbede has joined #idempiere | 09:08 | |
| *** hieplq has joined #idempiere | 09:14 | |
| *** tsvikruha has quit IRC | 10:30 | |
| *** norbertbede has quit IRC | 10:41 | |
| *** norbertbede has joined #idempiere | 10:43 | |
| *** norbertbede has quit IRC | 11:01 | |
| *** norbertbede has joined #idempiere | 11:26 | |
| *** a42niem_ has joined #idempiere | 11:31 | |
| *** is-mw has quit IRC | 11:35 | |
| *** tsvikruha has joined #idempiere | 11:35 | |
| *** norbertbede has quit IRC | 11:42 | |
| *** ocurieles has joined #idempiere | 11:47 | |
| *** ocurieles has quit IRC | 11:51 | |
| *** norbertbede has joined #idempiere | 11:59 | |
| *** druiz has joined #idempiere | 12:19 | |
| *** norbertbede1 has joined #idempiere | 12:40 | |
| *** norbertbede has quit IRC | 12:43 | |
| *** norbertbede has joined #idempiere | 12:54 | |
| *** norbertbede1 has quit IRC | 12:56 | |
| *** ocurieles has joined #idempiere | 13:12 | |
| *** ocurieles has quit IRC | 13:30 | |
| *** CarlosRuiz has joined #idempiere | 13:58 | |
| *** ChanServ sets mode: +o CarlosRuiz | 13:58 | |
| druiz | Hi @CarlosRuiz | 13:58 |
|---|---|---|
| CarlosRuiz | Hi druiz | 14:00 |
| druiz | We created this ticket today https://idempiere.atlassian.net/browse/IDEMPIERE-2630 | 14:00 |
| druiz | The solution in @tbayen 's patch has a good impact in the time. It works in postgres and does not break Oracle functionality | 14:01 |
| druiz | What do you think abt that? | 14:02 |
| CarlosRuiz | checking ... | 14:02 |
| *** norbertbede has quit IRC | 14:03 | |
| druiz | I'm testing it now with and without the asynchronous commit, and checking the time impact with a file with 103222 records | 14:03 |
| *** norbertbede has joined #idempiere | 14:03 | |
| druiz | so we can show real data on the performance | 14:04 |
| *** norbertbede has quit IRC | 14:06 | |
| druiz | @CarlosRuiz we're also working on an improvement to show the progress in that process. When it takes too long you don't know what's happening behind, and yesterday once it was not running but we couldn't notice because the current Processing... Please Wait... kept going. | 14:11 |
| druiz | Should that go in the same patch or should I create a new one for it? | 14:12 |
| CarlosRuiz | different patch | 14:12 |
| druiz | It's the same case as here https://idempiere.atlassian.net/browse/IDEMPIERE-432 | 14:13 |
| druiz | Is it better to reopne that one? | 14:13 |
| CarlosRuiz | well Eugen was talking there about the old importers | 14:14 |
| CarlosRuiz | and it was closed as potential idea - better to open a new one | 14:14 |
| CarlosRuiz | about 2630 | 14:14 |
| CarlosRuiz | I don't think is a good idea to improve performance | 14:15 |
| CarlosRuiz | if you're ok with the trade-off mentioned in the postgresql page - then is ok - you can implement it as a plugin I think | 14:17 |
| *** norbertbede has joined #idempiere | 14:17 | |
| tbayen | We are measuring with a bigger data size. But with our first test it was an improvement of 30%. In my opinion the slow and long-running process is a problem. | 14:17 |
| CarlosRuiz | --> "at the cost that the most recent transactions may be lost if the database should crash. In many applications this is an acceptable trade-off." | 14:18 |
| CarlosRuiz | --> "As an example, a bank would certainly not use asynchronous commit for a transaction recording an ATM's dispensing of cash" | 14:19 |
| CarlosRuiz | --> "But in many scenarios, such as event logging, there is no need for a strong guarantee of this kind" | 14:19 |
| tbayen | I don't see a trade-off. The import is done as single commits anyways - independent of problems in the importer log. So if something does not work or creash you are in trouble anyways. You always need to check if everything finished and be prepared to delete the table again. | 14:19 |
| tbayen | The importer is far away from being bullet-proof. This new feature does not change that fact. | 14:20 |
| CarlosRuiz | it changes that makes postgresql transaction not bullet proof | 14:20 |
| CarlosRuiz | what I meant about that can be implemented in a plugin is: I think we must keep core on the safest way | 14:21 |
| CarlosRuiz | cos people can use importers to load event logging as described above - but people also can use importer to load POS orders for example | 14:22 |
| CarlosRuiz | in the second case the trade-off is ok - but in the second is probably not | 14:22 |
| tbayen | afaiu the danger is if the postgres server crashes. In this case it can happen that the last commits are not recorded. But in case of this crash you do not know how far the importer process was. The data will be committet until one record. I see no difference if there is one more lost or not. | 14:23 |
| tbayen | When the Server crashes inside a file of POS Orders noone knows which orders are done and which not. There is no log file in this case. | 14:24 |
| *** ocurieles has joined #idempiere | 14:25 | |
| tbayen | If we want a guarantee for that case we need to have a single commit for the while file. Perhaps that is the solution: To allow "one single commit" mode with a guarantee and "line by line" mode where it tries to write as good as possible and tries to give a logfile back. | 14:25 |
| tbayen | whole | 14:26 |
| CarlosRuiz | tbayen, I understand your reasoning - is just that I think that reasoning cannot be generalized into core | 14:26 |
| CarlosRuiz | I would probably apply it in some cases - but not in all cases | 14:28 |
| druiz | What about creating a field in the import template to allow the user in which cases to apply it | 14:28 |
| druiz | So it's the user decision to "take the risk" or not | 14:29 |
| CarlosRuiz | AFAIR importing a file is an one single big commit | 14:29 |
| druiz | It's several single commits | 14:29 |
| CarlosRuiz | and if you use /K it makes commits per unique key ? | 14:29 |
| CarlosRuiz | that's my understanding | 14:29 |
| CarlosRuiz | still not checking code | 14:30 |
| druiz | The trx.commit is inside the for loop. And even before the process ends you can see data in the table you're committing | 14:31 |
| CarlosRuiz | I see - every line is a transaction | 14:38 |
| CarlosRuiz | unless is master - detail | 14:39 |
| *** norbertbede has quit IRC | 14:39 | |
| CarlosRuiz | in which case the master plus its details are a single trx | 14:39 |
| CarlosRuiz | my test here is showing a warning in console on every line "TrxName - Value too long - truncated to length=60" | 14:40 |
| CarlosRuiz | annoying | 14:40 |
| druiz | When do you get that message? | 14:41 |
| CarlosRuiz | eclipse consol | 14:41 |
| CarlosRuiz | console | 14:41 |
| druiz | But do you get that with 2630 patch? | 14:42 |
| tbayen | I makes sense in some cases to have a single trx for the whole file. And it makes sense in other cases to accept to loose some data. We can allow the user to choose between both cases. | 14:42 |
| CarlosRuiz | no, without patch | 14:43 |
| CarlosRuiz | I think we can trick the trx using the master detail approach | 14:43 |
| CarlosRuiz | that would make the process faster - commiting per line is adding big slowness | 14:44 |
| CarlosRuiz | the issue is that every commit is checked for success to record in the output log - so to "pack" commits in lots it would need a big refactoring | 14:45 |
| CarlosRuiz | but with master detail maybe you can do the trick | 14:45 |
| tbayen | I remember that I had problems with the "single trx" approach and a very long file. There is a trx timeout in Postgres that says that a transaction can not last longer than about one hour. That is an example for the "line by line" mode. That is the default actually. But with most small files the "one single commit" mode looks better. | 14:45 |
| *** norbertbede has joined #idempiere | 14:46 | |
| tbayen | CarlosRuiz, I realize that we need to commit to get a possible error message. Hmmm... | 14:47 |
| CarlosRuiz | tbayen, to make a single commit you can create a window for that | 14:52 |
| CarlosRuiz | suppose all your 5.000 records share the same organization | 14:53 |
| CarlosRuiz | you create a window with organization on first tab and your second tab is the table you're importing | 14:53 |
| CarlosRuiz | and do a master detail import | 14:53 |
| tbayen | Thats a hack. :-) | 14:55 |
| CarlosRuiz | I remember we thought about how to manage transactions | 14:56 |
| CarlosRuiz | and when made the master/detail as a trx - we thought - ok - there is a possible way here | 14:56 |
| tbayen | In the files that I used in the past I had errors because of wrong foreign keys and such. But these errors are caught by java code, not in the commit. I think that most human errors are caught there. Perhaps we can accept (in some cases) that we get sql errors for the whole file. | 14:57 |
| tbayen | I think we agree that it is not so easy to decide how to do it as it looks. It depends heavy on the use case, length of data, importance of data, etc. | 14:58 |
| tbayen | We can solve this either by makiing it heavy configurable or by using different plugins for different cases. | 14:58 |
| *** hieplq has quit IRC | 15:00 | |
| CarlosRuiz | just tested my theory :-) | 15:00 |
| CarlosRuiz | and it's correct - you can make it work as I said in a single trx - per org for example | 15:01 |
| CarlosRuiz | but you must use in such case the Merge option | 15:01 |
| *** tsvikruha has quit IRC | 15:01 | |
| CarlosRuiz | I think a single commit must speed up the import process more than the asynchronous commit | 15:05 |
| tbayen | We will test that. | 15:05 |
| *** nmicoud has quit IRC | 15:18 | |
| *** CarlosRuiz has quit IRC | 15:19 | |
| *** CarlosRuiz has joined #idempiere | 15:20 | |
| *** ChanServ sets mode: +o CarlosRuiz | 15:20 | |
| *** norbertbede has quit IRC | 15:24 | |
| *** KermitTheFragger has quit IRC | 15:32 | |
| *** norbertbede has joined #idempiere | 15:34 | |
| *** nmicoud has joined #idempiere | 15:40 | |
| *** nmicoud has quit IRC | 15:45 | |
| *** norbertbede has quit IRC | 15:52 | |
| *** a42niem_ has quit IRC | 15:54 | |
| *** norbertbede has joined #idempiere | 16:25 | |
| *** CarlosRuiz has quit IRC | 16:31 | |
| *** CarlosRuiz has joined #idempiere | 16:35 | |
| *** ChanServ sets mode: +o CarlosRuiz | 16:35 | |
| *** norbertbede has quit IRC | 16:59 | |
| *** CarlosRuiz has quit IRC | 17:01 | |
| *** CarlosRuiz has joined #idempiere | 17:04 | |
| *** ChanServ sets mode: +o CarlosRuiz | 17:04 | |
| *** norbertbede has joined #idempiere | 17:05 | |
| *** nmicoud has joined #idempiere | 17:06 | |
| *** is-mw has joined #idempiere | 17:49 | |
| CarlosRuiz | druiz, tbayen, still there? | 17:53 |
| *** CarlosRuiz has quit IRC | 18:05 | |
| *** norbertbede has quit IRC | 18:09 | |
| *** nmicoud has quit IRC | 18:13 | |
| *** ocurieles has quit IRC | 18:21 | |
| *** CarlosRuiz has joined #idempiere | 18:22 | |
| *** ChanServ sets mode: +o CarlosRuiz | 18:22 | |
| *** norbertbede has joined #idempiere | 18:26 | |
| *** norbertbede has quit IRC | 19:02 | |
| *** norbertbede has joined #idempiere | 19:03 | |
| *** ocurieles has joined #idempiere | 20:19 | |
| *** banym has joined #idempiere | 21:16 | |
| *** banym has quit IRC | 21:19 | |
| *** banym has joined #idempiere | 21:19 | |
| *** CarlosRuiz has quit IRC | 21:51 | |
| *** a42niem has quit IRC | 22:17 | |
| *** norbertbede has quit IRC | 22:18 | |
| *** CarlosRuiz has joined #idempiere | 22:23 | |
| *** ChanServ sets mode: +o CarlosRuiz | 22:23 | |
| *** norbertbede has joined #idempiere | 22:30 | |
| *** ocurieles has quit IRC | 22:34 | |
| *** norbertbede has quit IRC | 22:46 | |
| *** is-mw has quit IRC | 23:51 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!