Complete system hangs (likely out of memory)
Hi, When editing with Cinelerra (current git) I experience frequently that my entire system hangs after scratching through either the timeline or the viewer. As experienced before I have noticed that Cinerella eats a lot of memory. The total amount of assets I have currently in my project is about 70GB, my system memory is 16GB, and yesterday I have added 8GB of swap (normally no swap). It seems that with every video that I preview my memory usage significantly increases, but is never released. I think we have discussed this behavior on the mailinglist before being that the indices never get released. But still the amount does not make sense to me. As user my expectation would be that when I would close the *viewer*, the memory would get released, especially when the content is not placed on the timeline. Are there any statistics available on the memory usage what goes where? Practical question: is it possible to have a different path for the emergency backups than /tmp? -- Stefan
вс, 22 янв. 2023 г., 17:54 Stefan de Konink via Cin < cin@lists.cinelerra-gg.org>:
Hi,
When editing with Cinelerra (current git) I experience frequently that my entire system hangs after scratching through either the timeline or the viewer. As experienced before I have noticed that Cinerella eats a lot of memory.
The total amount of assets I have currently in my project is about 70GB, my system memory is 16GB, and yesterday I have added 8GB of swap (normally no swap).
It seems that with every video that I preview my memory usage significantly increases, but is never released. I think we have discussed this behavior on the mailinglist before being that the indices never get released. But still the amount does not make sense to me.
As user my expectation would be that when I would close the *viewer*, the memory would get released, especially when the content is not placed on the timeline.
Are there any statistics available on the memory usage what goes where?
can you use 'top' to see if memory eaten by cin process or X server?
Practical question: is it possible to have a different path for the emergency backups than /tmp?
hm, from code ... char string[BCTEXTLEN]; // Delete extra backups fs.set_filter("backup*.prev_*"); fs.complete_path(preferences->index_directory); fs.update(preferences->index_directory); so check preferences-> index directory path in GUI?
-- Stefan -- Cin mailing list Cin@lists.cinelerra-gg.org https://lists.cinelerra-gg.org/mailman/listinfo/cin
On Sunday, January 22, 2023 4:59:17 PM CET, Andrew Randrianasulu wrote:
can you use 'top' to see if memory eaten by cin process or X server?
This is rendering; 15761 skinkie 20 0 17.5g 9.7g 167088 S 625.6 62.2 33:24.43 cineler+ 772 skinkie 20 0 1161336 168508 85956 S 2.3 1.0 4:46.48 Xorg
Practical question: is it possible to have a different path for the emergency backups than /tmp?
hm, from code ...
char string[BCTEXTLEN];
// Delete extra backups fs.set_filter("backup*.prev_*"); fs.complete_path(preferences->index_directory); fs.update(preferences->index_directory);
so check preferences-> index directory path in GUI?
That goes into /home/skinkie/.bcast5 -- Stefan
вс, 22 янв. 2023 г., 19:13 Stefan de Konink <stefan@konink.de>:
On Sunday, January 22, 2023 4:59:17 PM CET, Andrew Randrianasulu wrote:
can you use 'top' to see if memory eaten by cin process or X server?
This is rendering;
15761 skinkie 20 0 17.5g 9.7g 167088 S 625.6 62.2 33:24.43 cineler+ 772 skinkie 20 0 1161336 168508 85956 S 2.3 1.0 4:46.48 Xorg
--
quite a lot, is this with just fullhd/rgba-8 going into h264, or something bigger? I looked at sourci in viewer and there slight difference in handling object destruction vs close event. Are quo closing by 'x' button in window corner or keyboard key? diff void VWindow::handle_close_event(int result) { delete playback_engine; playback_engine = 0; } vs VWindow::~VWindow() { close_window(); //printf("VWindow::~VWindow 1\n"); delete playback_engine; //printf("VWindow::~VWindow 1\n"); delete playback_cursor; delete_source(1, 0); delete clip_edit; //printf("VWindow::~VWindow 2\n"); } you can try to add lines from second to first, and see how it goes ...
Practical question: is it possible to have a different path for the emergency backups than /tmp?
hm, from code ...
char string[BCTEXTLEN];
// Delete extra backups fs.set_filter("backup*.prev_*"); fs.complete_path(preferences->index_directory); fs.update(preferences->index_directory);
so check preferences-> index directory path in GUI?
That goes into /home/skinkie/.bcast5 -- Stefan
On Sunday, January 22, 2023 5:22:15 PM CET, Andrew Randrianasulu wrote:
quite a lot, is this with just fullhd/rgba-8 going into h264, or something bigger?
My input comes from a hacked Nikon D3200 giving me FullHD, 64Mbit in h264. With only previewing these files I can already push towards the boundaries of my internal memory.
I looked at sourci in viewer and there slight difference in handling object destruction vs close event. Are quo closing by 'x' button in window corner or keyboard key?
The x in the corner.
you can try to add lines from second to first, and see how it goes ...
Interesting approach :) -- Stefan
On 1/22/23 11:31, Stefan de Konink via Cin wrote:
My input comes from a hacked Nikon D3200 giving me FullHD, 64Mbit in h264. With only previewing these files I can already push towards the boundaries of my internal memory.
Since resource utilization is kind of a thing with me, I'm curiously watching this thread. fullHD? What does that mean in terms of resolution and frame rate, and D3200 native codec format? What is the container format you are pulling from the camera? Also, what happens if you render to a different format? mjpeg? does the memory bloat happen slowly as rendering proceeds? is it a linear increase or does it happen after some amount of rendering frames? common cinelerra transcoding/rendering uses ffmpeg/libavcodec as a backend. I'd be curious if you see similar issue if you transcode your content directly, independently of cinelerra ffmpeg -i source -vf scale=1280:720 -c:v x264 -r:v 29.97 -c:a aac -o out.mp4
On Sunday, January 22, 2023 7:05:51 PM CET, Rob Prowel wrote:
On 1/22/23 11:31, Stefan de Konink via Cin wrote:
My input comes from a hacked Nikon D3200 giving me FullHD, 64Mbit in h264. With only previewing these files I can already push towards the boundaries of my internal memory.
Since resource utilization is kind of a thing with me, I'm curiously watching this thread. fullHD? What does that mean in terms of resolution and frame rate, and D3200 native codec format? What is the container format you are pulling from the camera?
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '_DSC0397.MOV': Metadata: major_brand : qt minor_version : 537331968 compatible_brands: qt niko creation_time : 2023-01-21T16:43:02.000000Z Duration: 00:01:50.52, start: 0.000000, bitrate: 61237 kb/s Stream #0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, smpte170m/bt709/bt470m, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 59681 kb/s, 25 fps, 25 tbr, 25k tbn (default) Metadata: creation_time : 2023-01-21T16:43:02.000000Z vendor_id : [0][0][0][0] Stream #0:1[0x2](eng): Audio: pcm_s16le (sowt / 0x74776F73), 48000 Hz, stereo, s16, 1536 kb/s (default) Metadata: creation_time : 2023-01-21T16:43:02.000000Z vendor_id : [0][0][0][0]
Also, what happens if you render to a different format? mjpeg?
No experience with mjpeg in this context yet.
does the memory bloat happen slowly as rendering proceeds? is it a linear increase or does it happen after some amount of rendering frames?
It has nothing to do with rendering. This occurs when being in Resources > Media and start a few viewers to review my material and make some clips out of them. -- Stefan
On 1/22/23 13:14, Stefan de Konink wrote:
It has nothing to do with rendering. This occurs when being in Resources
Media and start a few viewers to review my material and make some clips out of them.
"start a few viewers"? I believe that you're going to quickly eat up a lot of memory. each viewer times the hugemoungous resolution and bitrate of your clips, and assuming that a portion/window of that content will need to be buffered "uncompressed" so that it can be viewed in real-time. I think that if you pre-transcode your video (using ffemp) to a smaller size and lower bitrate then you won't see these problems. Andrew can better answer whether multiple concurrent viewers of hi-res/hi-bitrate content will be memory hogs. That's where I'd lay my money.
On Sun, 22 Jan 2023, Rob Prowel via Cin wrote:
My input comes from a hacked Nikon D3200 giving me FullHD, 64Mbit in h264. With only previewing these files I can already push towards the
Somewhen while trying to play videos taken by Nikon (D7100) I observed that mplayer is extremely slow on them (it even complained that I have too slow processor etc. and then seemed to hang) while some other players had no problem with them. This appeared to be due to the occurence of some frames with very special properties which mplayer tried to interpret but the other players perhaps ignored them. I do not know video formats so deep and cannot say which particular property it was. The following option to mplayer overcame the problem, the videos were played smoothly (and without any visual difference): mplayer -lavdopts skiploopfilter=nonkey ... To be able to use such source videos in Cinelerra I simply preprocessed them with ffmpeg (recoding into x264 with default options) before loading into Cinelerra. _______________________________________________________________________________ Georgy Salnikov NMR Group Novosibirsk Institute of Organic Chemistry Lavrentjeva, 9, 630090 Novosibirsk, Russia Phone +7-383-3307864 Email sge@nmr.nioch.nsc.ru _______________________________________________________________________________
пн, 23 янв. 2023 г., 06:15 Georgy Salnikov via Cin < cin@lists.cinelerra-gg.org>:
On Sun, 22 Jan 2023, Rob Prowel via Cin wrote:
My input comes from a hacked Nikon D3200 giving me FullHD, 64Mbit in h264. With only previewing these files I can already push towards the
Somewhen while trying to play videos taken by Nikon (D7100) I observed that mplayer is extremely slow on them (it even complained that I have too slow processor etc. and then seemed to hang) while some other players had no problem with them. This appeared to be due to the occurence of some frames with very special properties which mplayer tried to interpret but the other players perhaps ignored them. I do not know video formats so deep and cannot say which particular property it was.
The following option to mplayer overcame the problem, the videos were played smoothly (and without any visual difference):
mplayer -lavdopts skiploopfilter=nonkey ...
I wonder if just putting "skiploopfilter=nokey" in filename.opts alongside original filename will be enough for cingg to do the same, without re-encoding?
To be able to use such source videos in Cinelerra I simply preprocessed them with ffmpeg (recoding into x264 with default options) before loading into Cinelerra.
_______________________________________________________________________________
Georgy Salnikov NMR Group Novosibirsk Institute of Organic Chemistry Lavrentjeva, 9, 630090 Novosibirsk, Russia Phone +7-383-3307864 Email sge@nmr.nioch.nsc.ru
_______________________________________________________________________________
-- Cin mailing list Cin@lists.cinelerra-gg.org https://lists.cinelerra-gg.org/mailman/listinfo/cin
On Mon, 23 Jan 2023, Andrew Randrianasulu wrote:
mplayer -lavdopts skiploopfilter=nonkey ...
I wonder if just putting "skiploopfilter=nokey" in filename.opts alongside original filename will be enough for cingg to do the same, without re-encoding?
The same codec options in ffmpeg and in mplayer/mencoder can have different names. That person can give the answer who knows what the option really means in the technical sense, not me. _______________________________________________________________________________ Georgy Salnikov NMR Group Novosibirsk Institute of Organic Chemistry Lavrentjeva, 9, 630090 Novosibirsk, Russia Phone +7-383-3307864 Email sge@nmr.nioch.nsc.ru _______________________________________________________________________________
пн, 23 янв. 2023 г., 07:44 Georgy Salnikov via Cin < cin@lists.cinelerra-gg.org>:
On Mon, 23 Jan 2023, Andrew Randrianasulu wrote:
mplayer -lavdopts skiploopfilter=nonkey ...
I wonder if just putting "skiploopfilter=nokey" in filename.opts alongside original filename will be enough for cingg to do the same, without re-encoding?
The same codec options in ffmpeg and in mplayer/mencoder can have different names. That person can give the answer who knows what the option really means in the technical sense, not me.
oh, you are right, sorry! "Skip loop filtering for selected frames." enum AVDiscard skip_loop_filter; https://ffmpeg-devel.ffmpeg.narkive.com/1eHu7Ali/patch-lavc-extend-documenta... still present in thirdparty/ffmpeg-5.1/libavcodec/avcodec.h
_______________________________________________________________________________
Georgy Salnikov NMR Group Novosibirsk Institute of Organic Chemistry Lavrentjeva, 9, 630090 Novosibirsk, Russia Phone +7-383-3307864 Email sge@nmr.nioch.nsc.ru
_______________________________________________________________________________
-- Cin mailing list Cin@lists.cinelerra-gg.org https://lists.cinelerra-gg.org/mailman/listinfo/cin
Hi Andrew, Below dooes not give me any significant difference. I have opened 5 viewers, the memory rose from 4GB to 5.2GB, after closing the windows the memory reduced to 5.1GB. diff --git a/cinelerra-5.1/cinelerra/vwindow.C b/cinelerra-5.1/cinelerra/vwindow.C index 7be15eb8..d477dab1 100644 --- a/cinelerra-5.1/cinelerra/vwindow.C +++ b/cinelerra-5.1/cinelerra/vwindow.C @@ -160,6 +160,9 @@ void VWindow::handle_close_event(int result) { delete playback_engine; playback_engine = 0; + delete playback_cursor; + delete_source(1, 0); + delete clip_edit; } With the change I have also hit the issue below upon closing the main application with a viewer open; valgrind is pretty convincing this was caused by the change. My guess 'delete clip_edit' is at least one of the issues. double free or corruption (!prev) After reverting the change, I tried some more valgrinding. Cinelerra is rather slugish with valgrind on, but I have attached, what I see on viewing some smaller files. -- Stefan
вс, 22 янв. 2023 г., 21:14 Stefan de Konink <stefan@konink.de>:
Hi Andrew,
Below dooes not give me any significant difference. I have opened 5 viewers, the memory rose from 4GB to 5.2GB, after closing the windows the memory reduced to 5.1GB.
diff --git a/cinelerra-5.1/cinelerra/vwindow.C b/cinelerra-5.1/cinelerra/vwindow.C index 7be15eb8..d477dab1 100644 --- a/cinelerra-5.1/cinelerra/vwindow.C +++ b/cinelerra-5.1/cinelerra/vwindow.C @@ -160,6 +160,9 @@ void VWindow::handle_close_event(int result) { delete playback_engine; playback_engine = 0; + delete playback_cursor; + delete_source(1, 0); + delete clip_edit; }
With the change I have also hit the issue below upon closing the main application with a viewer open; valgrind is pretty convincing this was caused by the change. My guess 'delete clip_edit' is at least one of the issues.
double free or corruption (!prev)
yeah, then it was bad idea (without additional checks at keast)
After reverting the change, I tried some more valgrinding. Cinelerra is rather slugish with valgrind on, but I have attached, what I see on viewing some smaller files.
==128034== 3,315,172 (112 direct, 3,315,060 indirect) bytes in 1 blocks are definitely lost in loss record 2,957 of 2,962 ==128034== at 0x4843003: operator new(unsigned long) (vg_replace_malloc.c:422) ==128034== by 0x985E79: CICache::check_out(Asset*, EDL*, int) (in /opt/cingg/bin/cin) ==128034== by 0x9304A4: AModule::import_samples(AEdit*, long, long, long, int, int, Samples*, long) (in /opt/cingg/bin/cin) ==128034== by 0x930FCE: AModule::render(Samples*, long, long, int, int, int) (in /opt/cingg/bin/cin) ==128034== by 0xBA389D: VirtualANode::read_data(Samples*, long, long, long) (in /opt/cingg/bin/cin) ==128034== by 0xBA370A: VirtualANode::render_as_module(Samples**, Samples*, long, long, long) (in /opt/cingg/bin/cin) ==128034== by 0xBA375A: VirtualANode::render(Samples*, long, long, long) (in /opt/cingg/bin/cin) ==128034== by 0xBA2442: VirtualAConsole::process_buffer(long, long) (in /opt/cingg/bin/cin) ==128034== by 0x93AA05: ARender::run() (in /opt/cingg/bin/cin) ==128034== by 0xC84DD4: Thread::entrypoint(void*) (in /opt/cingg/bin/cin) ==128034== by 0x67598FC: start_thread (pthread_create.c:442) ==128034== by 0x67DAF33: clone (clone.S:100) ==128034== ==128034== 12,190,657 (480 direct, 12,190,177 indirect) bytes in 1 blocks are definitely lost in loss record 2,961 of 2,962 ==128034== at 0x4843003: operator new(unsigned long) (vg_replace_malloc.c:422) ==128034== by 0xA0E107: FFMPEG::open_decoder() (in /opt/cingg/bin/cin) ==128034== by 0xA1928B: FileFFMPEG::open_file(int, int) (in /opt/cingg/bin/cin) ==128034== by 0xA2A933: File::open_file(Preferences*, Asset*, int, int) (in /opt/cingg/bin/cin) so, something leaks but is it due to mussing free() on exit or also the case for normal operations ... it seems cache is involved, so may be I messed this up somehow. Does leak exist in previous cin-gg, like one from 2021 or 2020?
-- Stefan
Just a little feedback too. 1) "does closing the *viewer*, release the memory" -- I could be wrong, but I do not think closing the viewer releases the memory because it thinks you may want to use it again? 2) "backups on /tmp"? -- I am not sure how you are getting backups to go to /tmp. As far as I know they always go to $HOME/.bcast5. If you see files like: backup.prev_20230122_113922 backup.prev_20230122_114222 backup.prev backup.xml going to /tmp, then that could be a bug because I don't know how they got there. Index files, marker files, snapshot files, and nested proxy files can be written somewhere else via Settings->Preferences, but not backups. 3) I have found that adding swap is a complete waste of time - it never works for me and just hangs (and that has been true for me since the 1980's!) 4) in Valgrind, I always look at these lines at the end of the file: ==128034== LEAK SUMMARY: ==128034== definitely lost: 5,086 bytes in 40 blocks ==128034== i*ndirectly lost: 15,535,063 bytes *in 1,143 blocks ==128034== possibly lost: 350,864 bytes in 10 blocks The "definitely lost" line is reasonable BUT "indirectly lost" seems astronomical.
so, something leaks but is it due to mussing free() on exit or also the case for normal operations ... it seems cache is involved, so may be I messed this up somehow. Does leak exist in previous cin-gg, like one from 2021 or 2020?
I am going to test this too.
On Sunday, January 22, 2023 8:18:48 PM CET, Phyllis Smith wrote:
Just a little feedback too. 1) "does closing the *viewer*, release the memory" -- I could be wrong, but I do not think closing the viewer releases the memory because it thinks you may want to use it again?
That would be a futile attempt, if cache will grow beyond main memory.
2) "backups on /tmp"? -- I am not sure how you are getting backups to go to /tmp. As far as I know they always go to $HOME/.bcast5. If you see files like:
I think I remember the non-gg cinelerra then. There are indeed no files in /tmp.
3) I have found that adding swap is a complete waste of time - it never works for me and just hangs (and that has been true for me since the 1980's!)
It was worth the try, but I would agree with you. I never have it on any production system either.
4) in Valgrind, I always look at these lines at the end of the file: ==128034== LEAK SUMMARY: ==128034== definitely lost: 5,086 bytes in 40 blocks ==128034== indirectly lost: 15,535,063 bytes in 1,143 blocks ==128034== possibly lost: 350,864 bytes in 10 blocks The "definitely lost" line is reasonable BUT "indirectly lost" seems astronomical.
This were only a few small 3gp phone clips. I am happy to investigate further, but there is likely a serious bug that everyone must be hitting at some point. Or people just don't produce as much content as I do for one session ;) -- Stefan
Some testing results below.
Just a little feedback too.
1) "does closing the *viewer*, release the memory" -- I could be wrong, but I do not think closing the viewer releases the memory because it thinks you may want to use it again?
That would be a futile attempt, if cache will grow beyond main memory.
You are right - it does appear that closing the viewer does release the memory.
... This were only a few small 3gp phone clips. I am happy to investigate further, but there is likely a serious bug that everyone must be hitting at some point. Or people just don't produce as much content as I do for one session ;)
It does look like a very serious bug, but I do not think it is opening new viewers. I ran valgrind, loaded 2 smaller mp4 files into the Resources window, opened new viewer for the first one and played it by itself in the viewer, then opened another new viewer for the second mp4 file and then played that, and finally just Quit. Results were: ==180139== LEAK SUMMARY: ==180139== definitely lost: 0 bytes in 0 blocks ==180139== indirectly lost: 0 bytes in 0 blocks ==180139== possibly lost: 0 bytes in 0 blocks I have more tests to do yet.
Problem Solved and some final feedback: Memory leak fix has been checked into GIT. The problem started Dec. 27 so the AppImages created on Dec. 31 will have the problem but not very many users update theirs on a monthly basis. *All *- the Cinelerra program code is really quite complicated so problems like this happen. It will never be bug-free or easy to modify, but still we need to attempt changes to move forward. *Stefan* - thanks so very much for reporting this issue and your persistence in working on it. It is a BIG help when users do their own builds because we detect problems earlier that way. Initially my first attempt at creating the problem opening new viewers did not create the memory leak, apparently because the video contained no audio (so that was a bad test on my part). The EDL undo problem does not crash for me, but even crashing on other systems is not nice. To test on the current GIT: 1) Load a jpg image, it will expand itself as default 3 sec long timeline. 2) "Save as" with xml extension. 3) Load this xml as nested. 4) Hit 'undo' right away. For Andrew, it crashes. *Rob* - you brought up a lot of good questions to investigate which I will keep in mind going forward. Especially something to keep in mind is to "transcode your content directly, independently of cinelerra" - this can make the input more palatable to Cinelerra. *Georgy* - after reading your email about "some frames with very special properties", I downloaded a video sample and ran "strings" on it (because that is about all I knew to do) and certainly found a bunch of weird stuff after the initial attributes and before the actual data. And sections between as well as on the end. It must be some kind of filler but it is very strange looking (in following, lines were broken up): ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZ] )iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii iiiiiiiiiiiiiiiiiiiiiiiiiiiix! *Andrew *- I tried again to create the EDL undo crash and still could not, but I still have not booted the older distros and 32-bit to check them. I hope to have time tomorrow to do that and still test the netbsd patches from January 12 on those partitions. Maybe there is a better solution to fix the EDL undo crash because the hope is to never crash. *BTW*: filename.opts is one feature of CinGG that is underutilized. But it is not that well explained in the manual. https://cinelerra-gg.org/download/CinelerraGG_Manual/Raw_Input_Opts_File_Vid... https://cinelerra-gg.org/download/CinelerraGG_Manual/Speed_up_Ffmpeg_plugin_... On Sun, Jan 22, 2023 at 7:54 AM Stefan de Konink via Cin < cin@lists.cinelerra-gg.org> wrote:
Hi,
When editing with Cinelerra (current git) I experience frequently that my entire system hangs after scratching through either the timeline or the viewer. As experienced before I have noticed that Cinerella eats a lot of memory.
The total amount of assets I have currently in my project is about 70GB, my system memory is 16GB, and yesterday I have added 8GB of swap (normally no swap).
It seems that with every video that I preview my memory usage significantly increases, but is never released. I think we have discussed this behavior on the mailinglist before being that the indices never get released. But still the amount does not make sense to me.
As user my expectation would be that when I would close the *viewer*, the memory would get released, especially when the content is not placed on the timeline.
Are there any statistics available on the memory usage what goes where?
Practical question: is it possible to have a different path for the emergency backups than /tmp?
-- Stefan -- Cin mailing list Cin@lists.cinelerra-gg.org https://lists.cinelerra-gg.org/mailman/listinfo/cin
ср, 25 янв. 2023 г., 04:50 Phyllis Smith via Cin <cin@lists.cinelerra-gg.org
:
Problem Solved and some final feedback:
Memory leak fix has been checked into GIT. The problem started Dec. 27 so the AppImages created on Dec. 31 will have the problem but not very many users update theirs on a monthly basis.
*All *- the Cinelerra program code is really quite complicated so problems like this happen. It will never be bug-free or easy to modify, but still we need to attempt changes to move forward.
*Stefan* - thanks so very much for reporting this issue and your persistence in working on it. It is a BIG help when users do their own builds because we detect problems earlier that way. Initially my first attempt at creating the problem opening new viewers did not create the memory leak, apparently because the video contained no audio (so that was a bad test on my part). The EDL undo problem does not crash for me, but even crashing on other systems is not nice. To test on the current GIT: 1) Load a jpg image, it will expand itself as default 3 sec long timeline. 2) "Save as" with xml extension. 3) Load this xml as nested. 4) Hit 'undo' right away. For Andrew, it crashes.
*Rob* - you brought up a lot of good questions to investigate which I will keep in mind going forward. Especially something to keep in mind is to "transcode your content directly, independently of cinelerra" - this can make the input more palatable to Cinelerra.
*Georgy* - after reading your email about "some frames with very special properties", I downloaded a video sample and ran "strings" on it (because that is about all I knew to do) and certainly found a bunch of weird stuff after the initial attributes and before the actual data. And sections between as well as on the end. It must be some kind of filler but it is very strange looking (in following, lines were broken up):
ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZ] )iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii iiiiiiiiiiiiiiiiiiiiiiiiiiiix!
*Andrew *- I tried again to create the EDL undo crash and still could not, but I still have not booted the older distros and 32-bit to check them. I hope to have time tomorrow to do that and still test the netbsd patches from January 12 on those partitions. Maybe there is a better solution to fix the EDL undo crash because the hope is to never crash.
well, I think cin* tend to exploit threading in interesting ways, not obvious even to good programmer, just like it was with bcast2000: http://mstation.org/bcast.html ===== Threading offloaded all the concurrency requirements to the operating system, saving another chunk of coding time. Every window is a separate thread. Almost every object is a thread. It fills up your process table, not as elegant as classical signal handling, but it's fast to write. ===== nested edl a bit never feature exploiting same infrastructure. SO, yes, in hindsight I should be realize earlier my "fix" just prevent object deletion in most normal cases, I was too high on apparent "yay, FIX" feeling for checking memory consumption. My system a bit more weird than just 32-bit, it actually 64-bit kernel with 32 bit userspace, so you probably need to put "/" of 32bit distro somewhere on your normally 64-bit OS and then chroot into there. Yet, I am afraid even this might be not enough due to glibc/gcc/patches difference ....
*BTW*: filename.opts is one feature of CinGG that is underutilized. But it is not that well explained in the manual.
https://cinelerra-gg.org/download/CinelerraGG_Manual/Raw_Input_Opts_File_Vid...
https://cinelerra-gg.org/download/CinelerraGG_Manual/Speed_up_Ffmpeg_plugin_...
On Sun, Jan 22, 2023 at 7:54 AM Stefan de Konink via Cin < cin@lists.cinelerra-gg.org> wrote:
Hi,
When editing with Cinelerra (current git) I experience frequently that my entire system hangs after scratching through either the timeline or the viewer. As experienced before I have noticed that Cinerella eats a lot of memory.
The total amount of assets I have currently in my project is about 70GB, my system memory is 16GB, and yesterday I have added 8GB of swap (normally no swap).
It seems that with every video that I preview my memory usage significantly increases, but is never released. I think we have discussed this behavior on the mailinglist before being that the indices never get released. But still the amount does not make sense to me.
As user my expectation would be that when I would close the *viewer*, the memory would get released, especially when the content is not placed on the timeline.
Are there any statistics available on the memory usage what goes where?
Practical question: is it possible to have a different path for the emergency backups than /tmp?
-- Stefan -- Cin mailing list Cin@lists.cinelerra-gg.org https://lists.cinelerra-gg.org/mailman/listinfo/cin
-- Cin mailing list Cin@lists.cinelerra-gg.org https://lists.cinelerra-gg.org/mailman/listinfo/cin
Hi Phyllis, I would like to understand why you have removed the first unlock. // cache deleted during checkout, destroy this if( users == 1 ) { remove_user(); - total_lock->unlock(); return 0; } -- Stefan
ср, 25 янв. 2023 г., 11:48 Stefan de Konink via Cin < cin@lists.cinelerra-gg.org>:
Hi Phyllis,
I would like to understand why you have removed the first unlock.
It does not fix any bug I know of, so probably safest way to leave code as it was before me (because threads!) and probably revisit if some tool show issue or user report it.
// cache deleted during checkout, destroy this if( users == 1 ) { remove_user(); - total_lock->unlock(); return 0; }
-- Stefan -- Cin mailing list Cin@lists.cinelerra-gg.org https://lists.cinelerra-gg.org/mailman/listinfo/cin
On Wednesday, January 25, 2023 9:51:46 AM CET, Andrew Randrianasulu wrote:
It does not fix any bug I know of, so probably safest way to leave code as it was before me (because threads!) and probably revisit if some tool show issue or user report it.
Developing software is not equal to copy-pasting. I would like to see an explanation why someone would not want to unlock, while that happens in any other case. -- Stefan
ср, 25 янв. 2023 г., 11:56 Stefan de Konink <stefan@konink.de>:
On Wednesday, January 25, 2023 9:51:46 AM CET, Andrew Randrianasulu wrote:
It does not fix any bug I know of, so probably safest way to leave code as it was before me (because threads!) and probably revisit if some tool show issue or user report it.
Developing software is not equal to copy-pasting. I would like to see an explanation why someone would not want to unlock, while that happens in any other case.
Just for making things more clear: *I* added this unlock, not because I knew what I was doing but simply because yeah, it looked like this place might need it. In theory oicks protects shared resources, so one thread may leave object/variable around without worry another thread scheduled by kernel will overwrite it. But I do not know enough of details for carelessly altering locking like I did. So, Phyllis removed this line on my request, because, I reiterate, I put it there literally on a whim.
-- Stefan
On Wednesday, January 25, 2023 10:05:36 AM CET, Andrew Randrianasulu wrote:
Just for making things more clear: *I* added this unlock, not because I knew what I was doing but simply because yeah, it looked like this place might need it.
In theory oicks protects shared resources, so one thread may leave object/variable around without worry another thread scheduled by kernel will overwrite it.
But I do not know enough of details for carelessly altering locking like I did. So, Phyllis removed this line on my request, because, I reiterate, I put it there literally on a whim.
I think you did fix a bug at this place, and I wonder how total lock would get unlocked (other than maybe time out?). -- Stefan
ср, 25 янв. 2023 г., 12:08 Stefan de Konink <stefan@konink.de>:
On Wednesday, January 25, 2023 10:05:36 AM CET, Andrew Randrianasulu wrote:
Just for making things more clear: *I* added this unlock, not because I knew what I was doing but simply because yeah, it looked like this place might need it.
In theory oicks protects shared resources, so one thread may leave object/variable around without worry another thread scheduled by kernel will overwrite it.
But I do not know enough of details for carelessly altering locking like I did. So, Phyllis removed this line on my request, because, I reiterate, I put it there literally on a whim.
I think you did fix a bug at this place, and I wonder how total lock would get unlocked (other than maybe time out?).
I guess we can scratch our collective head over this for some more time, because IIRC Phyllis currently sleeping (hopefully). I do not remember any simple way to trigger this codepath, but may be I'll rediscover something .... sorry for being more clueless than everyone hoped.
-- Stefan
On Wednesday, January 25, 2023 10:13:11 AM CET, Andrew Randrianasulu wrote:
I do not remember any simple way to trigger this codepath, but may be I'll rediscover something ....
sorry for being more clueless than everyone hoped.
In my perspective the code need (a lot) more comments. And I would suggest also a developer document how the different files relate to eachother. I think a screenshot annotating what happens in the background and a flow diagram for parts would certainly help. Finding it out via backtraces is not the way it should be done. -- Stefan
ср, 25 янв. 2023 г., 12:16 Stefan de Konink <stefan@konink.de>:
On Wednesday, January 25, 2023 10:13:11 AM CET, Andrew Randrianasulu wrote:
I do not remember any simple way to trigger this codepath, but may be I'll rediscover something ....
sorry for being more clueless than everyone hoped.
In my perspective the code need (a lot) more comments. And I would suggest also a developer document how the different files relate to eachother. I think a screenshot annotating what happens in the background and a flow diagram for parts would certainly help. Finding it out via backtraces is not the way it should be done.
yeahhhh . Problem is, Adam is not very much interested in adding more comments than it was there originally, Bill for obvious reason can't answer any question anymore, and Einar mostly works on different codebase .... There is something autogenerated, but for me it hardly helpful! https://fossies.org/dox/cin_5.1.20221231-src/classColorWindow.html (for example) ps: /me putting misleading comments based on misunderstanding actually not great service ....
-- Stefan
participants (5)
-
Andrew Randrianasulu -
Georgy Salnikov -
Phyllis Smith -
Rob Prowel -
Stefan de Konink