Occasional random memory gobbling

For test results, bug reports, announcements of new builds etc.

Moderators: another_commander, winston, Getafix

User avatar
winston
Pirate
Pirate
Posts: 731
Joined: Mon Sep 27, 2004 10:21 pm
Location: Port St. Mary, Isle of Man
Contact:

Occasional random memory gobbling

Post by winston »

I'm occasionally seeing explanation-less and fairly random memory gobbling episodes. After several hours of uneventful play, occasionally Oolite will start allocating ridiculous amounts of memory. I just watched it going from smooth, 100fps lovelyness to causing the machine to hopelessly thrash until the kernel's oom killer killed the game. Once it started this memory usage increase, it just kept going up and up very rapidly.

I don't think it's specific to the Linux version - I suspect it's one of the post 1.55 changes (since it never used to happen). Unfortunately, every time things like this happens, it's always when I'm just trying to play a game rather than testing in the debugger!

oO
Competent
Competent
Posts: 32
Joined: Sat Oct 15, 2005 8:51 pm

Post by oO »

I saw an almost-OOM in rev 171, but i thought it was normal because i saw you added a few NSAutoreleasePool in 172.

The game went totally unresponsive and top reported 639Mb VIRT.
Then it became playable again for a while, then again unresponsive for a few times. Until it was frozen for a long time, top said 1.09Gb VIRT and remained there for some time until i got bored and killed it. (I got 0.5Gb ram + 1Gb swap).

This happened in the middle of a battle against a lot of thargoid warships, maybe it's because they release a lot of robofighters and they don't release memory or something when they die?

User avatar
winston
Pirate
Pirate
Posts: 731
Joined: Mon Sep 27, 2004 10:21 pm
Location: Port St. Mary, Isle of Man
Contact:

Post by winston »

No, the Thargoids have been a feature for a long time, and it's never done that (I think the bulk of oolite's allocated memory is cached textures - don't forget those PNGs will sit as *uncompressed* bitmaps in memory). Most texture-mapped games like allocating lots of memory.

This is different - my hunch is that it's not a memory leak in the traditional sense (i.e. whoops we forgot to free what we just made) but something triggering something that starts allocating memory in small chunks at a high rate. When it happened to me, the memory useage steadily climbed in megabytes per second from about 50% memory usage until the oom killer viciously knifed it in the back with signal 9. I have a hunch it may be related to the new wormhole stuff, but that's merely a hunch - it happens so infrequently that it's difficult to tell. The two times it's happened so far (in about 4 hours of play testing) has always been right after a wormhole event showed up in the console log. As I said it's merely a completely unscientific hunch. Wormholes seem to get created all the time without all my memory vanishing down them.

User avatar
JensAyton
Grand Admiral Emeritus
Grand Admiral Emeritus
Posts: 6657
Joined: Sat Apr 02, 2005 2:43 pm
Location: Sweden
Contact:

Post by JensAyton »

I’ve been playing with wormholes for a while and not seeing this. The Mac build did run up to 3.5 gigs or so, but that was when I had FoundationKit’s zombie tracking on (in which objects are never released) and played for a couple of hours…

User avatar
aegidian
Master and Commander
Master and Commander
Posts: 1152
Joined: Thu May 20, 2004 10:46 pm
Location: London UK
Contact:

Post by aegidian »

An out-of-control memory allocating loop or race-condition. That would explain a number of crashes on Macs I've been seeing - the NSScanner crashes in scanTokensFromString and some NSDictionary allocation crashes, I think that the Mac could be handling OOM problems rather quietly.

like this:

Code: Select all

Thread 0 Crashed:
0   <<00000000>> 	0xfffeff18 objc_msgSend_rtp + 24
1   com.apple.CoreFoundation       	0x9073d308 __CFDictionaryFindBuckets1b + 140
2   com.apple.CoreFoundation       	0x9073d24c CFDictionaryGetValue + 212
3   org.aegidian.oolite            	0x0001f400 -[TextureStore getTextureNameFor:] + 64 (TextureStore.m:86)
4   org.aegidian.oolite            	0x00022864 -[Entity initialiseTextures] + 104 (Entity.m:730)
5   org.aegidian.oolite            	0x0002251c -[Entity drawEntity::] + 652 (Entity.m:626)
6   org.aegidian.oolite            	0x00047a04 -[ShipEntity drawEntity::] + 156 (ShipEntity.m:2186)
7   org.aegidian.oolite            	0x0002270c -[Entity drawSubEntity::] + 192 (Entity.m:681)
8   org.aegidian.oolite            	0x00047a88 -[ShipEntity drawEntity::] + 288 (ShipEntity.m:2195)
9   org.aegidian.oolite            	0x00013b0c -[Universe drawFromEntity:] + 1572 (Universe.m:3333)
or this

Code: Select all

Thread 0 Crashed:
0   libSystem.B.dylib              	0x90003b24 szone_malloc + 200
1   libSystem.B.dylib              	0x900039fc malloc_zone_malloc + 100
2   com.apple.CoreFoundation       	0x90729c88 CFAllocatorAllocate + 296
3   com.apple.CoreFoundation       	0x90729a7c _CFRuntimeCreateInstance + 224
4   com.apple.CoreFoundation       	0x90778208 CFCharacterSetCreateMutable + 44
5   com.apple.CoreFoundation       	0x9077de24 __CFCharacterSetCreateCopy + 232
6   com.apple.CoreFoundation       	0x907931f0 CFCharacterSetCreateInvertedSet + 156
7   com.apple.Foundation           	0x928bf128 -[NSCFCharacterSet invertedSet] + 32
8   org.aegidian.oolite            	0x000219dc +[Entity scanTokensFromString:] + 148 (Entity.m:182)
9   org.aegidian.oolite            	0x000122b4 -[Universe getShipWithRole:] + 224 (Universe.m:2704)
10  org.aegidian.oolite            	0x00012d74 -[Universe getContainersOfScarceGoods:] + 276 (Universe.m:2938)
11  org.aegidian.oolite            	0x0004a05c -[ShipEntity becomeExplosion] + 1592 (ShipEntity.m:3220)
12  org.aegidian.oolite            	0x00050120 -[ShipEntity manageCollisions] + 196 (ShipEntity.m:5253)
13  org.aegidian.oolite            	0x00045d2c -[ShipEntity update:] + 236 (ShipEntity.m:1368)
14  org.aegidian.oolite            	0x0001884c -[Universe update:] + 1124 (Universe.m:4881)
I'm not sure what to do to investigate this properly.
"The planet Rear is scourged by well-intentioned OXZs."

Oolite models and gear? click here!

User avatar
JensAyton
Grand Admiral Emeritus
Grand Admiral Emeritus
Posts: 6657
Joined: Sat Apr 02, 2005 2:43 pm
Location: Sweden
Contact:

Post by JensAyton »

The szone_malloc things are not out-of-memory. I’ve had them and subsequently checked peak swap file usage; it’s always been less than free disk space and less than 4 gigs (the 32-bit process limit).

User avatar
winston
Pirate
Pirate
Posts: 731
Joined: Mon Sep 27, 2004 10:21 pm
Location: Port St. Mary, Isle of Man
Contact:

Post by winston »

Bump.
This is still occasaionally happening. There is no particular thing I can identify that triggers it, but it happened with the most recent SVN head on Linux tonight. It is perhaps happening once every 6 hours of play which makes it very difficult to tie to a particular event.

kaypy
Above Average
Above Average
Posts: 19
Joined: Sat Nov 19, 2005 5:16 am

Post by kaypy »

Hi

Got another few "oolite starts to sieze up" events today. What I wanted to add to the knowledge pool is:

Remembering what people said in one of the related threads about it possibly being the AI, I tried hyperspacing out when my system started to intermittently freeze. Once I got to a new system, the juddering stopped and I was able to get my ship home and save the game.

May be somewhat useful to debuggers and more useful to folks wanting to rescue their ships from the memory glitch of doom...

kp

kaypy
Above Average
Above Average
Posts: 19
Joined: Sat Nov 19, 2005 5:16 am

Post by kaypy »

And another one. This time I was taking dumps of "ps aux | grep oolite" periodically:

just started a new game
kaypy 6264 46.8 22.9 139196 118700 pts/1 SLl+ 17:19 0:35 ./oolite

still pretty normal
kaypy 6264 52.7 25.0 149748 129284 pts/1 SLl+ 17:19 2:47 ./oolite

about here would be the first inklings of trouble
kaypy 6264 57.6 34.0 196056 176008 pts/1 SLl+ 17:19 12:35 ./oolite

this would be juddering a fair bit... its time to bail out
kaypy 6264 57.4 37.1 212496 191620 pts/1 SLl+ 17:19 12:51 ./oolite

just before the warp hits. freezing badly now
kaypy 6264 57.6 65.7 366824kp 339684 pts/1 SLl+ 17:19 13:25 ./oolite

now in a new system. memory still up at 58%, but must all be no longer used and hence swapped out or something, because the freezing has stopped
kaypy 6264 57.3 58.1 327332 300244 pts/1 SLl+ 17:19 13:36 ./oolite

a bit later in the new system
kaypy 6264 58.0 57.0 321776 294696 pts/1 SLl+ 17:19 17:09 ./oolite

The freeze ups seem to happen when theres lots of fighting and looting. Ive had about 4 freeze ups today, all of which were "forget the salvage- the universe is about to die!" events...

kp

User avatar
TedJ
Deadly
Deadly
Posts: 131
Joined: Tue Jan 17, 2006 3:18 am
Location: Melbourne, Australia

Post by TedJ »

I've also been spotting this one occasionally (linux 1.62rc1&2), I've had several in the last 2-3 days... Usually happens to me after a large battle when I'm scooping cargo.

User avatar
winston
Pirate
Pirate
Posts: 731
Joined: Mon Sep 27, 2004 10:21 pm
Location: Port St. Mary, Isle of Man
Contact:

Post by winston »

Damn. I've not seen a recurrence since building 1.61. Can you take a look at 'top' next time it happens and report the game's size? (I've had separate performance issues with lave.oxp, but only with that one OXP, and it didn't involve excessive memory usage).

User avatar
Murgh
---- E L I T E ----
---- E L I T E ----
Posts: 454
Joined: Fri Sep 03, 2004 6:19 am
Location: Terra Firma

Post by Murgh »

winston wrote:I've had separate performance issues with lave.oxp, but only with that one OXP, and it didn't involve excessive memory usage
what can be the fault with it? all the crazy AI? too many AI and spawnings? the added moon?
The man next to you is your lunch

User avatar
winston
Pirate
Pirate
Posts: 731
Joined: Mon Sep 27, 2004 10:21 pm
Location: Port St. Mary, Isle of Man
Contact:

Post by winston »

I dunno yet, I really need to flush out my AddOns directory and get all the latest versions of the OXPs so I can make sure out of date or conflicting add-ons aren't causing the problems.

User avatar
TedJ
Deadly
Deadly
Posts: 131
Joined: Tue Jan 17, 2006 3:18 am
Location: Melbourne, Australia

Post by TedJ »

winston wrote:Damn. I've not seen a recurrence since building 1.61. Can you take a look at 'top' next time it happens and report the game's size? (I've had separate performance issues with lave.oxp, but only with that one OXP, and it didn't involve excessive memory usage).
Happened to me again last night, top reported vmem usage had shot to nearly 500MB (I only have 512MB swap), rss size was around 180 MB. I zapped X shortly thereafter, as the swapping was really thrashing my disk and we were having a very warm night (got to 43C yesterday).

EDIT: More details. Once again I was scooping cargo after tackling a Cobra 3 and an Asp, so I doubt it's an issue with an OXP, unless it's a problem with the nubarrel model in aegidian-special.

User avatar
TedJ
Deadly
Deadly
Posts: 131
Joined: Tue Jan 17, 2006 3:18 am
Location: Melbourne, Australia

Post by TedJ »

It just gets weirder... Oolite decided to dine heavily on my swap space again last night, only this time I was docked.

On Winston's advice, I've taken to keeping top running in a background terminal while playing, just to keep an eye on things. I have to admit being a little surprised how much CPU Oolite uses while paused (around 70-90% on my PIII 866).

Post Reply