View Issue Details

IDProjectCategoryView StatusLast Update
0001951FSSCPmultiplayerpublic2020-07-08 19:20
ReporterFUBAR-BDHR Assigned ToFSCyborg  
PriorityhighSeveritycrashReproducibilityrandom
Status closedResolutionunable to reproduce 
Product Version3.6.11 
Summary0001951: Standalone crashes - Ship <ship>does not have subsystem <subsystem> linked into the model file
DescriptionBeen getting a bunch of new standalone crashes in the last couple of weeks. Looks like it's 2 different ones but they may be connected since they both crash at the same line sometimes. First one is a bunch of assertions that always seem to start when a Lilith model is loaded. This usually leads to the second one which is a crash but this crash also happens on it's own. Attaching logs
TagsNo tags attached.

Relationships

related to 0002015 resolvedEchelon9 Assert: team != -1 in multi_respawn.cpp 
related to 0002052 closedFSCyborg Standalone - Int(3) ship_get_indexed_subsys - Player drop related? 

Activities

2009-07-05 20:32

 

1951.rar (46,094 bytes)

FUBAR-BDHR

2009-07-05 20:33

developer   ~0011053

Just a note that this was with retail FS2 data running on the standalone.

2009-07-08 20:02

 

fs2_open_vp_1733.log (190,919 bytes)

FUBAR-BDHR

2009-07-08 20:03

developer   ~0011062

Attached additional log. First time seen on the MediaVP standalone. Slightly different set of problems but crashed at the same spot.

FUBAR-BDHR

2009-07-09 04:15

developer   ~0011063

OK this has officially become an epidemic. Attached another log. This one shows something that I don't know if the others do or not. The Lilith model loads fine at least one time before it goes nuts and starts with the assertions on another load.

2009-07-09 04:57

 

fs2_open_both.rar (6,320 bytes)

FUBAR-BDHR

2009-07-11 23:58

developer   ~0011075

No crashes in today's session using the RC3 no warnings build. Quite a few asserts but they are all the same. This was with the MediaVP. Attaching log just in case.

2009-07-11 23:59

 

fs2_open-07-11-09.log (184,591 bytes)

portej05

2009-07-14 08:22

reporter   ~0011083

Really need the callstack for this one.
The culprit is vm_project_point_onto_plane, which is only called from 3 places, but tracing through is going to be very hard without knowing which one is being called.

2009-07-19 07:20

 

fs2_open-07-18-09.log (88,164 bytes)

FUBAR-BDHR

2009-07-19 07:22

developer   ~0011091

Last edited: 2009-07-19 07:25

Well it seems the big problems are well patched. Standalone did crash once in yesterdays games. I've seen the same crash before. Log attached. I asked them if someone dropped before the game started and was told that someone was lost in mission load. Same mission was played after restart with no crash.

portej05

2009-07-19 07:31

reporter   ~0011092

I'd still like to see a callstack from when this occurs since it implies that a vector being passed to vm_project_point_onto_plane hasn't been normalised, which is really bad.
Are there any instructions for getting this to occur?

FUBAR-BDHR

2009-07-19 07:46

developer   ~0011094

Didn't seen any nul vec3d errors today just the the one crash.

FUBAR-BDHR

2009-07-25 22:19

developer   ~0011105

Still no luck on getting a debug trace. It's up for testing but since today was the normal Saturday games they were using 3.6.10. Did get another one of those dropped player crashes. Had the debug_filter turned on. I'd attach the file but it's 75meg and even zipped still over a meg. You can grab it here: http://fubar5.fubar.org/mantis/fs2_open-07-25-09_MVP_1733.rar

portej05

2009-07-27 02:08

reporter   ~0011106

FUBAR: With this one, it might be worth the inconvenience of dropping some players in order to get a callstack - Asserts should never be ignored (who stuck Cmdline_nowarn into Assert?)

FUBAR-BDHR

2009-07-27 04:32

developer   ~0011108

Already planned and the server to do that is up and running. Just waiting on our main tester to get unsick and give it a go. I didn't want to use the main Saturday games since you never know what version they are running, if they are using the MediaVPs or not, if they have any rogue tables, etc. That and there aren't that many multi players now. Hate turning new players off because of testing crashes.

FUBAR-BDHR

2009-08-07 00:17

developer   ~0011126

Well we got to hits with the debugger yesterday. One as a client but it was a null vec3d so it's possibly related. The other was on the standalone and involved the player drop problem (still have the debugger open on this one). The think is it doesn't look like it was actually a player drop. I just happened to check on the standalone at the right time to actually see it happen. 6 IP addresses where showing as connected on the standalone but I could only see 3 players in the status window. Right after that it crashed. I asked and there were 6 or 7 people in the game. Looking at the multi.log earlier today I noticed that there was a FS2NetD refresh right about the time of the crash. This makes me wonder if it is that refresh that is causing the issue. It does cause problems for normal hosting such as the awarding of all medals in debrief bug.

FUBAR-BDHR

2009-08-07 02:32

developer   ~0011127

OK I just saw this about to happen again. 3 players on server and even chatting but none showing in the player info pull down. Checked the debug log on the stnadalone and it had just done a FS2netD refresh. I hit reset all before it crashed and when the joined again the player info screen was fine. Definitely something going on here even if it's not related to the crash.

2009-08-15 21:51

 

players1.jpg (168,365 bytes)   

2009-08-15 21:52

 

players2.jpg (184,912 bytes)   

FUBAR-BDHR

2009-08-15 21:55

developer   ~0011136

Last edited: 2009-08-15 21:57

Attached 2 screen shots of what appears to be the cause of the crash. You'll notice 7 IP connections showing but only 4 players. Seems that when one of those "ghost" players goes to start a mission or respawn it crashes.

Almost positive this is related to 1827

FUBAR-BDHR

2009-08-25 22:50

developer   ~0011159

Last edited: 2009-08-25 22:52

While most of the stuff relating to this seems to be fixed or known I did get the Lilith problem again last night. We were discussing it last night but Wanderer had to leave and I felt the sudden need to get horizontal so we didn't make much progress. Took a second look at it today and it seems (big guess here)it's looking for an already cached subsystem at the wrong location. sip->subsystems[j].model_num is 515 and sip->model_num is 8583.

I'll attach the call stack and variables. I still have the debugger open so I can't get the fs2_open.log right now. Mission was cet_m03 which only has one Lilith. I'll be checking the SCP IRC channel periodically tonight so if you need more info and I'm logged in there just ask.

2009-08-25 22:50

 

lilith.txt (8,979 bytes)   

chief1983

2010-04-05 16:55

administrator   ~0011856

Changing the subject because this title could get confusing very soon.

FUBAR-BDHR

2010-04-05 22:32

developer   ~0011857

Maybe we should change it to the actual error that is left.

Anyway think I had this happen again last night. Saw the server was crashed with a similar error but didn't have time to check it yet.

chief1983

2010-04-05 23:03

administrator   ~0011858

That or open a new bug that's not cluttered and close this one as fixed. If an issue was fixed this should probably be closed, it was never really supposed to be a parent type ticket.

FUBAR-BDHR

2010-07-08 19:50

developer   ~0012205

Apparently not fixed. Just found a standalone crashed with it. Retail data. r6278.

2010-07-08 19:50

 

1951.txt (25,778 bytes)   

2010-07-09 22:47

 

1951_logs.rar (27,379 bytes)

FUBAR-BDHR

2010-07-09 22:51

developer   ~0012209

Changed the topic to the error and uploaded the logs from yesterday's crash.

Echelon9

2010-07-10 12:03

developer   ~0012214

This latest example looks to be a Debug build correctly catching a model problem with a Warning() call. Please open a new case for this, but I'd say it is a model bug (although if with Retail data we will have to look into it).

FUBAR-BDHR

2010-07-10 19:54

developer   ~0012224

Nope not a model bug. This happens on a reload of a model that is already loaded although it's usually the Lilith for some reason. Somewhere one of the indexes is getting changed or stepped on in memory.

Goober5000

2012-11-11 05:19

administrator   ~0014021

Is this bug still a problem?

FUBAR-BDHR

2012-11-11 16:23

developer   ~0014026

I have no clue at this point. The standalones were crashing so often with multiple issues I haven't tried to run one in over a year.

Goober5000

2012-11-11 18:07

administrator   ~0014029

The model code has been changed a lot in the past two years and the cause of this bug may have been fixed. It would help to run a standalone to be sure.

karajorma

2012-11-23 07:08

administrator   ~0014159

Has this one resurfaced?

FUBAR-BDHR

2012-11-23 17:17

developer   ~0014161

With all the new warnings and FS2netD server going down they haven't been up long enough to really test.

FUBAR-BDHR

2012-12-02 02:35

developer   ~0014245

Unfortunately it still lives. Just got it with retail data loading the Gany.

I'll pack up the logs and attach them as soon as I get out of the debugger.

FUBAR-BDHR

2012-12-02 03:56

developer  

1951_3_6_15.rar (83,714 bytes)

Goober5000

2012-12-02 05:13

administrator   ~0014250

Is that the complete log? Because it doesn't display the expected -mod information at the beginning. Can you post the command line you used?

FUBAR-BDHR

2012-12-02 05:38

developer   ~0014251

Standalone logs don't include that info for some reason.

Here it is. No mod was active. Retail data used.

C:\Games\FreeSpace2\fs2_open_3_6_15d_INF_SSE2.exe -nomotiondebris -missile_lighting -3dshockwave -post_process -3dwarp -warp_flash -snd_preload -standalone -fps -spec -glow -env

FUBAR-BDHR

2012-12-13 17:29

developer  

1951_3_6_15_2.rar (97,436 bytes)

FUBAR-BDHR

2012-12-13 17:29

developer   ~0014434

Another one this time with the Lilith again.

Goober5000

2012-12-20 04:23

administrator   ~0014548

Can you attach the mission that was loaded?

FUBAR-BDHR

2012-12-20 04:40

developer  

fubar_m18.fs2 (22,340 bytes)

FUBAR-BDHR

2012-12-20 04:41

developer   ~0014550

Sure but it's not specific to that mission.

Goober5000

2012-12-20 07:02

administrator   ~0014552

I've added some additional debug code. Please post the new warning message if it occurs again.

Goober5000

2013-01-04 15:07

administrator   ~0014614

Has anyone tested the debug code I added? Does this problem still occur?

FUBAR-BDHR

2013-01-04 18:52

developer   ~0014615

I've had the debug code up since it was added. Unfortunately I've only seen a few uses of the standalones in the last couple of weeks and most of the time it has been 1 player just sitting there waiting for other players not actually playing. Hoping things will pick back up a bit once people get tired of their xmas presents and go back to playing FS2.

FUBAR-BDHR

2013-03-10 08:42

developer   ~0014761

I have not seen an occurrence of this in months. Last one I had I believe was shortly before commits 9423 and 9424. Possible plugging those memory leaks fixed this issue?

Goober5000

2013-03-10 19:23

administrator   ~0014762

That's terrific news. I'm going to assume revision 9423 fixed this, as it's highly plausible that it's related.

FUBAR-BDHR

2013-03-29 02:57

developer   ~0014850

Unfortunately this has occurred again. With the changes it hit an assert in model_get(). Attaching new log, stack, and variables.

FUBAR-BDHR

2013-03-29 02:57

developer  

1951_3_6_19.rar (96,842 bytes)

Echelon9

2013-12-01 01:21

developer   ~0015471

Can we get clear steps for which mission will most typically trigger this bug?

I would really like to see an AddressSanitizer-enabled standalone server report this bug, as we will get a precise measure of any memory corruption going on.

Goober5000

2014-09-07 03:59

administrator   ~0016271

Bump because we need steps to reproduce this.

Echelon9

2014-09-10 14:08

developer   ~0016273

I'll try to reproduce, but any other assistance reproducing appreciated.

Goober5000

2014-09-11 02:50

administrator   ~0016274

Yeah, that comment was directed mainly at FUBAR.

FSCyborg

2020-07-08 19:10

developer   ~0017005

In the 6 months of working and testing multi, which includes standalones loading missions, I have not gotten this crash. Feel free to reopen if it appears again.

Related Changesets

fs2open: trunk r9423

2012-12-13 00:56

Goober5000


Ported: N/A

Details Diff
patch a memory leak Affected Issues
0001951
mod - /trunk/fs2_open/code/ship/ship.cpp Diff File

fs2open: trunk r9455

2012-12-20 02:34

Goober5000


Ported: N/A

Details Diff
enhance debug warning for Mantis 0001951 Affected Issues
0001951
mod - /trunk/fs2_open/code/ship/ship.cpp Diff File

Issue History

Date Modified Username Field Change
2009-07-05 20:30 FUBAR-BDHR New Issue
2009-07-05 20:32 FUBAR-BDHR File Added: 1951.rar
2009-07-05 20:33 FUBAR-BDHR Note Added: 0011053
2009-07-08 20:02 FUBAR-BDHR File Added: fs2_open_vp_1733.log
2009-07-08 20:03 FUBAR-BDHR Note Added: 0011062
2009-07-09 04:15 FUBAR-BDHR Note Added: 0011063
2009-07-09 04:25 chief1983 Priority normal => high
2009-07-09 04:57 FUBAR-BDHR File Added: fs2_open_both.rar
2009-07-11 23:58 FUBAR-BDHR Note Added: 0011075
2009-07-11 23:59 FUBAR-BDHR File Added: fs2_open-07-11-09.log
2009-07-14 08:22 portej05 Note Added: 0011083
2009-07-19 07:20 FUBAR-BDHR File Added: fs2_open-07-18-09.log
2009-07-19 07:22 FUBAR-BDHR Note Added: 0011091
2009-07-19 07:25 FUBAR-BDHR Note Edited: 0011091
2009-07-19 07:31 portej05 Note Added: 0011092
2009-07-19 07:46 FUBAR-BDHR Note Added: 0011094
2009-07-25 22:19 FUBAR-BDHR Note Added: 0011105
2009-07-27 02:08 portej05 Note Added: 0011106
2009-07-27 04:32 FUBAR-BDHR Note Added: 0011108
2009-08-07 00:17 FUBAR-BDHR Note Added: 0011126
2009-08-07 02:32 FUBAR-BDHR Note Added: 0011127
2009-08-15 21:51 FUBAR-BDHR File Added: players1.jpg
2009-08-15 21:52 FUBAR-BDHR File Added: players2.jpg
2009-08-15 21:55 FUBAR-BDHR Note Added: 0011136
2009-08-15 21:57 FUBAR-BDHR Note Edited: 0011136
2009-08-25 22:50 FUBAR-BDHR Note Added: 0011159
2009-08-25 22:50 FUBAR-BDHR File Added: lilith.txt
2009-08-25 22:52 FUBAR-BDHR Note Edited: 0011159
2009-11-05 23:26 chief1983 Status new => acknowledged
2009-11-05 23:26 chief1983 Product Version 3.6.9 => 3.6.11
2009-11-05 23:26 chief1983 Target Version => 3.6.12 RC1
2010-04-05 16:55 chief1983 Note Added: 0011856
2010-04-05 16:55 chief1983 Summary RC3 standalone crashes => 3.6.10 RC3 standalone crashes
2010-04-05 22:32 FUBAR-BDHR Note Added: 0011857
2010-04-05 23:03 chief1983 Note Added: 0011858
2010-07-08 19:50 FUBAR-BDHR Note Added: 0012205
2010-07-08 19:50 FUBAR-BDHR File Added: 1951.txt
2010-07-09 22:47 FUBAR-BDHR File Added: 1951_logs.rar
2010-07-09 22:50 FUBAR-BDHR Summary 3.6.10 RC3 standalone crashes => Standalone crashes - Ship <ship>does not have subsystem <subsystem> linked into the model file
2010-07-09 22:51 FUBAR-BDHR Note Added: 0012209
2010-07-10 12:03 Echelon9 Note Added: 0012214
2010-07-10 19:54 FUBAR-BDHR Note Added: 0012224
2010-08-05 20:03 chief1983 Target Version 3.6.12 RC1 => 3.7.2
2012-04-03 20:25 Echelon9 Relationship added related to 0002015
2012-11-11 05:19 Goober5000 Note Added: 0014021
2012-11-11 16:23 FUBAR-BDHR Note Added: 0014026
2012-11-11 18:07 Goober5000 Note Added: 0014029
2012-11-16 05:56 Goober5000 Target Version 3.7.2 => 3.6.16
2012-11-23 07:08 karajorma Note Added: 0014159
2012-11-23 17:17 FUBAR-BDHR Note Added: 0014161
2012-12-02 02:35 FUBAR-BDHR Note Added: 0014245
2012-12-02 03:56 FUBAR-BDHR File Added: 1951_3_6_15.rar
2012-12-02 05:13 Goober5000 Note Added: 0014250
2012-12-02 05:38 FUBAR-BDHR Note Added: 0014251
2012-12-13 17:29 FUBAR-BDHR File Added: 1951_3_6_15_2.rar
2012-12-13 17:29 FUBAR-BDHR Note Added: 0014434
2012-12-20 04:23 Goober5000 Note Added: 0014548
2012-12-20 04:40 FUBAR-BDHR File Added: fubar_m18.fs2
2012-12-20 04:41 FUBAR-BDHR Note Added: 0014550
2012-12-20 07:02 Goober5000 Changeset attached => fs2open trunk r9455
2012-12-20 07:02 Goober5000 Note Added: 0014552
2013-01-04 15:07 Goober5000 Note Added: 0014614
2013-01-04 18:52 FUBAR-BDHR Note Added: 0014615
2013-03-10 08:42 FUBAR-BDHR Note Added: 0014761
2013-03-10 19:23 Goober5000 Note Added: 0014762
2013-03-10 19:23 Goober5000 Assigned To => Goober5000
2013-03-10 19:23 Goober5000 Status acknowledged => resolved
2013-03-10 19:23 Goober5000 Resolution open => fixed
2013-03-29 02:57 FUBAR-BDHR Note Added: 0014850
2013-03-29 02:57 FUBAR-BDHR Status resolved => feedback
2013-03-29 02:57 FUBAR-BDHR Resolution fixed => reopened
2013-03-29 02:57 FUBAR-BDHR File Added: 1951_3_6_19.rar
2013-04-19 21:19 chief1983 Target Version 3.6.16 => 3.7.0
2013-12-01 01:21 Echelon9 Note Added: 0015471
2014-09-07 03:57 Goober5000 Changeset attached => fs2open trunk r9423
2014-09-07 03:59 Goober5000 Note Added: 0016271
2014-09-07 03:59 Goober5000 Assigned To Goober5000 => Echelon9
2014-09-07 03:59 Goober5000 Target Version 3.7.0 =>
2014-09-10 14:08 Echelon9 Note Added: 0016273
2014-09-11 02:50 Goober5000 Note Added: 0016274
2020-01-14 06:37 FSCyborg Relationship added related to 0002052
2020-07-08 19:10 FSCyborg Status feedback => closed
2020-07-08 19:10 FSCyborg Resolution reopened => unable to reproduce
2020-07-08 19:10 FSCyborg Note Added: 0017005
2020-07-08 19:20 Goober5000 Assigned To Echelon9 => FSCyborg