Source Code Project Mantis - FSSCP
View Issue Details
0002971FSSCPmultiplayerpublic2013-12-03 13:252015-05-01 02:33
Reporterchief1983 
Assigned ToMageKing17 
PrioritynormalSeveritycrashReproducibilitysometimes
StatusresolvedResolutionfixed 
Platformx86OSGentoo LinuxOS Version3.5.7
Product Version3.7.1 
Target Version3.7.4Fixed in Version 
Summary0002971: Standalone AddressSanitizer crash SEGV in hud_shield_quadrant_hit
Description=================================================================
==18747== ERROR: AddressSanitizer: SEGV on unknown address 0x0000000c (pc 0x082fc5c5 sp 0xbfea81f0 bp 0xbfea8228 T0)
AddressSanitizer can not provide additional info.
    #0 0x82fc5c4 in hud_shield_quadrant_hit(object*, int) /home/chief1983/fs2_open_build/code/hud/hudshield.cpp:573 (discriminator 1)
    0000001 0x8a2e50b in ship_weapon_do_hit_stuff(object*, object*, vec3d*, vec3d*, int, int, vec3d) /home/chief1983/fs2_open_build/code/object/collideshipweapon.cpp:97
    0000002 0x8a30565 in ship_weapon_check_collision(object*, object*, float, int*) /home/chief1983/fs2_open_build/code/object/collideshipweapon.cpp:403
    0000003 0x8a30eab in collide_ship_weapon(obj_pair*) /home/chief1983/fs2_open_build/code/object/collideshipweapon.cpp:479
    0000004 0x86bf397 in obj_collide_pair(object*, object*) /home/chief1983/fs2_open_build/code/object/objcollide.cpp:1645
    0000005 0x86bca02 in obj_find_overlap_colliders(SCP_vector<int>*, SCP_vector<int>*, int, bool) /home/chief1983/fs2_open_build/code/object/objcollide.cpp:1265
    0000006 0x86bc62f in obj_sort_and_collide() /home/chief1983/fs2_open_build/code/object/objcollide.cpp:1230
    0000007 0x86ceccf in obj_move_all(float) /home/chief1983/fs2_open_build/code/object/object.cpp:1502
    0000008 0x805f16b in game_simulation_frame() /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:4006 (discriminator 2)
    0000009 0x80603e2 in game_frame(bool) /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:4399
    0000010 0x806124c in game_do_frame() /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:4814
    #11 0x8063f5b in game_do_state(int) /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:6494
    0000012 0x81c54ee in gameseq_process_events() /home/chief1983/fs2_open_build/code/gamesequence/gamesequence.cpp:409
    0000013 0x8065468 in game_main(char*) /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:7061
    0000014 0x806589d in main /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:7195
    0000015 0xb5c03595 in ?? ??:0
==18747== ABORTING
Steps To ReproduceHow I have reproduced it:

Connect two players to an AddressSanitizer-enabled Standalone
Fire up Demon Dogfight Light
Shoot around at the demon or the other player for a while. Took longer to create than previous ASan crashes but it happened to me within a minute still.
Additional InformationSecond time I reproduced it was the one that seemed to take longer, and had a different code path:

#0 0x082fc660 in hud_shield_quadrant_hit (objp=0x940c8bc <Objects+508>, quadrant=-1) at hud/hudshield.cpp:575
0000001 0x086b45c3 in collide_ship_ship (pair=0xbfffe6d0) at object/collideshipship.cpp:1242
0000002 0x086bf398 in obj_collide_pair (A=0x940c8bc <Objects+508>, B=0x940c6c0 <Objects>) at object/objcollide.cpp:1645
0000003 0x086bca03 in obj_find_overlap_colliders (overlap_list_out=0xbfffe820, list=0xbfffe860, axis=2, collide=true) at object/objcollide.cpp:1265
0000004 0x086bc630 in obj_sort_and_collide () at object/objcollide.cpp:1230
0000005 0x086cecd0 in obj_move_all (frametime=0.0330047607) at object/object.cpp:1502
0000006 0x0805f16c in game_simulation_frame () at freespace2/freespace.cpp:4006
0000007 0x080603e3 in game_frame (paused=false) at freespace2/freespace.cpp:4399
0000008 0x0806124d in game_do_frame () at freespace2/freespace.cpp:4814
0000009 0x08063f5c in game_do_state (state=2) at freespace2/freespace.cpp:6494
0000010 0x081c54ef in gameseq_process_events () at gamesequence/gamesequence.cpp:409
#11 0x08065469 in game_main (cmdline=0xb6100610 "-standalone") at freespace2/freespace.cpp:7061
0000012 0x0806589e in main (argc=2, argv=0xbfffedb4) at freespace2/freespace.cpp:7195

Additional stack info from that crash:

(frame 0)
(gdb) info locals
shi = 0x9181a48 <Shield_hit_data+40>
num = 141278518

(gdb) frame 1
0000001 0x086b45c3 in collide_ship_ship (pair=0xbfffe6d0) at object/collideshipship.cpp:1242
(gdb) info locals
dam2 = 3.17000103
quadrant_num = -1
damage = 6.8274188
a_override = false
b_override = false
hit = 1
LightOne = 0x940c8bc <Objects+508>
light_sip = 0x99f02dc <Ship_info+109500>
ship_ship_hit_info = {heavy = 0x940c6c0 <Objects>, light = 0x940c8bc <Objects+508>, heavy_collision_cm_pos = {{xyz = {x = 0, y = 0, z = 0}, a1d = {0, 0, 0}}}, light_collision_cm_pos = {
{xyz = {x = -24.7636738, y = 252.959991, z = -178.660339}, a1d = {-24.7636738, 252.959991, -178.660339}}}, r_heavy = {{xyz = {x = -22.50424, y = 247.505051, z = -178.703674}, a1d = {-22
.50424, 247.505051, -178.703674}}}, r_light = {{xyz = {x = 2.25943375, y = -5.4549408, z = -0.0433349609}, a1d = {2.25943375, -5.4549408, -0.0433349609}}}, hit_pos = {{xyz = {x = -22.50
424, y = 247.505051, z = -178.703674}, a1d = {-22.50424, 247.505051, -178.703674}}}, collision_normal = {{xyz = {x = -0.382660955, y = 0.923859656, z = 0.00733927637}, a1d = {-0.3826609
55, 0.923859656, 0.00733927637}}}, hit_time = 0.39586851, impulse = 7192.90283, light_rel_vel = {{xyz = {x = -44.084198, y = -51.2995834, z = 21.8710117}, a1d = {-44.084198, -51.2995834
, 21.8710117}}}, collide_rotate = 1, submodel_num = 0, edge_hit = 0, submodel_rot_hit = 0, is_landing = false}
world_hit_pos = {{xyz = {x = -37.1646347, y = 247.511673, z = 955.386658}, a1d = {-37.1646347, 247.511673, 955.386658}}}
HeavyOne = 0x940c6c0 <Objects>
player_involved = 0
dist = 309.667145
A = 0x940c8bc <Objects+508>
B = 0x940c6c0 <Objects>

(gdb) frame 2
0000002 0x086bf398 in obj_collide_pair (A=0x940c8bc <Objects+508>, B=0x940c6c0 <Objects>) at object/objcollide.cpp:1645
(gdb) info locals
ctype = 257
check_collision = 0x86b2eb9 <collide_ship_ship(obj_pair*)>
key = 4096
swapped = 0
collision_info = 0xa850f4f4
valid = true
new_pair = {a = 0x940c8bc <Objects+508>, b = 0x940c6c0 <Objects>, check_collision = 0x86b2eb9 <collide_ship_ship(obj_pair*)>, next_check_time = 1, next = 0xbfffe790}
TagsNo tags attached.
Attached Filespatch assert-mantis-2971.patch (559) 2013-12-05 03:12
http://scp.indiegames.us/mantis/file_download.php?file_id=2304&type=bug
patch hudshield.cpp-Mantis-2971.patch (1,321) 2014-09-24 02:34
http://scp.indiegames.us/mantis/file_download.php?file_id=2570&type=bug
patch hudshield.cpp.patch (424) 2015-04-29 21:29
http://scp.indiegames.us/mantis/file_download.php?file_id=2707&type=bug

Notes
(0015500)
chief1983   
2013-12-03 13:29   
zookeeper indicated that Shield_hit_data must not be init'd right but can't figure out why it wouldn't be at that point on the standalone.
(0015507)
chief1983   
2013-12-05 02:46   
=================================================================
==6463== ERROR: AddressSanitizer: SEGV on unknown address 0x00000008 (pc 0x082fc5cd sp 0xbfc69950 bp 0xbfc69988 T0)
AddressSanitizer can not provide additional info.
    #0 0x82fc5cc in hud_shield_quadrant_hit(object*, int) /home/chief1983/fs2_open_build/code/hud/hudshield.cpp:573 (discriminator 1)
    0000001 0x8a2f1fb in ship_weapon_do_hit_stuff(object*, object*, vec3d*, vec3d*, int, int, vec3d) /home/chief1983/fs2_open_build/code/object/collideshipweapon.cpp:97
    0000002 0x8a31255 in ship_weapon_check_collision(object*, object*, float, int*) /home/chief1983/fs2_open_build/code/object/collideshipweapon.cpp:403
    0000003 0x8a31b9b in collide_ship_weapon(obj_pair*) /home/chief1983/fs2_open_build/code/object/collideshipweapon.cpp:479
    0000004 0x86bfc87 in obj_collide_pair(object*, object*) /home/chief1983/fs2_open_build/code/object/objcollide.cpp:1645
    0000005 0x86bd2f2 in obj_find_overlap_colliders(SCP_vector<int>*, SCP_vector<int>*, int, bool) /home/chief1983/fs2_open_build/code/object/objcollide.cpp:1265
    0000006 0x86bcf1f in obj_sort_and_collide() /home/chief1983/fs2_open_build/code/object/objcollide.cpp:1230
    0000007 0x86cf5bf in obj_move_all(float) /home/chief1983/fs2_open_build/code/object/object.cpp:1502
    0000008 0x805f170 in game_simulation_frame() /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:4007 (discriminator 2)
    0000009 0x80603e7 in game_frame(bool) /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:4400
    0000010 0x8061251 in game_do_frame() /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:4815
    #11 0x8063f60 in game_do_state(int) /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:6495
    0000012 0x81c54f6 in gameseq_process_events() /home/chief1983/fs2_open_build/code/gamesequence/gamesequence.cpp:409
    0000013 0x806546d in game_main(char*) /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:7062
    0000014 0x80658a2 in main /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:7196
    0000015 0xb5c0f595 in ?? ??:0
==6463== ABORTING
(0015532)
Echelon9   
2013-12-26 02:07   
Have builds with the assert patch (attached) reported this issue yet?
(0015535)
chief1983   
2013-12-26 10:02   
I hadn't noticed the patch I guess, zookeeper had been planning on taking a look at this one but he's been engrossed in other projects, so maybe I'll take another look at this one here soon.
(0015552)
chief1983   
2013-12-31 19:44   
(Last edited: 2013-12-31 19:45)
Ok, didn't hit the Assert with a patched standalone, still got this SEGV on the otherwise current trunk revision.

ASAN:SIGSEGV
=================================================================
==10391== ERROR: AddressSanitizer: SEGV on unknown address 0x00000000 (pc 0x08328859 sp 0xbf923f90 bp 0xbf923fd8 T0)
AddressSanitizer can not provide additional info.
    #0 0x8328858 in hud_shield_quadrant_hit(object*, int) /home/chief1983/fs2_open_build/code/hud/hudshield.cpp:573 (discriminator 1)
    0000001 0x8a60729 in ship_weapon_do_hit_stuff(object*, object*, vec3d*, vec3d*, int, int, vec3d) /home/chief1983/fs2_open_build/code/object/collideshipweapon.cpp:97
    0000002 0x8a6277d in ship_weapon_check_collision(object*, object*, float, int*) /home/chief1983/fs2_open_build/code/object/collideshipweapon.cpp:403
    0000003 0x8a630b7 in collide_ship_weapon(obj_pair*) /home/chief1983/fs2_open_build/code/object/collideshipweapon.cpp:479
    0000004 0x86f06a1 in obj_collide_pair(object*, object*) /home/chief1983/fs2_open_build/code/object/objcollide.cpp:1645
    0000005 0x86edd3c in obj_find_overlap_colliders(SCP_vector<int>*, SCP_vector<int>*, int, bool) /home/chief1983/fs2_open_build/code/object/objcollide.cpp:1265
    0000006 0x86ed969 in obj_sort_and_collide() /home/chief1983/fs2_open_build/code/object/objcollide.cpp:1230
    0000007 0x86ff104 in obj_move_all(float) /home/chief1983/fs2_open_build/code/object/object.cpp:1502
    0000008 0x805f14f in game_simulation_frame() /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:4007 (discriminator 2)
    0000009 0x80603c6 in game_frame(bool) /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:4400
    0000010 0x806123a in game_do_frame() /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:4816
    #11 0x8063f49 in game_do_state(int) /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:6496
    0000012 0x81c709a in gameseq_process_events() /home/chief1983/fs2_open_build/code/gamesequence/gamesequence.cpp:409
    0000013 0x8065456 in game_main(char*) /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:7063
    0000014 0x806588b in main /home/chief1983/fs2_open_build/code/freespace2/freespace.cpp:7197
    0000015 0xb5c85595 in ?? ??:0
==10391== ABORTING

(0015745)
chief1983   
2014-05-13 16:59   
Anything else that we can try to sort this out for 3.7.2?
(0016244)
Echelon9   
2014-08-20 09:36   
I'll give this another go with an AddressSanitizer instrumented standalone server and client.
(0016266)
chief1983   
2014-08-30 08:59   
Any input from that? Need any help testing with an ASan server? I can probably get mine running again.
(0016291)
Goober5000   
2014-09-24 02:32   
Based on the stack dump, and narrowing down the code paths based on variable values, the culprit appears to be this line:

shi->shield_hit_timers[shi->hull_hit_index] = timestamp(SHIELD_HIT_DURATION_SHORT);

Now, on standalone servers Player_obj is specifically set to NULL (per multiteamselect.cpp:937). I think what is happening is that shield_info_reset is clearing the vectors but leaving the hull_hit_index set to 0. Trying to access position 0 in the vector causes an access violation.

(The wrinkle in that hypothesis is that Player_obj being NULL should only affect the player's shield data, yet the code dump indicated the crash occurred on the target data. But I may not be following the initialization 100% correctly.)

I would recommend littering the hud_shield_quadrant_hit function with logging statements to try to narrow this down. However, the shield_info_reset function needs fixing in any case, and based on my understanding of the situation it is the most likely culprit. I am attaching a patch.
(0016312)
chief1983   
2014-09-28 11:50   
Program received signal SIGSEGV, Segmentation fault.
0x0835aaa1 in hud_shield_quadrant_hit (objp=0x940a73c <Objects+508>, quadrant=1) at hud/hudshield.cpp:568
(gdb) bt
#0 0x0835aaa1 in hud_shield_quadrant_hit (objp=0x940a73c <Objects+508>, quadrant=1) at hud/hudshield.cpp:568
0000001 0x08a99498 in ship_weapon_do_hit_stuff (ship_obj=0x940a73c <Objects+508>, weapon_obj=0x940c304 <Objects+7620>, world_hitpos=
0xbfffe3e8, hitpos=0xbfffe3dc, quadrant_num=1, submodel_num=-1, hit_dir=...) at object/collideshipweapon.cpp:97
0000002 0x08a9b4e7 in ship_weapon_check_collision (ship_objp=0x940a73c <Objects+508>, weapon_objp=0x940c304 <Objects+7620>, time_lim
it=0, next_hit=0x0) at object/collideshipweapon.cpp:410
0000003 0x08a9be47 in collide_ship_weapon (pair=0xbfffe6c0) at object/collideshipweapon.cpp:486
0000004 0x0872fa17 in obj_collide_pair (A=0x940a73c <Objects+508>, B=0x940c304 <Objects+7620>) at object/objcollide.cpp:1645
0000005 0x0872d0a4 in obj_find_overlap_colliders (overlap_list_out=0xbfffe810, list=0xbfffe850, axis=2, collide=true) at object/objc
ollide.cpp:1265
0000006 0x0872ccd3 in obj_sort_and_collide () at object/objcollide.cpp:1230
0000007 0x0873e7f3 in obj_move_all (frametime=0.0330047607) at object/object.cpp:1507
0000008 0x08060b2a in game_simulation_frame () at freespace2/freespace.cpp:4085
0000009 0x08061da1 in game_frame (paused=false) at freespace2/freespace.cpp:4478
0000010 0x08062bf0 in game_do_frame () at freespace2/freespace.cpp:4894
#11 0x080659bb in game_do_state (state=2) at freespace2/freespace.cpp:6577
0000012 0x081e93fb in gameseq_process_events () at gamesequence/gamesequence.cpp:409
0000013 0x08066f4f in game_main (cmdline=0xb6100610 "-standalone") at freespace2/freespace.cpp:7144
0000014 0x0806739c in main (argc=2, argv=0xbfffedb4) at freespace2/freespace.cpp:7278

Was I supposed to run the patch on the standalone only, or also all the clients? For this run I just had it on the standalone.
(0016313)
Goober5000   
2014-09-28 13:32   
Good question... I would have thought standalone would be sufficient. Try it on all clients. If that doesn't work, then we'll have to go the logging route.
(0016318)
Goober5000   
2014-09-29 23:56   
I committed the patch because that function needs fixing anyway, but it sounds like it doesn't fix the whole problem. (Though, make sure that you didn't accidentally test with an unpatched build.)
(0016656)
chief1983   
2015-04-23 13:11   
I'll set up a dedicated with 3.7.2 final and some 3.7.2 clients after I compile them and give them all a go.
(0016671)
chief1983   
2015-04-25 16:10   
Crash still happens, AddressSanitizer caught a seg fault. This one isn't much use, I don't have symbols for some reason.
(0016680)
Echelon9   
2015-04-26 23:49   
Yes, AddressSanitizer is still appearing to correctly flag a memory corruption issue here in multi
(0016685)
MageKing17   
2015-04-30 13:44   
So I think I've got this one figured out (thanks to a lot of help from chief1983 and his server). When model point shields were added (r10135), shi->shield_hit_timers was changed from an array to a vector. The errors are due to trying to access the uninitialized vector. The reason r11099 didn't affect the crash is because the standalone would never call hud_shield_hit_reset() except for the player object (which would never be the object called for hud_shield_quadrant_hit() due to being NULL). I think the correct solution here, rather than making hud_shield_quadrant_hit() work on the standalone, is to simply not call it on the standalone; it's HUD-related code that probably shouldn't have been called in the first place, it simply used to work due to the fixed-size array being used.

(That being said, some assertions to avoid similar errors in the future are undoubtedly a good idea.)

After a little more testing to make sure I've got it sorted out, I'll make a pull request for the fix.
(0016687)
Goober5000   
2015-04-30 22:56   
Ah, excellent sleuthing!
(0016688)
MageKing17   
2015-05-01 01:22   
Pull request: https://github.com/scp-fs2open/fs2open.github.com/pull/47
(0016689)
MageKing17   
2015-05-01 02:33   
And merged.

Issue History
2013-12-03 13:25chief1983New Issue
2013-12-03 13:29chief1983Note Added: 0015500
2013-12-05 02:46chief1983Note Added: 0015507
2013-12-05 02:59chief1983Reproducibilityalways => sometimes
2013-12-05 03:12Echelon9File Added: assert-mantis-2971.patch
2013-12-26 02:07Echelon9Note Added: 0015532
2013-12-26 10:02chief1983Note Added: 0015535
2013-12-31 19:44chief1983Note Added: 0015552
2013-12-31 19:45chief1983Note Edited: 0015552bug_revision_view_page.php?bugnote_id=15552#r732
2014-04-21 02:55Echelon9Assigned To => Echelon9
2014-04-21 02:55Echelon9Statusnew => assigned
2014-05-13 16:59chief1983Note Added: 0015745
2014-08-20 09:36Echelon9Note Added: 0016244
2014-08-30 08:59chief1983Note Added: 0016266
2014-08-31 11:34MageKing17Assigned ToEchelon9 => MageKing17
2014-08-31 11:35MageKing17Assigned ToMageKing17 => Echelon9
2014-09-24 02:32Goober5000Note Added: 0016291
2014-09-24 02:32Goober5000Assigned ToEchelon9 => Goober5000
2014-09-24 02:34Goober5000File Added: hudshield.cpp-Mantis-2971.patch
2014-09-24 02:34Goober5000Statusassigned => code review
2014-09-28 11:50chief1983Note Added: 0016312
2014-09-28 13:32Goober5000Note Added: 0016313
2014-09-29 23:55Goober5000Changeset attached => fs2open trunk r11099
2014-09-29 23:56Goober5000Note Added: 0016318
2015-04-16 00:25Goober5000Target Version3.7.2 => 3.7.4
2015-04-23 13:11chief1983Note Added: 0016656
2015-04-25 16:10chief1983Note Added: 0016671
2015-04-26 23:49Echelon9Note Added: 0016680
2015-04-29 21:29MageKing17File Added: hudshield.cpp.patch
2015-04-30 13:44MageKing17Note Added: 0016685
2015-04-30 13:44MageKing17Assigned ToGoober5000 => MageKing17
2015-04-30 13:44MageKing17Statuscode review => assigned
2015-04-30 22:56Goober5000Note Added: 0016687
2015-05-01 01:22MageKing17Note Added: 0016688
2015-05-01 01:22MageKing17Statusassigned => code review
2015-05-01 02:33MageKing17Note Added: 0016689
2015-05-01 02:33MageKing17Statuscode review => resolved
2015-05-01 02:33MageKing17Resolutionopen => fixed