All Projects FSSCP  wxFRED

View Issue Details

Related ChangesetsJump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0002861FSSCPpublic2013-05-01 09:532013-07-08 16:35
ReporterEchelon9 Assigned ToValathil  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
Product Version3.6.19 
Target Version3.7.0 
Summary0002861: AddressSanitizer: heap-buffer-overflow in opengl_shader_set_current()
DescriptionReported by AddressSanitizer, a memory error detector for C/C++, in FS2Open builds based on trunk r9668.

ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61d000102c40 at pc 0x1003d1413 bp 0x7fff5fbfb490 sp 0x7fff5fbfb488
READ of size 8 at 0x61d000102c40 thread T0
    #0 0x1003d1412 in opengl_shader_set_current gropenglshader.cpp:93
    0000001 0x10037dbfd in gr_opengl_render_effect gropengldraw.cpp:1505
    0000002 0x10099c854 in geometry_batcher::render grbatch.cpp:526
    0000003 0x1009a3ca2 in batch_render_geometry_map_bitmaps grbatch.cpp:779
    0000004 0x1009a481d in batch_render_all grbatch.cpp:830
    0000005 0x101a594c3 in obj_render_all, bool*) objectsort.cpp:337
    0000006 0x1002efb3b in game_render_frame freespace.cpp:3732
    0000007 0x1002f9681 in game_frame freespace.cpp:4535
    0000008 0x1002fe74b in game_do_frame freespace.cpp:4917
    0000009 0x10030a286 in game_do_state freespace.cpp:6593
    0000010 0x100913798 in gameseq_process_events gamesequence.cpp:405
    #11 0x100310b26 in game_main freespace.cpp:7160
    0000012 0x1003121c4 in SDL_main freespace.cpp:7294
    ...
0x61d000102c40 is located 64 bytes to the left of 2048-byte region [0x61d000102c80,0x61d000103480)
allocated by thread T0 here:
    #0 0x105a51e05 in wrap_malloc (in libclang_rt.asan_osx_dynamic.dylib) + 53
    0000001 0x102626088 in _vm_malloc stubs.cpp:571
    0000002 0x1003eda0a in SCP_vm_allocator<opengl_shader_t>::allocate vmallocator.h:59
    0000003 0x1003e8ed0 in std::_Vector_base<opengl_shader_t, SCP_vm_allocator<opengl_shader_t> >::_M_allocate stl_vector.h:131
    0000004 0x1003e7920 in opengl_shader_t* std::vector<opengl_shader_t, SCP_vm_allocator<opengl_shader_t> >::_M_allocate_and_copy<opengl_shader_t*> stl_vector.h:766
    0000005 0x1003e00c6 in std::vector<opengl_shader_t, SCP_vm_allocator<opengl_shader_t> >::reserve vector.tcc:76
    0000006 0x1003d9904 in opengl_shader_init gropenglshader.cpp:494
    0000007 0x100a673e4 in gr_opengl_init gropengl.cpp:2014
    0000008 0x100963543 in gr_init_sub 2d.cpp:485
    0000009 0x100960d2c in gr_init 2d.cpp:615
    0000010 0x1002cc6ac in game_init freespace.cpp:1809
    #11 0x1003105e9 in game_main freespace.cpp:7103
    0000012 0x1003121c4 in SDL_main freespace.cpp:7294


/**
 * Set the currently active shader
 * @param shader_obj Pointer to an opengl_shader_t object. This function calls glUseProgramARB with parameter 0 if shader_obj is NULL or if function is called without parameters, causing OpenGL to revert to fixed-function processing
 */
void opengl_shader_set_current(opengl_shader_t *shader_obj)
{
    if (shader_obj != NULL) {
        if(!Current_shader || (Current_shader->program_id != shader_obj->program_id)) {
    Current_shader = shader_obj;
        vglUseProgramObjectARB(Current_shader->program_id);
Steps To Reproduce1. Utilise a version of Clang that supports AddressSantizer (https://code.google.com/p/address-sanitizer/wiki/AddressSanitizer [^]).
2. Build with -fsanitize=address C/C++ flag
3. Run the game with mediavps 3.6.12, using hardware where Use_GLSL = 1 i.e. that incompletely supports OpenGL (i.e. Legacy Context Mac OS 10.8 at present)
Additional InformationERROR: AddressSanitizer: heap-buffer-overflow on address 0x61d000102c40 at pc 0x1003d1413 bp 0x7fff5fbfb490 sp 0x7fff5fbfb488
READ of size 8 at 0x61d000102c40 thread T0
    #0 0x1003d1412 in opengl_shader_set_current gropenglshader.cpp:93
    0000001 0x10037dbfd in gr_opengl_render_effect gropengldraw.cpp:1505
    0000002 0x10099c854 in geometry_batcher::render grbatch.cpp:526
    0000003 0x1009a3ca2 in batch_render_geometry_map_bitmaps grbatch.cpp:779
    0000004 0x1009a481d in batch_render_all grbatch.cpp:830
    0000005 0x101a594c3 in obj_render_all, bool*) objectsort.cpp:337
    0000006 0x1002efb3b in game_render_frame freespace.cpp:3732
    0000007 0x1002f9681 in game_frame freespace.cpp:4535
    0000008 0x1002fe74b in game_do_frame freespace.cpp:4917
    0000009 0x10030a286 in game_do_state freespace.cpp:6593
    0000010 0x100913798 in gameseq_process_events gamesequence.cpp:405
    #11 0x100310b26 in game_main freespace.cpp:7160
    0000012 0x1003121c4 in SDL_main freespace.cpp:7294
    0000013 0x100002c83 in -[SDLMain applicationDidFinishLaunching:] SDLMain.m:300
    0000014 0x7fff8fc3bed9 in _CFXNotificationPost (in CoreFoundation) + 2553
    0000015 0x7fff91ce9e25 in -[NSNotificationCenter postNotificationName:object:userInfo:] (in Foundation) + 63
    0000016 0x7fff8c67e55c in -[NSApplication _postDidFinishNotification] (in AppKit) + 291
    0000017 0x7fff8c67e295 in -[NSApplication _sendFinishLaunchingNotification] (in AppKit) + 215
    0000018 0x7fff8c67b481 in -[NSApplication(NSAppleEventHandling) _handleAEOpenEvent:] (in AppKit) + 565
    0000019 0x7fff8c67b07b in -[NSApplication(NSAppleEventHandling) _handleCoreEvent:withReplyEvent:] (in AppKit) + 350
    0000020 0x7fff91d0370a in -[NSAppleEventManager dispatchRawAppleEvent:withRawReply:handlerRefCon:] (in Foundation) + 307
    0000021 0x7fff91d0356c in _NSAppleEventManagerGenericHandler (in Foundation) + 105
    0000022 0x7fff90c4b077 in aeDispatchAppleEvent(AEDesc const*, AEDesc*, unsigned int, unsigned char*) (in AE) + 306
    0000023 0x7fff90c4aed8 in dispatchEventAndSendReply(AEDesc const*, AEDesc*) (in AE) + 36
    0000024 0x7fff90c4ad98 in aeProcessAppleEvent (in AE) + 317
    0000025 0x7fff8ec71708 in AEProcessAppleEvent (in HIToolbox) + 99
    0000026 0x7fff8c677865 in _DPSNextEvent (in AppKit) + 1455
    0000027 0x7fff8c676e21 in -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] (in AppKit) + 127
    0000028 0x7fff8c66e1d2 in -[NSApplication run] (in AppKit) + 516
    0000029 0x100004de2 in CustomApplicationMain SDLMain.m:227
    0000030 0x10000484f in main SDLMain.m:377
    0000031 0x1000019a3 in start (in FS2_Open (debug)) + 51
    0000032 0x0 in 0x0
0x61d000102c40 is located 64 bytes to the left of 2048-byte region [0x61d000102c80,0x61d000103480)
allocated by thread T0 here:
    #0 0x105a51e05 in wrap_malloc (in libclang_rt.asan_osx_dynamic.dylib) + 53
    0000001 0x102626088 in _vm_malloc stubs.cpp:571
    0000002 0x1003eda0a in SCP_vm_allocator<opengl_shader_t>::allocate vmallocator.h:59
    0000003 0x1003e8ed0 in std::_Vector_base<opengl_shader_t, SCP_vm_allocator<opengl_shader_t> >::_M_allocate stl_vector.h:131
    0000004 0x1003e7920 in opengl_shader_t* std::vector<opengl_shader_t, SCP_vm_allocator<opengl_shader_t> >::_M_allocate_and_copy<opengl_shader_t*> stl_vector.h:766
    0000005 0x1003e00c6 in std::vector<opengl_shader_t, SCP_vm_allocator<opengl_shader_t> >::reserve vector.tcc:76
    0000006 0x1003d9904 in opengl_shader_init gropenglshader.cpp:494
    0000007 0x100a673e4 in gr_opengl_init gropengl.cpp:2014
    0000008 0x100963543 in gr_init_sub 2d.cpp:485
    0000009 0x100960d2c in gr_init 2d.cpp:615
    0000010 0x1002cc6ac in game_init freespace.cpp:1809
    #11 0x1003105e9 in game_main freespace.cpp:7103
    0000012 0x1003121c4 in SDL_main freespace.cpp:7294
    0000013 0x100002c83 in -[SDLMain applicationDidFinishLaunching:] SDLMain.m:300
    0000014 0x7fff8fc3bed9 in _CFXNotificationPost (in CoreFoundation) + 2553
    0000015 0x7fff91ce9e25 in -[NSNotificationCenter postNotificationName:object:userInfo:] (in Foundation) + 63
    0000016 0x7fff8c67e55c in -[NSApplication _postDidFinishNotification] (in AppKit) + 291
    0000017 0x7fff8c67e295 in -[NSApplication _sendFinishLaunchingNotification] (in AppKit) + 215
    0000018 0x7fff8c67b481 in -[NSApplication(NSAppleEventHandling) _handleAEOpenEvent:] (in AppKit) + 565
    0000019 0x7fff8c67b07b in -[NSApplication(NSAppleEventHandling) _handleCoreEvent:withReplyEvent:] (in AppKit) + 350
    0000020 0x7fff91d0370a in -[NSAppleEventManager dispatchRawAppleEvent:withRawReply:handlerRefCon:] (in Foundation) + 307
    0000021 0x7fff91d0356c in _NSAppleEventManagerGenericHandler (in Foundation) + 105
    0000022 0x7fff90c4b077 in aeDispatchAppleEvent(AEDesc const*, AEDesc*, unsigned int, unsigned char*) (in AE) + 306
    0000023 0x7fff90c4aed8 in dispatchEventAndSendReply(AEDesc const*, AEDesc*) (in AE) + 36
    0000024 0x7fff90c4ad98 in aeProcessAppleEvent (in AE) + 317
    0000025 0x7fff8ec71708 in AEProcessAppleEvent (in HIToolbox) + 99
    0000026 0x7fff8c677865 in _DPSNextEvent (in AppKit) + 1455
    0000027 0x7fff8c676e21 in -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] (in AppKit) + 127
    0000028 0x7fff8c66e1d2 in -[NSApplication run] (in AppKit) + 516
    0000029 0x100004de2 in CustomApplicationMain SDLMain.m:227
Shadow bytes around the buggy address:
  0x1c3a00020530: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x1c3a00020540: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x1c3a00020550: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
  0x1c3a00020560: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c3a00020570: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x1c3a00020580: fa fa fa fa fa fa fa fa[fa]fa fa fa fa fa fa fa
  0x1c3a00020590: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c3a000205a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c3a000205b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c3a000205c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c3a000205d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable: 00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone: fa
  Heap right redzone: fb
  Freed heap region: fd
  Stack left redzone: f1
  Stack mid redzone: f2
  Stack right redzone: f3
  Stack partial redzone: f4
  Stack after return: f5
  Stack use after scope: f8
  Global redzone: f9
  Global init order: f6
  Poisoned by user: f7
  ASan internal: fe
==31322==ABORTING
TagsNo tags attached.

Activities

Echelon9

2013-05-01 10:52

developer   ~0015011

It is caused where the preceding call to gr_opengl_maybe_create_shader() returns -1, which occurs in all cases where Use_GLSL variable equals 1.

Echelon9

2013-05-01 13:08

developer   ~0015013

This is because the following is being reached in gropenglshader.cpp:897

    if (in_error) {
        // We died on a lighting shader, probably due to instruction count.
        // Drop down to a special var that will use fixed-function rendering
        // but still allow for post-processing to work
        mprintf((" Shader in_error! Disabling GLSL model rendering!\n"));
        Use_GLSL = 1;
        Cmdline_height = 0;
        Cmdline_normal = 0;
    }

---

Compiling deferred light shader...
   Loading built-in default shader for: deferred-v.sdr
   Loading built-in default shader for: deferred-f.sdr
Fragment shader failed to compile:
ERROR: 0:11: Invalid qualifiers 'in' in global variable context
ERROR: 0:13: Invalid qualifiers 'in' in global variable context
ERROR: 0:37: Use of undeclared identifier 'lightposition'
ERROR: 0:38: Use of undeclared identifier 'lightDir'
ERROR: 0:39: Use of undeclared identifier 'dist'
ERROR: 0:42: Use of undeclared identifier 'lightDir'
ERROR: 0:44: Use of undeclared identifier 'conedot'
ERROR: 0:47: Use of undeclared identifier 'attenuation'
ERROR: 0:47: Use of undeclared identifier 'conedot'
ERROR: 0:49: Use of undeclared identifier 'conedot'
ERROR: 0:52: Use of undeclared identifier 'attenuation'
ERROR: 0:52: Use of undeclared identifier 'conedot'
ERROR: 0:55: Use of undeclared identifier 'dist'
ERROR: 0:63: Use of undeclared identifier 'beamvec'
ERROR: 0:64: Use of undeclared identifier 'beamvec'
ERROR: 0:64: Use of undeclared identifier 'beamlength'
ERROR: 0:66: Use of undeclared identifier 'lightDir'
ERROR: 0:66: Use of undeclared identifier 'beamDir'
ERROR: 0:66: Use of undeclared identifier 'beamlength'
ERROR: 0:69: Use of undeclared identifier 'lightposition'
ERROR: 0:69: Use of undeclared identifier 'beamDir'
ERROR: 0:69: Use of undeclared identifier 'neardist'
ERROR: 0:70: Use of undeclared identifier 'lightDir'
ERROR: 0:70: Use of undeclared identifier 'nearest'
ERROR: 0:71: Use of undeclared identifier 'dist'
ERROR: 0:71: Use of undeclared identifier 'lightDir'
ERROR: 0:72: Use of undeclared identifier 'dist'
ERROR: 0:75: Use of undeclared identifier 'lightDir'
ERROR: 0:75: Use of undeclared identifier 'dist'
ERROR: 0:76: Use of undeclared identifier 'lightDir'
ERROR: 0:77: Use of undeclared identifier 'half_vec'
ERROR: 0:78: Use of undeclared identifier 'lightDir'
ERROR: 0:78: Use of undeclared identifier 'attenuation'
ERROR: 0:79: Use of undeclared identifier 'NdotHV'
ERROR: 0:79: Use of undeclared identifier 'attenuation'

ERROR! Unable to create fragment shader!
  Shader in_error! Disabling GLSL model rendering!

Echelon9

2013-05-01 13:17

developer   ~0015014

Last edited: 2013-05-01 13:26

 So in summary, this is caused by uses of gr_opengl_maybe_create_shader() inside gr_opengl_render_effect() not considering the potential -1 return value and dealing with it accordingly.

The_E

2013-07-08 16:35

administrator   ~0015176

Fix committed to trunk@9718.

Related Changesets

fs2open: trunk r9718

2013-07-08 13:40

The_E


Ported: N/A

Details Diff		 Fix for Mantis 2861: Make sure we switch off shader-based effect rendering if the GPU can't handle it
 Affected Issues
0002861
mod - /trunk/fs2_open/code/graphics/grbatch.cpp Diff File
mod - /trunk/fs2_open/code/graphics/gropengldraw.cpp Diff File
mod - /trunk/fs2_open/code/graphics/gropengldraw.h Diff File

Issue History

Date Modified Username Field Change
2013-05-01 09:53 Echelon9 New Issue
2013-05-01 09:53 Echelon9 Status new => assigned
2013-05-01 09:53 Echelon9 Assigned To => Echelon9
2013-05-01 10:51 Echelon9 Steps to Reproduce Updated
2013-05-01 10:52 Echelon9 Note Added: 0015011
2013-05-01 13:08 Echelon9 Note Added: 0015013
2013-05-01 13:08 Echelon9 Assigned To Echelon9 =>
2013-05-01 13:08 Echelon9 Status assigned => confirmed
2013-05-01 13:17 Echelon9 Note Added: 0015014
2013-05-01 13:26 Echelon9 Note Edited: 0015014
2013-05-07 13:44 Echelon9 Assigned To => Valathil
2013-05-07 13:44 Echelon9 Status confirmed => assigned
2013-07-08 16:35 The_E Changeset attached => fs2open trunk r9718
2013-07-08 16:35 The_E Note Added: 0015176
2013-07-08 16:35 The_E Status assigned => resolved
2013-07-08 16:35 The_E Resolution open => fixed