View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0003049 | FSSCP | tables | public | 2014-05-28 21:45 | 2014-08-18 08:32 |
Reporter | Yarn | Assigned To | Yarn | ||
Priority | high | Severity | minor | Reproducibility | always |
Status | resolved | Resolution | reopened | ||
Platform | x64 | OS | Windows 7 | ||
Product Version | 3.7.2 | ||||
Target Version | 3.7.2 | ||||
Summary | 0003049: Characters in tstrings.tbl not being remapped properly | ||||
Description | When FreeSpace 2 loads most mission, campaign, and table files, it does so using either read_file_text() or read_file_text_from_array() (located in parselo.cpp), both of which call process_raw_file_text(). This latter function performs a few tasks, but the one pertaining to this report is character remapping. Most of this work is accomplished via a call to maybe_convert_foreign_characters(), which remaps characters like ç, ü, and é so that they can be displayed using the German and French fonts. The process_raw_file_text() function itself also converts ß to ss since the former character does not exist in the default fonts. Tstrings.tbl is the exception to this rule, however. This file is not loaded with read_file_text() or read_file_text_from_array(), so the normal character remapping does not occur. Instead, the game relies on lcl_fix_umlauts() (which is used only with tstrings.tbl) to do this job, but only if the game language is set to German. However, this function remaps only the umlaut characters (specifically, Ä, Ö, Ü, ä, ö, and ü). It also converts ß in a way that isn't compatible with the default fonts. Thus, non-ASCII characters that aren't also umlaut characters aren't displayed, even if the appropriate font is used. This issue also exists in retail, as does the lcl_fix_umlauts() function. Based on the code comments, it's obvious that the Volition programmers also knew about this issue but couldn't figure out how to fix it properly. | ||||
Steps To Reproduce | 1. Change the in-game language to German. 2. Install and play the attached mod, which includes the German font. It also includes two table files that have certain strings modified so they begin with "ß äöü áóú". 3. On the pilot selection screen, look at the line of text at the very bottom, which comes from char_remap_test-lcl.tbm. It should begin with "ss äöü áóú". According to my tests, character remapping does work properly here. 4. Go to the cutscenes screen in the Tech Room, select "Introduction," and look at the description. As with step 3, the text should start with "ss äöü áóú". Since this text is loaded from tstrings.tbl, it will look like " äöü " until this issue is fixed. | ||||
Tags | No tags attached. | ||||
|
|
|
This should be fixed in 3.7.2 because we have translation support in FSPort 3.4.1. I've assigned it to myself, but if someone else is able to fix it first, please take it and do so. |
|
Reassigning to Yarn, in case Yarn has time to look into this. Probably more time than me, at least. ;) |
|
I finally got a good look at this. As mentioned in this report, most tables (including strings.tbl) are loaded using read_file_text() or read_file_text_from_array(). Once this is done, though, all relevant information in the table (including strings) is loaded into memory. With tstrings.tbl, however, the translated strings are not initially loaded into memory; instead, FSO maintains an array of pointers into the actual file, reading the strings from the file when necessary (for example, when loading a mission). It looks like fixing this means treating tstrings.tbl like any other table and loading all of the strings of a language section into memory. Because the table is only about a megabyte (and only half of that is used for German), and considering that PCs in use today generally have far more RAM than those in 1999, I don't think that memory usage is a big concern here. |
|
Interesting. Sounds like a good plan. |
|
All right, I got the patch done. Everything in parse_stringstbl() was moved to parse_stringstbl_common() and then modified to support tstrings.tbl as well. A boolean switch is used to set how the function loads the files. Lots of now-obsolete stuff was also removed, including the open/close functions for tstrings.tbl (which is why lots of files were changed). The change also made it easy to add modular support to tstrings.tbl, so I did so. The modular tstrings.tbl suffix is *-tlc.tbm. |
|
mantis3049.patch (50,310 bytes)
Index: code/ai/ai_profiles.cpp =================================================================== --- code/ai/ai_profiles.cpp (revision 10869) +++ code/ai/ai_profiles.cpp (working copy) @@ -75,13 +75,9 @@ char *saved_Mp = NULL; char buf[NAME_LENGTH]; - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", (filename) ? filename : "<default ai_profiles.tbl>", rval)); - lcl_ext_close(); return; } @@ -514,9 +510,6 @@ // add tbl/tbm to multiplayer validation list extern void fs2netd_add_table_validation(const char *tblname); fs2netd_add_table_validation(filename); - - // close localization - lcl_ext_close(); } void ai_profiles_init() Index: code/ai/aicode.cpp =================================================================== --- code/ai/aicode.cpp (revision 10869) +++ code/ai/aicode.cpp (working copy) @@ -781,9 +781,6 @@ #define AI_CLASS_INCREMENT 10 void parse_aitbl() { - // open localization - lcl_ext_open(); - read_file_text("ai.tbl", CF_TYPE_TABLES); reset_parse(); @@ -815,9 +812,6 @@ reset_ai_class_names(); } } - - // close localization - lcl_ext_close(); atexit(free_ai_stuff); } @@ -837,7 +831,6 @@ if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", "ai.tbl", rval)); - lcl_ext_close(); } else { parse_aitbl(); } Index: code/asteroid/asteroid.cpp =================================================================== --- code/asteroid/asteroid.cpp (revision 10869) +++ code/asteroid/asteroid.cpp (working copy) @@ -1896,12 +1896,8 @@ "are no species for them to belong to." ); - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", "asteroid.tbl", rval)); - lcl_ext_close(); return; } @@ -2030,9 +2026,6 @@ } else { Asteroid_icon_closeup_zoom = 0.5f; // magic number from retail } - - // close localization - lcl_ext_close(); } /** Index: code/autopilot/autopilot.cpp =================================================================== --- code/autopilot/autopilot.cpp (revision 10869) +++ code/autopilot/autopilot.cpp (working copy) @@ -1274,13 +1274,9 @@ int rval; SCP_vector<SCP_string> lines; - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", (filename) ? filename : "<default autopilot.tbl>", rval)); - lcl_ext_close(); return; } @@ -1337,9 +1333,6 @@ required_string("#END"); - - // close localization - lcl_ext_close(); } Index: code/cutscene/cutscenes.cpp =================================================================== --- code/cutscene/cutscenes.cpp (revision 10869) +++ code/cutscene/cutscenes.cpp (working copy) @@ -55,12 +55,8 @@ int rval; cutscene_info cutinfo; - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", "cutscenes.tbl", rval)); - lcl_ext_close(); return; } @@ -107,9 +103,6 @@ } required_string("#End"); - - // close localization - lcl_ext_close(); } // function to return 0 based index of which CD a particular movie is on Index: code/fireball/fireballs.cpp =================================================================== --- code/fireball/fireballs.cpp (revision 10869) +++ code/fireball/fireballs.cpp (working copy) @@ -163,12 +163,8 @@ lod_checker lod_check; color fb_color; - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", filename, rval)); - lcl_ext_close(); return; } @@ -241,9 +237,6 @@ } required_string("#End"); - - // close localization - lcl_ext_close(); } void fireball_parse_tbl() Index: code/gamehelp/contexthelp.cpp =================================================================== --- code/gamehelp/contexthelp.cpp (revision 10869) +++ code/gamehelp/contexthelp.cpp (working copy) @@ -334,13 +334,9 @@ SCP_vector<help_left_bracket> lbracket_temp; help_left_bracket lbracket_temp2; vec3d vec3d_temp; - - // open localization - lcl_ext_open(); if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", filename, rval)); - lcl_ext_close(); return; } @@ -488,9 +484,6 @@ } // end while } // end while - - // close localization - lcl_ext_close(); } Index: code/gamesnd/eventmusic.cpp =================================================================== --- code/gamesnd/eventmusic.cpp (revision 10869) +++ code/gamesnd/eventmusic.cpp (working copy) @@ -1380,12 +1380,8 @@ if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", filename, rval)); - lcl_ext_close(); } else { - // open localization - lcl_ext_open(); - read_file_text(filename, CF_TYPE_TABLES); reset_parse(); @@ -1404,9 +1400,6 @@ } } } - - // close localization - lcl_ext_close(); } } Index: code/gamesnd/gamesnd.cpp =================================================================== --- code/gamesnd/gamesnd.cpp (revision 10869) +++ code/gamesnd/gamesnd.cpp (working copy) @@ -777,16 +777,10 @@ */ void gamesnd_parse_soundstbl() { - // open localization - lcl_ext_open(); - parse_sound_table("sounds.tbl"); parse_modular_table("*-snd.tbm", parse_sound_table); - // close localization - lcl_ext_close(); - // if we are missing any species then report if (missingFlybySounds.size() > 0) { Index: code/hud/hudparse.cpp =================================================================== --- code/hud/hudparse.cpp (revision 10869) +++ code/hud/hudparse.cpp (working copy) @@ -174,12 +174,8 @@ color *ship_clr_p = NULL; bool scale_gauge = true; - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", filename, rval)); - lcl_ext_close(); return; } @@ -455,9 +451,6 @@ required_string("$End Gauges"); required_string("#End"); } - - // close localization - lcl_ext_close(); } void hud_positions_init() Index: code/lab/wmcgui.cpp =================================================================== --- code/lab/wmcgui.cpp (revision 10869) +++ code/lab/wmcgui.cpp (working copy) @@ -149,12 +149,8 @@ DestroyClassInfo(); } - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("WMCGUI: Unable to parse '%s'! Error code = %i.\n", filename, rval)); - lcl_ext_close(); return; } @@ -171,9 +167,6 @@ } } while(flag); - // close localization - lcl_ext_close(); - ClassInfoParsed = true; } Index: code/localization/localize.cpp =================================================================== --- code/localization/localize.cpp (revision 10869) +++ code/localization/localize.cpp (working copy) @@ -75,45 +75,16 @@ // table/mission externalization stuff -------------------- - -#define TABLE_STRING_FILENAME "tstrings.tbl" -// filename of the file to use when localizing table strings -char *Lcl_ext_filename = NULL; -CFILE *Lcl_ext_file = NULL; - -// for scanning/parsing tstrings.tbl (from ExStr) #define PARSE_TEXT_BUF_SIZE PARSE_BUF_SIZE #define PARSE_ID_BUF_SIZE 5 -#define TS_SCANNING 0 // scanning for a line of text -#define TS_ID_STRING 1 // reading in an id string -#define TS_OPEN_QUOTE 2 // looking for an open quote -#define TS_STRING 3 // reading in the text string itself -int Ts_current_state = 0; -char Ts_text[PARSE_TEXT_BUF_SIZE]; // string we're currently working with -char Ts_id_text[PARSE_ID_BUF_SIZE]; // id string we're currently working with -size_t Ts_text_size; -size_t Ts_id_text_size; +#define LCL_MAX_STRINGS 4500 +char *Lcl_ext_str[LCL_MAX_STRINGS]; -// file pointers for optimized string lookups -// some example times for FreeSpace2 startup with granularities (mostly .tbl files, ~500 strings in the table file, many looked up more than once) -// granularity 20 : 13 secs -// granularity 10 : 11 secs -// granularity 5 : 9 secs -// granularity 2 : 7-8 secs -#define LCL_GRANULARITY 1 // how many strings between each pointer (lower granularities should give faster lookup times) -#define LCL_MAX_POINTERS 4500 // max # of pointers -#define LCL_MAX_STRINGS (LCL_GRANULARITY * LCL_MAX_POINTERS) -int Lcl_pointers[LCL_MAX_POINTERS]; -int Lcl_pointer_count = 0; - // ------------------------------------------------------------------------------------------------------------ // LOCALIZE FORWARD DECLARATIONS // -// associate table file externalization with the specified input file -void lcl_ext_associate(const char *filename); - // given a valid XSTR() tag piece of text, extract the string portion, return it in out, nonzero on success int lcl_ext_get_text(const char *xstr, char *out); int lcl_ext_get_text(const SCP_string &xstr, SCP_string &out); @@ -122,23 +93,9 @@ int lcl_ext_get_id(const char *xstr, int *out); int lcl_ext_get_id(const SCP_string &xstr, int *out); -// given a valid XSTR() id#, lookup the string in tstrings.tbl, filling in out if found, nonzero on success -int lcl_ext_lookup(char *out, int id); - // if the char is a valid char for a signed integer value string int lcl_is_valid_numeric_char(char c); -// sub-parse function for individual lines of tstrings.tbl (from Exstr) -// returns : integer with the low bits having the following values : -// 0 on fail, 1 on success, 2 if found a matching id/string pair, 3 if end of language has been found -// for cases 1 and 2 : the high bit (1<<31) will be set if the parser detected the beginning of a new string id on this line -// so be sure to mask this value out to get the low portion of the return value -// -int lcl_ext_lookup_sub(const char *text, char *out, int id); - -// initialize the pointer array into tstrings.tbl (call from lcl_ext_open() ONLY) -void lcl_ext_setup_pointers(); - // parses the string.tbl and reports back only on the languages it found void parse_stringstbl_quick(const char *filename); @@ -211,26 +168,10 @@ lang = lang_init; } - // language markers - Lcl_pointer_count = 0; - - // associate the table string file - lcl_ext_associate(TABLE_STRING_FILENAME); - // set the language (this function takes care of setting up file pointers) - lcl_set_language(lang); + lcl_set_language(lang); } -// added 2.2.99 by NeilK to take care of fs2 launcher memory leaks -// shutdown localization -void lcl_close() -{ - // if the filename exists, free it up - if(Lcl_ext_filename != NULL){ - vm_free(Lcl_ext_filename); - } -} - // determine what language we're running in, see LCL_* defines above int lcl_get_language() { @@ -244,9 +185,6 @@ int lang_idx; int i; - // make sure localization is NOT running - lcl_ext_close(); - read_file_text(filename, CF_TYPE_TABLES); reset_parse(); @@ -253,9 +191,9 @@ if (optional_string("#Supported Languages")) { while (required_string_either("#End","$Language:")) { required_string("$Language:"); - stuff_string(language.lang_name, F_NAME, LCL_LANG_NAME_LEN + 1); + stuff_string(language.lang_name, F_RAW, LCL_LANG_NAME_LEN + 1); required_string("+Extension:"); - stuff_string(language.lang_ext, F_NAME, LCL_LANG_NAME_LEN + 1); + stuff_string(language.lang_ext, F_RAW, LCL_LANG_NAME_LEN + 1); required_string("+Special Character Index:"); stuff_ubyte(&language.special_char_indexes[0]); for (i = 1; i < MAX_FONTS; ++i) { @@ -288,7 +226,9 @@ } } -void parse_stringstbl(const char *filename) +// Unified function for loading strings.tbl and tstrings.tbl (and their modular versions). +// The "external" parameter controls which format to load: true for tstrings.tbl, false for strings.tbl +void parse_stringstbl_common(const char *filename, const bool external) { char chr, buf[4096]; char language_tag[512]; @@ -296,9 +236,6 @@ char *p_offset = NULL; int offset_lo = 0, offset_hi = 0; - // make sure localization is NOT running - lcl_ext_close(); - read_file_text(filename, CF_TYPE_TABLES); reset_parse(); @@ -305,7 +242,11 @@ // move down to the proper section memset(language_tag, 0, sizeof(language_tag)); strcpy_s(language_tag, "#"); - strcat_s(language_tag, Lcl_languages[Lcl_current_lang].lang_name); + if (external && Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE){ + strcat_s(language_tag, "default"); + } else { + strcat_s(language_tag, Lcl_languages[Lcl_current_lang].lang_name); + } if ( skip_to_string(language_tag) != 1 ) { mprintf(("Current language not found in %s\n", filename)); @@ -317,44 +258,41 @@ int num_offsets_on_this_line = 0; stuff_int(&index); - stuff_string(buf, F_NAME, sizeof(buf)); + if (external) { + ignore_white_space(); + get_string(buf, sizeof(buf)); + drop_trailing_white_space(buf); + } else { + stuff_string(buf, F_RAW, sizeof(buf)); + } - if (index < 0 || index >= XSTR_SIZE) { + if (external && (index < 0 || index >= LCL_MAX_STRINGS)) { + Error(LOCATION, "Invalid tstrings table index specified (%i). Please increment LCL_MAX_STRINGS in localize.cpp.", index); + } else if (!external && (index < 0 || index >= XSTR_SIZE)) { Error(LOCATION, "Invalid strings table index specified (%i)", index); } if (Lcl_pl) lcl_fix_polish(buf); + + if (!external) { + i = strlen(buf); - i = strlen(buf); + while (i--) { + if ( !isspace(buf[i]) ) + break; + } - while (i--) { - if ( !isspace(buf[i]) ) - break; - } + // trim unnecessary end of string + if (i >= 0) { + // Assert(buf[i] == '"'); + if (buf[i] != '"') { + // probably an offset on this entry - // trim unneccesary end of string - if (i >= 0) { - // Assert(buf[i] == '"'); - if (buf[i] != '"') { - // probably an offset on this entry + // drop down a null terminator (prolly unnecessary) + buf[i+1] = 0; - // drop down a null terminator (prolly unnecessary) - buf[i+1] = 0; - - // back up over the potential offset - while ( !is_white_space(buf[i]) ) - i--; - - // now back up over intervening spaces - while ( is_white_space(buf[i]) ) - i--; - - num_offsets_on_this_line = 1; - - if (buf[i] != '"') { - // could have a 2nd offset value (one for 640, one for 1024) - // so back up again + // back up over the potential offset while ( !is_white_space(buf[i]) ) i--; @@ -362,49 +300,80 @@ while ( is_white_space(buf[i]) ) i--; - num_offsets_on_this_line = 2; + num_offsets_on_this_line = 1; + + if (buf[i] != '"') { + // could have a 2nd offset value (one for 640, one for 1024) + // so back up again + while ( !is_white_space(buf[i]) ) + i--; + + // now back up over intervening spaces + while ( is_white_space(buf[i]) ) + i--; + + num_offsets_on_this_line = 2; + } + + p_offset = &buf[i+1]; // get ptr to string section with offset in it + + if (buf[i] != '"') + Error(LOCATION, "%s is corrupt", filename); // now its an error } - p_offset = &buf[i+1]; // get ptr to string section with offset in it - - if (buf[i] != '"') - Error(LOCATION, "%s is corrupt", filename); // now its an error + buf[i] = 0; } - buf[i] = 0; - } + // copy string into buf + z = 0; + for (i = 1; buf[i]; i++) { + chr = buf[i]; - // copy string into buf - z = 0; - for (i = 1; buf[i]; i++) { - chr = buf[i]; + if (chr == '\\') { + chr = buf[++i]; - if (chr == '\\') { - chr = buf[++i]; + if (chr == 'n') + chr = '\n'; + else if (chr == 'r') + chr = '\r'; + } - if (chr == 'n') - chr = '\n'; - else if (chr == 'r') - chr = '\r'; + buf[z++] = chr; } - buf[z++] = chr; + // null terminator on buf + buf[z] = 0; } - // null terminator on buf - buf[z] = 0; - - // write into Xstr_table - if ( Parsing_modular_table && (Xstr_table[index].str != NULL) ) { - vm_free((void *) Xstr_table[index].str); - Xstr_table[index].str = NULL; + // write into Xstr_table (for strings.tbl) or Lcl_ext_str (for tstrings.tbl) + if (Parsing_modular_table) { + if ( external && (Lcl_ext_str[index] != NULL) ) { + vm_free((void *) Lcl_ext_str[index]); + Lcl_ext_str[index] = NULL; + } else if ( !external && (Xstr_table[index].str != NULL) ) { + vm_free((void *) Xstr_table[index].str); + Xstr_table[index].str = NULL; + } } - if (Xstr_table[index].str != NULL) + if (external && (Lcl_ext_str[index] != NULL)) { + Warning(LOCATION, "Tstrings table index %d used more than once", index); + } else if (!external && (Xstr_table[index].str != NULL)) { Warning(LOCATION, "Strings table index %d used more than once", index); + } - Xstr_table[index].str = vm_strdup(buf); + if (external) { + Lcl_ext_str[index] = vm_strdup(buf); + } else { + Xstr_table[index].str = vm_strdup(buf); + } + // the rest of this loop applies only to strings.tbl, + // so we can move on to the next line if we're reading from tstrings.tbl + if (external) { + continue; + } + // read offset information, assume 0 if nonexistant if (p_offset != NULL) { if (sscanf(p_offset, "%d%d", &offset_lo, &offset_hi) < num_offsets_on_this_line) { @@ -427,14 +396,29 @@ } } +void parse_stringstbl(const char *filename) +{ + parse_stringstbl_common(filename, false); +} + +void parse_tstringstbl(const char *filename) +{ + parse_stringstbl_common(filename, true); +} + // initialize the xstr table void lcl_xstr_init() { int i, rval; + for (i = 0; i < XSTR_SIZE; i++) Xstr_table[i].str = NULL; + for (i = 0; i < LCL_MAX_STRINGS; i++) + Lcl_ext_str[i] = NULL; + + if ( (rval = setjmp(parse_abort)) != 0 ) mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", "strings.tbl", rval)); else @@ -442,6 +426,15 @@ parse_modular_table(NOX("*-lcl.tbm"), parse_stringstbl); + + if ( (rval = setjmp(parse_abort)) != 0 ) + mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", "tstrings.tbl", rval)); + else + parse_tstringstbl("tstrings.tbl"); + + parse_modular_table(NOX("*-tlc.tbm"), parse_tstringstbl); + + Xstr_inited = 1; } @@ -455,7 +448,13 @@ Xstr_table[i].str = NULL; } } - vm_free(Lcl_ext_filename); + + for (int i=0; i<LCL_MAX_STRINGS; i++){ + if (Lcl_ext_str[i] != NULL) { + vm_free((void *) Lcl_ext_str[i]); + Lcl_ext_str[i] = NULL; + } + } } @@ -483,14 +482,6 @@ } else if (!strcmp(Lcl_languages[Lcl_current_lang].lang_name, Lcl_builtin_languages[LCL_POLISH].lang_name)) { Lcl_pl = 1; } - - // set to 0, so lcl_ext_open() knows to reset file pointers - Lcl_pointer_count = 0; - - // reset file pointers to the proper language-section - if(Lcl_current_lang != FS2_OPEN_DEFAULT_LANGUAGE){ - lcl_ext_setup_pointers(); - } } ubyte lcl_get_font_index(int font_num) @@ -571,42 +562,6 @@ // externalization of table/mission files ----------------------- -// open the externalization file for use during parsing (call before parsing a given file) -void lcl_ext_open() -{ - // if the file is already open, do nothing - Assert(Lcl_ext_file == NULL); - - // if we're running in the default language, do nothing - if(Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE){ - return; - } - - // otherwise open the file - Lcl_ext_file = cfopen(Lcl_ext_filename, "rt"); - if(Lcl_ext_file == NULL){ - return; - } -} - -// close the externalization file (call after parsing a given file) -void lcl_ext_close() -{ - // if the file is not open, do nothing - if(Lcl_ext_file == NULL){ - return; - } - - // if we're running in the default language, do nothing - if(Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE){ - return; - } - - // otherwise close it - cfclose(Lcl_ext_file); - Lcl_ext_file = NULL; -} - void lcl_replace_stuff(char *text, size_t max_len) { if (Fred_running) @@ -681,7 +636,6 @@ void lcl_ext_localize_sub(const char *in, char *out, size_t max_len, int *id) { char text_str[PARSE_BUF_SIZE]=""; - char lookup_str[PARSE_BUF_SIZE]=""; int str_id; size_t str_len; @@ -747,7 +701,7 @@ } // if the localization file is not open, or we're running in the default language, return the original string - if ( (Lcl_ext_file == NULL) || (str_id < 0) || (Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE) ) { + if ( !Xstr_inited || (str_id < 0) || (Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE) ) { if ( strlen(text_str) > max_len ) error_display(0, "Token too long: [%s]. Length = %i. Max is %i.\n", text_str, strlen(text_str), max_len); @@ -759,13 +713,13 @@ return; } - // attempt to find the string - if (lcl_ext_lookup(lookup_str, str_id)) { + // get the string if it exists + if (Lcl_ext_str[str_id] != NULL) { // copy to the outgoing string - if ( strlen(lookup_str) > max_len ) - error_display(0, "Token too long: [%s]. Length = %i. Max is %i.\n", lookup_str, strlen(lookup_str), max_len); + if ( strlen(Lcl_ext_str[str_id]) > max_len ) + error_display(0, "Token too long: [%s]. Length = %i. Max is %i.\n", Lcl_ext_str[str_id], strlen(Lcl_ext_str[str_id]), max_len); - strncpy(out, lookup_str, max_len); + strncpy(out, Lcl_ext_str[str_id], max_len); } // otherwise use what we have - probably should Int3() or assert here else { @@ -785,7 +739,6 @@ void lcl_ext_localize_sub(const SCP_string &in, SCP_string &out, int *id) { SCP_string text_str = ""; - char lookup_str[PARSE_BUF_SIZE]=""; int str_id; // default (non-external string) value @@ -833,7 +786,7 @@ } // if the localization file is not open, or we're running in the default language, return the original string - if ( (Lcl_ext_file == NULL) || (str_id < 0) || (Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE) ) { + if ( !Xstr_inited || (str_id < 0) || (Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE) ) { out = text_str; if (id != NULL) @@ -843,9 +796,9 @@ } // attempt to find the string - if (lcl_ext_lookup(lookup_str, str_id)) { + if (Lcl_ext_str[str_id] != NULL) { // copy to the outgoing string - out = lookup_str; + out = Lcl_ext_str[str_id]; } // otherwise use what we have - probably should Int3() or assert here else { @@ -916,18 +869,6 @@ // LOCALIZE FORWARD DEFINITIONS // -// associate table file externalization with the specified input file -void lcl_ext_associate(const char *filename) -{ - // if the filename already exists, free it up - if(Lcl_ext_filename != NULL){ - vm_free(Lcl_ext_filename); - } - - // set the new filename - Lcl_ext_filename = vm_strdup(filename); -} - // given a valid XSTR() tag piece of text, extract the string portion, return it in out, nonzero on success int lcl_ext_get_text(const char *xstr, char *out) { @@ -1162,192 +1103,6 @@ return 1; } -// given a valid XSTR() id#, lookup the string in tstrings.tbl, filling in out if found, nonzero on success -int lcl_ext_lookup(char *out, int id) -{ - char text[1024]; - int ret; - int pointer; - - Assert(Lcl_pointer_count >= 0); - Assert(Lcl_pointers[0] >= 0); - Assert(Lcl_pointers[Lcl_pointer_count - 1] >= 0); - Assert(Lcl_ext_file != NULL); - Assert(id >= 0); - - // seek to the closest pointer <= the id# we're looking for - pointer = id / LCL_GRANULARITY; - cfseek(Lcl_ext_file, Lcl_pointers[pointer], CF_SEEK_SET); - - // reset parsing vars and go to town - Ts_current_state = TS_SCANNING; - Ts_id_text_size = 0; -// Ts_text_size; - memset(Ts_text, 0, PARSE_TEXT_BUF_SIZE); - memset(Ts_id_text, 0, PARSE_ID_BUF_SIZE); - while((cftell(Lcl_ext_file) < Lcl_pointers[Lcl_pointer_count - 1]) && cfgets(text, 1024, Lcl_ext_file)){ - ret = lcl_ext_lookup_sub(text, out, id); - - // run the line parse function - switch(ret & 0x0fffffff){ - // error - case 0 : - Int3(); // should never get here - it means the string doens't exist in the table!! - return 0; - - // success parsing the line - please continue - case 1 : - break; - - // found a matching string/id pair - case 2 : - // success - if (Lcl_gr) { - // this is because tstrings.tbl reads in as ANSI for some reason - // opening tstrings with "rb" mode didnt seem to help, so its now still "rt" like before - lcl_fix_umlauts(out, LCL_TO_ASCII); - } - return 1; - - // end of language found - case 3 : - Int3(); // should never get here - it means the string doens't exist in the table!! - return 0; - } - } - - Int3(); // should never get here - it means the string doens't exist in the table!! - return 0; -} - -// sub-parse function for individual lines of tstrings.tbl (from Exstr) -// returns : integer with the low bits having the following values : -// 0 on fail, 1 on success, 2 if found a matching id/string pair, 3 if end of language has been found -// for cases 1 and 2 : the high bit (1<<31) will be set if the parser detected the beginning of a new string id on this line -// -int lcl_ext_lookup_sub(const char *text, char *out, int id) -{ - const char *p; // current ptr - int len = strlen(text); - int count; - char text_copy[1024]; - char *tok; - int found_new_string_id = 0; - - p = text; - count = 0; - while(count < len){ - // do something useful - switch(Ts_current_state){ - // scanning for a line of text - case TS_SCANNING: - // if the first word is #end, we're done with the file altogether - strcpy_s(text_copy, text); - tok = strtok(text_copy, " \n"); - if((tok != NULL) && !stricmp(tok, "#end")){ - return 3; - } - // if its a commented line, skip it - else if((text[0] == ';') || (text[0] == ' ') || (text[0] == '\n')){ - return 1; - } - // otherwise we should have an ID #, so stuff it and move to the proper state - else { - if(lcl_is_valid_numeric_char(*p)){ - memset(Ts_id_text, 0, PARSE_ID_BUF_SIZE); - Ts_id_text_size = 0; - Ts_id_text[Ts_id_text_size++] = *p; - Ts_current_state = TS_ID_STRING; - - found_new_string_id = 1; - } - // error - else { - Int3(); - return 0; - } - } - break; - - // scanning in an id string - case TS_ID_STRING: - // if we have another valid char - if(lcl_is_valid_numeric_char(*p)) { - if (Ts_id_text_size >= PARSE_ID_BUF_SIZE - 1) { - error_display(0, "XSTR id %s too long!\n", Ts_id_text); - return 0; - } - Ts_id_text[Ts_id_text_size++] = *p; - } - // if we found a comma, our id# is finished, look for the open quote - else if(*p == ','){ - Ts_current_state = TS_OPEN_QUOTE; - } else { - Int3(); - return 0; - } - break; - - case TS_OPEN_QUOTE: - // valid space or an open quote - if((*p == ' ') || (*p == '\"')){ - if(*p == '\"'){ - memset(Ts_text, 0, PARSE_TEXT_BUF_SIZE); - Ts_text_size = 0; - Ts_current_state = TS_STRING; - } - } else { - Int3(); - return 0; - } - break; - - case TS_STRING: - // if we have an end quote, we need to look for a comma - if((*p == '\"') /*&& (Ts_text_size > 0)*/ && (Ts_text[Ts_text_size - 1] != '\\')){ - // we're now done - we have a string - Ts_current_state = TS_SCANNING; - - // if the id#'s match, copy the string and return "string found" - if((atoi(Ts_id_text) == id) && (out != NULL)){ - // this is redundant to the PARSE_TEXT_BUF_SIZE, but let's be future proof - if (strlen(Ts_text) > PARSE_BUF_SIZE - 1) { - error_display(0, "XSTR text result exceeds output buffer size!\n\n%s\n", Ts_text); - return 0; - } - strcpy(out, Ts_text); - - return found_new_string_id ? (1<<1) | (1<<31) : (1<<1); - } - - // otherwise, just continue parsing - return found_new_string_id ? (1<<0) | (1<<31) : (1<<0); - } - // otherwise add to the string - else { - if (Ts_text_size >= PARSE_TEXT_BUF_SIZE - 1) { - error_display(0, "XSTR text too long!\n\n%s\n", Ts_text); - return 0; - } - Ts_text[Ts_text_size++] = *p; - } - break; - } - - // if we have a newline, return success, we're done with this line - if(*p == '\n'){ - return found_new_string_id ? (1<<0) | (1<<31) : (1<<0); - } - - // next char in the line - p++; - count++; - } - - // success - return found_new_string_id ? (1<<0) | (1<<31) : (1<<0); -} - // if the char is a valid char for a signed integer value int lcl_is_valid_numeric_char(char c) { @@ -1355,98 +1110,6 @@ (c == '5') || (c == '6') || (c == '7') || (c == '8') || (c == '9') ) ? 1 : 0; } -// initialize the pointer array into tstrings.tbl (call from lcl_ext_open() ONLY) -void lcl_ext_setup_pointers() -{ - char language_string[128]; - char line[1024]; - char *tok; - int string_count; - int ret; - int found_start = 0; - - // open the localization file - lcl_ext_open(); - if(Lcl_ext_file == NULL){ - error_display(0, "Error opening externalization file! File likely does not exist or could not be found\n"); - return; - } - - // seek to the currently active language - memset(language_string, 0, 128); - strcpy_s(language_string, "#"); - if(Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE){ - strcat_s(language_string, "default"); - } else { - strcat_s(language_string, Lcl_languages[Lcl_current_lang].lang_name); - } - memset(line, 0, 1024); - - // reset seek variables and begin - Lcl_pointer_count = 0; - while(cfgets(line, 1024, Lcl_ext_file)){ - tok = strtok(line, " \n"); - if(tok == NULL){ - continue; - } - - // if the language matches, we're good to start parsing strings - if(!stricmp(language_string, tok)){ - found_start = 1; - break; - } - } - - // if we didn't find the language specified, error - if(found_start <= 0){ - error_display(0, "Could not find specified language in tstrings.tbl!\n"); - lcl_ext_close(); - return; - } - - string_count = 0; - while(cfgets(line, 1024, Lcl_ext_file)){ - ret = lcl_ext_lookup_sub(line, NULL, -1); - - // do stuff - switch(ret & 0x0fffffff){ - // error - case 0 : - lcl_ext_close(); - return; - - // end of language found - case 3 : - // mark one final pointer - Lcl_pointers[Lcl_pointer_count++] = cftell(Lcl_ext_file) - strlen(line) - 1; - lcl_ext_close(); - return; - } - - // the only other case we care about is the beginning of a new id# - if(ret & (1<<31)){ - if((string_count % LCL_GRANULARITY) == 0){ - // mark the pointer down - Lcl_pointers[Lcl_pointer_count++] = cftell(Lcl_ext_file) - strlen(line) - 1; - - // if we're out of pointer slots - if(Lcl_pointer_count >= LCL_MAX_POINTERS){ - error_display(0, "Out of pointers for tstrings.tbl lookup. Please increment LCL_MAX_POINTERS in localize.cpp\n"); - lcl_ext_close(); - return; - } - } - // increment string count - string_count++; - } - } - - // should never get here. we should always be exiting through case 3 (end of language section) of the above switch - // statement - Int3(); - lcl_ext_close(); -} - void lcl_get_language_name(char *lang_name) { Assert(Lcl_current_lang < (int)Lcl_languages.size()); @@ -1454,94 +1117,6 @@ strcpy(lang_name, Lcl_languages[Lcl_current_lang].lang_name); } -// converts german umlauted chars from ASCII to ANSI -// so they appear in the launcher -// how friggin lame is this -// pass in a null terminated string, foo! -// returns ptr to string you sent in -char* lcl_fix_umlauts(char *str, int which_way) -{ - int i=0; - - if (which_way == LCL_TO_ANSI) { - // moving to ANSI charset - // run thru string and perform appropriate conversions - while (str[i] != '\0') { - switch (str[i]) { - case '\x81': - // lower umlaut u - str[i] = '\xFC'; - break; - case '\x84': - // lower umlaut a - str[i] = '\xE4'; - break; - case '\x94': - // lower umlaut o - str[i] = '\xF6'; - break; - case '\x9A': - // upper umlaut u - str[i] = '\xDC'; - break; - case '\x8E': - // upper umlaut a - str[i] = '\xC4'; - break; - case '\x99': - // upper umlaut o - str[i] = '\xD6'; - break; - case '\xE1': - // beta-lookin thing that means "ss" - str[i] = '\xDF'; - break; - } - - i++; - } - } else { - // moving to ASCII charset - // run thru string and perform appropriate conversions - while (str[i] != '\0') { - switch (str[i]) { - case '\xFC': - // lower umlaut u - str[i] = '\x81'; - break; - case '\xE4': - // lower umlaut a - str[i] = '\x84'; - break; - case '\xF6': - // lower umlaut o - str[i] = '\x94'; - break; - case '\xDC': - // upper umlaut u - str[i] = '\x9A'; - break; - case '\xC4': - // upper umlaut a - str[i] = '\x8E'; - break; - case '\xD6': - // upper umlaut o - str[i] = '\x99'; - break; - case '\xDF': - // beta-lookin thing that means "ss" - str[i] = '\xE1'; - break; - } - - i++; - } - } - - return str; -} - // convert some of the polish characters void lcl_fix_polish(char *str) { Index: code/localization/localize.h =================================================================== --- code/localization/localize.h (revision 10869) +++ code/localization/localize.h (working copy) @@ -92,12 +92,6 @@ // maybe add localized directory to full path with file name when opening a localized file int lcl_add_dir_to_path_with_filename(char *current_path, size_t path_max); -// open the externalization file for use during parsing (call before parsing a given file) -void lcl_ext_open(); - -// close the externalization file (call after parsing a given file) -void lcl_ext_close(); - // Goober5000 void lcl_replace_stuff(char *text, size_t max_len); void lcl_replace_stuff(SCP_string &text); @@ -120,23 +114,10 @@ const char *XSTR(const char *str, int index); int lcl_get_xstr_offset(int index, int res); -// translate umlauted chars from ascii to ansi codes -// used in launcher -#define LCL_TO_ANSI 0 -#define LCL_TO_ASCII 1 -char* lcl_fix_umlauts(char *str, int which_way); - // covert some polish characters void lcl_fix_polish(char *str); void lcl_fix_polish(SCP_string &str); -// macro for launcher xstrs -#if defined(GERMAN_BUILD) -#define LXSTR(str, i) (lcl_fix_umlauts(XSTR(str, i), LCL_TO_ANSI)) -#else -#define LXSTR(str, i) (XSTR(str, i)) -#endif // defined(GERMAN_BUILD) - void lcl_translate_wep_name_gr(char *name); void lcl_translate_ship_name_gr(char *name); void lcl_translate_brief_icon_name_gr(char *name); Index: code/menuui/credits.cpp =================================================================== --- code/menuui/credits.cpp (revision 10869) +++ code/menuui/credits.cpp (working copy) @@ -415,17 +415,11 @@ void credits_parse() { - // open localization - lcl_ext_open(); - // Parse main table credits_parse_table("credits.tbl"); // Parse modular tables parse_modular_table("*-crd.tbm", credits_parse_table); - - // close localization - lcl_ext_close(); } void credits_init() Index: code/menuui/playermenu.cpp =================================================================== --- code/menuui/playermenu.cpp (revision 10869) +++ code/menuui/playermenu.cpp (working copy) @@ -1314,12 +1314,8 @@ Num_player_tips = 0; - // begin external localization stuff - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", "tips.tbl", rval)); - lcl_ext_close(); return; } @@ -1334,9 +1330,6 @@ } Player_tips[Num_player_tips++] = stuff_and_malloc_string(F_NAME, NULL); } - - // stop externalizing, homey - lcl_ext_close(); } // close out player tips - *only call from game_shutdown()* Index: code/menuui/snazzyui.cpp =================================================================== --- code/menuui/snazzyui.cpp (revision 10869) +++ code/menuui/snazzyui.cpp (working copy) @@ -188,16 +188,10 @@ *num_regions=0; - // open localization - lcl_ext_open(); - fp = cfopen( NOX("menu.tbl"), "rt" ); if (fp == NULL) { Error(LOCATION, "menu.tbl could not be opened\n"); - // close localization - lcl_ext_close(); - return; } @@ -208,9 +202,6 @@ p1 = p3 = strchr( tmp_line, '[' ); if (p3 && state == 1) { - // close localization - lcl_ext_close(); - cfclose(fp); return; } @@ -234,9 +225,6 @@ if (!p2) { nprintf(("Warning","Error parsing menu file\n")); - // close localization - lcl_ext_close(); - return; } *p2 = 0; @@ -279,9 +267,6 @@ } } cfclose(fp); - - // close localization - lcl_ext_close(); } // snazzy_menu_close() is called when the menu using a snazzy interface is exited Index: code/menuui/techmenu.cpp =================================================================== --- code/menuui/techmenu.cpp (revision 10869) +++ code/menuui/techmenu.cpp (working copy) @@ -1032,12 +1032,8 @@ if (inited) return; - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", "species.tbl", rval)); - lcl_ext_close(); return; } @@ -1075,9 +1071,6 @@ } inited = 1; - - // close localization - lcl_ext_close(); } void techroom_init() Index: code/mission/missionbriefcommon.cpp =================================================================== --- code/mission/missionbriefcommon.cpp (revision 10869) +++ code/mission/missionbriefcommon.cpp (working copy) @@ -305,12 +305,8 @@ Assert(!Species_info.empty()); const size_t max_icons = Species_info.size() * MIN_BRIEF_ICONS; - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", "icons.tbl", rval)); - lcl_ext_close(); return; } @@ -351,10 +347,7 @@ } required_string("#End"); - // close localization - lcl_ext_close(); - // now assign the icons to their species const size_t num_species_covered = Briefing_icon_info.size() / MIN_BRIEF_ICONS; size_t bii_index = 0; Index: code/mission/missioncampaign.cpp =================================================================== --- code/mission/missioncampaign.cpp (revision 10869) +++ code/mission/missioncampaign.cpp (working copy) @@ -113,9 +113,6 @@ } Assert(fname_len < MAX_FILENAME_LEN); - // open localization - lcl_ext_open(); - *type = -1; do { if ((rval = setjmp(parse_abort)) != 0) { @@ -167,9 +164,6 @@ } } while (0); - // close localization - lcl_ext_close(); - Assert(success); return success; } @@ -425,9 +419,6 @@ filename = cf_add_ext(filename, FS_CAMPAIGN_FILE_EXT); - // open localization - lcl_ext_open(); - if ( pl == NULL ) pl = Player; @@ -440,9 +431,6 @@ if ((rval = setjmp(parse_abort)) != 0) { mprintf(("Error parsing '%s'\r\nError code = %i.\r\n", filename, rval)); - // close localization - lcl_ext_close(); - Campaign.filename[0] = 0; Campaign.num_missions = 0; @@ -565,9 +553,6 @@ } else { if ( cm->formula == -1 ){ - // close localization - lcl_ext_close(); - Campaign_load_failure = CAMPAIGN_ERROR_SEXP_EXHAUSTED; return CAMPAIGN_ERROR_SEXP_EXHAUSTED; } @@ -605,9 +590,6 @@ } else { if ( cm->mission_loop_formula == -1 ){ - // close localization - lcl_ext_close(); - Campaign_load_failure = CAMPAIGN_ERROR_SEXP_EXHAUSTED; return CAMPAIGN_ERROR_SEXP_EXHAUSTED; } @@ -648,9 +630,6 @@ } } - // close localization - lcl_ext_close(); - // set up the other variables for the campaign stuff. After initializing, we must try and load // the campaign save file for this player. Since all campaign loads go through this routine, I // think this place should be the only necessary place to load the campaign save stuff. The campaign @@ -1447,13 +1426,9 @@ int i, z, rval, event_count, count = 0; filename = Campaign.missions[num].name; - - // open localization - lcl_ext_open(); if ((rval = setjmp(parse_abort)) != 0) { mprintf(("MISSIONCAMPAIGN: Unable to parse '%s'! Error code = %i.\n", filename, rval)); - lcl_ext_close(); return; } @@ -1552,9 +1527,6 @@ } // Goober5000 - variables do not need to be read here - - // close localization - lcl_ext_close(); } /** Index: code/mission/missionmessage.cpp =================================================================== --- code/mission/missionmessage.cpp (revision 10869) +++ code/mission/missionmessage.cpp (working copy) @@ -533,9 +533,6 @@ { int i, j; - // open localization - lcl_ext_open(); - //speed things up a little by setting the capacities for the message vectors to roughly the FS2 amounts Messages.reserve(500); Message_waves.reserve(300); @@ -646,10 +643,6 @@ required_string("#End"); } - - - // close localization - lcl_ext_close(); } // this is called at the start of each level Index: code/mission/missionparse.cpp =================================================================== --- code/mission/missionparse.cpp (revision 10869) +++ code/mission/missionparse.cpp (working copy) @@ -5828,9 +5828,6 @@ if ( mission_p == NULL ) mission_p = &The_mission; - // open localization - lcl_ext_open(); - do { CFILE *ftemp = cfopen(real_fname, "rt"); if (!ftemp) { @@ -5857,9 +5854,6 @@ parse_mission_info(mission_p, basic); } while (0); - // close localization - lcl_ext_close(); - return rval; } @@ -5900,9 +5894,6 @@ for (i = 0; i < Num_ship_classes; i++) Ship_class_names[i] = Ship_info[i].name; - - // open localization - lcl_ext_open(); do { // don't do this for imports @@ -5943,9 +5934,6 @@ display_parse_diagnostics(); } while (0); - // close localization - lcl_ext_close(); - if (!Fred_running) strcpy_s(Mission_filename, mission_name); @@ -6302,9 +6290,6 @@ if ( filelength == 0 ) return 0; - // open localization - lcl_ext_open(); - game_type = 0; do { if ((rval = setjmp(parse_abort)) != 0) { @@ -6328,9 +6313,6 @@ stuff_int(&game_type); } while (0); - // close localization - lcl_ext_close(); - return (game_type & MISSION_TYPE_MULTI) ? game_type : 0; } Index: code/missionui/missiondebrief.cpp =================================================================== --- code/missionui/missiondebrief.cpp (revision 10869) +++ code/missionui/missiondebrief.cpp (working copy) @@ -1059,12 +1059,8 @@ int rval; int stage_num; - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", "traitor.tbl", rval)); - lcl_ext_close(); return; } @@ -1103,9 +1099,6 @@ stuff_string( stagep->recommendation_text, F_MULTITEXT, NULL); inited = 1; - - // close localization - lcl_ext_close(); } // disable the accept button if in single player and I am a traitor Index: code/mod_table/mod_table.cpp =================================================================== --- code/mod_table/mod_table.cpp (revision 10869) +++ code/mod_table/mod_table.cpp (working copy) @@ -37,13 +37,9 @@ int rval; // SCP_vector<SCP_string> lines; - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", (filename) ? filename : "<default game_settings.tbl>", rval)); - lcl_ext_close(); return; } @@ -254,9 +250,6 @@ } required_string("#END"); - - // close localization - lcl_ext_close(); } void mod_table_init() Index: code/parse/parselo.cpp =================================================================== --- code/parse/parselo.cpp (revision 10869) +++ code/parse/parselo.cpp (working copy) @@ -1131,12 +1131,17 @@ /** * Stuff a string (" chars ") into *str, return length. + * Accepts an optional max length parameter. If it is omitted or negative, then no max length is enforced. */ -int get_string(char *str) +int get_string(char *str, int max) { int len; len = strcspn(Mp + 1, "\""); + + if (max >= 0 && len >= max) + error_display(0, "String too long. Length = %i. Max is %i.\n", len, max); + strncpy(str, Mp + 1, len); str[len] = 0; @@ -1246,7 +1251,7 @@ copy_to_eoln(read_str, terminators, Mp, read_len); drop_trailing_white_space(read_str); advance_to_eoln(terminators); - break; + break; default: Error(LOCATION, "Unhandled string type %d in stuff_string!", type); @@ -1356,7 +1361,7 @@ copy_to_eoln(read_str, terminators, Mp); drop_trailing_white_space(read_str); advance_to_eoln(terminators); - break; + break; default: Error(LOCATION, "Unhandled string type %d in stuff_string!", type); Index: code/parse/parselo.h =================================================================== --- code/parse/parselo.h (revision 10869) +++ code/parse/parselo.h (working copy) @@ -159,7 +159,7 @@ extern int match_and_stuff(int f_type, char *strlist[], int max, char *description); extern void find_and_stuff_or_add(char *id, int *addr, int f_type, char *strlist[], int *total, int max, char *description); -extern int get_string(char *str); +extern int get_string(char *str, int max = -1); extern void get_string(SCP_string &str); extern void stuff_parenthesized_vec3d(vec3d *vp); extern void stuff_boolean(int *i, bool a_to_eol=true); Index: code/ship/ship.cpp =================================================================== --- code/ship/ship.cpp (revision 10869) +++ code/ship/ship.cpp (working copy) @@ -4123,12 +4123,8 @@ { int rval; - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", filename, rval)); - lcl_ext_close(); return; } @@ -4165,9 +4161,6 @@ // add tbl/tbm to multiplayer validation list fs2netd_add_table_validation(filename); - - // close localization - lcl_ext_close(); } // The E - Simple lookup function for FRED. @@ -4224,14 +4217,10 @@ void parse_shiptbl(const char *filename) { int rval; - - // open localization - lcl_ext_open(); if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", filename, rval)); - lcl_ext_close(); return; } @@ -4288,9 +4277,6 @@ // add tbl/tbm to multiplayer validation list fs2netd_add_table_validation(filename); - - // close localization - lcl_ext_close(); } int ship_show_velocity_dot = 0; @@ -17662,12 +17648,8 @@ { int rval; - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", filename, rval)); - lcl_ext_close(); return; } @@ -17686,9 +17668,6 @@ // add tbl/tbm to multiplayer validation list fs2netd_add_table_validation(filename); - - // close localization - lcl_ext_close(); } void armor_init() Index: code/species_defs/species_defs.cpp =================================================================== --- code/species_defs/species_defs.cpp (revision 10869) +++ code/species_defs/species_defs.cpp (working copy) @@ -156,13 +156,9 @@ int i, rval; char species_name[NAME_LENGTH]; - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", (filename) ? filename : NOX("<default species_defs.tbl>"), rval)); - lcl_ext_close(); return; } @@ -354,9 +350,6 @@ // add tbl/tbm to multiplayer validation list extern void fs2netd_add_table_validation(const char *tblname); fs2netd_add_table_validation(filename); - - // close localization - lcl_ext_close(); } int Species_initted = 0; Index: code/stats/medals.cpp =================================================================== --- code/stats/medals.cpp (revision 10869) +++ code/stats/medals.cpp (working copy) @@ -246,12 +246,8 @@ { int rval, i; - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", "medals.tbl", rval)); - lcl_ext_close(); return; } @@ -429,9 +425,6 @@ if (Medals[i].kills_needed > 0) prev_badge_kills = Medals[i].kills_needed; } - - // close localization - lcl_ext_close(); } // replacement for -gimmemedals Index: code/stats/scoring.cpp =================================================================== --- code/stats/scoring.cpp (revision 10869) +++ code/stats/scoring.cpp (working copy) @@ -61,12 +61,8 @@ char buf[MULTITEXT_LENGTH]; int rval, idx, persona; - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", "rank.tbl", rval)); - lcl_ext_close(); return; } @@ -118,9 +114,6 @@ Int3(); } #endif - - // close localization - lcl_ext_close(); } // initialize a nice blank scoring element Index: code/weapon/weapons.cpp =================================================================== --- code/weapon/weapons.cpp (revision 10869) +++ code/weapon/weapons.cpp (working copy) @@ -324,12 +324,8 @@ uint i; lod_checker lod_check; - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", filename, rval)); - lcl_ext_close(); return; } @@ -368,9 +364,6 @@ } } required_string("#End"); - - // close localization - lcl_ext_close(); } /** @@ -2781,13 +2774,9 @@ { int rval; - // open localization - lcl_ext_open(); - if ((rval = setjmp(parse_abort)) != 0) { mprintf(("TABLES: Unable to parse '%s'! Error code = %i.\n", filename, rval)); - lcl_ext_close(); return; } @@ -2860,9 +2849,6 @@ // add tbl/tbm to multiplayer validation list fs2netd_add_table_validation(filename); - - // close localization - lcl_ext_close(); } //uses a simple bucket sort to sort weapons, order of importance is: |
|
Splendid. I gave the patch a look and didn't see anything obviously out of place with it, although it would take some time to dig through the changes to localize.[h|cpp]. But if you've tested it with both English and German, I'm satisfied. Patch committed. |
|
I noticed two things in the original patch that should be fixed: 1) In the event that an entry index in tstrings.tbl is 4500 or greater, a fatal error is thrown. This did not happen in prior revisions (which would throw a warning instead), nor did it happen in retail. 2) If index of an XSTR structure is 4500 or greater, there is nothing preventing an out-of-bounds read of the Lcl_ext_str array. The attached patch (mantis3049_2.patch) fixes both of these issues. |
|
Is it not desirable to add a Warning() in the second case? Since it sounds like an XSTR() with an ID >= LCL_MAX_STRINGS would be caused by bad user data, we might want to let the user know it happened... |
|
mantis3049_2.patch (1,961 bytes)
Index: code/localization/localize.cpp =================================================================== --- code/localization/localize.cpp (revision 11007) +++ code/localization/localize.cpp (working copy) @@ -267,7 +267,8 @@ } if (external && (index < 0 || index >= LCL_MAX_STRINGS)) { - Error(LOCATION, "Invalid tstrings table index specified (%i). Please increment LCL_MAX_STRINGS in localize.cpp.", index); + error_display(0, "Invalid tstrings table index specified (%i). Please increment LCL_MAX_STRINGS in localize.cpp.", index); + return; } else if (!external && (index < 0 || index >= XSTR_SIZE)) { Error(LOCATION, "Invalid strings table index specified (%i)", index); } @@ -711,7 +712,7 @@ } // get the string if it exists - if (Lcl_ext_str[str_id] != NULL) { + if ((str_id < LCL_MAX_STRINGS) && (Lcl_ext_str[str_id] != NULL)) { // copy to the outgoing string if ( strlen(Lcl_ext_str[str_id]) > max_len ) error_display(0, "Token too long: [%s]. Length = %i. Max is %i.\n", Lcl_ext_str[str_id], strlen(Lcl_ext_str[str_id]), max_len); @@ -723,6 +724,9 @@ if ( strlen(text_str) > max_len ) error_display(0, "Token too long: [%s]. Length = %i. Max is %i.\n", text_str, strlen(text_str), max_len); + if (str_id >= LCL_MAX_STRINGS) + error_display(0, "Invalid XSTR ID: [%d]. (Must be less than %d.)\n", str_id, LCL_MAX_STRINGS); + strncpy(out, text_str, max_len); } @@ -792,13 +796,16 @@ return; } - // attempt to find the string - if (Lcl_ext_str[str_id] != NULL) { + // get the string if it exists + if ((str_id < LCL_MAX_STRINGS) && (Lcl_ext_str[str_id] != NULL)) { // copy to the outgoing string out = Lcl_ext_str[str_id]; } // otherwise use what we have - probably should Int3() or assert here else { + if (str_id >= LCL_MAX_STRINGS) + error_display(0, "Invalid XSTR ID: [%d]. (Must be less than %d.)\n", str_id, LCL_MAX_STRINGS); + out = text_str; } |
|
I updated the patch to add the warning. |
|
Fix committed to trunk@11008. |
fs2open: trunk r10894 2014-07-07 00:56 Ported: N/A Details Diff |
Yarn's patch for Mantis 0003049 (character remapping in tstrings.tbl) |
Affected Issues 0003049 |
|
mod - /trunk/fs2_open/code/species_defs/species_defs.cpp | Diff File | ||
mod - /trunk/fs2_open/code/lab/wmcgui.cpp | Diff File | ||
mod - /trunk/fs2_open/code/localization/localize.cpp | Diff File | ||
mod - /trunk/fs2_open/code/hud/hudparse.cpp | Diff File | ||
mod - /trunk/fs2_open/code/gamesnd/gamesnd.cpp | Diff File | ||
mod - /trunk/fs2_open/code/mission/missionparse.cpp | Diff File | ||
mod - /trunk/fs2_open/code/menuui/snazzyui.cpp | Diff File | ||
mod - /trunk/fs2_open/code/menuui/credits.cpp | Diff File | ||
mod - /trunk/fs2_open/code/localization/localize.h | Diff File | ||
mod - /trunk/fs2_open/code/gamesnd/eventmusic.cpp | Diff File | ||
mod - /trunk/fs2_open/code/autopilot/autopilot.cpp | Diff File | ||
mod - /trunk/fs2_open/code/mission/missioncampaign.cpp | Diff File | ||
mod - /trunk/fs2_open/code/stats/scoring.cpp | Diff File | ||
mod - /trunk/fs2_open/code/stats/medals.cpp | Diff File | ||
mod - /trunk/fs2_open/code/cutscene/cutscenes.cpp | Diff File | ||
mod - /trunk/fs2_open/code/weapon/weapons.cpp | Diff File | ||
mod - /trunk/fs2_open/code/ship/ship.cpp | Diff File | ||
mod - /trunk/fs2_open/code/mission/missionmessage.cpp | Diff File | ||
mod - /trunk/fs2_open/code/menuui/techmenu.cpp | Diff File | ||
mod - /trunk/fs2_open/code/mod_table/mod_table.cpp | Diff File | ||
mod - /trunk/fs2_open/code/ai/aicode.cpp | Diff File | ||
mod - /trunk/fs2_open/code/gamehelp/contexthelp.cpp | Diff File | ||
mod - /trunk/fs2_open/code/parse/parselo.cpp | Diff File | ||
mod - /trunk/fs2_open/code/missionui/missiondebrief.cpp | Diff File | ||
mod - /trunk/fs2_open/code/menuui/playermenu.cpp | Diff File | ||
mod - /trunk/fs2_open/code/fireball/fireballs.cpp | Diff File | ||
mod - /trunk/fs2_open/code/mission/missionbriefcommon.cpp | Diff File | ||
mod - /trunk/fs2_open/code/asteroid/asteroid.cpp | Diff File | ||
mod - /trunk/fs2_open/code/parse/parselo.h | Diff File | ||
mod - /trunk/fs2_open/code/ai/ai_profiles.cpp | Diff File | ||
fs2open: trunk r11008 2014-08-18 05:02 Ported: N/A Details Diff |
Followup fix for mantis 3049 (from Yarn) Revert an Error back to a Warning Prevent an out-of-bounds read of the Lcl_ext_str array |
Affected Issues 0003049 |
|
mod - /trunk/fs2_open/code/localization/localize.cpp | Diff File |
Date Modified | Username | Field | Change |
---|---|---|---|
2014-05-28 21:45 | Yarn | New Issue | |
2014-05-28 21:45 | Yarn | File Added: char_remap_test.zip | |
2014-05-29 05:45 | Goober5000 | Note Added: 0015770 | |
2014-05-29 05:45 | Goober5000 | Assigned To | => Goober5000 |
2014-05-29 05:45 | Goober5000 | Priority | normal => high |
2014-05-29 05:45 | Goober5000 | Status | new => assigned |
2014-05-29 05:45 | Goober5000 | Target Version | => 3.7.2 |
2014-06-29 23:13 | Goober5000 | Note Added: 0015941 | |
2014-06-29 23:13 | Goober5000 | Assigned To | Goober5000 => Yarn |
2014-07-03 05:55 | Yarn | Note Added: 0016014 | |
2014-07-04 00:32 | Goober5000 | Note Added: 0016017 | |
2014-07-04 05:47 | Yarn | File Added: mantis3049.patch | |
2014-07-04 05:47 | Yarn | Note Added: 0016019 | |
2014-07-04 05:47 | Yarn | Status | assigned => code review |
2014-07-04 06:11 | Yarn | File Deleted: mantis3049.patch | |
2014-07-04 06:11 | Yarn | File Added: mantis3049.patch | |
2014-07-07 04:34 | Goober5000 | Changeset attached | => fs2open trunk r10894 |
2014-07-07 04:48 | Goober5000 | Note Added: 0016031 | |
2014-07-07 04:48 | Goober5000 | Status | code review => resolved |
2014-07-07 04:48 | Goober5000 | Resolution | open => fixed |
2014-08-17 05:06 | Yarn | Note Added: 0016229 | |
2014-08-17 05:06 | Yarn | Status | resolved => feedback |
2014-08-17 05:06 | Yarn | Resolution | fixed => reopened |
2014-08-17 05:06 | Yarn | File Added: mantis3049_2.patch | |
2014-08-17 05:06 | Yarn | Status | feedback => code review |
2014-08-17 07:29 | MageKing17 | Note Added: 0016230 | |
2014-08-17 16:39 | Yarn | File Deleted: mantis3049_2.patch | |
2014-08-17 16:39 | Yarn | File Added: mantis3049_2.patch | |
2014-08-17 16:39 | Yarn | Note Added: 0016234 | |
2014-08-18 08:32 | niffiwan | Changeset attached | => fs2open trunk r11008 |
2014-08-18 08:32 | niffiwan | Note Added: 0016235 | |
2014-08-18 08:32 | niffiwan | Status | code review => resolved |