View Issue Details

IDProjectCategoryView StatusLast Update
0003049FSSCPtablespublic2014-08-18 08:32
ReporterYarn Assigned ToYarn  
PriorityhighSeverityminorReproducibilityalways
Status resolvedResolutionreopened 
Platformx64OSWindows 7 
Product Version3.7.2 
Target Version3.7.2 
Summary0003049: Characters in tstrings.tbl not being remapped properly
DescriptionWhen FreeSpace 2 loads most mission, campaign, and table files, it does so using either read_file_text() or read_file_text_from_array() (located in parselo.cpp), both of which call process_raw_file_text(). This latter function performs a few tasks, but the one pertaining to this report is character remapping. Most of this work is accomplished via a call to maybe_convert_foreign_characters(), which remaps characters like ç, ü, and é so that they can be displayed using the German and French fonts. The process_raw_file_text() function itself also converts ß to ss since the former character does not exist in the default fonts.

Tstrings.tbl is the exception to this rule, however. This file is not loaded with read_file_text() or read_file_text_from_array(), so the normal character remapping does not occur. Instead, the game relies on lcl_fix_umlauts() (which is used only with tstrings.tbl) to do this job, but only if the game language is set to German. However, this function remaps only the umlaut characters (specifically, Ä, Ö, Ü, ä, ö, and ü). It also converts ß in a way that isn't compatible with the default fonts. Thus, non-ASCII characters that aren't also umlaut characters aren't displayed, even if the appropriate font is used.

This issue also exists in retail, as does the lcl_fix_umlauts() function. Based on the code comments, it's obvious that the Volition programmers also knew about this issue but couldn't figure out how to fix it properly.
Steps To Reproduce1. Change the in-game language to German.
2. Install and play the attached mod, which includes the German font. It also includes two table files that have certain strings modified so they begin with "ß äöü áóú".
3. On the pilot selection screen, look at the line of text at the very bottom, which comes from char_remap_test-lcl.tbm. It should begin with "ss äöü áóú". According to my tests, character remapping does work properly here.
4. Go to the cutscenes screen in the Tech Room, select "Introduction," and look at the description. As with step 3, the text should start with "ss äöü áóú". Since this text is loaded from tstrings.tbl, it will look like " äöü " until this issue is fixed.
TagsNo tags attached.

Activities

Yarn

2014-05-28 21:45

developer  

char_remap_test.zip (294,322 bytes)

Goober5000

2014-05-29 05:45

administrator   ~0015770

This should be fixed in 3.7.2 because we have translation support in FSPort 3.4.1. I've assigned it to myself, but if someone else is able to fix it first, please take it and do so.

Goober5000

2014-06-29 23:13

administrator   ~0015941

Reassigning to Yarn, in case Yarn has time to look into this. Probably more time than me, at least. ;)

Yarn

2014-07-03 05:55

developer   ~0016014

I finally got a good look at this.

As mentioned in this report, most tables (including strings.tbl) are loaded using read_file_text() or read_file_text_from_array(). Once this is done, though, all relevant information in the table (including strings) is loaded into memory. With tstrings.tbl, however, the translated strings are not initially loaded into memory; instead, FSO maintains an array of pointers into the actual file, reading the strings from the file when necessary (for example, when loading a mission).

It looks like fixing this means treating tstrings.tbl like any other table and loading all of the strings of a language section into memory. Because the table is only about a megabyte (and only half of that is used for German), and considering that PCs in use today generally have far more RAM than those in 1999, I don't think that memory usage is a big concern here.

Goober5000

2014-07-04 00:32

administrator   ~0016017

Interesting. Sounds like a good plan.

Yarn

2014-07-04 05:47

developer   ~0016019

All right, I got the patch done. Everything in parse_stringstbl() was moved to parse_stringstbl_common() and then modified to support tstrings.tbl as well. A boolean switch is used to set how the function loads the files. Lots of now-obsolete stuff was also removed, including the open/close functions for tstrings.tbl (which is why lots of files were changed).

The change also made it easy to add modular support to tstrings.tbl, so I did so. The modular tstrings.tbl suffix is *-tlc.tbm.

Yarn

2014-07-04 06:11

developer  

mantis3049.patch (50,310 bytes)   
Index: code/ai/ai_profiles.cpp
===================================================================
--- code/ai/ai_profiles.cpp	(revision 10869)
+++ code/ai/ai_profiles.cpp	(working copy)
@@ -75,13 +75,9 @@
 	char *saved_Mp = NULL;
 	char buf[NAME_LENGTH];
 
-	// open localization
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0)
 	{
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", (filename) ? filename : "<default ai_profiles.tbl>", rval));
-		lcl_ext_close();
 		return;
 	}
 
@@ -514,9 +510,6 @@
 	// add tbl/tbm to multiplayer validation list
 	extern void fs2netd_add_table_validation(const char *tblname);
 	fs2netd_add_table_validation(filename);
-
-	// close localization
-	lcl_ext_close();
 }
 
 void ai_profiles_init()
Index: code/ai/aicode.cpp
===================================================================
--- code/ai/aicode.cpp	(revision 10869)
+++ code/ai/aicode.cpp	(working copy)
@@ -781,9 +781,6 @@
 #define AI_CLASS_INCREMENT		10
 void parse_aitbl()
 {
-	// open localization
-	lcl_ext_open();
-
 	read_file_text("ai.tbl", CF_TYPE_TABLES);
 	reset_parse();
 
@@ -815,9 +812,6 @@
 			reset_ai_class_names();
 		}
 	}
-
-	// close localization
-	lcl_ext_close();
 	
 	atexit(free_ai_stuff);
 }
@@ -837,7 +831,6 @@
 
 		if ((rval = setjmp(parse_abort)) != 0) {
 			mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", "ai.tbl", rval));
-			lcl_ext_close();
 		} else {			
 			parse_aitbl();			
 		}
Index: code/asteroid/asteroid.cpp
===================================================================
--- code/asteroid/asteroid.cpp	(revision 10869)
+++ code/asteroid/asteroid.cpp	(working copy)
@@ -1896,12 +1896,8 @@
 		"are no species for them to belong to."
 		);
 
-	// open localization
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0) {
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", "asteroid.tbl", rval));
-		lcl_ext_close();
 		return;
 	}
 
@@ -2030,9 +2026,6 @@
 	} else {
 		Asteroid_icon_closeup_zoom = 0.5f;	// magic number from retail
 	}
-
-	// close localization
-	lcl_ext_close();
 }
 
 /**
Index: code/autopilot/autopilot.cpp
===================================================================
--- code/autopilot/autopilot.cpp	(revision 10869)
+++ code/autopilot/autopilot.cpp	(working copy)
@@ -1274,13 +1274,9 @@
 	int rval;
 	SCP_vector<SCP_string> lines;
 
-	// open localization
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0)
 	{
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", (filename) ? filename : "<default autopilot.tbl>", rval));
-		lcl_ext_close();
 		return;
 	}
 
@@ -1337,9 +1333,6 @@
 
 
 	required_string("#END");
-
-	// close localization
-	lcl_ext_close();
 }
 
 
Index: code/cutscene/cutscenes.cpp
===================================================================
--- code/cutscene/cutscenes.cpp	(revision 10869)
+++ code/cutscene/cutscenes.cpp	(working copy)
@@ -55,12 +55,8 @@
 	int rval;
     cutscene_info cutinfo;
 
-	// open localization
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0) {
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", "cutscenes.tbl", rval));
-		lcl_ext_close();
 		return;
 	}
 
@@ -107,9 +103,6 @@
 	}
 
 	required_string("#End");
-
-	// close localization
-	lcl_ext_close();
 }
 
 // function to return 0 based index of which CD a particular movie is on
Index: code/fireball/fireballs.cpp
===================================================================
--- code/fireball/fireballs.cpp	(revision 10869)
+++ code/fireball/fireballs.cpp	(working copy)
@@ -163,12 +163,8 @@
 	lod_checker lod_check;
 	color fb_color;
 
-	// open localization
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0) {
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", filename, rval));
-		lcl_ext_close();
 		return;
 	}
 
@@ -241,9 +237,6 @@
 	}
 
 	required_string("#End");
-
-	// close localization
-	lcl_ext_close();
 }
 
 void fireball_parse_tbl()
Index: code/gamehelp/contexthelp.cpp
===================================================================
--- code/gamehelp/contexthelp.cpp	(revision 10869)
+++ code/gamehelp/contexthelp.cpp	(working copy)
@@ -334,13 +334,9 @@
 	SCP_vector<help_left_bracket> lbracket_temp;
 	help_left_bracket lbracket_temp2;
 	vec3d vec3d_temp;
-
-	// open localization
-	lcl_ext_open();
 	
 	if ((rval = setjmp(parse_abort)) != 0) {
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", filename, rval));
-		lcl_ext_close();
 		return;
 	} 
 
@@ -488,9 +484,6 @@
 
 		}		// end while
 	}		// end while
-
-	// close localization
-	lcl_ext_close();
 }
 
 
Index: code/gamesnd/eventmusic.cpp
===================================================================
--- code/gamesnd/eventmusic.cpp	(revision 10869)
+++ code/gamesnd/eventmusic.cpp	(working copy)
@@ -1380,12 +1380,8 @@
 
 	if ((rval = setjmp(parse_abort)) != 0) {
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", filename, rval));
-		lcl_ext_close();
 
 	} else {
-		// open localization
-		lcl_ext_open();
-
 		read_file_text(filename, CF_TYPE_TABLES);
 		reset_parse();
 
@@ -1404,9 +1400,6 @@
 				}
 			}
 		}
-
-		// close localization
-		lcl_ext_close();
 	}
 }
 
Index: code/gamesnd/gamesnd.cpp
===================================================================
--- code/gamesnd/gamesnd.cpp	(revision 10869)
+++ code/gamesnd/gamesnd.cpp	(working copy)
@@ -777,16 +777,10 @@
  */
 void gamesnd_parse_soundstbl()
 {
-	// open localization
-	lcl_ext_open();
-
 	parse_sound_table("sounds.tbl");
 
 	parse_modular_table("*-snd.tbm", parse_sound_table);
 
-	// close localization
-	lcl_ext_close();
-
 	// if we are missing any species then report 
 	if (missingFlybySounds.size() > 0)
 	{
Index: code/hud/hudparse.cpp
===================================================================
--- code/hud/hudparse.cpp	(revision 10869)
+++ code/hud/hudparse.cpp	(working copy)
@@ -174,12 +174,8 @@
 	color *ship_clr_p = NULL;
 	bool scale_gauge = true;
 
-	// open localization
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0) {
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", filename, rval));
-		lcl_ext_close();
 		return;
 	}
 
@@ -455,9 +451,6 @@
 		required_string("$End Gauges");
 		required_string("#End");
 	}
-
-	// close localization
-	lcl_ext_close();
 }
 
 void hud_positions_init()
Index: code/lab/wmcgui.cpp
===================================================================
--- code/lab/wmcgui.cpp	(revision 10869)
+++ code/lab/wmcgui.cpp	(working copy)
@@ -149,12 +149,8 @@
 		DestroyClassInfo();
 	}
 
-	// open localization
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0) {
 		mprintf(("WMCGUI: Unable to parse '%s'!  Error code = %i.\n", filename, rval));
-		lcl_ext_close();
 		return;
 	}
 
@@ -171,9 +167,6 @@
 		}
 	} while(flag);
 
-	// close localization
-	lcl_ext_close();
-
 	ClassInfoParsed = true;
 }
 
Index: code/localization/localize.cpp
===================================================================
--- code/localization/localize.cpp	(revision 10869)
+++ code/localization/localize.cpp	(working copy)
@@ -75,45 +75,16 @@
 
 
 // table/mission externalization stuff --------------------
-
-#define TABLE_STRING_FILENAME						"tstrings.tbl"
-// filename of the file to use when localizing table strings
-char *Lcl_ext_filename = NULL;
-CFILE *Lcl_ext_file = NULL;
-
-// for scanning/parsing tstrings.tbl (from ExStr)
 #define PARSE_TEXT_BUF_SIZE			PARSE_BUF_SIZE
 #define PARSE_ID_BUF_SIZE			5
-#define TS_SCANNING						0				// scanning for a line of text
-#define TS_ID_STRING						1				// reading in an id string
-#define TS_OPEN_QUOTE					2				// looking for an open quote
-#define TS_STRING							3				// reading in the text string itself
-int Ts_current_state = 0;
-char Ts_text[PARSE_TEXT_BUF_SIZE];				// string we're currently working with
-char Ts_id_text[PARSE_ID_BUF_SIZE];				// id string we're currently working with
-size_t Ts_text_size;
-size_t Ts_id_text_size;
+#define LCL_MAX_STRINGS					4500
+char *Lcl_ext_str[LCL_MAX_STRINGS];
 
-// file pointers for optimized string lookups
-// some example times for FreeSpace2 startup with granularities (mostly .tbl files, ~500 strings in the table file, many looked up more than once)
-// granularity 20			:		13 secs
-// granularity 10			:		11 secs
-// granularity 5			:		9 secs
-// granularity 2			:		7-8 secs
-#define LCL_GRANULARITY					1				// how many strings between each pointer (lower granularities should give faster lookup times)
-#define LCL_MAX_POINTERS				4500			// max # of pointers
-#define LCL_MAX_STRINGS					(LCL_GRANULARITY * LCL_MAX_POINTERS)
-int Lcl_pointers[LCL_MAX_POINTERS];
-int Lcl_pointer_count = 0;
 
-
 // ------------------------------------------------------------------------------------------------------------
 // LOCALIZE FORWARD DECLARATIONS
 //
 
-// associate table file externalization with the specified input file
-void lcl_ext_associate(const char *filename);
-
 // given a valid XSTR() tag piece of text, extract the string portion, return it in out, nonzero on success
 int lcl_ext_get_text(const char *xstr, char *out);
 int lcl_ext_get_text(const SCP_string &xstr, SCP_string &out);
@@ -122,23 +93,9 @@
 int lcl_ext_get_id(const char *xstr, int *out);
 int lcl_ext_get_id(const SCP_string &xstr, int *out);
 
-// given a valid XSTR() id#, lookup the string in tstrings.tbl, filling in out if found, nonzero on success
-int lcl_ext_lookup(char *out, int id);
-
 // if the char is a valid char for a signed integer value string
 int lcl_is_valid_numeric_char(char c);
 
-// sub-parse function for individual lines of tstrings.tbl (from Exstr)
-// returns : integer with the low bits having the following values :
-// 0 on fail, 1 on success, 2 if found a matching id/string pair, 3 if end of language has been found
-// for cases 1 and 2 : the high bit (1<<31) will be set if the parser detected the beginning of a new string id on this line
-// so be sure to mask this value out to get the low portion of the return value
-//
-int lcl_ext_lookup_sub(const char *text, char *out, int id);
-
-// initialize the pointer array into tstrings.tbl (call from lcl_ext_open() ONLY)
-void lcl_ext_setup_pointers();
-
 // parses the string.tbl and reports back only on the languages it found
 void parse_stringstbl_quick(const char *filename);
 
@@ -211,26 +168,10 @@
 		lang = lang_init;
 	}
 
-	// language markers
-	Lcl_pointer_count = 0;
-
-	// associate the table string file
-	lcl_ext_associate(TABLE_STRING_FILENAME);		
-
 	// set the language (this function takes care of setting up file pointers)
-	lcl_set_language(lang);		
+	lcl_set_language(lang);
 }
 
-// added 2.2.99 by NeilK to take care of fs2 launcher memory leaks
-// shutdown localization
-void lcl_close()
-{
-	// if the filename exists, free it up
-	if(Lcl_ext_filename != NULL){
-		vm_free(Lcl_ext_filename);
-	}
-}
-
 // determine what language we're running in, see LCL_* defines above
 int lcl_get_language()
 {
@@ -244,9 +185,6 @@
 	int lang_idx;
 	int i;
 
-	// make sure localization is NOT running
-	lcl_ext_close();
-
 	read_file_text(filename, CF_TYPE_TABLES);
 	reset_parse();
 
@@ -253,9 +191,9 @@
 	if (optional_string("#Supported Languages")) {
 		while (required_string_either("#End","$Language:")) {			
 			required_string("$Language:");
-			stuff_string(language.lang_name, F_NAME, LCL_LANG_NAME_LEN + 1);
+			stuff_string(language.lang_name, F_RAW, LCL_LANG_NAME_LEN + 1);
 			required_string("+Extension:");
-			stuff_string(language.lang_ext, F_NAME, LCL_LANG_NAME_LEN + 1);
+			stuff_string(language.lang_ext, F_RAW, LCL_LANG_NAME_LEN + 1);
 			required_string("+Special Character Index:");
 			stuff_ubyte(&language.special_char_indexes[0]);
 			for (i = 1; i < MAX_FONTS; ++i) {
@@ -288,7 +226,9 @@
 	}
 }
 
-void parse_stringstbl(const char *filename)
+// Unified function for loading strings.tbl and tstrings.tbl (and their modular versions).
+// The "external" parameter controls which format to load: true for tstrings.tbl, false for strings.tbl
+void parse_stringstbl_common(const char *filename, const bool external)
 {
 	char chr, buf[4096];
 	char language_tag[512];
@@ -296,9 +236,6 @@
 	char *p_offset = NULL;
 	int offset_lo = 0, offset_hi = 0;
 
-	// make sure localization is NOT running
-	lcl_ext_close();
-
 	read_file_text(filename, CF_TYPE_TABLES);
 	reset_parse();
 
@@ -305,7 +242,11 @@
 	// move down to the proper section		
 	memset(language_tag, 0, sizeof(language_tag));
 	strcpy_s(language_tag, "#");
-	strcat_s(language_tag, Lcl_languages[Lcl_current_lang].lang_name);
+	if (external && Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE){
+		strcat_s(language_tag, "default");
+	} else {
+		strcat_s(language_tag, Lcl_languages[Lcl_current_lang].lang_name);
+	}
 
 	if ( skip_to_string(language_tag) != 1 ) {
 		mprintf(("Current language not found in %s\n", filename));
@@ -317,44 +258,41 @@
 		int num_offsets_on_this_line = 0;
 
 		stuff_int(&index);
-		stuff_string(buf, F_NAME, sizeof(buf));
+		if (external) {
+			ignore_white_space();
+			get_string(buf, sizeof(buf));
+			drop_trailing_white_space(buf);
+		} else {
+			stuff_string(buf, F_RAW, sizeof(buf));
+		}
 
-		if (index < 0 || index >= XSTR_SIZE) {
+		if (external && (index < 0 || index >= LCL_MAX_STRINGS)) {
+			Error(LOCATION, "Invalid tstrings table index specified (%i). Please increment LCL_MAX_STRINGS in localize.cpp.", index);
+		} else if (!external && (index < 0 || index >= XSTR_SIZE)) {
 			Error(LOCATION, "Invalid strings table index specified (%i)", index);
 		}
 
 		if (Lcl_pl)
 			lcl_fix_polish(buf);
+		
+		if (!external) {
+			i = strlen(buf);
 
-		i = strlen(buf);
+			while (i--) {
+				if ( !isspace(buf[i]) )
+					break;
+			}
 
-		while (i--) {
-			if ( !isspace(buf[i]) )
-				break;
-		}
+			// trim unnecessary end of string
+			if (i >= 0) {
+				// Assert(buf[i] == '"');
+				if (buf[i] != '"') {
+					// probably an offset on this entry
 
-		// trim unneccesary end of string
-		if (i >= 0) {
-			// Assert(buf[i] == '"');
-			if (buf[i] != '"') {
-				// probably an offset on this entry
+					// drop down a null terminator (prolly unnecessary)
+					buf[i+1] = 0;
 
-				// drop down a null terminator (prolly unnecessary)
-				buf[i+1] = 0;
-
-				// back up over the potential offset
-				while ( !is_white_space(buf[i]) )
-					i--;
-
-				// now back up over intervening spaces
-				while ( is_white_space(buf[i]) )
-					i--;
-
-				num_offsets_on_this_line = 1;
-
-				if (buf[i] != '"') {
-					// could have a 2nd offset value (one for 640, one for 1024)
-					// so back up again
+					// back up over the potential offset
 					while ( !is_white_space(buf[i]) )
 						i--;
 
@@ -362,49 +300,80 @@
 					while ( is_white_space(buf[i]) )
 						i--;
 
-					num_offsets_on_this_line = 2;
+					num_offsets_on_this_line = 1;
+
+					if (buf[i] != '"') {
+						// could have a 2nd offset value (one for 640, one for 1024)
+						// so back up again
+						while ( !is_white_space(buf[i]) )
+							i--;
+
+						// now back up over intervening spaces
+						while ( is_white_space(buf[i]) )
+							i--;
+
+						num_offsets_on_this_line = 2;
+					}
+
+					p_offset = &buf[i+1];			// get ptr to string section with offset in it
+
+					if (buf[i] != '"')
+						Error(LOCATION, "%s is corrupt", filename);		// now its an error
 				}
 
-				p_offset = &buf[i+1];			// get ptr to string section with offset in it
-
-				if (buf[i] != '"')
-					Error(LOCATION, "%s is corrupt", filename);		// now its an error
+				buf[i] = 0;
 			}
 
-			buf[i] = 0;
-		}
+			// copy string into buf
+			z = 0;
+			for (i = 1; buf[i]; i++) {
+				chr = buf[i];
 
-		// copy string into buf
-		z = 0;
-		for (i = 1; buf[i]; i++) {
-			chr = buf[i];
+				if (chr == '\\') {
+					chr = buf[++i];
 
-			if (chr == '\\') {
-				chr = buf[++i];
+					if (chr == 'n')
+						chr = '\n';
+					else if (chr == 'r')
+						chr = '\r';
+				}
 
-				if (chr == 'n')
-					chr = '\n';
-				else if (chr == 'r')
-					chr = '\r';
+				buf[z++] = chr;
 			}
 
-			buf[z++] = chr;
+			// null terminator on buf
+			buf[z] = 0;
 		}
 
-		// null terminator on buf
-		buf[z] = 0;
-
-		// write into Xstr_table
-		if ( Parsing_modular_table && (Xstr_table[index].str != NULL) ) {
-			vm_free((void *) Xstr_table[index].str);
-			Xstr_table[index].str = NULL;
+		// write into Xstr_table (for strings.tbl) or Lcl_ext_str (for tstrings.tbl)
+		if (Parsing_modular_table) {
+			if ( external && (Lcl_ext_str[index] != NULL) ) {
+				vm_free((void *) Lcl_ext_str[index]);
+				Lcl_ext_str[index] = NULL;
+			} else if ( !external && (Xstr_table[index].str != NULL) ) {
+				vm_free((void *) Xstr_table[index].str);
+				Xstr_table[index].str = NULL;
+			}
 		}
 
-		if (Xstr_table[index].str != NULL)
+		if (external && (Lcl_ext_str[index] != NULL)) {
+			Warning(LOCATION, "Tstrings table index %d used more than once", index);
+		} else if (!external && (Xstr_table[index].str != NULL)) {
 			Warning(LOCATION, "Strings table index %d used more than once", index);
+		}
 
-		Xstr_table[index].str = vm_strdup(buf);
+		if (external) {
+			Lcl_ext_str[index] = vm_strdup(buf);
+		} else {
+			Xstr_table[index].str = vm_strdup(buf);
+		}
 
+		// the rest of this loop applies only to strings.tbl,
+		// so we can move on to the next line if we're reading from tstrings.tbl
+		if (external) {
+			continue;
+		}
+
 		// read offset information, assume 0 if nonexistant
 		if (p_offset != NULL) {
 			if (sscanf(p_offset, "%d%d", &offset_lo, &offset_hi) < num_offsets_on_this_line) {
@@ -427,14 +396,29 @@
 	}
 }
 
+void parse_stringstbl(const char *filename)
+{
+	parse_stringstbl_common(filename, false);
+}
+
+void parse_tstringstbl(const char *filename)
+{
+	parse_stringstbl_common(filename, true);
+}
+
 // initialize the xstr table
 void lcl_xstr_init()
 {
 	int i, rval;
 
+
 	for (i = 0; i < XSTR_SIZE; i++)
 		Xstr_table[i].str = NULL;
 
+	for (i = 0; i < LCL_MAX_STRINGS; i++)
+		Lcl_ext_str[i] = NULL;
+
+
 	if ( (rval = setjmp(parse_abort)) != 0 )
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", "strings.tbl", rval));
 	else
@@ -442,6 +426,15 @@
 
 	parse_modular_table(NOX("*-lcl.tbm"), parse_stringstbl);
 
+
+	if ( (rval = setjmp(parse_abort)) != 0 )
+		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", "tstrings.tbl", rval));
+	else
+		parse_tstringstbl("tstrings.tbl");
+
+	parse_modular_table(NOX("*-tlc.tbm"), parse_tstringstbl);
+
+
 	Xstr_inited = 1;
 }
 
@@ -455,7 +448,13 @@
 			Xstr_table[i].str = NULL;
 		}
 	}
-	vm_free(Lcl_ext_filename);
+
+	for (int i=0; i<LCL_MAX_STRINGS; i++){
+		if (Lcl_ext_str[i] != NULL) {
+			vm_free((void *) Lcl_ext_str[i]);
+			Lcl_ext_str[i] = NULL;
+		}
+	}
 }
 
 
@@ -483,14 +482,6 @@
 	} else if (!strcmp(Lcl_languages[Lcl_current_lang].lang_name, Lcl_builtin_languages[LCL_POLISH].lang_name)) {
 		Lcl_pl = 1;
 	}
-
-	// set to 0, so lcl_ext_open() knows to reset file pointers
-	Lcl_pointer_count = 0;
-
-	// reset file pointers to the proper language-section
-	if(Lcl_current_lang != FS2_OPEN_DEFAULT_LANGUAGE){
-		lcl_ext_setup_pointers();
-	}
 }
 
 ubyte lcl_get_font_index(int font_num)
@@ -571,42 +562,6 @@
 
 // externalization of table/mission files ----------------------- 
 
-// open the externalization file for use during parsing (call before parsing a given file)
-void lcl_ext_open()
-{
-	// if the file is already open, do nothing
-	Assert(Lcl_ext_file == NULL);	
-
-	// if we're running in the default language, do nothing
-	if(Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE){
-		return;
-	}
-
-	// otherwise open the file
-	Lcl_ext_file = cfopen(Lcl_ext_filename, "rt");
-	if(Lcl_ext_file == NULL){
-		return;
-	}		
-}
-
-// close the externalization file (call after parsing a given file)
-void lcl_ext_close()
-{
-	// if the file is not open, do nothing
-	if(Lcl_ext_file == NULL){
-		return;
-	}
-
-	// if we're running in the default language, do nothing
-	if(Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE){
-		return;
-	}
-		
-	// otherwise close it
-	cfclose(Lcl_ext_file);
-	Lcl_ext_file = NULL;
-}
-
 void lcl_replace_stuff(char *text, size_t max_len)
 {
 	if (Fred_running)
@@ -681,7 +636,6 @@
 void lcl_ext_localize_sub(const char *in, char *out, size_t max_len, int *id)
 {
 	char text_str[PARSE_BUF_SIZE]="";
-	char lookup_str[PARSE_BUF_SIZE]="";
 	int str_id;
 	size_t str_len;
 
@@ -747,7 +701,7 @@
 	}
 	
 	// if the localization file is not open, or we're running in the default language, return the original string
-	if ( (Lcl_ext_file == NULL) || (str_id < 0) || (Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE) ) {
+	if ( !Xstr_inited || (str_id < 0) || (Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE) ) {
 		if ( strlen(text_str) > max_len )
 			error_display(0, "Token too long: [%s].  Length = %i.  Max is %i.\n", text_str, strlen(text_str), max_len);
 
@@ -759,13 +713,13 @@
 		return;
 	}
 
-	// attempt to find the string
-	if (lcl_ext_lookup(lookup_str, str_id)) {
+	// get the string if it exists
+	if (Lcl_ext_str[str_id] != NULL) {
 		// copy to the outgoing string
-		if ( strlen(lookup_str) > max_len )
-			error_display(0, "Token too long: [%s].  Length = %i.  Max is %i.\n", lookup_str, strlen(lookup_str), max_len);
+		if ( strlen(Lcl_ext_str[str_id]) > max_len )
+			error_display(0, "Token too long: [%s].  Length = %i.  Max is %i.\n", Lcl_ext_str[str_id], strlen(Lcl_ext_str[str_id]), max_len);
 
-		strncpy(out, lookup_str, max_len);
+		strncpy(out, Lcl_ext_str[str_id], max_len);
 	}
 	// otherwise use what we have - probably should Int3() or assert here
 	else {
@@ -785,7 +739,6 @@
 void lcl_ext_localize_sub(const SCP_string &in, SCP_string &out, int *id)
 {
 	SCP_string text_str = "";
-	char lookup_str[PARSE_BUF_SIZE]="";
 	int str_id;
 
 	// default (non-external string) value
@@ -833,7 +786,7 @@
 	}
 	
 	// if the localization file is not open, or we're running in the default language, return the original string
-	if ( (Lcl_ext_file == NULL) || (str_id < 0) || (Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE) ) {
+	if ( !Xstr_inited || (str_id < 0) || (Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE) ) {
 		out = text_str;
 
 		if (id != NULL)
@@ -843,9 +796,9 @@
 	}
 
 	// attempt to find the string
-	if (lcl_ext_lookup(lookup_str, str_id)) {
+	if (Lcl_ext_str[str_id] != NULL) {
 		// copy to the outgoing string
-		out = lookup_str;
+		out = Lcl_ext_str[str_id];
 	}
 	// otherwise use what we have - probably should Int3() or assert here
 	else {
@@ -916,18 +869,6 @@
 // LOCALIZE FORWARD DEFINITIONS
 //
 
-// associate table file externalization with the specified input file
-void lcl_ext_associate(const char *filename)
-{
-	// if the filename already exists, free it up
-	if(Lcl_ext_filename != NULL){
-		vm_free(Lcl_ext_filename);
-	}
-
-	// set the new filename
-	Lcl_ext_filename = vm_strdup(filename);
-}
-
 // given a valid XSTR() tag piece of text, extract the string portion, return it in out, nonzero on success
 int lcl_ext_get_text(const char *xstr, char *out)
 {
@@ -1162,192 +1103,6 @@
 	return 1;
 }
 
-// given a valid XSTR() id#, lookup the string in tstrings.tbl, filling in out if found, nonzero on success
-int lcl_ext_lookup(char *out, int id)
-{
-	char text[1024];
-	int ret;
-	int pointer;
-	
-	Assert(Lcl_pointer_count >= 0);
-	Assert(Lcl_pointers[0] >= 0);
-	Assert(Lcl_pointers[Lcl_pointer_count - 1] >= 0);
-	Assert(Lcl_ext_file != NULL);
-	Assert(id >= 0);
-
-	// seek to the closest pointer <= the id# we're looking for
-	pointer = id / LCL_GRANULARITY;
-	cfseek(Lcl_ext_file, Lcl_pointers[pointer], CF_SEEK_SET);
-
-	// reset parsing vars and go to town
-	Ts_current_state = TS_SCANNING;
-	Ts_id_text_size = 0;
-//	Ts_text_size;
-	memset(Ts_text, 0, PARSE_TEXT_BUF_SIZE);
-	memset(Ts_id_text, 0, PARSE_ID_BUF_SIZE);
-	while((cftell(Lcl_ext_file) < Lcl_pointers[Lcl_pointer_count - 1]) && cfgets(text, 1024, Lcl_ext_file)){
-		ret = lcl_ext_lookup_sub(text, out, id);
-			
-		// run the line parse function		
-		switch(ret & 0x0fffffff){
-		// error
-		case 0 :
-			Int3();			// should never get here - it means the string doens't exist in the table!!
-			return 0;
-
-		// success parsing the line - please continue
-		case 1 :
-			break;
-
-		// found a matching string/id pair
-		case 2 :			
-			// success
-			if (Lcl_gr) {
-				// this is because tstrings.tbl reads in as ANSI for some reason
-				// opening tstrings with "rb" mode didnt seem to help, so its now still "rt" like before
-				lcl_fix_umlauts(out, LCL_TO_ASCII);
-			}
-			return 1;
-
-		// end of language found
-		case 3 :
-			Int3();			// should never get here - it means the string doens't exist in the table!!
-			return 0;		
-		}
-	}
-	
-	Int3();			// should never get here - it means the string doens't exist in the table!!
-	return 0;
-}
-
-// sub-parse function for individual lines of tstrings.tbl (from Exstr)
-// returns : integer with the low bits having the following values :
-// 0 on fail, 1 on success, 2 if found a matching id/string pair, 3 if end of language has been found
-// for cases 1 and 2 : the high bit (1<<31) will be set if the parser detected the beginning of a new string id on this line
-//
-int lcl_ext_lookup_sub(const char *text, char *out, int id)
-{
-	const char *p;					// current ptr
-	int len = strlen(text);
-	int count;	
-	char text_copy[1024];	
-	char *tok;
-	int found_new_string_id = 0;
-
-	p = text;
-	count = 0;
-	while(count < len){
-		// do something useful
-		switch(Ts_current_state){		
-		// scanning for a line of text
-		case TS_SCANNING:
-			// if the first word is #end, we're done with the file altogether
-			strcpy_s(text_copy, text);
-			tok = strtok(text_copy, " \n");
-			if((tok != NULL) && !stricmp(tok, "#end")){
-				return 3;
-			}
-			// if its a commented line, skip it
-			else if((text[0] == ';') || (text[0] == ' ') || (text[0] == '\n')){
-				return 1;
-			}
-			// otherwise we should have an ID #, so stuff it and move to the proper state
-			else {
-				if(lcl_is_valid_numeric_char(*p)){
-					memset(Ts_id_text, 0, PARSE_ID_BUF_SIZE);
-					Ts_id_text_size = 0;
-					Ts_id_text[Ts_id_text_size++] = *p;
-					Ts_current_state = TS_ID_STRING;
-
-					found_new_string_id = 1;
-				}
-				// error
-				else {
-					Int3();
-					return 0;
-				}
-			}
-			break;
-
-		// scanning in an id string
-		case TS_ID_STRING:
-			// if we have another valid char
-			if(lcl_is_valid_numeric_char(*p)) {
-				if (Ts_id_text_size >= PARSE_ID_BUF_SIZE - 1) {
-					error_display(0, "XSTR id %s too long!\n", Ts_id_text);
-					return 0;
-				}
-				Ts_id_text[Ts_id_text_size++] = *p;
-			}
-			// if we found a comma, our id# is finished, look for the open quote
-			else if(*p == ','){
-				Ts_current_state = TS_OPEN_QUOTE;
-			} else {
-				Int3();
-				return 0;
-			}
-			break;
-
-		case TS_OPEN_QUOTE:
-			// valid space or an open quote
-			if((*p == ' ') || (*p == '\"')){
-				if(*p == '\"'){
-					memset(Ts_text, 0, PARSE_TEXT_BUF_SIZE);
-					Ts_text_size = 0;
-					Ts_current_state = TS_STRING;
-				}
-			} else {
-				Int3();
-				return 0;
-			}
-			break;
-
-		case TS_STRING:
-			// if we have an end quote, we need to look for a comma
-			if((*p == '\"') /*&& (Ts_text_size > 0)*/ && (Ts_text[Ts_text_size - 1] != '\\')){
-				// we're now done - we have a string
-				Ts_current_state = TS_SCANNING;
-
-				// if the id#'s match, copy the string and return "string found"
-				if((atoi(Ts_id_text) == id) && (out != NULL)){
-					// this is redundant to the PARSE_TEXT_BUF_SIZE, but let's be future proof
-					if (strlen(Ts_text) > PARSE_BUF_SIZE - 1) {
-						error_display(0, "XSTR text result exceeds output buffer size!\n\n%s\n", Ts_text);
-						return 0;
-					}
-					strcpy(out, Ts_text);
-
-					return found_new_string_id ? (1<<1) | (1<<31) : (1<<1);					
-				}
-				
-				// otherwise, just continue parsing				
-				return found_new_string_id ? (1<<0) | (1<<31) : (1<<0);
-			} 
-			// otherwise add to the string
-			else {
-				if (Ts_text_size >= PARSE_TEXT_BUF_SIZE - 1) {
-					error_display(0, "XSTR text too long!\n\n%s\n", Ts_text);
-					return 0;
-				}
-				Ts_text[Ts_text_size++] = *p;
-			}
-			break;
-		}		
-
-		// if we have a newline, return success, we're done with this line
-		if(*p == '\n'){
-			return found_new_string_id ? (1<<0) | (1<<31) : (1<<0);
-		}
-
-		// next char in the line
-		p++;
-		count++;
-	}	
-
-	// success
-	return found_new_string_id ? (1<<0) | (1<<31) : (1<<0);
-}
-
 // if the char is a valid char for a signed integer value
 int lcl_is_valid_numeric_char(char c)
 {
@@ -1355,98 +1110,6 @@
 				(c == '5') || (c == '6') || (c == '7') || (c == '8') || (c == '9') ) ? 1 : 0;
 }
 
-// initialize the pointer array into tstrings.tbl (call from lcl_ext_open() ONLY)
-void lcl_ext_setup_pointers()
-{
-	char language_string[128];
-	char line[1024];
-	char *tok;	
-	int string_count;
-	int ret;
-	int found_start = 0;
-
-	// open the localization file
-	lcl_ext_open();
-	if(Lcl_ext_file == NULL){
-		error_display(0, "Error opening externalization file! File likely does not exist or could not be found\n");
-		return;
-	}
-
-	// seek to the currently active language
-	memset(language_string, 0, 128);
-	strcpy_s(language_string, "#");
-	if(Lcl_current_lang == FS2_OPEN_DEFAULT_LANGUAGE){
-		strcat_s(language_string, "default");
-	} else {
-		strcat_s(language_string, Lcl_languages[Lcl_current_lang].lang_name);
-	}
-	memset(line, 0, 1024);
-
-	// reset seek variables and begin		
-	Lcl_pointer_count = 0;
-	while(cfgets(line, 1024, Lcl_ext_file)){
-		tok = strtok(line, " \n");
-		if(tok == NULL){
-			continue;			
-		}
-		
-		// if the language matches, we're good to start parsing strings
-		if(!stricmp(language_string, tok)){
-			found_start = 1;			
-			break;
-		}		
-	}
-
-	// if we didn't find the language specified, error
-	if(found_start <= 0){
-		error_display(0, "Could not find specified language in tstrings.tbl!\n");
-		lcl_ext_close();
-		return;
-	}
-
-	string_count = 0;	
-	while(cfgets(line, 1024, Lcl_ext_file)){
-		ret = lcl_ext_lookup_sub(line, NULL, -1);
-
-		// do stuff
-		switch(ret & 0x0fffffff){
-		// error
-		case 0 :
-			lcl_ext_close();
-			return;		
-
-		// end of language found
-		case 3 :
-			// mark one final pointer
-			Lcl_pointers[Lcl_pointer_count++] = cftell(Lcl_ext_file) - strlen(line) - 1;
-			lcl_ext_close();
-			return;
-		}
-
-		// the only other case we care about is the beginning of a new id#
-		if(ret & (1<<31)){		
-			if((string_count % LCL_GRANULARITY) == 0){
-				// mark the pointer down
-				Lcl_pointers[Lcl_pointer_count++] = cftell(Lcl_ext_file) - strlen(line) - 1;
-
-				// if we're out of pointer slots
-				if(Lcl_pointer_count >= LCL_MAX_POINTERS){
-					error_display(0, "Out of pointers for tstrings.tbl lookup. Please increment LCL_MAX_POINTERS in localize.cpp\n");
-					lcl_ext_close();
-					return;
-				}
-			}
-			// increment string count
-			string_count++;			
-		}
-	}
-
-	// should never get here. we should always be exiting through case 3 (end of language section) of the above switch
-	// statement
-	Int3();
-	lcl_ext_close();
-}
-
 void lcl_get_language_name(char *lang_name)
 {
 	Assert(Lcl_current_lang < (int)Lcl_languages.size());
@@ -1454,94 +1117,6 @@
 	strcpy(lang_name, Lcl_languages[Lcl_current_lang].lang_name);
 }
 
-// converts german umlauted chars from ASCII to ANSI
-// so they appear in the launcher
-// how friggin lame is this
-// pass in a null terminated string, foo!
-// returns ptr to string you sent in
-char* lcl_fix_umlauts(char *str, int which_way)
-{
-	int i=0;
-
-	if (which_way == LCL_TO_ANSI) {
-		// moving to ANSI charset
-		// run thru string and perform appropriate conversions
-		while (str[i] != '\0') {
-			switch (str[i]) {
-			case '\x81':
-				// lower umlaut u
-				str[i] = '\xFC';
-				break;
-			case '\x84':
-				// lower umlaut a
-				str[i] = '\xE4';
-				break;
-			case '\x94':
-				// lower umlaut o
-				str[i] = '\xF6';
-				break;
-			case '\x9A':
-				// upper umlaut u
-				str[i] = '\xDC';
-				break;
-			case '\x8E':
-				// upper umlaut a
-				str[i] = '\xC4';
-				break;
-			case '\x99':
-				// upper umlaut o
-				str[i] = '\xD6';
-				break;
-			case '\xE1':
-				// beta-lookin thing that means "ss"
-				str[i] = '\xDF';
-				break;
-			}
-
-			i++;
-		}
-	} else {
-		// moving to ASCII charset
-		// run thru string and perform appropriate conversions
-		while (str[i] != '\0') {
-			switch (str[i]) {
-			case '\xFC':
-				// lower umlaut u
-				str[i] = '\x81';
-				break;
-			case '\xE4':
-				// lower umlaut a
-				str[i] = '\x84';
-				break;
-			case '\xF6':
-				// lower umlaut o
-				str[i] = '\x94';
-				break;
-			case '\xDC':
-				// upper umlaut u
-				str[i] = '\x9A';
-				break;
-			case '\xC4':
-				// upper umlaut a
-				str[i] = '\x8E';
-				break;
-			case '\xD6':
-				// upper umlaut o
-				str[i] = '\x99';
-				break;
-			case '\xDF':
-				// beta-lookin thing that means "ss"
-				str[i] = '\xE1';
-				break;
-			}
-
-			i++;
-		}
-	}
-
-	return str;
-}
-
 // convert some of the polish characters
 void lcl_fix_polish(char *str)
 {
Index: code/localization/localize.h
===================================================================
--- code/localization/localize.h	(revision 10869)
+++ code/localization/localize.h	(working copy)
@@ -92,12 +92,6 @@
 // maybe add localized directory to full path with file name when opening a localized file
 int lcl_add_dir_to_path_with_filename(char *current_path, size_t path_max);
 
-// open the externalization file for use during parsing (call before parsing a given file)
-void lcl_ext_open();
-
-// close the externalization file (call after parsing a given file)
-void lcl_ext_close();
-
 // Goober5000
 void lcl_replace_stuff(char *text, size_t max_len);
 void lcl_replace_stuff(SCP_string &text);
@@ -120,23 +114,10 @@
 const char *XSTR(const char *str, int index);
 int lcl_get_xstr_offset(int index, int res);
 
-// translate umlauted chars from ascii to ansi codes
-// used in launcher
-#define LCL_TO_ANSI	0
-#define LCL_TO_ASCII	1
-char* lcl_fix_umlauts(char *str, int which_way);
-
 // covert some polish characters
 void lcl_fix_polish(char *str);
 void lcl_fix_polish(SCP_string &str);
 
-// macro for launcher xstrs
-#if defined(GERMAN_BUILD)
-#define LXSTR(str, i)		(lcl_fix_umlauts(XSTR(str, i), LCL_TO_ANSI))
-#else
-#define LXSTR(str, i)		(XSTR(str, i))
-#endif	// defined(GERMAN_BUILD)
-
 void lcl_translate_wep_name_gr(char *name);
 void lcl_translate_ship_name_gr(char *name);
 void lcl_translate_brief_icon_name_gr(char *name);
Index: code/menuui/credits.cpp
===================================================================
--- code/menuui/credits.cpp	(revision 10869)
+++ code/menuui/credits.cpp	(working copy)
@@ -415,17 +415,11 @@
 
 void credits_parse()
 {
-	// open localization
-	lcl_ext_open();
-
 	// Parse main table
 	credits_parse_table("credits.tbl");
 
 	// Parse modular tables
 	parse_modular_table("*-crd.tbm", credits_parse_table);
-
-	// close localization
-	lcl_ext_close();
 }
 
 void credits_init()
Index: code/menuui/playermenu.cpp
===================================================================
--- code/menuui/playermenu.cpp	(revision 10869)
+++ code/menuui/playermenu.cpp	(working copy)
@@ -1314,12 +1314,8 @@
 
 	Num_player_tips = 0;
 
-	// begin external localization stuff
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0) {
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", "tips.tbl", rval));
-		lcl_ext_close();
 		return;
 	}
 
@@ -1334,9 +1330,6 @@
 		}
 		Player_tips[Num_player_tips++] = stuff_and_malloc_string(F_NAME, NULL);
 	}
-
-	// stop externalizing, homey
-	lcl_ext_close();
 }
 
 // close out player tips - *only call from game_shutdown()*
Index: code/menuui/snazzyui.cpp
===================================================================
--- code/menuui/snazzyui.cpp	(revision 10869)
+++ code/menuui/snazzyui.cpp	(working copy)
@@ -188,16 +188,10 @@
 
 	*num_regions=0;
 
-	// open localization
-	lcl_ext_open();
-
 	fp = cfopen( NOX("menu.tbl"), "rt" );
 	if (fp == NULL) {
 		Error(LOCATION, "menu.tbl could not be opened\n");
 
-		// close localization
-		lcl_ext_close();
-
 		return;
 	}
 
@@ -208,9 +202,6 @@
 		p1 = p3 = strchr( tmp_line, '[' );
 
 		if (p3 && state == 1) {	
-			// close localization
-			lcl_ext_close();
-
 			cfclose(fp);
 			return;
 		}
@@ -234,9 +225,6 @@
 				if (!p2) {
 					nprintf(("Warning","Error parsing menu file\n"));
 
-					// close localization
-					lcl_ext_close();
-
 					return;
 				}
 				*p2 = 0;
@@ -279,9 +267,6 @@
 		}
 	}	
 	cfclose(fp);
-	
-	// close localization
-	lcl_ext_close();
 }
 
 // snazzy_menu_close() is called when the menu using a snazzy interface is exited
Index: code/menuui/techmenu.cpp
===================================================================
--- code/menuui/techmenu.cpp	(revision 10869)
+++ code/menuui/techmenu.cpp	(working copy)
@@ -1032,12 +1032,8 @@
 	if (inited)
 		return;
 
-	// open localization
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0) {
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", "species.tbl", rval));
-		lcl_ext_close();
 		return;
 	}
 	
@@ -1075,9 +1071,6 @@
 	}
 
 	inited = 1;
-
-	// close localization
-	lcl_ext_close();
 }
 
 void techroom_init()
Index: code/mission/missionbriefcommon.cpp
===================================================================
--- code/mission/missionbriefcommon.cpp	(revision 10869)
+++ code/mission/missionbriefcommon.cpp	(working copy)
@@ -305,12 +305,8 @@
 	Assert(!Species_info.empty());
 	const size_t max_icons = Species_info.size() * MIN_BRIEF_ICONS;
 
-	// open localization
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0) {
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", "icons.tbl", rval));
-		lcl_ext_close();
 
 		return;
 	}
@@ -351,10 +347,7 @@
 	}
 	required_string("#End");
 
-	// close localization
-	lcl_ext_close();
 
-
 	// now assign the icons to their species
 	const size_t num_species_covered = Briefing_icon_info.size() / MIN_BRIEF_ICONS;
 	size_t bii_index = 0;
Index: code/mission/missioncampaign.cpp
===================================================================
--- code/mission/missioncampaign.cpp	(revision 10869)
+++ code/mission/missioncampaign.cpp	(working copy)
@@ -113,9 +113,6 @@
 	}
 	Assert(fname_len < MAX_FILENAME_LEN);
 
-	// open localization
-	lcl_ext_open();
-
 	*type = -1;
 	do {
 		if ((rval = setjmp(parse_abort)) != 0) {
@@ -167,9 +164,6 @@
 		}
 	} while (0);
 
-	// close localization
-	lcl_ext_close();
-
 	Assert(success);
 	return success;
 }
@@ -425,9 +419,6 @@
 
 	filename = cf_add_ext(filename, FS_CAMPAIGN_FILE_EXT);
 
-	// open localization
-	lcl_ext_open();	
-
 	if ( pl == NULL )
 		pl = Player;
 
@@ -440,9 +431,6 @@
 	if ((rval = setjmp(parse_abort)) != 0) {
 		mprintf(("Error parsing '%s'\r\nError code = %i.\r\n", filename, rval));
 
-		// close localization
-		lcl_ext_close();
-
 		Campaign.filename[0] = 0;
 		Campaign.num_missions = 0;
 
@@ -565,9 +553,6 @@
 
 				} else {
 					if ( cm->formula == -1 ){
-						// close localization
-						lcl_ext_close();
-
 						Campaign_load_failure = CAMPAIGN_ERROR_SEXP_EXHAUSTED;
 						return CAMPAIGN_ERROR_SEXP_EXHAUSTED;
 					}
@@ -605,9 +590,6 @@
 
 				} else {
 					if ( cm->mission_loop_formula == -1 ){
-						// close localization
-						lcl_ext_close();
-
 						Campaign_load_failure = CAMPAIGN_ERROR_SEXP_EXHAUSTED;
 						return CAMPAIGN_ERROR_SEXP_EXHAUSTED;
 					}
@@ -648,9 +630,6 @@
 		}
 	}
 
-	// close localization
-	lcl_ext_close();
-
 	// set up the other variables for the campaign stuff.  After initializing, we must try and load
 	// the campaign save file for this player.  Since all campaign loads go through this routine, I
 	// think this place should be the only necessary place to load the campaign save stuff.  The campaign
@@ -1447,13 +1426,9 @@
 	int i, z, rval, event_count, count = 0;
 
 	filename = Campaign.missions[num].name;
-
-	// open localization
-	lcl_ext_open();	
 	
 	if ((rval = setjmp(parse_abort)) != 0) {
 		mprintf(("MISSIONCAMPAIGN: Unable to parse '%s'!  Error code = %i.\n", filename, rval));
-		lcl_ext_close();
 		return;
 	}
 
@@ -1552,9 +1527,6 @@
 	}
 
 	// Goober5000 - variables do not need to be read here
-
-	// close localization
-	lcl_ext_close();
 }
 
 /**
Index: code/mission/missionmessage.cpp
===================================================================
--- code/mission/missionmessage.cpp	(revision 10869)
+++ code/mission/missionmessage.cpp	(working copy)
@@ -533,9 +533,6 @@
 {
 	int i, j;
 
-	// open localization
-	lcl_ext_open();
-
 	//speed things up a little by setting the capacities for the message vectors to roughly the FS2 amounts
 	Messages.reserve(500);
 	Message_waves.reserve(300);
@@ -646,10 +643,6 @@
 
 		required_string("#End");
 	}
-
-	
-	// close localization
-	lcl_ext_close();
 }
 
 // this is called at the start of each level
Index: code/mission/missionparse.cpp
===================================================================
--- code/mission/missionparse.cpp	(revision 10869)
+++ code/mission/missionparse.cpp	(working copy)
@@ -5828,9 +5828,6 @@
 	if ( mission_p == NULL )
 		mission_p = &The_mission;
 
-	// open localization
-	lcl_ext_open();
-
 	do {
 		CFILE *ftemp = cfopen(real_fname, "rt");
 		if (!ftemp) {
@@ -5857,9 +5854,6 @@
 		parse_mission_info(mission_p, basic);
 	} while (0);
 
-	// close localization
-	lcl_ext_close();
-
 	return rval;
 }
 
@@ -5900,9 +5894,6 @@
 
 	for (i = 0; i < Num_ship_classes; i++)
 		Ship_class_names[i] = Ship_info[i].name;
-
-	// open localization
-	lcl_ext_open();
 	
 	do {
 		// don't do this for imports
@@ -5943,9 +5934,6 @@
 		display_parse_diagnostics();
 	} while (0);
 
-	// close localization
-	lcl_ext_close();
-
 	if (!Fred_running)
 		strcpy_s(Mission_filename, mission_name);
 
@@ -6302,9 +6290,6 @@
 	if ( filelength == 0 )
 		return 0;
 
-	// open localization
-	lcl_ext_open();
-
 	game_type = 0;
 	do {
 		if ((rval = setjmp(parse_abort)) != 0) {
@@ -6328,9 +6313,6 @@
 		stuff_int(&game_type);
 	} while (0);
 
-	// close localization
-	lcl_ext_close();
-
 	return (game_type & MISSION_TYPE_MULTI) ? game_type : 0;
 }
 
Index: code/missionui/missiondebrief.cpp
===================================================================
--- code/missionui/missiondebrief.cpp	(revision 10869)
+++ code/missionui/missiondebrief.cpp	(working copy)
@@ -1059,12 +1059,8 @@
 		int rval;
 		int stage_num;
 
-		// open localization
-		lcl_ext_open();
-
 		if ((rval = setjmp(parse_abort)) != 0) {
 			mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", "traitor.tbl", rval));
-			lcl_ext_close();
 			return;
 		}
 
@@ -1103,9 +1099,6 @@
 		stuff_string( stagep->recommendation_text, F_MULTITEXT, NULL);
 
 		inited = 1;
-
-		// close localization
-		lcl_ext_close();
 	}
 
 	// disable the accept button if in single player and I am a traitor
Index: code/mod_table/mod_table.cpp
===================================================================
--- code/mod_table/mod_table.cpp	(revision 10869)
+++ code/mod_table/mod_table.cpp	(working copy)
@@ -37,13 +37,9 @@
 	int rval;
 	// SCP_vector<SCP_string> lines;
 
-	// open localization
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0)
 	{
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", (filename) ? filename : "<default game_settings.tbl>", rval));
-		lcl_ext_close();
 		return;
 	}
 
@@ -254,9 +250,6 @@
 	}
 
 	required_string("#END");
-
-	// close localization
-	lcl_ext_close();
 }
 
 void mod_table_init()
Index: code/parse/parselo.cpp
===================================================================
--- code/parse/parselo.cpp	(revision 10869)
+++ code/parse/parselo.cpp	(working copy)
@@ -1131,12 +1131,17 @@
 
 /**
  * Stuff a string (" chars ") into *str, return length.
+ * Accepts an optional max length parameter. If it is omitted or negative, then no max length is enforced.
  */
-int get_string(char *str)
+int get_string(char *str, int max)
 {
 	int	len;
 
 	len = strcspn(Mp + 1, "\"");
+
+	if (max >= 0 && len >= max)
+		error_display(0, "String too long.  Length = %i.  Max is %i.\n", len, max);
+
 	strncpy(str, Mp + 1, len);
 	str[len] = 0;
 
@@ -1246,7 +1251,7 @@
 			copy_to_eoln(read_str, terminators, Mp, read_len);
 			drop_trailing_white_space(read_str);
 			advance_to_eoln(terminators);
-			break;		
+			break;
 
 		default:
 			Error(LOCATION, "Unhandled string type %d in stuff_string!", type);
@@ -1356,7 +1361,7 @@
 			copy_to_eoln(read_str, terminators, Mp);
 			drop_trailing_white_space(read_str);
 			advance_to_eoln(terminators);
-			break;		
+			break;	
 
 		default:
 			Error(LOCATION, "Unhandled string type %d in stuff_string!", type);
Index: code/parse/parselo.h
===================================================================
--- code/parse/parselo.h	(revision 10869)
+++ code/parse/parselo.h	(working copy)
@@ -159,7 +159,7 @@
 extern int match_and_stuff(int f_type, char *strlist[], int max, char *description);
 extern void find_and_stuff_or_add(char *id, int *addr, int f_type, char *strlist[], int *total,
 	int max, char *description);
-extern int get_string(char *str);
+extern int get_string(char *str, int max = -1);
 extern void get_string(SCP_string &str);
 extern void stuff_parenthesized_vec3d(vec3d *vp);
 extern void stuff_boolean(int *i, bool a_to_eol=true);
Index: code/ship/ship.cpp
===================================================================
--- code/ship/ship.cpp	(revision 10869)
+++ code/ship/ship.cpp	(working copy)
@@ -4123,12 +4123,8 @@
 {
 	int rval;
 
-	// open localization
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0) {
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", filename, rval));
-		lcl_ext_close();
 		return;
 	}
 
@@ -4165,9 +4161,6 @@
 
 	// add tbl/tbm to multiplayer validation list
 	fs2netd_add_table_validation(filename);
-
-	// close localization
-	lcl_ext_close();
 }
 
 // The E - Simple lookup function for FRED.
@@ -4224,14 +4217,10 @@
 void parse_shiptbl(const char *filename)
 {
 	int rval;
-
-	// open localization
-	lcl_ext_open();
 	
 	if ((rval = setjmp(parse_abort)) != 0)
 	{
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", filename, rval));
-		lcl_ext_close();
 		return;
 	}
 
@@ -4288,9 +4277,6 @@
 
 	// add tbl/tbm to multiplayer validation list
 	fs2netd_add_table_validation(filename);
-
-	// close localization
-	lcl_ext_close();
 }
 
 int ship_show_velocity_dot = 0;
@@ -17662,12 +17648,8 @@
 {
 	int rval;
 
-	// open localization
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0) {
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", filename, rval));
-		lcl_ext_close();
 		return;
 	}
 
@@ -17686,9 +17668,6 @@
 
 	// add tbl/tbm to multiplayer validation list
 	fs2netd_add_table_validation(filename);
-
-	// close localization
-	lcl_ext_close();
 }
 
 void armor_init()
Index: code/species_defs/species_defs.cpp
===================================================================
--- code/species_defs/species_defs.cpp	(revision 10869)
+++ code/species_defs/species_defs.cpp	(working copy)
@@ -156,13 +156,9 @@
 	int i, rval;
 	char species_name[NAME_LENGTH];
 
-	// open localization
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0)
 	{
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", (filename) ? filename : NOX("<default species_defs.tbl>"), rval));
-		lcl_ext_close();
 		return;
 	}
 
@@ -354,9 +350,6 @@
 	// add tbl/tbm to multiplayer validation list
 	extern void fs2netd_add_table_validation(const char *tblname);
 	fs2netd_add_table_validation(filename);
-
-	// close localization
-	lcl_ext_close();
 }
 
 int Species_initted = 0;
Index: code/stats/medals.cpp
===================================================================
--- code/stats/medals.cpp	(revision 10869)
+++ code/stats/medals.cpp	(working copy)
@@ -246,12 +246,8 @@
 {
 	int rval, i;
 
-	// open localization
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0) {
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", "medals.tbl", rval));
-		lcl_ext_close();
 		return;
 	}
 
@@ -429,9 +425,6 @@
 		if (Medals[i].kills_needed > 0)
 			prev_badge_kills = Medals[i].kills_needed;
 	}
-
-	// close localization
-	lcl_ext_close();
 }
 
 // replacement for -gimmemedals
Index: code/stats/scoring.cpp
===================================================================
--- code/stats/scoring.cpp	(revision 10869)
+++ code/stats/scoring.cpp	(working copy)
@@ -61,12 +61,8 @@
 	char buf[MULTITEXT_LENGTH];
 	int rval, idx, persona;
 
-	// open localization
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0) {
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", "rank.tbl", rval));
-		lcl_ext_close();
 		return;
 	} 
 
@@ -118,9 +114,6 @@
 			Int3();
 	}
 #endif
-
-	// close localization
-	lcl_ext_close();
 }
 
 // initialize a nice blank scoring element
Index: code/weapon/weapons.cpp
===================================================================
--- code/weapon/weapons.cpp	(revision 10869)
+++ code/weapon/weapons.cpp	(working copy)
@@ -324,12 +324,8 @@
 	uint i;
 	lod_checker lod_check;
 
-	// open localization
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0) {
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", filename, rval));
-		lcl_ext_close();
 		return;
 	}
 
@@ -368,9 +364,6 @@
 		}
 	}
 	required_string("#End");
-
-	// close localization
-	lcl_ext_close();
 }
 
 /**
@@ -2781,13 +2774,9 @@
 {
 	int rval;
 
-	// open localization
-	lcl_ext_open();
-
 	if ((rval = setjmp(parse_abort)) != 0)
 	{
 		mprintf(("TABLES: Unable to parse '%s'!  Error code = %i.\n", filename, rval));
-		lcl_ext_close();
 		return;
 	}
 
@@ -2860,9 +2849,6 @@
 
 	// add tbl/tbm to multiplayer validation list
 	fs2netd_add_table_validation(filename);
-
-	// close localization
-	lcl_ext_close();
 }
 
 //uses a simple bucket sort to sort weapons, order of importance is:
mantis3049.patch (50,310 bytes)   

Goober5000

2014-07-07 04:48

administrator   ~0016031

Splendid. I gave the patch a look and didn't see anything obviously out of place with it, although it would take some time to dig through the changes to localize.[h|cpp]. But if you've tested it with both English and German, I'm satisfied. Patch committed.

Yarn

2014-08-17 05:06

developer   ~0016229

I noticed two things in the original patch that should be fixed:

1) In the event that an entry index in tstrings.tbl is 4500 or greater, a fatal error is thrown. This did not happen in prior revisions (which would throw a warning instead), nor did it happen in retail.

2) If index of an XSTR structure is 4500 or greater, there is nothing preventing an out-of-bounds read of the Lcl_ext_str array.

The attached patch (mantis3049_2.patch) fixes both of these issues.

MageKing17

2014-08-17 07:29

developer   ~0016230

Is it not desirable to add a Warning() in the second case? Since it sounds like an XSTR() with an ID >= LCL_MAX_STRINGS would be caused by bad user data, we might want to let the user know it happened...

Yarn

2014-08-17 16:39

developer  

mantis3049_2.patch (1,961 bytes)   
Index: code/localization/localize.cpp
===================================================================
--- code/localization/localize.cpp	(revision 11007)
+++ code/localization/localize.cpp	(working copy)
@@ -267,7 +267,8 @@
 		}
 
 		if (external && (index < 0 || index >= LCL_MAX_STRINGS)) {
-			Error(LOCATION, "Invalid tstrings table index specified (%i). Please increment LCL_MAX_STRINGS in localize.cpp.", index);
+			error_display(0, "Invalid tstrings table index specified (%i). Please increment LCL_MAX_STRINGS in localize.cpp.", index);
+			return;
 		} else if (!external && (index < 0 || index >= XSTR_SIZE)) {
 			Error(LOCATION, "Invalid strings table index specified (%i)", index);
 		}
@@ -711,7 +712,7 @@
 	}
 
 	// get the string if it exists
-	if (Lcl_ext_str[str_id] != NULL) {
+	if ((str_id < LCL_MAX_STRINGS) && (Lcl_ext_str[str_id] != NULL)) {
 		// copy to the outgoing string
 		if ( strlen(Lcl_ext_str[str_id]) > max_len )
 			error_display(0, "Token too long: [%s].  Length = %i.  Max is %i.\n", Lcl_ext_str[str_id], strlen(Lcl_ext_str[str_id]), max_len);
@@ -723,6 +724,9 @@
 		if ( strlen(text_str) > max_len )
 			error_display(0, "Token too long: [%s].  Length = %i.  Max is %i.\n", text_str, strlen(text_str), max_len);
 
+		if (str_id >= LCL_MAX_STRINGS)
+			error_display(0, "Invalid XSTR ID: [%d]. (Must be less than %d.)\n", str_id, LCL_MAX_STRINGS);
+
 		strncpy(out, text_str, max_len);
 	}
 
@@ -792,13 +796,16 @@
 		return;
 	}
 
-	// attempt to find the string
-	if (Lcl_ext_str[str_id] != NULL) {
+	// get the string if it exists
+	if ((str_id < LCL_MAX_STRINGS) && (Lcl_ext_str[str_id] != NULL)) {
 		// copy to the outgoing string
 		out = Lcl_ext_str[str_id];
 	}
 	// otherwise use what we have - probably should Int3() or assert here
 	else {
+		if (str_id >= LCL_MAX_STRINGS)
+			error_display(0, "Invalid XSTR ID: [%d]. (Must be less than %d.)\n", str_id, LCL_MAX_STRINGS);
+
 		out = text_str;
 	}
 
mantis3049_2.patch (1,961 bytes)   

Yarn

2014-08-17 16:39

developer   ~0016234

I updated the patch to add the warning.

niffiwan

2014-08-18 08:32

developer   ~0016235

Fix committed to trunk@11008.

Related Changesets

fs2open: trunk r10894

2014-07-07 00:56

Goober5000


Ported: N/A

Details Diff
Yarn's patch for Mantis 0003049 (character remapping in tstrings.tbl) Affected Issues
0003049
mod - /trunk/fs2_open/code/species_defs/species_defs.cpp Diff File
mod - /trunk/fs2_open/code/lab/wmcgui.cpp Diff File
mod - /trunk/fs2_open/code/localization/localize.cpp Diff File
mod - /trunk/fs2_open/code/hud/hudparse.cpp Diff File
mod - /trunk/fs2_open/code/gamesnd/gamesnd.cpp Diff File
mod - /trunk/fs2_open/code/mission/missionparse.cpp Diff File
mod - /trunk/fs2_open/code/menuui/snazzyui.cpp Diff File
mod - /trunk/fs2_open/code/menuui/credits.cpp Diff File
mod - /trunk/fs2_open/code/localization/localize.h Diff File
mod - /trunk/fs2_open/code/gamesnd/eventmusic.cpp Diff File
mod - /trunk/fs2_open/code/autopilot/autopilot.cpp Diff File
mod - /trunk/fs2_open/code/mission/missioncampaign.cpp Diff File
mod - /trunk/fs2_open/code/stats/scoring.cpp Diff File
mod - /trunk/fs2_open/code/stats/medals.cpp Diff File
mod - /trunk/fs2_open/code/cutscene/cutscenes.cpp Diff File
mod - /trunk/fs2_open/code/weapon/weapons.cpp Diff File
mod - /trunk/fs2_open/code/ship/ship.cpp Diff File
mod - /trunk/fs2_open/code/mission/missionmessage.cpp Diff File
mod - /trunk/fs2_open/code/menuui/techmenu.cpp Diff File
mod - /trunk/fs2_open/code/mod_table/mod_table.cpp Diff File
mod - /trunk/fs2_open/code/ai/aicode.cpp Diff File
mod - /trunk/fs2_open/code/gamehelp/contexthelp.cpp Diff File
mod - /trunk/fs2_open/code/parse/parselo.cpp Diff File
mod - /trunk/fs2_open/code/missionui/missiondebrief.cpp Diff File
mod - /trunk/fs2_open/code/menuui/playermenu.cpp Diff File
mod - /trunk/fs2_open/code/fireball/fireballs.cpp Diff File
mod - /trunk/fs2_open/code/mission/missionbriefcommon.cpp Diff File
mod - /trunk/fs2_open/code/asteroid/asteroid.cpp Diff File
mod - /trunk/fs2_open/code/parse/parselo.h Diff File
mod - /trunk/fs2_open/code/ai/ai_profiles.cpp Diff File

fs2open: trunk r11008

2014-08-18 05:02

niffiwan


Ported: N/A

Details Diff
Followup fix for mantis 3049 (from Yarn)

Revert an Error back to a Warning
Prevent an out-of-bounds read of the Lcl_ext_str array
Affected Issues
0003049
mod - /trunk/fs2_open/code/localization/localize.cpp Diff File

Issue History

Date Modified Username Field Change
2014-05-28 21:45 Yarn New Issue
2014-05-28 21:45 Yarn File Added: char_remap_test.zip
2014-05-29 05:45 Goober5000 Note Added: 0015770
2014-05-29 05:45 Goober5000 Assigned To => Goober5000
2014-05-29 05:45 Goober5000 Priority normal => high
2014-05-29 05:45 Goober5000 Status new => assigned
2014-05-29 05:45 Goober5000 Target Version => 3.7.2
2014-06-29 23:13 Goober5000 Note Added: 0015941
2014-06-29 23:13 Goober5000 Assigned To Goober5000 => Yarn
2014-07-03 05:55 Yarn Note Added: 0016014
2014-07-04 00:32 Goober5000 Note Added: 0016017
2014-07-04 05:47 Yarn File Added: mantis3049.patch
2014-07-04 05:47 Yarn Note Added: 0016019
2014-07-04 05:47 Yarn Status assigned => code review
2014-07-04 06:11 Yarn File Deleted: mantis3049.patch
2014-07-04 06:11 Yarn File Added: mantis3049.patch
2014-07-07 04:34 Goober5000 Changeset attached => fs2open trunk r10894
2014-07-07 04:48 Goober5000 Note Added: 0016031
2014-07-07 04:48 Goober5000 Status code review => resolved
2014-07-07 04:48 Goober5000 Resolution open => fixed
2014-08-17 05:06 Yarn Note Added: 0016229
2014-08-17 05:06 Yarn Status resolved => feedback
2014-08-17 05:06 Yarn Resolution fixed => reopened
2014-08-17 05:06 Yarn File Added: mantis3049_2.patch
2014-08-17 05:06 Yarn Status feedback => code review
2014-08-17 07:29 MageKing17 Note Added: 0016230
2014-08-17 16:39 Yarn File Deleted: mantis3049_2.patch
2014-08-17 16:39 Yarn File Added: mantis3049_2.patch
2014-08-17 16:39 Yarn Note Added: 0016234
2014-08-18 08:32 niffiwan Changeset attached => fs2open trunk r11008
2014-08-18 08:32 niffiwan Note Added: 0016235
2014-08-18 08:32 niffiwan Status code review => resolved