In this post we’ll cover TeX’s handing of strings and explain Here’s a small fragment of As you can see from the above fragment, the In addition to the string constants defined in From TeX.WEB<\/strong>: In vanilla C, a simple 8-bit string is an array of characters terminated by the null character ( From TeX.WEB<\/strong>: It is worth noting that when TANGLE produces Pascal code (from the WEB source) it strips out all underscores from variables defined in the WEB code. For example, the |str_pool| variable mentioned above is called strpool in the final C code produced from the Pascal.<\/p>\n After processing via Web2C, the WEB variables The types Stripping away all To see what’s going on, i.e., how TeX identifies a string, let’s first look at the global variable where and A Pascal legacy<\/strong>: In many places within The allocation of memory for – which looks very much like one huge C string. And, of course, it is. We can declare a variable Here, we have concatenated the 6 strings So, if we define an array of integers, Then for some string identified by a number For example, let us use this method to calculate the length of the string with number 4 ( Of course there is one minor complication – calculating the length of string 5, but we have other variables ( We started this discussion by noting that running the TANGLE program on The next stage in the discussion covers the mechanisms for processing If you just want to see the inputs\/outputs you can download the files I produced during my private build of Knuthian TeX:<\/p>\n Once you have generated Of course, when you build TeX you will need to compile The function loadpoolstrings(…) depends on a few of TeX’s internal global variables and the function makestring() (we’ll discuss that shortly), notably we need to declare the following vaiables as extern to Here is my slightly modified version of .pool<\/code> files. Using Web2C to build (Knuthian) TeX from Knuth’s TeX.WEB<\/code> source code involves many steps as explained elsewhere on this site<\/a>. One of the initial steps when building TeX is combining Knuth’s master source file (TeX.WEB<\/code>) with a “change file” (TeX.CH<\/code>) to produce a modified WEB source file (let’s call it TeXk.WEB<\/code>) which can be processed via the Web2C process. The TeX.CH<\/code> change file applies many modifications to the master TeX.WEB<\/code> source code – i.e., in preparation for conversion to C code and adding support for the kpathsea file-seaching library. After the change file has been applied, the next step is to process our modified TeX.WEB<\/code> (i.e., TeXk.WEB<\/code>) via the TANGLE program. If TANGLE successfully parses our TeXk.WEB source code it will output two files (download links are provided for the inquisitive):<\/p>\n\n
TeXk.p<\/a><\/code>: the source code of TeX (in Pascal).<\/li>\nTeXk.pool<\/a><\/code>: a file containing the string constants defined in TeXk.WEB<\/code><\/li>\n<\/ul>\nTeXk.pool<\/code> as produced during my Web2C process:<\/p>\n\r\n....\r\n11expandafter\r\n04font\r\n09fontdimen\r\n06halign\r\n05hrule\r\n12ignorespaces\r\n10mathaccent\r\n08mathchar\r\n10mathchoice\r\n08multiply\r\n07noalign\r\n10noboundary\r\n08noexpand\r\n04omit\r\n07penalty\r\n08prevgraf\r\n07radical\r\n04read\r\n05relax\r\n06setbox\r\n03the\r\n06valign\r\n07vcenter\r\n05vrule\r\n09save size\r\n15grouping levels\r\n08curlevel\r\n09retaining\r\n09restoring\r\n05SAVE(\r\n28Incompatible magnification (\r\n02);\r\n36 the previous value will be retained\r\n58I can handle only one magnification ratio per job. So I've\r\n59reverted to the magnification you used earlier on this run.\r\n46Illegal magnification has been changed to 1000\r\n52The magnification ratio must be between 1 and 32768.\r\n...\r\n*413816964\r\n<\/pre>\n
TeXk.pool<\/code> consists of many lines of the format [string length][string text][end_of_line]<\/code> and final containing *CHECKSUM<\/code>, where CHECKSUM<\/code> in the above example is 413816964<\/code>. Once upon a time, .pool<\/code> files had to be preserved as an external file for use when building .fmt<\/code> files via INITEX but in 2008 this was changed and the .pool<\/code> file is now compiled into the TeX binaries – I’ll explain this below. For example, the following note is contained in more recent texmf.cnf<\/code> files:<\/p>\nAs of 2008, pool files don't exist any more (the strings are compiled into the binaries), but just in case something expects to find these:
\nTEXPOOL = .;$TEXMF\/web2c
\nMFPOOL = ${TEXPOOL}
\nMPPOOL = ${TEXPOOL}
\n<\/code><\/p>\nTeXk.pool<\/code> file contains string constants for TeX’s primitive commands plus all the strings contained in help\/error messages that TeX outputs to the terminal and\/or log file. <\/p>\nTeX’s internal handling of strings<\/H1><\/p>\n
TeXk.pool<\/code>, TeX will, of course, encounter new strings – for example, when you define new macro names; consequently, TeX needs a way to store the string constants in TeXk.pool<\/code> and the strings it encounters during its run-time processing of your TeX files. It should not be a surprise that TeX’s internal handling of strings is achieved through methods designed to ensure portability.<\/p>\nThe TEX system does nearly all of its own memory allocation, so that it can readily be transported into environments that do not have automatic facilities for strings, garbage collection, etc., and so that it can be in control of what error messages the user receives... Control sequence names and diagnostic messages are variable-length strings of eight-bit characters. Since PASCAL does not have a well-developed string mechanism, TeX does all of its string processing by homegrown methods.<\/code><\/p><\/blockquote>\nHow does TeX use\/store strings?<\/H2><\/p>\n
'\\0'<\/code>). TeX does not store is strings as individually named string variables but allocates a single large array and uses integer offsets into that array to identify strings (and calculate lengths). Here’s how it works.<\/p>\n The array |str_pool| contains all of the (eight-bit) ASCII codes in all of the strings, and the array |str_start| contains indices of the starting points of each string. Strings are referred to by integer numbers, so that string number |s| comprises the characters |str_pool[j]| for |str_start[s]<=j<str_start[s+1]|. Additional integer variables |pool_ptr| and |str_ptr| indicate the number of entries used so far in |str_pool| and |str_start|, respectively; locations |str_pool[pool_ptr]| and |str_start[str_ptr]| are ready for the next string to be allocated.<\/code><\/p><\/blockquote>\n|str_pool|<\/code>, |str_start|<\/code>, |pool_ptr|<\/code> and |str_ptr|<\/code> are global variables declared as follows (near the start of TeX.C<\/code>): <\/p>\n
\npackedASCIIcode * strpool ;
\npoolpointer * strstart ;
\npoolpointer poolptr ;
\nstrnumber strptr
\n<\/code><\/p>\npackedASCIIcode<\/code> and poolpointer<\/code> are simply typedefs:<\/p>\n
\ntypedef unsigned char packedASCIIcode ;
\ntypedef int integer;
\ntypedef integer poolpointer ;
\n<\/code><\/p>\ntypedef<\/code>s introduced by Web2C gives:<\/p>\n
\nunsigned char* strpool ;
\nint* strstart ;
\nint poolptr ;
\nint strptr ;
\n<\/code><\/p>\nstrpool<\/code> (practically all key variables are declared with global scope in TeX.C<\/code>…!). During initialization (in INITEX mode, and when TeX is reading\/unpacking a .fmt<\/code> file to initialize a particular format (plain.fmt<\/code>, latex.fmt<\/code> etc)) the strpool<\/code> and strstart<\/code> variables are initialized as follows:<\/p>\nstrpool = xmallocarray (packedASCIIcode , poolsize) ;<\/code>
\nstrstart = xmallocarray (poolpointer , maxstrings) ;<\/code><\/p>\nxmallocarray<\/code> is a #define<\/code>:<\/p>\n
\n\/* Allocate an array of a given type. Add 1 to size to account for the fact that Pascal arrays are used from [1..size], unlike C arrays which use [0..size]. *\/
\n#define xmallocarray(type,size) ((type*)xmalloc((size+1)*sizeof(type)))
\n<\/code><\/p>\nxmalloc(...)<\/code> is a small utility function wrapped around the standard C function malloc(...)<\/code>.<\/p>\nTeX.C<\/code> you have to account for that fact that Pascal arrays start at index 1 but C arrays start at index 0. This is a consequence that Knuthian TeX is written in Pascal, not C. <\/p><\/blockquote>\nstrpool<\/code> uses an integer variable called poolsize<\/code>: the value of poolsize<\/code> is calculated at run-time from the value of other variables – including some variables whose value can be defined by settings in texmf.cnf<\/code>. So, in essence:<\/p>\nstrpool = (char *) malloc(sizeof(unsigned char)*(poolsize +1));<\/code><\/p>\nstrpool<\/code> stores all TeX’s strings BUT within strpool<\/code> all strings are contiguous (stored end-to-end) without any delimiter characters between them (such as NULL, ('\\0'<\/code>), space, etc). Clearly, there needs to be a mechanism to define where each individual string starts and stops: i.e., to partition strpool<\/code> into individual strings. That mechanism is the task of the integer array variable called strstart<\/code>. Perhaps an example will make this clearer.<\/p>\nmyfakestrpool<\/code> as follows:<\/p>\nunsigned char fakestrpool[]=\"ThisismyfakeTeXstrpool\";<\/code><\/p>\n\"This\"<\/code>, \"is\"<\/code>, \"my\"<\/code>, \"fake\"<\/code>,\"TeX\"<\/code> and \"strpool\"<\/code> into one long string. These 6 strings start at the following offsets in fakestrpool<\/code>:<\/p>\n
\nstring 0 (\"This\"): offsets 0
\nstring 1 (\"is\"): offset 4
\nstring 2 (\"my\"): offset 6
\nstring 3 (\"fake\"): offset 8
\nstring 4 (\"TeX\"): offset 12
\nstring 5 (\"strpool\") offset 15
\n<\/code><\/p>\nstrstart<\/code>, to record these offsets:<\/p>\nint strstart[6] ; \/\/ for 6 strings numbered 0 to 5<\/code><\/p>\n
\nstrstart[0]=0
\nstrstart[1]=4
\nstrstart[2]=6
\nstrstart[3]=8
\nstrstart[4]=12
\nstrstart[5]=15
\n<\/code><\/p>\nk<\/code> (where 0 =< k <= 5<\/code>), strstart[k]<\/code> gives the offset into fakestrpool<\/code> where the kth string starts. And this is exactly how TeX identifies strings: it identifies them using some integer value, k<\/code>, say, where strstart[k]<\/code> tells you where that string starts (in strpool<\/code>) and allows the length (length(k)<\/code>, of string number k<\/code>) to be easily be calculated using<\/p>\nlength(k) = strstart[k + 1] - strstart[k]<\/code><\/p>\nk=4<\/code>) (\"TeX\"<\/code> in our test array fakestrpool<\/code>).<\/p>\n
\nlength(4) = strstart[5] - strstart[4]
\nlength(5) = 15 - 12 = 3
\n<\/code><\/p>\npoolptr<\/code> and strptr<\/code>) to solve issues like this.<\/p>\nBack to
.pool<\/code> files<\/H1><\/p>\nTeXk.WEB<\/code> produces two output files:<\/p>\n\n
TeXk.p<\/a><\/code>: the source code of TeX (in Pascal).<\/li>\nTeXk.pool<\/a><\/code>: a file containing the string constants defined in TeXk.WEB<\/code><\/li>\n<\/ul>\n.pool<\/code> files – introduced in circa 2008. Prior to (circa) 2008, you needed to keep .pool<\/code> files available (part of the TeX distribution) as separate files for use whenever you ran INITEX to generate a new .fmt<\/code> file. As noted, the contents of the .pool<\/code> files are string constants generated by TANGLE from string constants defined in main WEB source code to TeX. Given that those strings they don’t change (they are constants), it makes more sense to build them into the TeX executable file rather than having to access them each time a new .fmt<\/code> file created by INITEX. Part of the Web2C process now involves using a small utility program called makecpool.exe<\/code> (on Windows) – makecpool.C<\/code> was written by Taco Hoekwater. The input to makecpool.exe<\/code> is the TeXk.pool<\/code> file and the output is another C file (called texpool.C<\/code> or similar) which defines a function called loadpoolstrings(...)<\/code>:<\/p>\nint loadpoolstrings (int spare_size)<\/code><\/p>\nDownloads<\/H2><\/p>\n
\n
TeXk.pool<\/code><\/a>: The .pool file input for makecpool.exe<\/code><\/li>\ntexpool.C<\/code><\/a>: The C file output by makecpool.exe<\/code>, defining the function loadpoolstrings(...)<\/code>.<\/li>\n<\/ul>\ntexpool.c<\/code> you no longer need the original TeXk.pool<\/code> file because the contents of TeXk.pool<\/code> are now stored within texpool.C<\/code>, stored as array of strings:<\/p>\n\r\nstatic const char *poolfilearr[] = {\r\n "buffer size",\r\n "pool size",\r\n "number of strings",\r\n "" "?" "?" "?",\r\n "m2d5c2l5x2v5i",\r\n "End of file on the terminal!",\r\n "! ",\r\n "(That makes 100 errors; please try again.)",\r\n "" "? ",\r\n "Type <return> to proceed, S to scroll future error messages,",\r\n "R to run without stopping, Q to run quietly,",\r\n "I to insert something, ",\r\n...\r\n...\r\n...\r\nNULL };\r\n<\/pre>\nTeXk.C<\/code> and texpool.C<\/code> so that the function loadpoolstrings(...)<\/code> is made available. The function loadpoolstrings(...)<\/code> is called from TeX.C<\/code> when TeX is in INITEX mode (i.e., the --ini<\/code> option is set on the command line). Specifically, loadpoolstrings(...)<\/code> function is called by the function getstringsstarted(...)<\/code> just after it has initialized the first 256 strings in TeX’s main string container: the strpool<\/code> array discussed above.<\/p>\nModifying loadpoolstrings (…) to see what it does<\/H2><\/p>\n
texpool.C<\/code>:<\/p>\n
\nextern int makestring ( void ) ;
\nextern unsigned char * strpool;
\nextern int poolptr;
\n<\/code><\/p>\nloadpoolstrings(...)<\/code> which outputs a file called \"datadump.txt\"<\/code> to list the strings and corresponding string numbers generated by makestring()<\/code>:<\/p>\n\r\nint loadpoolstrings (int spare_size) {\r\n const char *s;\r\n int g=0;\r\n FILE* dumpvals;\r\n int i=0,j=0;\r\n dumpvals=fopen("datadump.txt", "wb");\r\n\r\n while ((s = poolfilearr[j++])) {\r\n int l = strlen (s);\r\n\tfprintf(dumpvals, "\/\/string \\"%s\\" = number ", s);\r\n i += l;\r\n if (i>=spare_size) return 0;\r\n while (l-- > 0) strpool[poolptr++] = *s++;\r\n g = makestring();\r\n\tfprintf(dumpvals, "%ld\\n", g);\r\n }\r\n fclose(dumpvals);\r\n return g;\r\n}\r\n<\/pre>\ndatadump.txt<\/code><\/H2><\/p>\n