I've been reading about SIL International's Graphite engine and it looks really interesting. I downloaded the code and ran the CMake-based build process through the CMake graphical interface. It didn't work. Eventually, I found some instructions to build it from the command line, so here's the way I did it.

1. Make sure the cmake.exe is in your Windows PATH.
2. Download the Graphite source code and unpack into a directory (e.g, called Graphite).
3. Change directory to the one containing the Graphite source code.
4. I use Microsoft Visual Studio 2008 so you'll need to adjust the -G parameter (below) to your build environment (cmake --help tells you the ones it supports).
5. Run the command (all on one line): cmake -G "Visual Studio 9 2008" -DCMAKE_BUILD_TYPE=Release -DGRAPHITE2_COMPARE_RENDERER:BOOL=OFF

If all goes well you should see something like the following, together with a generated Visual Studio Solution file graphite2.sln.

-- Build: Release
-- Segment Cache support: enabled
-- File Face support: enabled
-- Tracing support: enabled
CMake Warning at CMakeLists.txt:54 (message):
  vm machine type direct can only be built using GCC

-- Using vm machine type: call
-- Configuring done
-- Generating done
-- Build files have been written to: E:/SILgraide/Graphite

Your Visual Studio Solution should look something like this:

Download PDF

This is a lengthy post which covers numerous topics on using fonts with TeX and DVIPS. It was fun to write and program but it certainly absorbed many hours of my evenings and weekends. In some areas I've had to omit some finer details because it would make the article way too long and I'd probably run out of steam and never finish it: think of it as a "getting started" tutorial. I hope it is useful and interesting. Now to get on with some of those household tasks I've put off whilst writing this – and thanks to my partner, Alison Tovey, who has waited patiently (well, almost ) whilst I was glued to WordPress!

Introduction

Modern TeX(-based) engines, such as XeTeX and LuaTeX, provide direct access to using OpenType fonts, albeit using different philosophies/methods. This post looks at just one way to use TrueType-flavoured OpenType fonts with the traditional TeX–DVIPS–PostScript–PDF workflow which is usually associated with the 8-bit world of Type 1 PostScript fonts. The idea is that we'll convert TrueType-flavoured OpenType fonts to Type 42 PostScript fonts and include the Type 42 font data into DVIPS's PostScript output stream using the DVIPS -h filename mechanism. In addition, we'll look at using font encoding and the creation of TeX Font Metrics to enable access to the rich set of glyphs in a modern TrueType-flavour OpenType font.

Many Truetype-flavoured OpenType fonts (and thus the resulting Type 42 PostScript font) contain hundreds, if not thousands, of glyphs – making the 8-bit world of the traditional PostScript Encoding Vector little more than a tiny window into the rich array of available glyphs. By re-encoding the base Type 42 font we can generate a range of 256-character fonts for TeX and DVIPS to exploit the full range of glyphs in the original TrueType font – such as a true small caps font if the TrueType font has them.

We will also need to create the TeX Font Metrics (TFMs) so that TeX can access the metric data describing our fonts – the width, height, depth plus any kerning and linatures we care to add. Of course, the virtual font mechanism is also a valid approach – see Virtual Fonts: More Fun for Grand Wizards for more details. Much of what we're doing here uses a number of freely available software tools to extract key data from the actual OpenType font files for onward processing into a form suitable for TeX.

Context of these experiments

Over the past few weeks I've spent some evenings and weekends building TeX and friends from WEB source code, using Microsoft's Visual Studio. At the moment, this all resides in a large Visual Studio project containing all the various applications and is a little "Heath Robinson" at the moment, although it does work. Within each of my builds of TeX and friends I've replaced the venerable Kpathsea path/file-searching library with my one of own creation – which does a direct search using recursive directory traversal. I'm also toying with using database-lookup approach, hence the appearance of SQLite in the list of C libries within the screenshot.

Turning to Eddie Kohler's marvellous LCDF Typetools collection, I used MinGW/MSYS to build this. LCDF Typetools contains some incredibly useful tools for working with fonts via TeX/DVIPS – including ttftotype42 which can generate a Type 42 PostScript font from TrueType-flavoured OpenType fonts. You can think of a Type 42 font as a PostScript "wrapper" around the native TrueType font data, allowing you to insert TrueType fonts into PostScript code.

Characters, glyphs, glyph names, encodings and glyph IDs

Firstly, we need to review several interrelated topics: characters, glyphs, glyph names, encodings and glyph IDs (contained in OpenType fonts). Let's begin by thinking about characters. A character can be considered as the fundamental building block of a language: it is, if you like, an "atomic unit of communication" (spoken or not) which has a defined role and purpose: the character's meaning (semantics). Most characters usually need some form visual representation; however, that visual representation may not be fixed: most characters of a human spoken/written language can be represented in different forms. For example, the character 'capital H' (H) can take on different visual appearances depending on the font you use to display it. Fonts come in different designs and each design of our 'capital H' is called a glyph: a specific visual design which is particular to the font used to represent the 'capital H'. Each character that a font is capable of displaying will have a glyph designed to to represent it – not only that but you may have a fancy font that contains multiple representations for a particular character: small caps, italic, bold and so forth. Each of these variants uses a different glyph to represent the same character: they still represent the same fundamental "unit of meaning" (a character) just using different visual forms of expression (glyphs).

If we look around us we see, of course, that there are hundreds of languages in our world and if we break these languages down into their core units of expression/meaning we soon find that many thousands of characters are needed to "define" or encompass these languages. So, how do we go about listing these characters and, more to the point, communicating in these languages through e-mails, text files, printed documents and so forth? As humans we refer to characters by a name (e.g., 'capital H') but computers, obviously, deal with numbers. To communicate our characters by computer we need a way to allocate an agreed set of numbers to those characters so that we can store or transmit them electronically. And that's called the encoding. An encoding is simply an agreed set of numbers assigned to an agreed set of characters – so that we can store those numbers and know that our software will eventually display the correct glyphs to provide visual expression of our characters. To communicate using numbers to represent characters both sides have to agree on the encoding (mapping of numbers to characters) being used. If I save my text file (a bunch of numbers) and you open it up then your software must interpret those numbers in the same way I did when I wrote the text. Clearly, it's essential for encoding standards to exist and perhaps the most well known is, of course, the Unicode standard which allocates a unique number to well over 100,000 characters (at present), with new characters being added from time-to-time as the Uniciode standard is updated.

Let's take closer at fonts. We've seen that the job of a font is to provide the glyphs which represent a certain set of characters. Naturally, any particular font will only contain glyphs to represent a small subset of the world's characters: there are just too many for any single font to contain them all. We've also said that some fonts may contain multiple glyphs to represent the same character. Considering OpenType fonts for the moment, within each font the individual glyphs (designs representing a specific chartacter) are each given a name and a numeric identifier, called the glyph identifier (also called the index or glyph ID). Each glyph is thus described by a (name, glyph ID) pair. It's really important to realise that the glyph ID has nothing to do with encoding of characters: it is just an internal bookkeeping number used within the font and assigned to each glyph by the font's creator. The numeric IDs assigned to a particular glyph are not defined by a global standard. Furthermore, the names given to glyphs also show a great deal of variation too, although there are some attempts at standardizing them: see the Adobe Glyph List which aims to provide a standard naming convention.

Let's recap. We've seen that the fundamental "unit of communication" is the character and that characters are encoded by assigning each one to a number. We've also seen that fonts contain the designs, called glyphs, which represent the characters supported by the font. Internally, each (OpenType) font assigns every glyph an identifier (glyph ID) and a glyph name which may, or may not, be "standard".

So, the next question we need to think about is: given a text file containing characters represented (stored) according to a specific encoding (a set of numbers), how does any font actually know how to map from a certain character in the text file to the correct glyph to represent it? After all, the encoding in the text file is usually based on a standard but the data in our font, glyph IDs and glyph names, are not standard? Well, not surprisingly there is indeed some extra bit of data inside the font which provides the glue and this is called the Encoding Vector (in older PostScript fonts) or character map (CMAP) table within the modern world of Unicode and OpenType fonts. The job of the Encoding Vector (or character map (CMAP)) is to provide the link between the standard world of encoded characters to the (relatively) non-standard inner font world of glyph IDs and glyph names.

A sneak peek at GentiumPlus-R: 5586 glyphs in a single font

For the remainder of this post I'll use the free Gentium OpenType font (GentiumPlus-R) as an example because I do not want to inadvertantly infringe any commercial licence conditions in the work below. To help solidify the ideas described above I generated a table of all the glyphs (plus glyph ID and glyph name) contained within the GentiumPlus-R TrueType-flavour OpenType font.

GentiumPlus-R glyph chart

Technical details: To generate these glyph tables I wrote a command-line utility (in C) which used the FreeType library to extract the low-level data from inside the OpenType font. This data was written out as a PostScript program which loops over all the glyphs: drawing each glyph together with its glyph ID and name. This PostScript program was combined with the GentiumPlus (TrueType) font after converting it to a Type 42 PostScript font using ttftotype42 compiled from the source code distributed as part of the wonderful LCDF Typetools collection.

Download PDF

PostScript Encoding Vectors

Let's recap on our objectives. We've explored the idea of glyphs, characters and encodings and seen that OpenType fonts can contain many thousands of glyphs to display thousands of characters. However, OpenType fonts can't easily be used within the traditional TeX–DVIPS–PostScriptS–PDF workflow: most traditional TeX workflows use 8-bit characters and Type 1 PostScript fonts. As yet, we've still not explained exctly how a character code is "mapped" to a specific glyph in a font. So, it's time to look at this, focussing on Type 1 and Type 42 PostScript fonts, ignoring OpenType fonts. The "magic glue" we need to explore is the so-called Encoding Vector present in Type 1 and Type 42 fonts. The job of the Encoding Vector is to map from character codes in the input to glyphs contained in the font. Let's look at an example to make this clearer. I'll assume that you have access to the ttftotype42 utility from the LCDF Typetools collection. If you don't have it, or can't compile it, contact me and I'll e-mail my compiled version to you.

Using ttftotype42

If you run ttftotype42 on a TrueType-flavour OpenType font it will generate a fairly large plain text file which you can inspect with any text editor, so let's do that. In these examples I'll use the free Gentium OpenType font.

If you download GentiumPlus and place the GentiumPlus-R.ttf file in the same directory as ttftotype42 and run

ttftotype42 GentiumPlus-R.ttf GentiumPlus.t42

you should generate a file GentiumPlus.t42 which is a little over 2MB in size – remember, the GentiumPlus font contains over 5,500 glyphs! Loosely speaking you can think of the Type 42 font generated by ttftotype42 as being made up from the following sections:

• PostScript header
• Encoding Vector
• /sfnts glyph data array
• /CharStrings dictionary
• PostScript trailer

Download GentiumPlus.t42: I uploaded the Type 42 font file GentiumPlus.t42 created by ttftotype42 onto this site: you can download it here.

Here's an extract from the Type 42 font version of GentiumPlus-R.ttf with vast amouts of data snipped out for brevity:

%!PS-TrueTypeFont-65536-98828-1
%%VMusage: 0 0
11 dict begin
/FontName /GentiumPlus def
/FontType 42 def
/FontMatrix [1 0 0 1 0 0] def
/FontBBox [-0.676758 -0.463867 1.49951 1.26953] readonly def
/PaintType 0 def
/XUID [42 16#30C4BB 16#E5CA1A 16#75CC0A 16#BE5D07 16#47E1FB 16#4C] def
/FontInfo 10 dict dup begin
/version (Version 1.510) readonly def
/Notice (Gentium is a trademark of SIL International.) readonly def
/Copyright (Copyright \(c\) 2003-2012, SIL International \(http://scripts.sil.org/\)) readonly def
/FullName (Gentium Plus) readonly def
/FamilyName (Gentium Plus) readonly def
/Weight (Regular) readonly def
/isFixedPitch false def
/ItalicAngle 0 def
/UnderlinePosition -0.146484 def
/UnderlineThickness 0.0488281 def
end readonly def
/Encoding 256 array
0 1 255{1 index exch/.notdef put}for
dup 13 /nonmarkingreturn put
dup 32 /space put
dup 33 /exclam put
dup 34 /quotedbl put
dup 35 /numbersign put
dup 36 /dollar put
dup 37 /percent put
dup 38 /ampersand put
...
...
-- snipped lots of lines of the encoding vector --
...
...
dup 254 /thorn put
dup 255 /ydieresis put
readonly def
/sfnts[
<00010000.......
...
...
-- snipped vast amounts of glyph data --
...
...
] def
/CharStrings 5586 dict dup begin
/.notdef 0 def
/.null 1 def
/nonmarkingreturn 2 def
/space 3 def
/exclam 4 def
/quotedbl 5 def
/numbersign 6 def
...
...
-- snipped vast amounts of CharStrings data --
...
...
end readonly def
FontName currentdict end definefont pop

The section of interest here is the Encoding Vector which is reproduced in full:

/Encoding 256 array
0 1 255{1 index exch/.notdef put}for
dup 13 /nonmarkingreturn put
dup 32 /space put
dup 33 /exclam put
dup 34 /quotedbl put
dup 35 /numbersign put
dup 36 /dollar put
dup 37 /percent put
dup 38 /ampersand put
dup 39 /quotesingle put
dup 40 /parenleft put
dup 41 /parenright put
dup 42 /asterisk put
dup 43 /plus put
dup 44 /comma put
dup 45 /hyphen put
dup 46 /period put
dup 47 /slash put
dup 48 /zero put
dup 49 /one put
dup 50 /two put
dup 51 /three put
dup 52 /four put
dup 53 /five put
dup 54 /six put
dup 55 /seven put
dup 56 /eight put
dup 57 /nine put
dup 58 /colon put
dup 59 /semicolon put
dup 60 /less put
dup 61 /equal put
dup 62 /greater put
dup 63 /question put
dup 64 /at put
dup 65 /A put
dup 66 /B put
dup 67 /C put
dup 68 /D put
dup 69 /E put
dup 70 /F put
dup 71 /G put
dup 72 /H put
dup 73 /I put
dup 74 /J put
dup 75 /K put
dup 76 /L put
dup 77 /M put
dup 78 /N put
dup 79 /O put
dup 80 /P put
dup 81 /Q put
dup 82 /R put
dup 83 /S put
dup 84 /T put
dup 85 /U put
dup 86 /V put
dup 87 /W put
dup 88 /X put
dup 89 /Y put
dup 90 /Z put
dup 91 /bracketleft put
dup 92 /backslash put
dup 93 /bracketright put
dup 94 /asciicircum put
dup 95 /underscore put
dup 96 /grave put
dup 97 /a put
dup 98 /b put
dup 99 /c put
dup 100 /d put
dup 101 /e put
dup 102 /f put
dup 103 /g put
dup 104 /h put
dup 105 /i put
dup 106 /j put
dup 107 /k put
dup 108 /l put
dup 109 /m put
dup 110 /n put
dup 111 /o put
dup 112 /p put
dup 113 /q put
dup 114 /r put
dup 115 /s put
dup 116 /t put
dup 117 /u put
dup 118 /v put
dup 119 /w put
dup 120 /x put
dup 121 /y put
dup 122 /z put
dup 123 /braceleft put
dup 124 /bar put
dup 125 /braceright put
dup 126 /asciitilde put
dup 160 /uni00A0 put
dup 161 /exclamdown put
dup 162 /cent put
dup 163 /sterling put
dup 164 /currency put
dup 165 /yen put
dup 166 /brokenbar put
dup 167 /section put
dup 168 /dieresis put
dup 169 /copyright put
dup 170 /ordfeminine put
dup 171 /guillemotleft put
dup 172 /logicalnot put
dup 173 /uni00AD put
dup 174 /registered put
dup 175 /macron put
dup 176 /degree put
dup 177 /plusminus put
dup 178 /twosuperior put
dup 179 /threesuperior put
dup 180 /acute put
dup 181 /mu put
dup 182 /paragraph put
dup 183 /periodcentered put
dup 184 /cedilla put
dup 185 /onesuperior put
dup 186 /ordmasculine put
dup 187 /guillemotright put
dup 188 /onequarter put
dup 189 /onehalf put
dup 190 /threequarters put
dup 191 /questiondown put
dup 192 /Agrave put
dup 193 /Aacute put
dup 194 /Acircumflex put
dup 195 /Atilde put
dup 196 /Adieresis put
dup 197 /Aring put
dup 198 /AE put
dup 199 /Ccedilla put
dup 200 /Egrave put
dup 201 /Eacute put
dup 202 /Ecircumflex put
dup 203 /Edieresis put
dup 204 /Igrave put
dup 205 /Iacute put
dup 206 /Icircumflex put
dup 207 /Idieresis put
dup 208 /Eth put
dup 209 /Ntilde put
dup 210 /Ograve put
dup 211 /Oacute put
dup 212 /Ocircumflex put
dup 213 /Otilde put
dup 214 /Odieresis put
dup 215 /multiply put
dup 216 /Oslash put
dup 217 /Ugrave put
dup 218 /Uacute put
dup 219 /Ucircumflex put
dup 220 /Udieresis put
dup 221 /Yacute put
dup 222 /Thorn put
dup 223 /germandbls put
dup 224 /agrave put
dup 225 /aacute put
dup 226 /acircumflex put
dup 227 /atilde put
dup 228 /adieresis put
dup 229 /aring put
dup 230 /ae put
dup 231 /ccedilla put
dup 232 /egrave put
dup 233 /eacute put
dup 234 /ecircumflex put
dup 235 /edieresis put
dup 236 /igrave put
dup 237 /iacute put
dup 238 /icircumflex put
dup 239 /idieresis put
dup 240 /eth put
dup 241 /ntilde put
dup 242 /ograve put
dup 243 /oacute put
dup 244 /ocircumflex put
dup 245 /otilde put
dup 246 /odieresis put
dup 247 /divide put
dup 248 /oslash put
dup 249 /ugrave put
dup 250 /uacute put
dup 251 /ucircumflex put
dup 252 /udieresis put
dup 253 /yacute put
dup 254 /thorn put
dup 255 /ydieresis put
readonly def

The Encoding Vector is an array indexed by a number which runs from 0 to 255 and the value stored at each index position is the name of a glyph contained in the font. You have probably guessed that the index (0 to 255) is the numeric value of an input character. So, via the Encoding Vector with 256 potential character values as input, we can reach up to 256 individual glyphs contained in the font. (Note: I'm ignoring the PostScript glyphshow operator which allows access to any glyph if you know its name).

The full story (quoting from the Type 42 font specification) "The PostScript interpreter uses the /Encoding array to look up the character name, which is then used to access the /Charstrings entry with that name. The value of that entry is the glyph index, which is then used to retrieve the glyph description."

However, there are 5586 glyphs in GentiumPlus so does this mean the remaining 5330 glyphs are wasted and unreachable? Of course that's not true but we can only reach 256 glyphs via each individual Encoding Vector: the trick we need is font re-encoding. The Encoding Vector is not a fixed entity: you can amend it or replace it entirely with a new one to map character codes 0 to 255 to different glyphs within the font. I won't give the full details here, although it's quite simple to understand. What you do, in effect, is a bit of PostScript programming to "clone" some of the font data structures, give this "clone" a new PostScript font name and a new Encoding Vector which maps the 256 character codes to totally different glyphs. For some excellent tutorials on PostScript programming, including font re-encoding, I highly recommend reading the truly excellent Acumen Training Journal which is completely free. Specifically, November 2001 and December 2001 issues.

If you want a simple example to explore the ideas behind Encoding Vectors you can download this code example (with PDF) to see the results of re-encoding Times-Roman.

Hooking this up to TeX and DVIPS

Having discussed fonts, encoding and glyphs at some length we now move to the next task: how do we use these ideas with TeX and DVIPS? Let's start with TeX. Here, I'm referring to the traditional TeX workflows that use TeX Font Metric (TFM) files. So what is a TFM? To do its typesetting work TeX's algorithms need only some basic information about the font you want to use: it needs the metrics. TeX does not care about the actual glyphs in your font or what they look like, it needs a set of data that describes how big each glyph is: to TeX your glyphs are boxes with a certain width, depth and height. That's not the whole story, of course, because TeX also needs some additional data called fontdimens which are a set of additional parameters that describe some overall characteristics of the font. For pure text fonts there are 7 of these fontdimens, for math fonts there are 13 or 22 depending on the type/role of the math font. These fontdimens are, of course, built into the TFM file.

Looking inside TFMs

TFM files are a highly compact binary file format and quite unsuitable for viewing or editing. However, you can convert a TFM file to a readable/editable text representation using a program called tftopl, which is part of most TeX distributions. The editable text version of a TFM is referred to as a property list file. At the start of a TFM file for a text font (e.g., cmr10.tfm) you should see the 7 fontdimens displayed like this:

(FONTDIMEN
   (SLANT R 0.0)
   (SPACE R 0.333334)
   (STRETCH R 0.166667)
   (SHRINK R 0.111112)
   (XHEIGHT R 0.430555)
   (QUAD R 1.000003)
   (EXTRASPACE R 0.111112)
   )

If you run tftopl on cmex10.tfm (math font with extensible symbols) you see 13 fontdimens displayed like this:

(FONTDIMEN
   (SLANT R 0.0)
   (SPACE R 0.0)
   (STRETCH R 0.0)
   (SHRINK R 0.0)
   (XHEIGHT R 0.430556)
   (QUAD R 1.0)
   (EXTRASPACE R 0.0)
   (DEFAULTRULETHICKNESS R 0.04)
   (BIGOPSPACING1 R 0.111111)
   (BIGOPSPACING2 R 0.166667)
   (BIGOPSPACING3 R 0.2)
   (BIGOPSPACING4 R 0.6)
   (BIGOPSPACING5 R 0.1)
   )

If you run tftopl on cmsy10.tfm (math symbol font) you see 22 fontdimens displayed like this:

(FONTDIMEN
   (SLANT R 0.25)
   (SPACE R 0.0)
   (STRETCH R 0.0)
   (SHRINK R 0.0)
   (XHEIGHT R 0.430555)
   (QUAD R 1.000003)
   (EXTRASPACE R 0.0)
   (NUM1 R 0.676508)
   (NUM2 R 0.393732)
   (NUM3 R 0.443731)
   (DENOM1 R 0.685951)
   (DENOM2 R 0.344841)
   (SUP1 R 0.412892)
   (SUP2 R 0.362892)
   (SUP3 R 0.288889)
   (SUB1 R 0.15)
   (SUB2 R 0.247217)
   (SUPDROP R 0.386108)
   (SUBDROP R 0.05)
   (DELIM1 R 2.389999)
   (DELIM2 R 1.01)
   (AXISHEIGHT R 0.25)
   )

The role of these fontdimens within math fonts is extremely complex. If you want to read about this in depth you can find a list of excellent articles in this post. In addition to the glyph metrics (height, width, depth) and fontdimens TFM files contain constructs for kerning and ligatures. There's a lot of information already available on the inner details of TFMs so there's no point repeating it here.

The bulk of a TFM file is concerned with providing the height, width and depth of the characters encoded into the TFM. And that brings up a very important point: individual TFM files are tied to a particular encoding. For example, right at the start of a cmr10.tfm file you should see something like this:

(FAMILY CMR)
(FACE O 352)
(CODINGSCHEME TEX TEXT)
(DESIGNSIZE R 10.0)
(COMMENT DESIGNSIZE IS IN POINTS)
(COMMENT OTHER SIZES ARE MULTIPLES OF DESIGNSIZE)
(CHECKSUM O 11374260171)

It contains the line (CODINGSCHEME TEX TEXT) telling you that the TFM is encoded using the TeX Text encoding scheme. Let's examine this. Referring back to our discussion of PostScript Encoding Vectors, let's take a look at the first few lines of the Encoding Vector sitting inside the Type 1 font file for cmr10 – i.e., cmr10.pfb. The first 10 positions are encoded like this:

dup 0 /Gamma put
dup 1 /Delta put
dup 2 /Theta put
dup 3 /Lambda put
dup 4 /Xi put
dup 5 /Pi put
dup 6 /Sigma put
dup 7 /Upsilon put
dup 8 /Phi put
dup 9 /Psi put
dup 10 /Omega put

And this is the key point: the character encoding in your TFM file has to match the encoding of your PostScript font (or a re-encoded version of it). If we look at the metric data for the corresponding characters encoded in the cmr10.tfm file we find:

(CHARACTER O 0
   (CHARWD R 0.625002)
   (CHARHT R 0.683332)
   )
(CHARACTER O 1
   (CHARWD R 0.833336)
   (CHARHT R 0.683332)
   )
(CHARACTER O 2
   (CHARWD R 0.777781)
   (CHARHT R 0.683332)
   )
(CHARACTER O 3
   (CHARWD R 0.694446)
   (CHARHT R 0.683332)
   )
(CHARACTER O 4
   (CHARWD R 0.666669)
   (CHARHT R 0.683332)
   )
(CHARACTER O 5
   (CHARWD R 0.750002)
   (CHARHT R 0.683332)
   )
(CHARACTER O 6
   (CHARWD R 0.722224)
   (CHARHT R 0.683332)
   )
(CHARACTER O 7
   (CHARWD R 0.777781)
   (CHARHT R 0.683332)
   )
(CHARACTER O 10
   (CHARWD R 0.722224)
   (CHARHT R 0.683332)
   )
(CHARACTER O 11
   (CHARWD R 0.777781)
   (CHARHT R 0.683332)
   )
(CHARACTER O 12
   (CHARWD R 0.722224)
   (CHARHT R 0.683332)
   )

Statements such as CHARACTER O 0 describe the metrics (just width and height in these examples) for the character with octal value 0, CHARACTER O 12 describes character with octal value 12 (i.e., 10 in denary (base 10)). Note that the values are relative to the (DESIGNSIZE R 10.0) which means, for example, that CHARACTER O 12 has a width of 0.722224 × 10 = 7.22224 points – because the DESIGNSIZE is 10 points. So, it is clearly vital that the encoding of your TFM matches the encoding of your PostScript font otherwise you'll get the wrong glyphs on output and the wrong widths, heights and depths used by TeX's typesetting calculations!

Using FreeType to generate raw metric data

FreeType is a superb C library which provides a rich set of functions to access many internals of a font, together, of course, with functions to rasterize fonts for screen display. Just to note, FreeType does not provide an OpenType shaping engine, for that you'll need to use the equally superb libotf C library (which also uses FreeType). However, I digress. Using FreeType you can create some extremely useful and simple utilities to extract a wide range of data from font files to generate raw data for creating the TFM files and Encoding Vectors you'll need to hook-up a Type 42 font to DVIPS and TeX. Let's look at this is a little detail. The task at hand is: given an OpenType (TrueType) font, how can do you obtain details of the glyphs it contains: the names and metrics (width, height, depth)?

FreeType's view of glyph metrics

The FreeType API provides access to the glyph metrics shown in the FreeType Glyph Conventions documentation. You should read this together with the Adobe's Type 1 Font Format Specification (Chapter 3) to make sure you understand what is meant by a glyph's width.

Simple examples of using the FreeType API

Here's some ultra-basic examples, without any proper error checking etc, to show how you might use FreeType. You start by initializing the FreeType library (FT_Init_FreeType(...)), then create a new face object (FT_New_Face(...)) and use this to access the font and glyph details you need. The first example writes metric data to STDOUT, the second example processes the font data to create an Encoding Vectors and a skeleton property list file for creating a TFM. Note that is a "bare bones" TFM and does not generate any ligatures or kerning data. To generate a binary TFM from a property list file you need another utility called pltotf which is also part of most TeX distributions.

#include <windows.h>
#include <ft2build.h>
#include <freetype/t1tables.h>
#include <freetype/ftoutln.h>
#include <freetype/ftbbox.h>
#include FT_GLYPH_H
#include FT_FREETYPE_H

int main (int  ac,  char** av)
{
    FT_Library font_library;
    FT_Face    font_face;
    FT_BBox    bbox;

    int  glyph_index;
    int  glyph_count;
    char char_name[256];
    const char* fontfilepath = "PUT THE PATH TO YOUR FONT HERE";
    char buf[5];
    int err=1;

   if (FT_Init_FreeType( &font_library ) )
	{
	// Failed to init library,
	} else
	{
		if ( FT_New_Face( font_library, fontfilepath, 0 , &font_face ) )
		{
			// Managed to open library but failedto open the font
			FT_Done_FreeType(font_library);
			return err;
		}
		else {
			// library and font opened OK
			// find out the number of glyphs and process each glyph
			glyph_count = font_face->num_glyphs;
			for ( glyph_index = 0 ; glyph_index < glyph_count; glyph_index++ )
				{
                                // NOTE: FT_Get_Glyph_Name can FAIL for some TrueType-flavour
                                // OpenType fonts so you *really* do need to check the value of err!!
				err = FT_Get_Glyph_Name(font_face, glyph_index, &char_name[0], 32 );
				_itoa(glyph_index, buf, 10);
				// load the glyph with no scaling etc to get raw data
				FT_Load_Glyph(font_face,  glyph_index,  FT_LOAD_NO_SCALE);
				// get the bounding box of the raw glyph data
				FT_Outline_Get_BBox(&(font_face->glyph->outline), &bbox);
				printf( "/%s %ld def ",   char_name, glyph_index);
				printf("width=%ld ", font_face->glyph->metrics.width);
				printf("height=%ld ", font_face->glyph->metrics.height);
				printf("horiAdvance=%ld ", font_face->glyph->metrics.horiAdvance);
				printf("horiBearingX=%ld ", font_face->glyph->metrics.horiBearingX);
				printf("horiBearingY=%ld ", font_face->glyph->metrics.horiBearingY);
				printf("vertAdvance=%ld ",  font_face->glyph->metrics.vertAdvance);
				printf("vertBearingX=%ld ", font_face->glyph->metrics.vertBearingX);
				printf("vertBearingY=%ld ", font_face->glyph->metrics.vertBearingY);
				printf("xMax=%ld ", bbox.xMax);
				printf("yMax=%ld ", bbox.yMax);
				printf("yMin=%ld ", bbox.yMin);
				printf("xMin=%ld \n", bbox.xMin);
			}
		}
		FT_Done_FreeType(font_library);
	}
}

Creating an Encoding Vector and property list file

The following simple-minded function shows how you might use FreeType to generate an Encoding Vector and property list file. Reflecting the unusual glyphs we're using, the output files are called weirdo.pl and weirdo.enc.

void makeweirdo(FT_Face font_face, char *name, int len)
{
FILE * vec;
FILE * plist;
int i;
FT_BBox bbox;
int k=32; // only encode positions 32--255

char *fname="your_path_here\\weirdo.enc";
char *pname="your_path_here\\weirdo.weirdo.pl";
char * header= "(COMMENT Created by Graham Douglas)\r\n\
(FAMILY WEIRDO)\r\n\
(CODINGSCHEME WEIRDO)\r\n\
(DESIGNSIZE R 10.0)\r\n\
(FONTDIMEN\r\n\
   (SLANT R 0.0)\r\n\
   (SPACE R 0.333334)\r\n\
   (STRETCH R 0.166667)\r\n\
   (SHRINK R 0.111112)\r\n\
   (XHEIGHT R 0.430555)\r\n\
   (QUAD R 1.000003)\r\n\
   (EXTRASPACE R 0.111112)\r\n\
   )\r\n";
	vec = fopen(fname, "wb");
	plist = fopen(pname, "wb");

	fprintf(vec,"%s", "/veccy 256 array 0 1 255 {1 index exch /.notdef put} for\r\n");
	fprintf(plist,"%s", header);

	// Here we are looping over GentiumPlus glyph IDs whose value is 5000 to 5223
	for (i=5000; i<5224; i++)
	{
		FT_Get_Glyph_Name(font_face, i, name, len);
		FT_Load_Glyph(font_face,  i,  FT_LOAD_NO_SCALE);
		FT_Outline_Get_BBox(&(font_face->glyph->outline), &bbox);
		fprintf(plist,"(CHARACTER O %o (COMMENT Glyph name is %s)\r\n", k, name);
		fprintf(plist,"    (CHARWD R %.5f)\r\n", font_face->glyph->metrics.horiAdvance/2048.0);
		fprintf(plist,"    (CHARHT R %.5f)\r\n", bbox.yMax/2048.0);
                // FreeType's depth values are negative, TeX Font Metrics are not
                // If bbox.yMin not negative then we don't output anything and TeX assumes zero depth
		if(bbox.yMin < 0)
		{
			fprintf(plist,"    (CHARDP R %.5f)\r\n", -1*bbox.yMin/2048.0);
		}
		fprintf(plist, "%s","    )\r\n"); 

	fprintf(vec, "dup %ld /%s put\r\n", k, name);
	k++;
	}
	fprintf(vec, "%s", " def\r\n");
	fclose(vec);
	fclose(plist);
}

Here is a small extract from weirdo.vec and weirdo.pl – if you wish to explore the output you can download them (and weirdo.tfm) in this zip file. (In the data below I followed the neat idea from LCDF Typetools and put the glyph name in as a comment).

weirdo.vec

/veccy
256 array 0 1 255 {1 index exch /.notdef put} for
dup 32 /uni1D9C.Dep50 put
dup 33 /uni1D9C.Dep41 put
dup 34 /uni023C.Dep51 put
dup 35 /uni023C.Dep50 put
....
dup 251 /uni024C.Dep51 put
dup 252 /uni024C.Dep50 put
dup 253 /uni2C64.Dep51 put
dup 254 /uni2C64.Dep50 put
dup 255 /uni1DB3.Dep51 put
def

weirdo.pl

(COMMENT Created by Graham Douglas)
(FAMILY WEIRDO)
(CODINGSCHEME WEIRDO)
(DESIGNSIZE R 10.0)
(FONTDIMEN
   (SLANT R 0.0)
   (SPACE R 0.333334)
   (STRETCH R 0.166667)
   (SHRINK R 0.111112)
   (XHEIGHT R 0.430555)
   (QUAD R 1.000003)
   (EXTRASPACE R 0.111112)
   )
(CHARACTER O 40 (COMMENT Glyph name is uni1D9C.Dep50)
    (CHARWD R 0.30566)
    (CHARHT R 0.59863)
    )
(CHARACTER O 41 (COMMENT Glyph name is uni1D9C.Dep41)
    (CHARWD R 0.30566)
    (CHARHT R 0.59863)
    )
(CHARACTER O 42 (COMMENT Glyph name is uni023C.Dep51)
    (CHARWD R 0.43701)
    (CHARHT R 0.55811)
    (CHARDP R 0.09033)
    )
(CHARACTER O 43 (COMMENT Glyph name is uni023C.Dep50)
    (CHARWD R 0.43701)
    (CHARHT R 0.55811)
    (CHARDP R 0.09033)
    )
(CHARACTER O 44 (COMMENT Glyph name is uni023C.Dep41)
    (CHARWD R 0.43701)
    (CHARHT R 0.55811)
    (CHARDP R 0.09033)
    )
(CHARACTER O 45 (COMMENT Glyph name is uni1D9D.Dep51)
    (CHARWD R 0.30566)
    (CHARHT R 0.59863)
    )
(CHARACTER O 46 (COMMENT Glyph name is uni1D9D.Dep50)
    (CHARWD R 0.30566)
    (CHARHT R 0.59863)
    )

...
...
...

(CHARACTER O 375 (COMMENT Glyph name is uni2C64.Dep51)
    (CHARWD R 0.56104)
    (CHARHT R 0.64453)
    (CHARDP R 0.20020)
    )
(CHARACTER O 376 (COMMENT Glyph name is uni2C64.Dep50)
    (CHARWD R 0.56104)
    (CHARHT R 0.64453)
    (CHARDP R 0.20020)
    )
(CHARACTER O 377 (COMMENT Glyph name is uni1DB3.Dep51)
    (CHARWD R 0.27002)
    (CHARHT R 0.59863)
    )

To generate the binary TFM file weirdo.tfm from weirdo.pl run pltotf:

pltotf weirdo.pl weirdo.tfm
I had to round some heights by 0.0002451 units.

I got a warning from pltotf, but I don't think it is too serious. To use the TFM you'll need to put it in a suitable location within your TEXMF tree.

Hooking up to DVIPS

We've covered a huge range of topics so it is time to recap. So far, we've generated an Encoding Vector (weirdo.vec) based on the names of glyphs (in the Gentium-Plus font) whose glyph IDs span the range 5000–5523. Within our Encoding Vector we mapped those glyph names to the character codes 32–255. We have also created a property list file, based on the same encoding, which simply contains the width, height and depth of the Gentium-Plus glyphs in the range 5000–5523. Our next task is to pull together the following items and convince DVIPS to use them.

1. Re-encode GentiumPlus.t42: We need to create a re-encoded font that uses our new Encoding Vector (weirdo.vec).
2. Update config.ps: We need to tell DVIPS how to use our new font by creating a .map file and making sure DVIPS can find that map file.
3. Command-line switches: We'll need to use some command-line switches to give DVIPS the info it needs to do its job.
4. Our Type 42 font: GentiumPlus.t42: We must tell DVIPS to embed that font into its PostScript output.

Our goal is to tell TeX to load a font (TFM) called weirdo and for DVIPS to know how to use and find the weirdo font data to generate the correct PostScript code to render our glyphs. We'll use our strange new weirdo font like this (in plain TeX):

\font\weird=weirdo {\weird HELLO}

Note that the displayed output will not be the English word "HELLO" because we've chosen some rather strange glyphs from Gentium-Plus. The key observation is the input character codes are the ASCII values of the string HELLO; i.e. (in base 10):


H = 72
E = 69
L = 76
L = 76
O = 79


and our weirdo.enc Encoding Vector maps these character codes to the following glyphs:


72 = uni1D94.Dep51
69 = uni1D9F.Dep51
76 = uni0511.Dep50
79 = uni0510.Dep51


So, we can expect some strange output in the final PostScript or PDF file!

How do we do the re-encoding?

The basic idea is that we tell DVIPS to embed the GentiumPlus.t42 PostScript Type 42 font data into its PostScript output stream. We will then write some short PostScript headers that will do the re-encoding to generate our newly re-encoded font: which we're calling weirdo. By using the DVIPS -h command-line switch we can get DVIPS to embed GentiumPlus.t42 and the header PostScript file to perform the re-encoding. For example:

DVIPS -h GentiumPlus.t42 -h weirdo.ps sometexfile.dvi

The actual re-encoding, and "creation", of our weirdo font will be taken care of by the file weirdo.ps, which will also need to contain the weirdo.enc data. If you wish, you can download weirdo.ps. Here is the tiny fragment of PostScript required within weirdo.ps to "create" the weirdo font by re-encoding our Type 42 font whose PostScript name is GentiumPlus.

/otfreencode
{
	findfont dup length dict copy
	dup 3 2 roll /Encoding exch put
	definefont
	pop
} bind def

/weirdo veccy /GentiumPlus otfreencode

Note, of course, you could create a header PostScript file to generate multiple new fonts each with their own unique Encoding Vectors containing a range of glyphs from the Type 42 font.

Telling DVIPS how to use our new font

So far we've built the TFM file for TeX so now we need to tell DVIPS how to use it – so that it can process the weirdo font name as it parses the DVI file. DVIPS uses .map files to associate TFM file names with PostScript font names, together the actions DVIPS needs to take in order to process the font files and get the right PostScript font data into its output. These actions include processing/parsing Type 1 font files (.pfb, .pfa) and re-encoding Type 1 fonts. For our weirdo font the .map file is very simple: all we need to do is create a file called weirdo.map with a single line:

weirdo weirdo

This super-simple .map file says that the TeX font name (TFM file) weirdo is mapped to a PostScript font called weirdo (as defined by the code in weirdo.ps). It also tells DVIPS that no other actions are needed because we're not doing the re-encoding, here nor are we asking DVIPS to process a Type 1 font file (.pfb) file associated with weirdo – because there isn't one! After you have created weirdo.map you'll need to edit the DVIPS's configuration file config.ps to tell DVIPS to use weirdo.map. Again, this is easy and all you need to do is add the following instruction to config.ps:

p +weirdo.map

Does it work?

Well, I'd have wasted many hours if it didn't . I used the following simple plain TeX example (test.tex) which I processed using my personal build of TeX for Windows (which does not use Kpathsea).

\hsize=300pt
\vsize=300pt
\font\smallweird=weirdo at 12pt
Dear \TeX\ I would like to say HELLO in weirdo so {\smallweird HELLO}. I would also like to see
a lot of strange glyphs so I'll input a text file containing  some of them: {\smallweird \input weirdchars }.
\bye

Download weirdchars.tex

The resulting DVI file was processed to PostScript using a standard build of DVIPS with the following command line:

DVIPS -h GentiumPlus.t42 -h weirdo.ps test.dvi

The resulting PostScript file is large because the GentiumPlus.t42 file is over 2MB. However, the PDF file produced by Acrobat Distiller was about 35KB because the Type 42 font (GentiumPlus.t42) was subsetted.

Download PDF

Concluding thoughts

"Alison, I'm ready to do the gardening. What?, it's too late. That's a shame."

This is just a short post to share a workaround to a problem I ran into when building Eddie Kohler's superb LCDF Typetools under Windows using MinGW. After running ./configure to create the make files I hit a problem during compilation with lots of error messages referring to undefined reference to `ntohl@4'


../typetools/libefont/otf.cc:863: undefined reference to `ntohl@4'
../typetools/libefont/otf.cc:861: undefined reference to `ntohs@4'
../typetools/libefont/otf.cc:861: undefined reference to `ntohs@4'


One solution

The cause of the error is failure to link to the library libwsock32.a (contained in the c:\MinGW\lib\ directory on my PC). The following workaround solves the problem but I'm sure there are better ways of doing it. Several tools within the Typetools collection depend on libwsock32.a to compile successfully. There are:

• otfinfo
• otftotfm
• cfftot1

To build these programs you need to make a small edit to the generated makefiles.

1. Create a directory called (say) libs within the Typetools directory tree.
2. Copy libwsock32.a into that directory.
3. For each application listed above, that depends on libwsock32.a, open the makefile in the appropriate application directory and look for a line starting with XXXXX_LDADD where XXXX is otfinfo or otftotfm or cfftot1
4. Edit that line to include libwsock32.a
5. Example: cfftot1_LDADD = ../libefont/libefont.a ../libs/libwsock32.a ../liblcdf/liblcdf.a

You should now be able to run make and achieve a successful compilation. It worked for me, I hope it works for you.

Over the last couple of evenings I've been looking at the C code for TeX generated by the tangle and Web2C conversion process. By default, the Web2C conversion process generates C source code which is almost completely devoid of comments and symbol strings are converted to numbers (etc), making the C source nearly impossible to read. However, by making a small change to the flex-generated source code (web2c-lexer.c) together with some careful use of regular expressions on the WEB sources (and/or some manual editing) you can get a lot of comments put into the C source. Note: I've not yet explored whether it is possible to use the changefile method to achieve the same (or similar) results. Here's an outline of my experimental technique.

Outline of the technique

Naturally, via the literate programming methodology, the WEB source files for TeX contain a full description of the TeX program. The Pascal code in the WEB source files is full of comments or short descriptions (enclosed in braces {....}) which, if preserved, (in the Web2C-generated C code), would make it much more readable. However, those Pascal comments are stripped out by tangle (but are used by weave); consequently, the Pascal generated by tangle, and fed into Web2C.exe, does not contain any useful comments. I say "useful" because the Pascal does contain some line-number comments but these are not really that helpful and they are removed by the comment-handling code in web2c-lexer.c.

Comments in WEB files

So, what are we looking to do? In essence, we need a way to convince tangle to output comments into the Pascal code it generates and find a way to ensure that those comments are processed and passed into the C code by Web2C.

Web2C and Pascal comments

Caveat: I am not a Pascal programmer and have no desire to become one! However, all you need to know is that within the Pascal generated by tangle the comments are simply enclosed in braces, like this: {This is a comment in Pascal}. These comments are filtered out by web2c-lexer.c. Another caveat: be extremely careful when making any changes whatsoever to the lexer code C (or .l sources), you can break things very badly (hmmm, wonder how I found that out...). I cannot stress the importance of being very, very careful in making any changes to web2c-lexer.l/web2c-parser.y or web2c-lexer.c/web2c-parser.c: these lexical analyser and parser-generator sources are critical to the C-generation process. OK, I think I made the point. The following description probably deserves nomination for an "Ugly Hack Award" and, no doubt, a flex/bison expert (which I'm definitely not) could design an elegant solution to incorporate comment-handling in proper context within the parsing process. OK, enough self-flagellation, let's move on.

If you look in Web2C-lexer.l the code which handles comments is simply:

"{" { while (webinput() != '}'); }

After running flex on Web2C-lexer.l this becomes (in Web2C-lexer.c)


case 2:
YY_RULE_SETUP
#line 53 "web2c-lexer.l"
{ while (webinput() != '}'); }
YY_BREAK


Basically, the lexical analyser is stripping out things like {This is a comment from Pascal}. To get comments into the C code generated for TeX you'll need to modify the lexer code to stop it skipping comments and process them to generate C comments such as /* This is a comment from Pascal */. There are a few points to consider here: firstly, you'll need to experiment to see exactly where the comments end up in your C code. Due to the "Ugly Hack" approach, we're not paying any real attention to the "context" of where we are in the parsing process when outputting our comments; again, a proper flex/bison implementation is required. For example, by the time your comment is seen by the lexer a newline (\n) may already have been output so your comments may end up on a new line – easily fixed by some manual tidy-up of the C code (or via the use of running regular-expression tools on the Web2C-generated C source code). So, just to note that you'll need to do some trial and error to see what happens.

Getting comments into WEB and surviving tangle

As noted, tangle strips out comments in the WEB sources and they don't even reach the Pascal code it outputs. So, can we coerce tangle to preserve comments in WEB soures and put them in the Pascal for Web2.exe to process/output? A quick reading of The WEB User Manual implies that there are two ways to get text output to the Pascal source code produced by tangle:

• use "control-text" such as @=your comment text here@> which causes the text to be output verbatim into the Pascal code, or
• use "meta-comments": such as @{your comment text here @} which, in the Pascal, results in a standard comment such as {your comment text here}.

Robby the Robot says Danger!

Sorry for the reference to Robby the Robot, indulge me..... Seriously, though, if you make edits to the WEB source to put in "control-text" or "meta-comments" you can very easily foul-up tangle's parser and break tangle's conversion process pretty badly. As yet, I'm not able to give precise rules on where it is safe to add "control-text" or "meta-comments" (I'm still experimenting) so I suggest you read The WEB User Manual to understand a little more about WEB syntax before attempting it.

Mind the pool file: Be careful inserting/using text with double quotes "..." because it can trigger tangle's parser to output that text in the tex.pool file which you don't want to do. I used single quotes '...' and that seems to be safe(er). I can't recall exactly what I did that caused this to happen, but just be sure to check that the .pool file does not become polluted with any of the text you insert into the WEB sources.

Getting to the point

So far we've seen that to get comments into the C source code we need to:

1. modify the behaviour of web2c-lexer.c and tell it (selectively) not to skip all Pascal's comment construct {...} (see use of '...', below).
2. coerce tangle to preserve comments and output them into the Pascal source so that Wb2C.exe sees them and the code in web2c-lexer.c can process them.

An example

Within the TeX WEB source code is a function which initializes TeX's "primitives". Here's a small extract of the raw WEB source code


@ The symbolic names for glue parameters are put into \TeX's hash table
by using the routine called |primitive|, defined below. Let us enter them
now, so that we don't have to list all those parameter names anywhere else.

@=
primitive("lineskip",assign_glue,glue_base+line_skip_code);@/
@!@:line_skip_}{\.{\\lineskip} primitive@>
primitive("baselineskip",assign_glue,glue_base+baseline_skip_code);@/
...


When this is translated to C the result looks something like this:


...
primitive ( 381 , 75 , 24527 ) ;
primitive ( 382 , 75 , 24528 ) ;
...


Not a string or comment in sight. tangle has also converted everything into integers: "lineskip" becomes 381 ... single-stepping through this C code with a debugger is not my idea of fun. So, what to do?

If you look at the form of code like

primitive("lineskip",assign_glue,glue_base+line_skip_code);

it is very amenable to processing with regular expressions. What you can do, for example, is pre-process the WEB source with your favourite regex tool to add "meta-comments" that will reach the Pascal and (with your modified lexer) make it into the C code. For example (should all be on one line):

primitive("lineskip",assign_glue,glue_base+line_skip_code); @{'lineskip,assign_glue,glue_base+line_skip_code'@};@/

Here we added the "meta-comment"

@{'lineskip,assign_glue,glue_base+line_skip_code'@}

just after the original Pascal code. Note that I have used single quotes '...' to delimit the text simply because I want to be able to detect the comments I introduced when the modified lexer is scanning my comments. To cut a long story short, through this technique you end up with C code that looks like this:


primitive ( 381 , 75 , 24527 ) ; /*lineskip,assign_glue,glue_base+line_skip_code*/
primitive ( 382 , 75 , 24528 ) ; /*baselineskip,assign_glue,glue_base+baseline_skip_code*/


Maybe not beautiful, but at least you now know what (some) of those tangle-generated numbers represent.

In conclusion

This techniqe is not "pretty" but, if used with care, you can get tangle to output a lot of useful comments, either through regular-expressions and pre-processing of the WEB code, or hand-editing the WEB to write summaries of the descriptions of the source code. I must stress that you can't put "meta-comments" just anywhere in the WEB source because you risk breaking tangle's parsing process: you'll need to experiment and proceed carefully with (say) small/minor manual edits to make sure tangle or Web2C don't "choke" on your changes.

This post is, once again, an aide-mémoire to record a work-in-progress: porting the tools that convert Knuth's original Pascal-based WEB source to C – to create a native build of Web2C.exe, fixwrites.exe and other tools using Microsoft's Visual Studio (and not using pipes). My apologies if this post is a little unstructured but the whole task is somewhat convoluted, which may be reflected in my writing style for this post! However, I'd like to record it whilst it is fresh in my memory.

Why would anyone want to do this when there are ready-made, reliable, TeX distributions freely available? Good question. Well, for me, it's nothing more than pure curiosity – and the fact that most British TV programs are now such mind-numbing drivel that I might as well do something productive in the evenings!

Join TUG: Just as an aside, I'm a member of the TeX User Group, TUG, so if you too would like to support TeX why not consider joining?

Another reason for writing this post is that I could not find much documentation on how to build Web2C.exe from source code – apart from these notes by Timothy Murphy, detailing the process for Macintosh-based port. Even though they were written in 1992 they were extremely helpful in filling in some of the details, so a belated thank you to Timothy Murphy – much of this post draws inspiration from that document. Piecing together the Web2c build process has been somewhat of a "programming jigsaw" – there are still gaps in my understanding but, I think, I can see the big picture even if it's still a little hazy in some areas.

The Big Picture

The source files for TeX, and other TeX-related programs and utilities, are written using Professor Donald Knuth's literate programming methodology. In essence, the program code (in Pascal) and documentation of the source code (in TeX) are contained within a single file, with extension .web. For example, Professor Knuth's source code of the latest version of TeX is contained in a file called tex.web. Similarly, within the TeXLive repository (see a previous post) or on CTAN, you can find the WEB source code for the latest versions of other programs; for example:

• bibtex.web: the source code/documentation of BiBTeX, for formatting and producing reference lists, as widely used within academic journal papers.
• mf.web: the source code/documentation of MetaFont.
• patgen.web: the source code/documentation of patgen which "... takes list of hyphenated words and generates a set of patterns that can be used by the TeX 82 hyphenation algorithm."
• tangle.web: the source code/documentation of tangle, which converts a WEB file to a Pascal (i.e., extracts the source code in Pascal, not in C – that's why Web2C exists).
• weave.web: the source code/documentation of weave, which converts a WEB file to TeX (i.e., extracts the documentation of the program's Pascal source code).

and other programs/utilities such as dvicopy.web, pltotf.web, tftopl.web and so forth.

What's in a name: tangle, web and weave? I've not researched to find out, but I cannot help thinking that Professor Knuth drew inspiration from Sir Walter Scott when naming these programs. Scott's poem Marmion contains the line(s) "O, what a tangled web we weave when we practice to decieve". Maybe these programs are as literary as they are literate?

TeXLive as the source of the files for building Web2C.exe

The files I reference throughout this post can be downloaded via SVN from the TeXLive repository. If you want to browse the TeXLive repository, using the TortoiseSVN program on Windows, this post may be of help. The following screenshots show the TeXLive folders you'll need to access for acquiring the various files I mention in this post.

• svn://tug.org/texlive/trunk/Build/source/texk/web2c: this folder contains, for example, tangleboot.pin (see below) and all the *.web files listed above, plus many other essential files.

  

• svn://tug.org/texlive/trunk/Build/source/texk/web2c/web2c: this folder contains the source files needed to build the actual Web2C.exe program. Note carefully it does not contain a file called Web2C.c, more on that below.

  

TeXLive has an advanced build-process for compiling/building all the tools and software it contains and I, for one, am in awe of the skills and expertise of its maintainers. In describing my explorations of building Web2C.exe as a Windows-based executable you need to realize that I am taking the source code files of Web2C.exe out of their "natural build environment". What do I mean by this? Building the Web2C executable program is usually part of the much bigger TeXLive build/compilation process so you should be prepared for a little extra complexity to create Web2C.exe as a "standalone" Windows program. Note that "standalone" is in quotes because converting WEB-generated Pascal into C code requires other tools in addition to Web2C.exe: it is not fully accomplished by Web2C.exe alone.

A note about Kpathsea

The Kpathsea (path-searching) C library in an integral part of most TeX-related software and the Web2C C source files #include a number of Kpathsea headers. However, for my own purposes/experiments I've decided to decouple my build of the Web2C.exe executable from the need to include Kpathsea's headers – the resulting C files generated by Web2C.exe will, of course, still depend on Kpathsea. If you grab the Web2C source files (see below) then "out of the box" you'll need to checkout the Kpathsea library from:

svn://tug.org/texlive/trunk/Build/source/texk/kpathsea

I've simply not got the time to document everything I had to do to decouple Kpathsea when building Web2C.exe. It mainly involved commenting out various #include lines that pulled in Kpathsea headers and placing a few #define statements into my local version of web2c.h – plus creating some typedefs and adding a few macros. If you're an experienced C programmer it is unlikely to present difficulties. As mentioned, this post describes a work-in-progress to satisfy my own curiosity and is meant to share a few of the things I've learnt, should they be useful to anyone as a starting point for their own work.

Web2C: so what is it?

Let me be clear that when I refer to Web2C I am referring to the executable program which undertakes the first (main) step in converting Pascal code into to C. So let's now start to take a look at the details but start with a summary of "Where are we?"

Where are we?

The starting point for generating C code is to extract the Pascal code from WEB source files and that is accomplished using the tangle program. However, where do we get a working tangle program from to start with – do we have a chicken and egg problem? tangle is itself distributed in WEB source code (tangle.web), so if I need tangle to extract tangle's source code from tangle.web, how do I create a working tangle program? Well, of course, this is solved by the distribution of tangle's Pascal code in a file called tangleboot.pin within the Web2C directory of the TeXLive repository (see above). In essence, tangleboot.pin let's you "bootstrap" the whole Web2C process by creating a working tangle.exe which you can use to generate the Pascal from WEB source files. Hence the name tangleboot.pin

So, how do I go from tangleboot.pin to a working tangle.exe? You need to build Web2C.exe and some associated utility programs (e.g., fixwrites.exe).

Where are the Web2C.exe source files?

As mentioned above, the TeXLive folder containing the source files needed to build Web2C.exe is

• svn://tug.org/texlive/trunk/Build/source/texk/web2c/web2c

The C source files you need to compile/build Web2C.exe are:

• kps.c
• main.c
• web2c-lexer.c
• web2c-parser.c

Some notes on these files

These C files #include a number of header files from the TeXLive distribution, notably from the Kpathsea library, so you should definitely look through them to determine any additional files you need.

The files web2c-parser.c and web2c-lexer.c are worthy of some explanation because they are the core files which drive the Pascal --> C conversion process. However, these two C source files are not hand-coded but are generated from two further source files with similar names. If you look among the source files you will also notice these two additional files:

• web2c-lexer.l
• web2c-parser.y

What are these files with similar names? As you may infer from their names, these files are a lexical analyser and a parser generator and require additional tools to process them:

• web2c-lexer.l --> web2c-lexer.c using a tool called flex.
• web2c-parser.y --> web2c-parser.c + web2c-parser.h using a tool called bison.

Are bison/flex available for Windows?

Fortunately they are and, at the time of writing (February 2013), you can download Windows ports of bison 2.7 and flex 2.5.37 from http://sourceforge.net/projects/winflexbison/. The executables are called win_bison.exe and win_flex.exe respectively. The win_flex.exe port of flex adds an extra command-line switch (--wincompat) so that the C code it generates uses the standard Windows header io.h instead of unistd.h (which is used on Linux). You can also download older versions of bison and flex for Windows from the GnuWin32 project.

I have not yet tried to use the code generated by win_flex.exe and win_bison.exe but to the best of my (current) knowledge the command-line options you need are:

• win_bison -y -d web2c-parser.y to generate the parser (you'll get different file names on output: y.tab.c and y.tab.h)
• win_flex --wincompat web2c-lexer.l to generate the lexical analyser (you'll get a different file name on output: lex.yy.c)

You need more than just Web2c.exe...

Assuming that you successfully build Web2c.exe, it is still not the end of the story. Although Web2c.exe does the bulk of the work in converting the Pascal to C, some initial pre-processing of the Pascal source file is needed before you can run it through Web2C.exe, and some further post-processing of the C code output by Web2C.exe is also needed. The details of how these pre- and post-processing steps actually work are contained within an important BASH shell script called convert (it has no extension) – convert is located within the TeXLive folder containing the Web2C source files. I readily confess that I know very little about Linux shell scripting so if you are well-versed in shell scripts no doubt you can easily understand what is going on in the convert file. However, here are pointers to get you started.

Pre-processing: adding the *.defines files to the Pascal file

Before you can actually run Web2C.exe on the Pascal file generated from WEB sources you need to concatenate the Pascal source file with some files having the extension ".defines": you add these files to the start of the Pascal file before running Web2C.exe. There are several .defines contained in the Web2C source directory including:

• common.defines
• mfmp.defines
• texmf.defines

The convert script checks which program, and its options, (TeX, MetaFont, BiBTeX etc) is being built and concatenates the appropriate *.defines file(s) to the start of the corresponding Pascal file. At this time, I don't quite fully understand how/why these files are needed, but for the full details you need to read convert. By way of an example, when processing tangleboot.pin I added the file common.defines to the beginning of tangleboot.pin.

Post-processing: fixwrites.exe

Web2C.exe's output is not quite pure C source code – it may still contain some fragments of Pascal which need a specialist post-processing step to fully convert them to C: enter fixwrites.exe. fixwrites.exe post-processes Web2C.exe's C output to "...convert Pascal write/writeln's into fprintf's or putc's" (see fixwrites.c).

Notes on web2c-parser.c, web2c-lexer.c and

main.c

Upon reading the convert script, and when I first ran Web2C.exe, it became readily apparent that the whole Pascal --> C tool chain (driven by convert) communicates using pipes) with stdout/stderr. The output of one program is "piped" into the input to another, rather than writing the data out to a physical disc file and then reading it back in. My personal preference, certainly whilst learning, is to output data to a file so that I can capture what's going on.

main.c and yyin

Without going into too much detail, I needed to make a number of changes in main.c so that the lexical analyzer web2c-lexer.c was set to read it's data from a disc file rather than through pipes/stdin. The FILE* variable you need to set/define is called yyin. For example, within main.c there is a function called initialize () which can be used to set yyin. For example:

void initialize (void)
{
register int i;
for (i = 0; i < hash_prime; hash_list[i++] = -1)
;

yyin = xfopen("your_path_to\\tangleboot.p","r");
...
...
}

In addition, within main.c there's a small function called normal () which does the following:


void normal (void)
{
out = stdout;
}

The normal () function is called from within web2c-parser.c to set the output file (FILE *out) to stdout. At present, I'm not sure precisely why this is done, but I guess it is part of the piping between programs as driven by the convert process. For example, code within convert uses sed (the stream editor).

Other output redirections happen in web2c-parser.c and you can search for these by looking for out = 0. Tracking down and locating these output redirections certainly helped me to better understand the flow of the programs.

In conclusion

This post is a little disjointed in places and light on detail in a number of areas, reflecting my own (currently) incomplete understanding of the relatively complex processes involved in converting WEB/Pascal to C. Nevertheless, I hope that it is of some use to someone, at some point. As my understanding develops I'll try to fill in the gaps with future posts.

Before continuing, I should say that this post is a sort of aide-mémoire for myself but I hope it is useful to others as well. Anyone who has looked into building TeX from the WEB source code soon finds that the process is somewhat "less than straightforward". Life can get a bit more complicated if, like me, you prefer to use Microsoft's Visual Studio rather than MSYS/MinGW – which gives you a cut down "Linux-like" build environment. I use MSYS/MinGW for building LuaTeX and it works really well, but I confess to being seduced by the nice IDE of Microsoft's Visual Studio. Having used Visual Studio to build a couple of C-based TeX distributions (Y&Y TeX, now open sourced), together with CXTeX, part of MetaTeX (by Taco Hoekwater), I have decided to "bite the bullet" and create a Visual Studio build for LuaTeX. I'm sure this will take quite some time but, you know, sometimes you get one of those itches you just have to scratch! And I've been meaning to attempt this for a long time, purely as an exercise.

CWEB

A lot of LuaTeX source code (apart from the libraries it uses) is written in CWEB, a dialect of literate programming by Silvio Levy and Donald Knuth. The original WEB sources of TeX use Pascal as the programming language, but for CWEB it is C – which, thankfully, saves you the painful process of converting Pascal to C via Web2C. Anyone who has looked at the C code generated by Web2C will, I'm sure, be dismayed because it's almost impossible to read. This is not so surprising given that it's machine generated. The joy of LuaTeX's C code, derived from CWEB, it that it is much, much more readable than the C code derived WEB --> Web2C --> C. Not suprising, of course, because CWEB C code is not generated mechanically.

So, how do you process CWEB code?

Enter CWEAVE and CTANGLE. What are these, you may well ask. Their task is to process a file written using CWEB (typically, with an extension ".w") and output the C source code (using CTANGLE) or the program's documentation in TeX (using CWEAVE). From http://sunburn.stanford.edu/~knuth/cweb.html:

• CTANGLE: converts a source file foo.w to a compilable program file foo.c;
• CWEAVE: converts a source file foo.w to a prettily-printable and cross-indexed document file foo.tex.

The LuaTeX build process (with MSYS/MinGW) also generates the executable ctangle.exe, so my first thought was "Great, I'll just use that to generate C source from the CWEB *.w files in the LuaTeX distribution". Running ctangle (from the LuaTeX build) using the command line (under the Windows cmd shell, or the MSYS BASH shell):

ctangle --help

you get the following output:

$ ctangle --help
Usage: ctangle [OPTIONS] WEBFILE[.w] [{CHANGEFILE[.ch]|-} [OUTFILE[.c]]]
Tangle WEBFILE with CHANGEFILE into a C/C++ program.
Default CHANGEFILE is /dev/null;
C output goes to the basename of WEBFILE extended with `.c'
unless otherwise specified by OUTFILE; in this case, '-' specifies
a null CHANGEFILE.

But first, change files and CTIE

From the output of ctangle --help you can see that its command line includes reference to CHANGEFILE.ch. So, what is that? Suppose that you have some program foo.w written in CWEB and you want to make some platform-specific modifications to foo.w. Rather than amending foo.w itself (e.g., to keep it platform-independent) you "merge" foo.w with a change file which, for example, may contain Windows-specific code. You would put your Windows CWEB code into, say, win32.ch and merge this code with foo.w. So how do you do this merge? There are two main ways:

1. you can combine foo.w and win32.ch using CWEAVE or CTANGLE, or
2. you can use another program called CTIE

What is CTIE?

The idea behind CTIE is that it lets you merge a master CWEB file with multiple change files, whereas CWEAVE or CTANGLE support only 1 change file on their command line. The source code of CTIE is also part of the LuaTeX distribution, in the directory ..\source\texk\web2c\ctiedir\. In there you'll find a file called ctie.c which compiles easily using Visual Studio to give you ctie.exe. If you want to read more about CTIE I have processed the documentation which you can download as a PDF.

Default CHANGEFILE is /dev/null

On reading the Usage information output by ctangle --help you should note that the Usage instructions state: Default CHANGEFILE is /dev/null. The explanation of /dev/null on http://en.wikipedia.org/wiki/Dev-null state that:

"In Unix-like operating systems, /dev/null or the null device is a special file that discards all data written to it."

The Usage instructions are a little cryptic but what it is saying is that if you want to run CWEAVE or CTANGLE without using a change file, you would run it like this:

ctangle foo.w -

where the hyphen (-) in effect says "don't use a changefile". Let's take an example CWEB file from the LuaTeX distribution, align.w, and try to generate the C source code using the version of ctangle built during the LuaTeX compilation process using MSYS/MinGW. Here we don't want to apply a change file so we'll use the hyphen option in place of a change file (the C output file will default to align.c):

ctangle align.w -

The resulting output is:

This is CTANGLE, Version 3.64 (TeX Live 2011)
! Cannot open change file NUL. (l. 0)

(That was a fatal error, my friend.)

That's a bit annoying, but the fix is very simple and there are a couple of ways to do it.

How to fix this?

To build CTANGLE you needs two files: ctangle.c and common.c, both of which are located in the source directory of LuaTeX (..\source\texk\web2c\cwebdir\). The "offending" code which causes the fatal error is located in common.c (and, of course, common.w).

In common.c (or common.w) you'll find the line:

if (found_change<=0) strcpy(change_file_name,"/dev/null");

and that needs changing for Windows. Fortunately, in the LuaTeX distribution (..\source\texk\web2c\cwebdir\) there is a change file, comm-w32.ch, taking care of this (written by Fabrice Popineau, in February 2002). In comm-w32.ch you'll find the above line replaced with:

if (found_change<=0) strcpy(change_file_name,"NUL");

Of course, the proper way to fix this is to apply a change file (such as comm-w32.ch) to the CWEB source of common.w and re-generate common.c with the above fix. You can fix common.c in at least two ways:

1. manually edit common.c to replace "/dev/null" with "NUL" in the line above, or
2. use the LuaTeX-build created version of ctangle but with the comm-w32.ch change file – it was the absence of a change file that we are trying to fix.

Note: If you are experimenting with these CWEB tools I strongly suggest you make a copy of all your *.w files into a working directory in case you make an error and accidentally overwrite any files.

Copy ctangle.exe, common.w and comm-w32.ch to a working directory away from your main source code, CD into that directory (make it the current directory), and run the following command line (it works under DOS and the MSYS BASH shell). the "./" simply tells ctangle to look in the current directory.

$ ctangle ./common.w ./comm-w32.ch ./mycommon.c

If successful, the output should be:

ctangle ./common.w ./comm-w32.ch ./mycommon.c
This is CTANGLE, Version 3.64 (TeX Live 2011)
*1*5*7*27*56*67*77*81
Writing the output file (./mycommon.c):.....500.....1000
Done.
(No errors were found.)


giving you a new version of common.c (which I called mycommon.c) with the fix applied by comm-w32.ch. If you look at the last lines of the mycommon.c file you just generated you should see something like this:

#line 78 "./comm-w32.ch"
if(found_change<=0)strcpy(change_file_name,"NUL");
#line 1283 "./common.w"

You can see that the line 78 of comm-w32.ch has been applied. Now, with the fixed file (mycommon.c) you can proceed to build CTANGLE using Visual Studio to generate an executable that accepts the "NULL" change file. We'll see that in a moment.

Let's use a different approach: using CTIE to merge common.w and comm-w32.ch into, say, mycommon.w. From mycommon.w we'll use our newly compiled CTANGLE to output mycommon.c. The the following CTIE command line does the trick:

ctie -m mycommon.w common.w comm-w32.ch

The -m option is documented here. If successful, you should see something like this:

This is CTIE, Version 1.1
Copyright 2002,2003 Julian Gilbey. All rights reserved. There is no warranty.
Run with the --version option for other important information.
(common.w)
(comm-w32.ch)
....500....1000....
(No errors were found.)

However, you can of course simply edit the file common.c directly to make the change. Once you fix common.c, both CWEAVE and CTANGLE compile nicely with Visual Studio and work perfectly when you use the "-" option to indicate no change file. With a working CTANGLE you can generate the C source of newcommon.w like this:

G:\CWEB\cwebtools\Debug>ctangle newcommon.w -
This is CTANGLE (Version 3.64)
*1*5*7*27*56*67*77*81
Writing the output file (newcommon.c):.....500.....1000
Done.
(No errors were found.)

With CTANGLE in place you can now run it on the CWEB *.w sources in LuaTeX to generate the C code. Clearly, for Visual Studio one way to proceed is to incorporate CWEB sources into your project and have a "Custom Build Step" for .w files, processing them with CTANGLE.

Happy TeXing, or should I say ctieing, ctangling and cweaving!

In this post I'm going to sketch out adding the free PCRE C library to LuaTeX through a DLL and outline how you can get PCRE to call LuaTeX! The following is just an outline of an experiment, not a tutorial on PCRE, and I've not tried this in a production environment. So, do please undertake all necessary testing and due diligence in your own code!

PCRE: Perl Compatible Regular Expressions

PCRE is a mature C library which provides a very powerful regular expression engine. It is also capable of working with UTF-8 encoded strings, which is, of course, very useful because LuaTeX uses UTF-8 input. I'm not going to cover the entire PCRE build process in this post because, frankly, it'll take too long. But in outline...

Building PCRE as a static library (.lib)

1. I used CMake to create a Visual Studio 2008 project via the PCRE-supplied CMakeLists.txt file. Using the CMake tool you can set the appropriate compile-time flags for UFT-8 support: PCRE_SUPPORT_UTF and PCRE_SUPPORT_UNICODE_PROPERTIES. The latter is very useful for seaching UTF-8 strings based on their Unicode character properties. Full details are in the PCRE documentation.



2. After you finish configuring the PCRE build, and have selected your build environment, press Generate and CMake will output a complete Visual Studio project that you can open and start working on. Wonderful!
3. However, getting PCRE to build as a static library was fine but I did have a few hassles getting the library to correctly link against the DLL I was building. It took me a bit of time to figure out which additional PCRE preprocessor directives I needed to set in the DLL C code to ensure everything was #define'd properly.

Building a DLL for LuaTeX

I wrote a very brief overview of building DLLs for LuaTeX in this post so I won't repeat the details here. Instead, I'll give a summary indicating how you can get PCRE to call LuaTeX. One word of advice, PCRE comes with a lot of documentation and you'll need to read through it very carefully! Asking PCRE to call LuaTeX sounds strange but indeed you can do it because PCRE provides the ability to register a callback function it will call each time it matches a string. Perl has a similar ability to execute Perl code on matching a string. From the PCRE documentation:

"PCRE provides a feature called 'callout', which is a means of temporarily passing control to the caller of PCRE in the middle of pattern matching. The caller of PCRE provides an external function by putting its entry point in the global variable pcre_callout."

Calling LuaTeX

OK, so how do we do that? There are two parts to this story: create a Lua function you want to call from C and create the C function which calls the Lua function.

1. From within LuaTeX, use \directlua{...} to create a simple Lua function printy that we are going to call from PCRE. This Lua function takes a string and sends it to LuaTeX via tex.print(). In these examples I sent LuaTeX a simple text string "Yo! I was called!", which LuaTeX then typeset. Of course, you could also send LuaTeX the string that was matched by PCRE!

          \directlua{
                 function printy (str)
                 tex.print(str)
                 end
          }
   

2. The next part is to create the C code to call a Lua function. This C function is the callout that PCRE will call when it matches a string.

          int mycallout(pcre_callout_block *cb){
          lua_State *L;
          L = cb->callout_data;
          if (L){
                 lua_getglobal(cb->callout_data, "printy");
                 if(!lua_isfunction(L,-1)) {
                        lua_pop(L,1);
                        return 0;
                  }
   
                 lua_pushstring(L, "Yo! I was called!");   /* push 1st argument */
                 /* Now make the call to printy with 1 argument and 0 results*/
                 if (lua_pcall(L, 1, 0, 0) != 0) {
                 // report your error
                  return 0;
                 }
       }
       return 0;
   }
   

   A few points here are worth noting.

   • From the PCRE documentation:
     

     "The external callout function returns an integer to PCRE. If the value is zero, matching proceeds as normal. If the value is greater than zero, matching fails at the current point, but the testing of other matching possibilities goes ahead, just as if a lookahead assertion had failed. If the value is less than zero, the match is abandoned, the matching function returns the negative value"

   • The lua_State variable, *L, is passed in via a mechanism I'll outline below.
   • The line lua_getglobal(cb->callout_data, "printy") does the main work of pushing the value of the gloabal variable printy onto Lua's stack. Of course, in effect this is a pointer to the function we defined in LuaTeX, and which we call through lua_pcall(...). Further details in the Lua documentation.
   • The above code does near-zero error checking, it is purely to demonstrate the ideas!

Other PCRE bits and pieces

There are a few other points to consider, namely how do you setup the callout and how do you pass lua_State *L to the callout? I'm not going to explain in great detail how all these parts hang together in a full application, simply point out some key pieces.

1. You have to set the PCRE global variable pcre_callout, a function pointer, to your callout function. Simply, pcre_callout = mycallout; Yes, it does work. Here, re represents our compiled regular expression pattern. Note that you must use the PCRE_UTF8 option if you are searching UTF-8 encoded text.
2. Before you can start searching, you need to "compile" your regular expression pattern.

                 re = pcre_compile(pattern,
   		      PCRE_UTF8|PCRE_UCP,
   		      &err_msg,
   		      &err,
   		      NULL);
   

3. Note, to use PCRE callouts you need to use the appropriate syntax in your regular expression; from the PCRE documentation, "Within a regular expression, (?C) indicates the points at which the external function is to be called." Once you have compiled your search pattern, and done your error checking, you need to run the search engine using the compiled pattern and your target string (s) in the code below.
4. The next step is to create a pointer to something called a pcre_callout_block, which is a struct. This struct has a field called callout_data which is a pointer into which you can store whatever you want to pass into the mycallout function: here, I'm setting it to the lua_State variable, L. By doing this, each time PCRE matches a string and calls the callout funtion, the lua_State variable, L will be available for our use! Clearly, you'll need to do this from within the appropriate function you call from LuaTeX. Once this is done you are ready to begin your searching using pcre_exec(...).

                 pcre_extra *p;
                 p = (pcre_extra*) malloc(sizeof(pcre_extra));
                 memset(p,0, sizeof(pcre_extra));
                 p->callout_data = L;
                 p->flags=PCRE_EXTRA_CALLOUT_DATA;
                        res = pcre_exec(re,
                               p,
                               s,
                               len,
                               0,
                               0,
                               offsets,
                        OVECMAX);
   

Summary

PCRE is a marvellous and powerful C library – with copious documentation that you'll need to read very carefully! The ability to provide LuaTeX with a UTF-8-enabled regex engine could open the way to some useful applications, particularly when combined with LuaTeX's own callback mechanism. In particular, the process_input_buffer callback which allows you to change the contents of the line input buffer just before LuaTeX actually starts looking at it. The mind boggles at the possibilities!

It's been a long time since I posted anything on this blog, mainly because my job has been keeping me very busy. As time permits I've been reading parts of the LuaTeX source code in an attempt to better understand how it all works: cross-referencing the source code to explanations in the LuaTeX Reference. A couple of days ago I stumbled on the NetBeans IDE – a free Integrated Development Environment. I was interested to see that NetBeans has a Subversion Checkout Wizard (i.e., built-in SVN capabilities), so you can checkout a copy of the LuaTeX code repository and import it directly into NetBeans as a new project. So, I downloaded NetBeans (with C/C++ support) and checked out a copy of the LuaTeX code base, directly from within NetBeans. After completing the download, NetBeans automatically imported the LuaTeX code to create a new project. Very nice!

However, I have not tried to build LuaTeX using NetBeans (because I need to understand more about the build process) but I have found that it provides excellent tools to search and browse the source code, allowing you to very quickly explore and probe some of the deeper mysteries of TeX.

Tip: tell NetBeans about .w files

Much of the LuaTeX code base is written in CWEB (integrated C source code and documentation); consequently, many of the source files have a .w extension. You'll need to configure NetBeans to tell it about .w files: see Tools --> Options --> Miscellaneous.

Here's a screenshot showing a search for the build_page() function, part of TeX's page-building machinery, showing you where and when TeX exercises the page builder.