Web2C.exe<\/code>, fixwrites.exe<\/code> and other tools using Microsoft’s Visual Studio (and not using pipes<\/a>). My apologies if this post is a little unstructured but the whole task is somewhat convoluted, which may be reflected in my writing style for this post! However, I’d like to record it whilst it is fresh in my memory.<\/p>\nWhy would anyone want to do this when there are ready-made, reliable, TeX distributions freely available? Good question. Well, for me, it’s nothing more than pure curiosity – and the fact that most British TV programs are now such mind-numbing drivel that I might as well do something productive in the evenings!<\/p>\n
Join TUG<\/strong>: Just as an aside, I’m a member of the TeX User Group, TUG, so if you too would like to support TeX why not consider joining<\/a>?<\/p><\/blockquote>\nAnother reason for writing this post is that I could not find much documentation on how<\/em> to build Web2C.exe<\/code> from source code – apart from these notes by Timothy Murphy<\/a>, detailing the process for Macintosh-based port. Even though they were written in 1992 they were extremely helpful in filling in some of the details, so a belated thank you to Timothy Murphy – much of this post draws inspiration from that document. Piecing together the Web2c build process has been somewhat of a “programming jigsaw” – there are still gaps in my understanding but, I think, I can see the big picture even if it’s still a little hazy in some areas.\n<\/p>\nThe Big Picture<\/h1>\n
The source files for TeX, and other TeX-related programs and utilities, are written using Professor Donald Knuth’s literate programming<\/a> methodology. In essence, the program code (in Pascal) and documentation of the source code (in TeX) are contained within a single file, with extension .web<\/code>. For example, Professor Knuth’s source code of the latest version of TeX is contained in a file called tex.web<\/code>. Similarly, within the TeXLive repository (see a previous post<\/a>) or on CTAN, you can find the WEB source code for the latest versions of other programs; for example: <\/p>\n\n- bibtex.web<\/strong>: the source code\/documentation of
BiBTeX<\/code>, for formatting and producing reference lists, as widely used within academic journal papers.<\/li>\n- mf.web<\/strong>: the source code\/documentation of MetaFont.<\/li>\n
- patgen.web<\/strong>: the source code\/documentation of
patgen<\/code> which “… takes list of hyphenated words and generates a set of patterns that can be used by the TeX 82 hyphenation algorithm.” <\/li>\n- tangle.web<\/strong>: the source code\/documentation of
tangle<\/code>, which converts a WEB file to a Pascal (i.e., extracts the source code in Pascal, not in C – that’s why Web2C exists).<\/li>\n- weave.web<\/strong>: the source code\/documentation of
weave<\/code>, which converts a WEB file to TeX (i.e., extracts the documentation of the program’s Pascal source code).<\/li>\n<\/ul>\nand other programs\/utilities such as dvicopy.web<\/code>, pltotf.web<\/code>, tftopl.web<\/code> and so forth.<\/p>\nWhat’s in a name: tangle, web and weave?<\/strong> I’ve not researched to find out, but I cannot help thinking that Professor Knuth drew inspiration from Sir Walter Scott when naming these programs. Scott’s poem Marmion<\/a> contains the line(s) “O, what a tangled web we weave when we practice to decieve”. Maybe these programs are as literary as they are literate?<\/p><\/blockquote>\nTeXLive as the source of the files for building Web2C.exe<\/code><\/h1>\nThe files I reference throughout this post can be downloaded via SVN from the TeXLive repository. If you want to browse the TeXLive repository, using the TortoiseSVN program on Windows, this post<\/a> may be of help. The following screenshots show the TeXLive folders you’ll need to access for acquiring the various files I mention in this post.<\/p>\n\n- svn:\/\/tug.org\/texlive\/trunk\/Build\/source\/texk\/web2c<\/strong>: this folder contains, for example,
tangleboot.pin<\/code> (see below) and all the *.web<\/code> files listed above, plus many other essential files.\n![](\"http:\/\/readytext.co.uk\/files\/texlive1.png\")
<\/p>\n<\/li>\n
- svn:\/\/tug.org\/texlive\/trunk\/Build\/source\/texk\/web2c\/web2c<\/strong>: this folder contains the source files needed to build the actual
Web2C.exe<\/code> program. Note carefully it does not<\/strong> contain a file called Web2C.c<\/code>, more on that below.\n![](\"http:\/\/readytext.co.uk\/files\/texlive2.png\")
<\/p>\n<\/li>\n<\/ul>\n
TeXLive has an advanced build-process for compiling\/building all the tools and software it contains and I, for one, am in awe of the skills and expertise of its maintainers. In describing my explorations of building Web2C.exe<\/code> as a Windows-based executable you need to realize that I am taking the source code files of Web2C.exe<\/code> out of their “natural build environment”. What do I mean by this? Building the Web2C executable program is usually part of the much bigger TeXLive build\/compilation process so you should be prepared for a little extra complexity to create Web2C.exe<\/code> as a “standalone” Windows program. Note that “standalone” is in quotes because converting WEB-generated Pascal into C code requires other tools in addition to Web2C.exe<\/code>: it is not fully accomplished by Web2C.exe<\/code> alone.<\/p>\nA note about Kpathsea<\/h1>\n
The Kpathsea (path-searching) C library<\/a> in an integral part of most TeX-related software and the Web2C C source files #include<\/code> a number of Kpathsea headers. However, for my own purposes\/experiments I’ve decided to decouple my build of the Web2C.exe<\/code> executable from the need to include Kpathsea’s headers – the resulting C files generated by Web2C.exe<\/code> will, of course, still depend on Kpathsea. If you grab the Web2C source files (see below) then “out of the box” you’ll need to checkout the Kpathsea library from:<\/p>\nsvn:\/\/tug.org\/texlive\/trunk\/Build\/source\/texk\/kpathsea<\/p>\n
I’ve simply not got the time to document everything I had to do to decouple Kpathsea when building Web2C.exe<\/code>. It mainly involved commenting out various #include<\/code> lines that pulled in Kpathsea headers and placing a few #define<\/code> statements into my local version of web2c.h<\/code> – plus creating some typedefs and adding a few macros. If you’re an experienced C programmer it is unlikely to present difficulties. As mentioned, this post describes a work-in-progress to satisfy my own curiosity and is meant to share a few of the things I’ve learnt, should they be useful to anyone as a starting point for their own work.<\/p>\nWeb2C: so what is it<\/em>?<\/h1>\nLet me be clear that when I refer to Web2C I am referring to the executable program which undertakes the first (main) step in converting Pascal code into to C. So let’s now start to take a look at the details but start with a summary of “Where are we?”<\/p>\n
Where are we?<\/h2>\n
The starting point for generating C code is to extract the Pascal code from WEB source files and that is accomplished using the tangle<\/code> program. However, where do we get a working tangle<\/code> program from to start with – do we have a chicken and egg<\/a> problem?<\/strong> tangle<\/code> is itself distributed in WEB source code (tangle.web<\/code>), so if I need tangle<\/code> to extract tangle’s source code from tangle.web<\/code>, how do I create a working tangle program? Well, of course, this is solved by the distribution of tangle’s Pascal code in a file called tangleboot.pin<\/code> within the Web2C directory of the TeXLive repository (see above). In essence, tangleboot.pin<\/code> let’s you “bootstrap” the whole Web2C process by creating a working tangle.exe<\/code> which you can use to generate the Pascal from WEB source files. Hence the name tangleboot<\/strong>.pin<\/p><\/blockquote>\nSo, how do I go from tangleboot.pin<\/code> to a working tangle.exe? You need to build Web2C.exe<\/code> and some associated utility programs (e.g., fixwrites.exe<\/code>).<\/p>\nWhere are the Web2C.exe<\/code> source files?<\/h2>\nAs mentioned above, the TeXLive folder containing the source files needed to build Web2C.exe<\/code> is<\/p>\n\n- svn:\/\/tug.org\/texlive\/trunk\/Build\/source\/texk\/web2c\/web2c<\/li>\n<\/ul>\n
The C source files you need to compile\/build Web2C.exe<\/code> are:<\/p>\n\n- kps.c<\/li>\n
- main.c<\/li>\n
- web2c-lexer.c<\/li>\n
- web2c-parser.c<\/li>\n<\/ul>\n
Some notes on these files<\/h2>\n
These C files #include<\/code> a number of header files from the TeXLive distribution, notably from the Kpathsea library, so you should definitely look through them to determine any additional files you need.<\/p>\nThe files web2c-parser.c<\/code> and web2c-lexer.c<\/code> are worthy of some explanation because they are the core files which drive the Pascal –> C conversion process. However, these two C source files are not hand-coded but are generated<\/em> from two further source files with similar names. If you look among the source files you will also notice these two additional files:<\/p>\n\nweb2c-lexer.l<\/code><\/li>\nweb2c-parser.y<\/code><\/li>\n<\/ul>\nWhat are these files with similar names? As you may infer from their names, these files are a lexical analyser and a parser generator and require additional tools to process them:<\/p>\n
\nweb2c-lexer.l --> web2c-lexer.c <\/code> using a tool called flex<\/a>.<\/li>\nweb2c-parser.y --> web2c-parser.c + web2c-parser.h<\/code> using a tool called bison<\/a>.<\/li>\n<\/ul>\nAre bison\/flex available for Windows?<\/h3>\n
Fortunately they are and, at the time of writing (February 2013), you can download Windows ports of bison 2.7<\/code> and flex 2.5.37<\/code> from http:\/\/sourceforge.net\/projects\/winflexbison\/<\/a>. The executables are called win_bison.exe<\/code> and win_flex.exe<\/code> respectively. The win_flex.exe<\/code> port of flex adds an extra command-line switch (--wincompat<\/code>) so that the C code it generates uses the standard Windows header io.h<\/code> instead of unistd.h<\/code> (which is used on Linux). You can also download older versions of bison and flex for Windows from the GnuWin32 project<\/a>.<\/p>\nI have not yet tried to use the code generated by win_flex.exe<\/code> and win_bison.exe<\/code> but to the best of my (current) knowledge the command-line options you need are:<\/p>\n\nwin_bison -y -d web2c-parser.y<\/code> to generate the parser (you’ll get different file names on output: y.tab.c<\/code> and y.tab.h<\/code>)<\/li>\nwin_flex --wincompat web2c-lexer.l<\/code> to generate the lexical analyser (you’ll get a different file name on output: lex.yy.c<\/code>)<\/li>\n<\/ul>\nYou need more than just Web2c.exe<\/code>…<\/h1>\nAssuming that you successfully build Web2c.exe<\/code>, it is still not the end of the story. Although Web2c.exe<\/code> does the bulk<\/em> of the work in converting the Pascal to C, some initial pre-processing of the Pascal source file is needed before you can run it through Web2C.exe<\/code>, and some further post-processing of the C code output by Web2C.exe<\/code> is also needed. The details of how these pre- and post-processing steps actually work are contained within an important BASH shell script called convert<\/code> (it has no extension) – convert<\/code> is located within the TeXLive folder containing the Web2C source files. I readily confess that I know very little about Linux shell scripting so if you are well-versed in shell scripts no doubt you can easily understand what is going on in the convert<\/code> file. However, here are pointers to get you started.<\/p>\nPre-processing: adding the *.defines<\/code> files to the Pascal file<\/h3>\nBefore you can actually run Web2C.exe<\/code> on the Pascal file generated from WEB sources you need to concatenate the Pascal source file with some files having the extension “.defines<\/code>“: you add these files to the start<\/strong> of the Pascal file before running Web2C.exe<\/code>. There are several .defines<\/code> contained in the Web2C source directory including:<\/p>\n\ncommon.defines<\/code><\/li>\nmfmp.defines<\/code><\/li>\ntexmf.defines<\/code><\/li>\n<\/ul>\nThe convert<\/code> script checks which program, and its options, (TeX, MetaFont, BiBTeX etc) is being built and concatenates the appropriate *.defines<\/code> file(s) to the start of the corresponding Pascal file. At this time, I don’t quite fully understand how\/why these files are needed, but for the full details you need to read convert<\/code>. By way of an example, when processing tangleboot.pin<\/code> I added the file common.defines<\/code> to the beginning of tangleboot.pin<\/code>. <\/p>\nPost-processing: fixwrites.exe<\/code><\/h3>\nWeb2C.exe<\/code>‘s output is not quite pure C source code – it may still contain some fragments of Pascal which need a specialist post-processing step to fully convert them to C: enter fixwrites.exe<\/code>. fixwrites.exe<\/code> post-processes Web2C.exe<\/code>‘s C output to “…convert Pascal write\/writeln’s into fprintf’s or putc’s” (see fixwrites.c<\/code>). <\/p>\nNotes on web2c-parser.c<\/code>, web2c-lexer.c<\/code> and <\/p>\n- main.c<\/li>\n<\/h3>\n
Upon reading the convert<\/code> script, and when I first ran Web2C.exe<\/code>, it became readily apparent that the whole Pascal –> C tool chain (driven by convert<\/code>) communicates using pipes<\/a>) with stdout\/stderr. The output of one program is “piped” into the input to another, rather than writing the data out to a physical disc file and then reading it back in. My personal preference, certainly whilst learning, is to output data to a file so that I can capture what’s going on. <\/p>\nmain.c<\/code> and yyin<\/code><\/h4>\nWithout going into too much detail, I needed to make a number of changes in main.c<\/code> so that the lexical analyzer web2c-lexer.c<\/code> was set to read it’s data from a disc file rather than through pipes\/stdin. The FILE*<\/code> variable you need to set\/define is called yyin<\/code>. For example, within main.c<\/code> there is a function called initialize ()<\/code> which can be used to set yyin<\/code>. For example:<\/p>\nvoid initialize (void)
\n{
\n register int i;
\n for (i = 0; i < hash_prime; hash_list[i++] = -1)\n ;\n\n yyin = xfopen(\"your_path_to\\\\tangleboot.p\",\"r\");\n ...\n ...\n}<\/code><\/p>\nIn addition, within main.c<\/code> there's a small function called normal ()<\/code> which does the following:<\/op><\/p>\n
\nvoid normal (void)
\n{
\n out = stdout;
\n}<\/code><\/p>\nThe normal ()<\/code> function is called from within web2c-parser.c<\/code> to set the output file (FILE *out<\/code>) to stdout. At present, I'm not sure precisely why this is done, but I guess it is part of the piping between programs as driven by the convert<\/code> process. For example, code within convert<\/code> uses sed<\/a> (the stream editor). <\/p>\nOther output redirections happen in web2c-parser.c<\/code> and you can search for these by looking for out = 0<\/code>. Tracking down and locating these output redirections certainly helped me to better understand the flow of the programs.<\/p>\nIn conclusion<\/h1>\n
This post is a little disjointed in places and light on detail in a number of areas, reflecting my own (currently) incomplete understanding of the relatively complex processes involved in converting WEB\/Pascal to C. Nevertheless, I hope that it is of some use to someone, at some point. As my understanding develops I'll try to fill in the gaps with future posts.<\/p>\n","protected":false},"excerpt":{"rendered":"
Introduction This post is, once again, an aide-m\u00e9moire to record a work-in-progress: porting the tools that convert Knuth’s original Pascal-based WEB source to C – to create a native build of Web2C.exe, fixwrites.exe and other tools using Microsoft’s Visual Studio (and not using pipes). My apologies if this post is a little unstructured but the […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[28,27],"tags":[],"class_list":["post-2529","post","type-post","status-publish","format-standard","hentry","category-c-programming-miscellaneous","category-web2c-tangle-weave-web"],"blocksy_meta":[],"_links":{"self":[{"href":"https:\/\/www.readytext.co.uk\/index.php?rest_route=\/wp\/v2\/posts\/2529","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.readytext.co.uk\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.readytext.co.uk\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.readytext.co.uk\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.readytext.co.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2529"}],"version-history":[{"count":82,"href":"https:\/\/www.readytext.co.uk\/index.php?rest_route=\/wp\/v2\/posts\/2529\/revisions"}],"predecessor-version":[{"id":2669,"href":"https:\/\/www.readytext.co.uk\/index.php?rest_route=\/wp\/v2\/posts\/2529\/revisions\/2669"}],"wp:attachment":[{"href":"https:\/\/www.readytext.co.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2529"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.readytext.co.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2529"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.readytext.co.uk\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2529"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}