Saturday, September 8, 2018

Code review: Using catgets/kitten to support different languages

This Code Review article is a repeat from last year, about how to use the Cats and Kitten libraries to support different spoken languages in your programs.

When you write a new program, you probably don't think about spoken languages other than your own. I am a native English speaker, so when I write a new program, all of my error messages and outputted text is in English. And that works well for the many people who have English as their native language, or who know enough English as a second language to get by. But what about others who don't speak English, or who only know a little English? They can't understand what my programs are saying.

The standard Unix method is with a set of C library functions built around language "catalogs." A catalog is just a file that contains all the error messages and other printed text from a program. In the Unix method, you have a different catalog for every language: English, German, Italian, Spanish, French, and so on.

The FreeDOS Cats library was a stripped-down implementation of the Unix library, using a very simple method. Every time you want to print some text in the user's preferred language, you first look up the message string from the catalog using the catgets() function—so named because it will get a string from a message catalog.

In Unix, you use catgets() this way:

  string = catgets(cat, set, num, "Hello world");

This fetches message string number num from message set set, from language catalog cat. The organization of messages into sets allows developers to group status messages into one set (say, set 1), error messages into another set (such as set 7), and so on.

Before calling catgets(), you need to open the appropriate language catalog with a previous call to catopen(). Typically, you have one catalog per language, so you have a different language file for English, another for Spanish, etc. Before your program exits, you close any message catalogs with calls to catclose().

If the string doesn't exist in the message catalog, catgets() returns a default string that you passed to it; in this case, the default string was "Hello world."

I implemented a simplified version of these functions in a FreeDOS Internationalization library called Cats. To save on memory, Cats supported only one open catalog at a time. If you tried to open a second message catalog, the call to catopen() would return an error (-1).

Message catalogs were very simple under Cats. Implemented as plain text files, Cats loaded the entire message catalog into memory at run-time. In this way, you didn't need to recompile the program just to support other languages; you just added another message catalog file for the new language. An English message catalog for a simple program might look like this:

  1.1:Hello world
  7.4:Failure writing to drive A:

The same message catalog in Spanish might look like this:

  1.1:Hola mundo
  7.4:Fallo al escribir en la unidad A:

For example, the string "Failure writing to drive A:" is message number 4 in set 7.

The Cats library was a simple way for developers to add support for different languages in their programs, written in C. And because Cats implemented a Unix standard, it made porting Unix tools to FreeDOS much easier. Once you added the calls to catgets(), all you needed to support other languages was a message catalog that someone translated to a different language. And I kept the Cats message catalogs very simple; they were plain text files.

Cats was a neat innovation, but loading the messages into memory was cumbersome because it used streams. Other FreeDOS developers improved on Cats to optimize the loading of catalogs, reduce memory footprint, and add other enhancements. The new library was noticeably smaller, so we renamed it Kitten.

Because of the optimizations, Kitten used a slightly different API. Since Cats only supported one message catalog at a time anyway, Kitten removed the cat catalog identifier. Once you open a message catalog with kittenopen(), all calls to kittengets() assume that message catalog. You only need to make a single call to kittenclose() before you end the program.

Using Kitten made it much easier to support different spoken languages in FreeDOS programs. Here's a trivial example to put it all together:

  /* test.c */

  #include <stdio.h>
  #include <stdlib.h>
  #include "kitten.h"

  int
  main(void)
  {
    char *s;
 
    kittenopen("test");
 
    s = kittengets(7, 4, "Failure writing to drive A:");
    puts(s);
 
    kittenclose();
    exit(0);
  }

This loads a message catalog "test" into memory, then retrieves message 4 from set 7 into a string pointer s. If message 4 in set 7 isn't found, kittengets() returns the default string "Failure writing to drive A:" into s. The program prints the message to the user, then closes the message catalog before exiting.

Typically, you name the message catalog after the program's name. So the message catalog for the FreeDOS CHOICE program is "choice". Kitten searches for the language file in a few locations on disk, and always appends the value of the %LANG% environment variable, which is typically set to the two-letter language abbreviation: "en" for English or "es" for Spanish. The DOS filename for the English version of the "choice" language catalog is CHOICE.EN, and the Spanish language version is CHOICE.ES.

A limitation to Cats and Kitten is that it can only support single-byte character sets supported by DOS. So Cats and Kitten cannot support Chinese or Japanese, but should do fine with most other languages.
You can find Cats and Kitten at the FreeDOS files archive on ibiblio, under devel/libs/cats/

We made three revisions to Kitten, so the latest version is Kitten revision C, which you can download directly as kitten-c.zip.

FreeDOS contributor Mateusz "Fox" Viste wrote a similar implementation for Pascal programs, called Cubs. You can also find it on ibiblio, under devel/libs/cats/cubs/

FreeDOS developer Eric Auer created a command-line version of Kitten, named Localize, so you can provide internationalization support for DOS Batch (BAT) files. Find it on ibiblio, under devel/libs/cats/localize/

Tuesday, August 21, 2018

New article series: code reviews

I've been thinking of ways to get new developers interested in working on FreeDOS, or at least contributing to FreeDOS in some way. I recognize that DOS is an old concept for some, and as a result, the way you construct programs for DOS can differ to how you'd construct programs for other operating systems such as Linux.

Sure, programs like Choice and Type and Find and other everyday command-line utilities are pretty straightforward. You'd probably write a DOS Type command in the same way you'd write a Unix ‘cat’ command (using streams). But other programs require different methods.

So I thought some kind of “code review” would be helpful, as a way to demonstrate certain DOS programming methods. I'll plan to examine a FreeDOS program and pull apart a specific function or method or programming trick to show how you can use these methods in your own programs.

It will be easiest for me to do a code review on my own programs, and I may start there. But I will try to explore different FreeDOS programs written by different people, so you can see a variety of methods. Every programmer is different, and this will highlight different ways to implement features or optimize execution or minimize footprint when programming in DOS.

Do you have a program you'd like to highlight? I welcome any guest posts in this series. If you have something you'd like to show off, please email me or leave a comment below. I'd be happy to include your article as a guest post here. (If you submit a guest post, it will be easiest if you can share your contribution under the Creative Commons Sharealike [cc-by] public license.)

What programs or methods would you like to see? If you have a suggestion for a “deep dive” into a FreeDOS utility, let me know! My expertise is C, so I am more likely to do code reviews for programs written in C.

Thursday, August 2, 2018

Expanding the FreeDOS COMMAND.COM

The FreeDOS COMMAND.COM (also known as "FreeCOM") is a compatible equivalent to MS-DOS COMMAND.COM. FreeCOM doesn't do anything special; it just creates a command prompt that lets you run programs.

FreeCOM also supports the standard MS-DOS batch programming syntax. Any batch script that runs in MS-DOS should run in FreeCOM. And any batch script you write for FreeCOM will run in MS-DOS COMMAND.COM.

From a recent discussion on the email list, I'd like to suggest an extension to FreeCOM. Maybe this would be better implemented as part of a new alternative shell, but it would be really interesting to see it added as a feature to FreeCOM.

Here's my idea: create a BASIC-like programming language that also supports MS-DOS batch scripts.

DOS batch scripts are simple affairs. Really, the idea behind the MS-DOS batch was to "batch up" a bunch of simple commands. DOS batch programming supports minimal flow control. You can query the exit status (ERRORLEVEL) of the previous command, and you can jump around the script (GOTO) but DOS batch programming doesn't support much else.

I'd love to see an extension to DOS batch programming. Something like a BASIC interpreter would be interesting, if it executed commands in the PATH if the instruction wasn't a keyword.

I wonder if someone is interested in writing a BASIC-like "shell"? Something that uses a combination of BASIC and COMMAND.COM syntax would make the interpreter a "superset" of COMMAND.COM, and an interesting new alternative shell.

Some suggestions for how to extend BASIC and DOS batch programming include:

Variables


Variables can store either numerical or string data. Variable names can be of any reasonable length, and can be referenced in uppercase or lowercase.

Use SET to assign values to variables, and BASIC's LET to assign values based on arithmetic. LET becomes a superset of SET:
SET A=4
SET DIR=C:\FDOS\BIN

And surround a variable with % to reference its value:
SET A=4
LET A=%A%​ ​+​ ​1
ECHO %A%​

Possibly allow %% as a prefix to reference the value from a variable, interchangeably: (sloppy, but see later)
SET A=4
LET A=%%A + 1
ECHO %%A

Output


Use ECHO to display output instead of PRINT:
ECHO Hello world

Tests


Support the standard DOS batch programming tests, such as == to test equality, and NOT to negate a test. Add extra constructs to test inequality, greater-than and less-than. One possible solution is to support simple DOS batch programming IF constructs and use Unix-style brackets for the extra constructs:
IF ERRORLEVEL 0 ECHO Success
IF NOT ERRORLEVEL 0 ECHO Fail
IF EXIST FILE.TXT ECHO Exists

And:
IF %VALUE%==1 ECHO Equal
IF NOT %VALUE%==1 ECHO Not equal

And:
IF [ %VALUE% EQ 1 ] ECHO Equal
IF [ %VALUE% NE 1 ] ECHO Not equal
IF [ %VALUE% GT 1 ] ECHO Greater
IF [ %VALUE% GE 1 ] ECHO Greater or equal
IF [ %STR% LT "ABC" ] ECHO Less

Looping


The FOR statement would need to be extended as a for-next loop:
FOR N IN 1 TO 10
ECHO %N%
NEXT

Or as the traditional DOS for-do one-line shortcut:
FOR N IN 1 TO 10 DO ECHO %N%

And add a special IN (…) construct to create an iteration over a set:
FOR %%N IN (1 2 3 4 5 6 7 8 9 10) DO ECHO %%N
FOR %%FILE IN (*.TXT) DO ECHO %%F

​(If the language allows %A% and %%A then DOS batch programs would probably work seamlessly. But it's sloppy coding. Other ideas?)

Branching


Similar to both BASIC and DOS batch programming, GOTO could use : to mark labels:
SET N=1
:LOOP
ECHO %N%
LET N=%N% + 1
IF %N% LT 10 GOTO LOOP

Comments


And of course, comments remain the same from BASIC and DOS batch programming:
REM This is a comment

Execution


If you called the BASIC-like "shell" using a special parameter (like /P) then it would be nice for anything e​lse you specify that isn't recognized as a shell statement (assuming other syntax is correct) get treated like an external command. That would make this more like a shell:
IF EXIST FILE.TXT EDIT FILE.TXT

Also other internal constructs for CALL and ERRORLEVEL and SHIFT and PATH, but you get the idea.

If you can preserve COMMAND.COM syntax and add a BASIC-like programming language behind it, that would make this an interesting extended shell. Maybe an idea for an interested developer?
Note: Tom points out that Microsoft similarly extended the COMMAND.COM batch programming language when creating CMD.EXE. For example, for /L %%N in (1,1,10) do echo %%N (iterative loops) and SET /A A=%%A + 1 (arithmetic). So this hybrid BASIC-like batch programming language idea is not new.

Sunday, July 22, 2018

Planning FreeDOS 1.3

We've started planning the FreeDOS 1.3 distribution! We previously decided the next release would be an iteration from FreeDOS 1.2. We wanted the next FreeDOS distribution to remain like classic DOS. For example, we won't "retire" any classic commands utilities from Base. But FreeDOS 1.3 is an opportunity to improve and update several things.

Compatibility remains key. FreeDOS 1.3 will remain 16-bit, and will focus on a single-user command line environment, just like classic DOS. The "Base" package group will still contain everything that replicates the functionality found in MS-DOS, although we plan to promote Zip and Unzip from "Util" to "Base." This is because all FreeDOS packages are really zip files, so it makes sense to include tools in "Base" that let you create and modify your own packages. And Zip and Unzip provide a handy way to backup and restore your system; while not compatible with MS-DOS Backup and Restore, Zip and Unzip provide a modern spin on that functionality.

FreeDOS 1.3 will follow a similar release schedule as used in FreeDOS 1.2. The deadlines are: No new packages after September 30, 2018. No new languages after October 31, 2018. Assuming these deadlines, the expected schedule is:

  • November 30, 2018 - FreeDOS 1.3 RC1
  • December 31, 2018 - FreeDOS 1.3 RC2
  • January 31, 2019 - FreeDOS 1.3 release


Please join the freedos-devel email list and contribute to the discussion! Also keep an eye on the FreeDOS Road Map on the FreeDOS Wiki for more updates. Looking for ideas to contribute to FreeDOS? See our Contribute page for several suggestions.

Updating the FreeDOS website

I just wanted to share a quick note that I am planning to update the FreeDOS website. I'm not planning any site redesigns; the website should still look the same. But I am moving to a new web host provider. Among other things, the updated web host will provide and manage https for me.

Dates are approximate, but I'll set up a test site on the new provider in early August, and plan to move the site in mid August. I'll leave the old web host for another week, while DNS changes propagate everywhere. We should be fully on the new web host by late August, and I'll retire the old web host at that time.

The new web host will also include a custom MediaWiki instance for us. Currently, the FreeDOS Wiki runs on a custom MediaWiki hosted at SourceForge. This has worked well, aside from a few unplanned downtimes. Also, the SourceForge-hosted MediaWiki instance doesn't allow outbound emails, which is the main stumbling block to let other users create their own accounts. (Wiki Admins have had to do this, and it's a huge pain; I never remember how to do it.)

I'll plan to move the FreeDOS Wiki in September, well after the website move.

Thursday, July 12, 2018

FreeDOS batch programming quick reference

A few weeks ago, I posted a "quick reference" for our new users. This reference showed the most common FreeDOS commands, including a few important notes for new users.

If you're looking to take the next step in FreeDOS, here's a handy quick reference guide to batch programming.

DOS batch programs are simple scripts that "batches up" several commands into one file. Batch programs have some basic flow control that you can use to do loops and jump to different segments of the program. In batch programs, reference variable values by enclosing the variable name with %, such as %PATH%

What do you want to do?How to do it in a batch program:
Execute another batch script from within a scriptCALL SCRIPT.BAT
Run a command for each file in a listFOR %%F IN (*.TXT) DO EDIT %%F

or at the command line:
C:\> FOR %F IN (*.TXT) DO EDIT %F

The loop variable name can only be one character.
Print a messageECHO Hello world
Jump to a label in a batch file:LOOP
GOTO LOOP
Test the value of a stringIF %VAR%==Y ECHO Yes
Test if a file existsIF EXIST TEMP.DAT DEL TEMP.DAT
Test the return value of the previous commandIF ERRORLEVEL 0 ECHO Success
Test the oppositeIF NOT ERRORLEVEL 0 ECHO Fail
Set the shell's search path for programsPATH C:\FDOS\BIN;C:\MY\BIN

or to reference the existing path:
PATH %PATH%;C:\MY\BIN

Use ; to separate paths.
Set a variableSET TEMPFILE=TEMP.DAT
Shift the command line optionsSHIFT or SHIFT 1 or any n

Reference command line options as %1, %2, and so on.
A commentREM This is a comment

Friday, June 29, 2018

FreeDOS commands quick-reference

FreeDOS turns 24 years old today (June 29)! That's a long time for any open source software project. Thanks to our FreeDOS community for keeping things going. And a special "thank you" to our developers, who continue to add new features and fix old bugs. You truly make FreeDOS better!

To celebrate this year's anniversary, I wrote a "quick reference" for our new users. This reference shows the most common FreeDOS commands, including a few important notes for new users.

You may also want to check out the free ebook we published last year: 23 Years of FreeDOS.

What do you want to do?How to do it on FreeDOS:
List directory contentsDIR
-in the directory “above”DIR ..
-in a different directoryDIR C:\FDOS\BIN
Change the current driveD:
Change the current directoryCD \FDOS\BIN
-“up” one directoryCD ..
Display the contents of a fileTYPE FILE.TXT
-one screen at a timeMORE FILE.TXT
Copy a fileCOPY FILE.TXT NEW.TXT
Delete a fileDEL FILE.TXT
Copy a directory and its contentsXCOPY DIR NEWDIR
Delete a directory and its contentsDELTREE MYFILES
Create a new directoryMKDIR NEWDIR
Remove an empty directoryRMDIR MYFILES
Rename a file or directoryREN FILE.TXT FILE.OLD
Show all lines in a file that contain “HelloFIND "Hello" FILE.TXT
-without regard for caseFIND /I "Hello" FILE.TXT
Clear the screenCLS
Edit a text fileEDIT FILE.TXT
View and set the dateDATE
View and set the timeTIME
Show the usage for a programDIR /? (for most programs)
Get systemwide helpHELP
Show the command historyHISTORY
Show the DOS versionVER

A few things to remember:
  • DOS commands and filenames can be upper or lowercase (DIR is the same as dir)
  • Pipes (|) are the same on DOS as on Linux
  • Output redirection (>) is the same too
  • . and .. are the same on DOS as on Linux
  • The directory separator is \ (such as C:\ or C:\FDOS or C:\FDOS\BIN)
  • File and directory names can only be 8.3 characters long (such as FILENAME.EXT)
  • DOS uses letters for each drive (C: is the first hard drive)
  • A full path is a drive letter and a directory path

Looking for more? I wrote a full FreeDOS cheat sheet for OpenSource, which you can find here: Celebrating 24 years of FreeDOS: Useful commands cheat sheet.