- Thinking Globally: The Earth Is Round
- Speaking the Same Language (in This Book!)
- Character Sets
- Language and Locale
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Speaking the Same Language (in This Book!)
Before going any further, I want to make sure everyone is on the same page regarding many of the basic terms in the book. As with describing any technology, I will be using a great deal of jargon to explain concepts. I hesitate to call this section a glossary because I will be explaining some of the terms at length. You might not yet understand how to actually make use of these terms, but at least you will know what I am talking about in later chapters!
Internationalization (i18N)
Internationalization, also known as globalization, is the process that converts an existing application to be globally aware. In an internationalized application, all the issues specific to other locales—such as changes in culture, convention, the input of data, the display of data, and the user interface elements—will work properly and will not be considered incorrect or offensive to the local user. Internationalization does not require an application's user interface to be in a different language from its original form (although that is the next step in the process, which is described further in the definitions of translation and localization). Internationalization does require that the user can type in data in her own language and to see it displayed properly in that language.
Internationalization is often abbreviated to i18N, which literally refers to the I at the beginning of the word, the N at the end, and the 18 letters in between (a lowercase i is used because an uppercase I can be mistaken for a lowercase l). Although i18N is not commonly used in Microsoft's documentation (it cannot be found at all in searches of current version of MSDN), in the world outside of Microsoft, i18N is the de facto term used to describe the concept. (Interestingly, inside Microsoft, i18N is used quite frequently in discussions about internationalization.)
In most cases, an internationalized application depends heavily on the default local settings of the machine on which it is running. This functionality enables me to write my application on a Japanese platform (probably using the localized version of Visual Basic); deploy it to customers in France, Hungary, or Korea; and expect the application to work properly. My users' French, Hungarian, and Korean data will display properly. Those users will be able to use their local keyboard or input method editor (IME, defined later in this chapter), and all dates, numbers, and currency values will be handled using the definitions in their regional settings.
i18N is the cornerstone of every other topic in this book, and understanding it is a prerequisite to understanding everything that follows. Many applications will never need to be converted beyond the step of internationalization, and, for Visual Basic 6, internationalization is one of the only readily supported scenarios.
Multinationalization (M18N)
If this book is describing a process, multinationalization is not necessarily a step that your Visual Basic applications need to take. Multinationalization is best defined as a large step beyond a merely internationalized application that brings your application to the world. A multinationalized application, as the name implies, brings the world to your application! A Visual Basic application can handle multiple languages, even crossing the code page barrier to support those different languages side by side. It does not depend on the machine's default language settings or user preferences to handle this data input and display.
The term multinationalization is one that I never saw used before I started using it in 1998. Therefore, I am going to borrow from the i18N example and refer to multinationalization as M18N throughout this book.
Visual Basic 6.0 does not support M18N out of the box because VB is so heavily tied to the default system code page and the user's regional settings. In all fairness, however, the strong need for this scenario was not created until almost a year after VB 6.0 shipped, with the introduction of Office 2000. With its Unicode support in all the major applications such as Word, Excel, and Access, and with excellent multiple code page support in programs such as FrontPage, Office 2000 was the first brick in the wall. The second was the Office 2000 multilanguage packs (sometimes known as langpacks), which were designed to allow the user interface elements of Office to display in any language at all, without degrading functionality. You can see in Figures 1 and 2 that, if you have CD2 of the multilanguage packs, you can change the Office user interface to Japanese. When you have done this, you will have a Japanese user interface in Office 2000 applications. For example, Access 2000 will now show a Japanese user interface, as seen in Figure 3.
){kind=link}
){kind=link}
){kind=link}
(It was, in fact, my own frustrations in trying to get VB6 add-ins designed for Office 2000 to work in this type of scenario that first led to my interest in M18N.)
Figure 1 The User Interface tab of the Office 2000 Language Settings applet enables you to choose a Japanese user interface for your Office applications.
Figure 2 The Enabled Languages tab enables you to choose what languages will be available for use in Microsoft Office.
Figure 3 Aside from the language difference, the U.S. English version of Access 2000 looks exactly the same as the localized Japanese product.
Even Office 2000 fell a little bit short of its goal, requiring a separate "South Asia" edition to handle some of the specific details for Thai, Vietnamese, and Hindi. But the capability became even more widespread with the release of Windows 2000, which is designed to support any language at all on any version of the product. Windows 2000 has its own version of the Office 2000 langpacks: the multilanguage user interface, or MUI. With this product, which can be installed over the English version of Windows 2000, it is possible to completely change the operating system's user interface from English to any locale you like. The MUI completely divorces the UI from the default system code page and the regional settings.
In fact, in Windows 2000, the goal of completely separating all three of these items is closer to being achieved than in any prior version of Windows. Figure 4 shows a U.S. Windows 2000 Server with a French UI and Russian regional settings. Figure 5 shows that same Windows 2000 Server with a Turkish UI and Korean regional settings. Note that although the user interface language changes, the behavior of dates is based on the regional settings (which you can see in the lower-right corner when the cursor hovers over the clock). Figure 6 shows where to set these three settings (the regional settings, the user interface language, and the default system language).
){kind=link}
){kind=link}
){kind=link}
NOTE
Before you get too excited about the capabilities of Windows 2000 MUI or the Office 2000 multilanguage pack, keep in mind that they are not included with Windows 2000 and Office 2000 by default. They are separate products that you must purchase and install. Note that these are add-ins to the original product that rely on the concept of worldwide EXE, an issue discussed later in this chapter.
Figure 4 Parlez-vous français? Hopefully, you do speak French if you are using the French user interface with the Windows 2000 MUI.
Figure 5 Even with a Turkish user interface, all it takes is Korean regional settings to cause dates and times to be shown in Korean when you hover over the date in the system tray.
Figure 6 Use the Regional Options Control Panel applet to set the user interface language and the default system code.
Unfortunately, this new scenario took Visual Basic by surprise. It created a need that VB simply is not designed to fulfill (at least, not without some help). In many senses, large parts of this book and especially of M18N are intended to help. The sections of the book related to multinationalization are an attempt to empower everyone to bring some of the same functionality of Windows 2000 and Office 2000 to their applications. Customers will very reasonably expect Microsoft's premier and best-selling rapid application development (RAD) tool to support the scenario that Office 2000 and Windows 2000 (more than half of Microsoft's revenue) have in large part created.
Translation
Translation is the process of representing the text of your application (the dialogs, menus, alerts, documentation, and so forth) in another language. Interestingly, the process of translation itself is becoming something that has even been automated with some success. Although a Web site such as http://babelfish.altavista.com is not perfect, it is very easy to get the overall meaning of text from its translations. However, there is a lot more to the process than the simple mapping of one word to another, as simple attempts to translate the same sentence repeatedly from French to English and back to French again will prove. That difference is best captured in our next term: localization.
Localization
Put simply, localization is when you move beyond the simple mapping of the translation process to make sure that the wording covers local customs and conventions. The process must also involve knowledge of the culture and expectations of speakers of both the source and target languages so that the process can effectively translate the differences between all these factors in a way that allows for total understanding. It is obvious why localization is so much more difficult than mere translation. A good localizer is key to producing a product (be it a software application or a book) that will succeed in other languages.
A good example of a localized VB application is the Replication Conflict Viewer that ships with Microsoft SQL Server 7.0 and Jet 4.0. The Swedish version of this application is shown in Figure 7. Other add-ins for Microsoft Office 2000 that were created in VB (and that are localized into between 20 and 26 languages) include the Word Web Page Wizard, the Office Web Chart Wizard, the MCE Cube Wizard, the Outlook Team Folders component, and the MultiSelect Objects add-in.
){kind=link}
Figure 7 If you have the Swedish version of Office 2000 and want to view and resolve your replication conflicts, there is a wizard created in Visual Basic that will display them for you.
CAUTION
Be careful not to insult a potential localizer by not fully explaining the distinction I am making here between translator and localizer. Many people who do an excellent job in the localization process are not using this exact terminology. It would seem that some of these terms make up a language of their own!
On the other hand, if you explain the concept properly, a localizer will probably be flattered by the distinction.
Code Pages
If you have ever used the Chr$() or Asc() functions in Visual Basic, you have already used code pages. A code page is a simple mapping between numbers (known as code points) and characters or glyphs. This standard allows the letter A to be code point 65, for example. The mapping enables you to represent all the languages that reside within that code page.
All the regular, single-byte code pages have 256 characters in them. The first 128 code points on all code pages are always the same; in a small bit of provincialism, the basic characters for English can be universally rendered on any code page. The other 128 characters are reserved for characters needed by other languages on the code page, such as [gb] for German or ç for French. Obviously, only so many languages can fit into each code page, which is where things start to get difficult. Those 256 code points are the same actual numbers, and it is up to the program you are running to properly choose the code page to use for interpreting them. To see this in action, you can open any of the 14 files in the code pages subdirectory on the CD in notepad.exe. These files represent the 14 Windows code pages. Only one of them will look right; the other 13 will look like gibberish (which one depends on your operating system's language). You can see the names of these different code pages in Table 1.
Table 1 Windows Code Pages
|
Code Page |
Windows Constant |
Windows Code Page Name |
|
1250 |
CP_EASTEUROPE |
Central European (Latin 2) |
|
1251 |
CP_RUSSIAN |
Cyrillic |
|
1252 |
CP_WESTEUROPE |
Western European (Latin 1) |
|
1253 |
CP_GREEK |
Greek |
|
1254 |
CP_TURKISH |
Turkish (Latin 5) |
|
1255 |
CP_HEBREW |
Hebrew |
|
1256 |
CP_ARABIC |
Arabic |
|
1257 |
CP_BALTIC |
Baltic Rim |
|
1258 |
CP_VIETNAMESE |
Vietnamese |
|
874 |
CP_THAI |
Thai |
|
932 |
CP_JAPAN |
Japanese |
|
936 |
CP_CHINA |
Simplified Chinese |
|
949 |
CP_KOREA |
Korean |
|
950 |
CP_TAIWAN |
Traditional Chinese |
The last four code pages in Table 1 are not limited to 256 characters—these Far East languages each have thousands of different characters. Each of these code pages uses double-byte character set (DBCS) encoding. All the rules about an application having to know the code page to use still apply because there is nothing intrinsic in a bunch of numbers that would tell you what language they are in.
Every Windows platform, even Windows 2000, has a single default code page that is used in most applications, including Visual Basic. Windows 2000 is the first member of the Windows family that has enabled you to change the default system code page.
The entire system of code pages gets more complex when you have to deal with all the alternative encodings for Far East languages, not to mention code pages for DOS, Macintosh, various ISO code pages, and more. The entire system is slowly being replaced by an encoding that is meant to include all languages. The ISO calls it 10646-1, and most of the rest of us know it as Unicode.
Unicode: UCS-2, UCS-4, UTF-7, UTF-8, UTF-16, and So On
Unicode was in many ways the answer to all the problems inherent in code pages. The Unicode standard is a 16-bit character set designed to encode all known characters. In practice, even 216 or 65536 characters are still not enough, which is why the 2.0 version of the Unicode standard defines a surrogate range. This range of characters turns into a pointer to the next 2 bytes, thus allowing a 32-bit character set. There are currently no languages in the surrogate range, and even Windows 2000 has only minimal support for the surrogate range. (It's hard to test surrogates because there are no characters in the surrogate range, yet.)
The Unicode standard is carefully kept in line with the ISO/IEC international encoding standard 10646-1. The changes to make this possible took place between versions 1.0 and 1.1 of the standard, and since then the Unicode Consortium has been very careful to stay in line with the ISO standard.
It is important to note that Unicode itself is not really an encoding; in fact, Unicode uses several different encodings that can represent it. In the world of Microsoft products, when you are talking about Unicode, you are generally talking about UCS-2/UTF-16, which is the standard that literally uses 2 bytes per character. This is what products such as SQL Server and Jet refer to when they mention Unicode, and it is the standard for strings that are used by COM and by the "W" or wide functions on Windows NT.
What the Heck Are the "W" and "A" Functions?
When Microsoft moved its operating systems to 32-bit, every API call that takes a string was given two separate APIs, one with an "A" suffix and one with a "W" suffix. The suffix choices are not really very intuitive, however. The "A" refers to ANSI but is really the multibyte character set (MBCS) version of the function, which assumes that the default code page for the system will be used. The "W" version, on the other hand, refers to "Wide" and assumes that the strings sent to this function will be Unicode strings in UCS-2/UTF-16 format. If you use the API Viewer in Visual Basic (see Figure 8), you will notice that it is always assuming that you want to use the "A" versions of the Windows API calls
){kind=link}
Figure 8 Visual Basic's API Viewer lists only the "A" versions of API calls.
Outside of Microsoft products and on the World Wide Web, usually when someone refers to Unicode he is talking about UTF-8. In fact, most databases (such as Oracle) seem to prefer UTF-8 to UTF-16 when it comes to encoding characters. To Windows, UTF-8 is just another multibyte encoding, one that just so happens to cover every language in the world. This is important to keep in mind when you need to translate information or data between your application and these other sources. One of the biggest uses of UTF-8 is in Extensible Markup Language (XML), whose default encoding is indeed UTF-8.
Glyphs
A glyph is simply a picture that represents a character; glyphs are the end result that users see once you deal with all of the issues with code pages and code points. There can be a great deal of variation with glyphs, even when they are meant to represent the same character, as you can see in Figure 9.
){kind=link}
Figure 9 How many ways can you say the letter A? There were more than 50 different ways on this machine!
Fonts
A font is a collection of glyphs that can map all of the various code points to the appropriate glyphs, as needed. It has been possible to create Unicode fonts since Windows first shipped 32-bit versions, and most font-making tools actually produce Unicode fonts, even if they represent characters only on a specific code page. Fonts can be very important when you need to display data, and they will be discussed in several parts of the book.
Strings and BSTRs
If you have been using Visual Basic at all, you know what a string is, but one of the things that VB hides from developers is how strings are represented and stored in memory. They are defined as BSTRs. The definition for a BSTR is simple: It is a wide-char (Unicode) pointer to a block of memory with the following characteristics:
-
A special contract for creation (allocation)
-
A special contract for destruction (deallocation)
-
An API (a set of functions), and prefixed by a length encoding that represents the size of the string
All of the details about BSTRs are handled automatically by Visual Basic, so, in theory, you do not need to worry about them.
In practice, because you are reading this book, you actually do need to worry about BSTRs and how they can be used. I will therefore be discussing BSTRs throughout the book.