gf-3.6: download/encoding-change.html
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML>
<HEAD>
<META NAME="generator" CONTENT="http://txt2tags.org">
<LINK REL="stylesheet" TYPE="text/css" HREF="../css/style.css">
<meta charset="UTF-8"><meta name = "viewport" content = "width = device-width"> <TITLE>GF character encoding changes</TITLE>
</HEAD><BODY BGCOLOR="white" TEXT="black">
<CENTER>
<H1><a href="../"><IMG src="../doc/Logos/gf0.png"></a>GF character encoding changes</H1>
<FONT SIZE="4"><I>Thomas Hallgren</I></FONT><BR>
<FONT SIZE="4">2014-06-23</FONT>
</CENTER>
<H2>Changes to character encodings in GF grammar files</H2>
<P>
Between the release of GF 3.5 and the next version, two changes were made
relating to character encodings in GF grammar files:
</P>
<OL>
<LI>The default character encoding was changed from Latin-1
(also known as iso-8859-1, cp1252) to UTF-8.
<P></P>
<LI>They way you specify alternate character encodings was changed. Instead of
using a <CODE>flags coding = ...</CODE> declaration in the source file, you should now
use a pragma <CODE>--# -coding=...</CODE> at the top of the file instead.
</OL>
<H2>Advantages</H2>
<P>
UTF-8 is the default encoding for text files on many systems these days, so
it makes sense to use it as the default for GF grammar files too.
</P>
<P>
Changing how alternate encodings are specified allows conversion to Unicode
to be done before parsing, which means that
</P>
<UL>
<LI>we can allow Unicode characters in identifiers, not just in string literals,
<LI>it makes accurate column positions in error messages possible,
<LI>and (an implementation detail) we can use Alex to generate the lexer again.
</UL>
<H2>How are my grammar files affected?</H2>
<P>
If your files still compile without errors after the change, you don't need
to do anything. (But see Known problems below!)
If you get one of the following errors,
</P>
<UL>
<LI><CODE>lexical error</CODE>,
<LI><CODE>encoding mismatch</CODE>,
<LI><CODE>Warning: default encoding has changed from Latin-1 to UTF-8</CODE>
</UL>
<P>
you need to add a
<CODE>--# -coding=...</CODE> pragma to your file (or convert it to UTF-8).
</P>
<UL>
<LI>For files containing only ASCII characters, no change is needed.
<LI>For files encoded in UTF-8 (and thus using a <CODE>flags coding=utf8</CODE>
declaration), no change is needed.
<LI>For files containing Latin-1 characters (e.g. characters like
å ä ö ü é), add a <CODE>#-- -coding=latin1</CODE> pragma at the top of the file.
<LI>For files using other encodings, copy the encoding specified in the
<CODE>flags coding=</CODE><I>enc</I> to a corresponding <CODE>--# -coding=</CODE><I>enc</I>.
</UL>
<P>
Grammars will still compile with GF-3.5 after these changes.
</P>
<P>
Note that GF only understands one option per pragma line. If you already
have a <CODE>--path=...</CODE> pragma, you can not put the <CODE>-coding=...</CODE> option on
the same line. Add it on a separate line:
</P>
<PRE>
--# -path=...
--# -coding=...
</PRE>
<P>
The recommendation for the future is to use UTF-8 for all source files.
</P>
<H2>Known problems</H2>
<P>
The intention is that if a grammar file is affected by the changed default
encoding, then you will see one of the messages listed in the previous
section when you compile the grammar. But there are a couple if issues to be
aware of:
</P>
<UL>
<LI>Alex 3.0 seems to be confused about the length of matched strings sometimes.
This can cause it to skip more than one line when it encounters a one-line
comment in a grammar file with character encoding problems. So instead of a
lexical error in the comment, you can get an odd syntax error
on a subsequent line.
<P></P>
<LI>If you explicitly specify -coding=utf8 for a file that is not in UTF-8, you
will not get an error, because the UTF-8 decoding function we currently use is
forgiving, substituting the Unicode replacement character �, instead of
reporting an error. Hopefully, we will be able to change this.
</UL>
<!-- html code generated by txt2tags 2.6 (http://txt2tags.org) -->
<!-- cmdline: txt2tags -thtml ./download/encoding-change.t2t -->
</BODY></HTML>