A PN2 Schemes Introduction
By Simon Steele
Wednesday, October 23, 2002
How schemes work in PN2:
Schemes in programmers notepad 2 are very different beasts to those seen in PN1. This is a brief article which explains the schemes mechanism in PN2.
What are schemes?
Schemes are the "things" that define what text looks like when displayed in PN. The following text:
char* myStr = "Stack String";
is transformed by the C++ scheme into:
char* myStr = "Stack String";
There are three parts to a PN2 scheme. The first of these is a piece of software which understands the language involved enough to specify, for example, that the word "void" is a keyword, and that characters between quote symbols (") form a string. This software is called a lexer - there are lexers available for a large number of languages, and more can be plugged-in. The important thing to note is that all that this software does is mark sections of a document as being of a certain type. As an aside here, generally the lexer will not know that "void" is a keyword, this is instead specified by the user.
Part two of the schemes system is the display engine. This engine (which is part of Scintilla) understands how to map from a type (such as "keyword") onto a colour, a font and other such style information. The display engine, fortunately, does not insist on keywords being blue - it is instead configured for each type of lexer with a list of mappings from types to visual styles. This enables user-defined colours. How are these colours specified?
For every language or scheme defined in PN2 there is a corresponding XML configuration block (or multiple blocks). These blocks live in the "schemes" directory where PN is installed, in *.scheme. The most important of these files is master.scheme. This file is read first, and contains simple instructions to the scheme parser on where to find the rest of the schemes. This is most usually in the form of:<imports>
The master file also defines a number of default styles to be used in other schemes. The idea behind this is that by default PN will use the same colour for "keywords" in every language giving a common look. This is, of course, overridable.
Each individual scheme is represented by a "language" XML block (note: Currently the Scheme element name is used for Scheme files - this may be changed, and language will become "Scheme"). This block defines which lexer to use for this particular scheme, the name of the language and configures some options related to lexer features. In addition, the language block links keyword lists to the scheme, and defines the colours for the different elements found by the lexer.
Almost all values in scheme definitions can use substitutions. Therefore, when you see a style configuration such as:
<stylename="STRING" key="6" fore="colour.string"/>
The colour.string is actually replaced with a value defined in master.scheme:
This is how the global values work. Styles can also be defined by class. For example:
<stylename="Keyword" key="5" class="keyword"/>
The visual style applied to keywords here is defined to be of the class "keyword". This is also defined in master.scheme:<style-classes>
<!-- All styles inherit the default style. Font fore and back colours are not specified - this way we inherit the system defaults -->
<style-class name="default" size="10" font="Lucida Console"/>
<style-class name="keyword" fore="colour.keyword" bold="true"/>
As hinted at in the above XML fragment, all styles automatically inherit the "default" class. This is the base font configuration for PN. This can, of course, be overridden on a scheme-by-scheme basis if necessary.
So, we can see that Lexer + Schemes + Display = Syntax Highlighted Text
Of course, somewhere in the middle these have to be slotted together. When PN2 loads, it compiles these scheme files into a binary form ready for transmission to the display component when necessary. These binary files are extremely quick to load and cut out the speed concerns of working with complex XML documents when defining schemes. The compiled schemes are stored in the user's application data directory as *.cscheme.
When a scheme is selected in PN2, the relevant compiled scheme is loaded and the settings are streamed into the display component. The display component then asks the selected lexer to mark out areas of text with their relevant styles, and applies its configured colours in order to display these sections of text.
The scheme files define not only the colours to be used for schemes, but also the available configurable styles for each scheme. Unfortunately, with only one set of these files users of a single computer or of a networked installation would have to share the scheme configuration. Instead, PN2 needs to provide user-by-user schemes. This is not currently available in PN2 and this feature is still on the drawing board. Current plans look something like this:
Because the scheme files define a nice "default" set of scheme configurations, it seems a shame to have an entire set of these for each user. The reason why this would be a bad idea:
When new schemes were added, it would be difficult to ensure that users got these without overwriting some of their settings.
If a scheme definition changed a user would lose their configuration in order to get the updated definition.
and some more...
Instead, a good solution seems to be to store a simple file per-user which defines changes to the default scheme configuration. Something like:
my keywords here
<!-- Get white on red keywords! -->
<Style key="5" fore="#ffffff" back="#ff0000" />
At compile time, the user settings would be read into a list first, and then used in an additional substitution process when the Scheme XML files are read in. When the user edits their settings, the only changes will be to the user settings file, allowing the user to reap the benefits of updated scheme XML files. note: This is implemented in PN 2 now in a similar form to that shown here.
Providing plug-in lexers:
Lexers can be developed in a large number of programming languages, but the easiest way is to develop one using C++. This article is not the place for discussion of lexer writing, but feel free to e-mail me about it.
A note on GUIs:
The XML format described is very powerful, but the GUI placed on top will probably make use of few of the features available in order to maintain usability. The reason for power in the Scheme XML is to ease scheme creation and initial PN configuration. Some of this power will hopefully make the GUI easier to use, and schemes easier to configure for the user. In the future I am planning on removing support for global replacement, this will speed up the schemes compiler somewhat with a minimal cost to configurability.
This document is a work in progress, but I hope it has helped the interested in understanding the scheme mechanisms in PN2. If you feel that anything has not been explained clearly enough, please let me know and I'll try to clarify the article.Back to top...