(Current as of 21 Feb 2007)
This document provides guidance on using SAFARI to define a customized
IDE
for a new language. It addresses the following topics:
- Prerequisites
- Identifying a project for the language
- Creating a language description
- Creating a grammar and parsing service
- Creating a token-coloring service
- Creating an outlining service
- Creating a text-folding service
- Creating a reference resolver and hyperlinking service
- Creating a content-proposal service
- Creating a builder service
- Creating a nature enabler
- Creating a preferences service and pages
- Using your SAFARI IDE
The discussion of the parsing service mainly addresses the case in which LPG (formerly and still occasionally known as JikesPG) will be used to define the language grammar and generate a lexer and parser. However, other approaches to grammar specification and parser development can be accommodated in SAFARI, and the treatment of most other topics here is independent of the manner in which the parser is provided.
A note on terminology: The text below will refer to several of the classes generated by SAFARI by simplified "generic" names, such as the "Plugin" class or the "ParseController" class. As these classes are actually generated, their names will typically be prefixed by the name of the language to which they apply. Thus, if your language is called "MyLanguage", SAFARI will generate classes for you that are called "MyLanguagePlugin", "MyLanguageParseController", and so on.
Prerequisites
SAFARI runs on Eclipse 3.1.2; we are working on adapting it to Eclipse 3.2. (SAFARI may also run on earlier 3.1 releases. It will almost certainly not (ever) run on any prior version of Eclipse.)SAFARI now requires a Java 5 VM, due to a dependency on the Java Concurrency Utilities within the x10.runtime project.
Parts of SAFARI depend on Polyglot, currently version 2.1.0, which can be downloaded from http://www.cs.cornell.edu/projects/polyglot/.
You must have the SAFARI features installed in your Eclipse workspace. As of 06 Feb 2007 the latest SAFARI release is number 1.0.45. SAFARI is distributed as four features:
- SAFARI (feature id org.eclipse.safari)
- SAFARI Runtime (feature id org.eclipse.safari.runtime)
- JikesPG IDE (feature id org.eclipse.safari.jikespg)
- X10 Development Toolkit (feature id org.eclipse.safari.x10dt)
For information on installing SAFARI, please look at the installation guide (which also lists SAFARI prerequisites).
Identifying a project for the language
The goal of this step is to set up an Eclipse project for the development of SAFARI IDE for your language.
A SAFARI IDE must be defined in a Eclipse plugin
project. Each SAFARI IDE must be defined in a separate
project. If you do not have a suitable plugin project
available as a home for the new IDE then you should create one.
(A simple plugin project is all that's required.)
When you use the Eclipse New Project wizard to create a new plugin project, the wizard by default creates a package with the name of the project. In that package the wizard creates a plugin class with a name of the form "<ProjectName>Plugin.java." The Safari New Programming Language wizard likewise creates a new package and plugin class, but one that by default is named based on the programming language rather than on the project. If you use the default values in each wizard, you can end up with two plugin classes, each in a different package.
There are two ways to avoid this. One is
just to give your project and language the same name. The other
is, during plugin-project creation, to override the default values and
give your plugin class and package a name that matches that of your
language. (With the latter approach it is possible to have the
name of the language be different from the name of the project in which
the language IDE is defined.)
(Note:
There is a SAFARI wizard called "New Project Wizard" but this is a
wizard for creating project wizards, not a wizard for creating
projects.)
Creating
a language description
The goal of this step is to provide basic information about the identity and characteristics of the new language.
This step is accomplished entirely within a SAFARI
wizard. To run this wizard invoke:
In the "Programming Language" wizard, you must provide information for the following required fields:
- Project: Browse to or type-in the name of your language project
- Language: Enter the name of the language (which for now must be the same as the name of the project). This is the canonical name for your language, by which it will be identified to various SAFARI services. Within SAFARI it is case-insensitive.
- Description: The “description” field is for the entry of a user-readable, user-sensible description of the language. Enter whatever description you think is appropriate.
- Extensions: The “extensions” field is for the entry of file-name extensions that may be used for source files in the language. Enter a non-empty, comma-separated list of extensions (omitting dots, and without spaces). The SAFARI services defined for your language will be available for files with these extensions.
You may also provide information in the following
optional
fields:
- Synonyms: This is a comma-separated list of alternative names for your language
- DerivedFrom: The “derivedFrom” field should be the canonical name of the programming language, if any, from which the new language is derived. The "derived from" language, if any, should be one for which there is already a SAFARI IDE. It is used within SAFARI for determining inheritance relationships when locating language services for a given language (for example, locating an appropriate editor if none is defined specifically for your language). If your language is derived from another SAFARI-supported language then this field should be filled in accordingly. If your language is not derived from another SAFARI-supported language then this field should be ignored.
- Icon: This should be the project-relative pathname of an icon that can be used in graphical interfaces to mark files in your language.
- URL: This is intended for the URL of a web page that contains information about your language.
- Validator: The “validator” field can be given the fully-qualified name of a class that can determine whether a given input file actually contains text in the given programming language. This class must extend the LanguageValidator class
Hit “Finish” when you are done editing the various
fields in
the wizard. A SAFARI language
description extension will be created using the above information and
added to
the project’s plugin.xml file. This step
also updates the projects Plugin class (to extend SAFARIPluginBase,
among other modifications), or it will create a Plugin class if one
does not exist.
(In earlier versions of SAFARI, this wizard also
generates a "preferences" package and some preferences-related
classes. In later versions this feature will probably be
eliminated as a separate wizard has been provided for generating a
preferences service and preference pages.)
Creating a grammar and parsing service
The goal of this step is to create a grammar for the language and a lexer and parser based on this grammar.To run the wizard to generate the grammar skeletons, invoke
- Be sure that the proper names are
entered into the project and language fields.
(These may be filled in automatically or you may have to set them
manually.) As these values are entered, reasonable
default values will be automatically entered into other fields.
- If you wish, edit the "class" field to specify a non-default fully-qualified name for the parse controller that will be generated by the wizard. This class is used to give access to the token stream, AST, and so on, to the various higher-level language services. (Note: A parse controller is necessary, so this field should be given a usable class name.)
- For the "templates" field most users will want to use the default setting of "UIDE" (which indicates to JikesPG where it can find the grammar file templates)
- For the "options", check or uncheck these according to your own needs:
- "Language has keywords" should be checked or not according to whether your language has keywords; this will affect the generation of a keyword lexer
- "Language requires backtracking" should be checked or not according to whether your language requires a backtracking parser; this will affect the ability of the generated pareser to backtrack
- "JikesPG auto-generated AST classes"
should be checked or not according to whether you want JikesPG to
automatically generate AST classes for your language. (This can
be a big convenience.)
- Hit "Finish" when done editing.
(It is also possible, by configuring the JikesPG options differently, to have the AST-related types generated as members within the parser class. This is controlled by the "automatic_ast" option within the ".g" file. With the option given as "automatic_ast=toplevel" a separate AST package is generated. If this option is changed to "automatic_ast=nested"--or just "automatic_ast"-- then the AST-related classes are generated as members of the parser class. For this document we assume that the AST is represented in its own package.)
Additionally upon finishing the wizard, the
JikesPG grammar file for your
language will open in the editor, and the structure of the file should
appear
in the "Outline" view.
Your language project should be configured to include the JikesPG Grammar File Builder. This should happen automatically when you run the parsing-service wizard. If for some reason your language project is not configured to include the JikesPG builder, then you can enable the builder by selecting the "Enable JikesPG Builder" in the context menu for the project. If for some reason you want to run JikesPG manually (which shouldn't usually be neecessary), you can find information about that here.
Notes about
compilation
The generated
parser classes will have a compilation dependency on the lpg plugin,
which is provided as part of the SAFARI release (lpg is actually an
open-source release of JikesPG). When the Jikes PG grammar and
parser wizard is run, the wizard adds a plugin dependency on lpg to
your language project. However, although the dependency on lpg
should then appear in the plugin dependency list for your project, it
may not actually be recognized. As a result, the ParseController
may not compile properly. To cause the new dependency to be
recognized, it seems that the plugin.xml file, or the dependencies
within it, must be manipulated somehow. For instance, if the
plugin.xml file is open, you can go to the dependencies tab, select the
lpg dependency, and move it up or down in the dependencies list,
afterwards saving the file. That should cause the dependency to
be recognized and trigger a recompilation of the project, which should
succeed.
If your grammar defines a type with a name that
duplicates one of the Java type names (such as “Number”), then you will
need to disambiguate references to this type in the parser (which can
be done by explicitly importing the corresponding AST node types for
your language). The Java compiler should make any problem like
this evident to you.
Other approaches
If you have your own parser: Since the goal of this step is to produce a lexer and parser, if you already have a lexer and parser that meet SAFARI requirements then these may be used directly. The "Hand-written Parser" wizard allows you to select an existing parser and creates a corresponding parser extension in the plugin.xml file. To work within SAFARI, any "hand written" or non-JikesPG-generated parser should, at a minimum, implement the ILexer, IParser, and IParseController interfaces in the org.eclipse.uide.parser package in the org.eclipse.uide.runtime project. This will entail providing implementations for several additional types defined by SAFARI or JikesPG. (Some of these, unfortunately, are currently defined by concrete types, although we expect to provide interfaces for these in the future.) Thus, at this time we recommend the use of JikesPG to generate parsers for use in SAFARI. NOTE: If you have a parser that was generated by JikesPG but outside of SAFARI, it should be relatively straightforward to adapt it to SAFARI.
If you have your own grammar: If JikesPG is to be used to generate the lexer and parser, but you already have grammars for the language to be parsed, then these may be substituted into, or in place of, the SAFARI-generated templates. Of course, the resulting grammar files must be compatible with JikesPG.
Creating a token-coloring service
As before, in the wizard assure that the names of
your project and language
have been correctly entered into the appropriate fields.
If desired, edit the default name of the implementation class or
its package. Hit "Finish" when done.
As you can see, getColoring() calls IToken.getKind() to determine the token kind (represented by an integer) and uses that to compute the correct text attributes. The set of valid token kinds is defined by the lexical analyzer; they are typically located in the JikesPG-generated "Parsersym" interface (in the parser package) and generally start with a prefix like "TK_".
The generated TokenColorer class is somewhat
dependent on JikesPG:
- The TokenColorer class implements a Parsersym interface that is
one of the standard, language-specific interfaces generated by JikesPG
- The getColoring(..) method takes an instance of an IToken, which is defined in lpg.lpgjavaruntime
- The getColoring(..) method also takes an instance of an IParseController; although this is defined by SAFARI, the default implementation makes use of several types generated by JikesPG
Creating an outlining service
The goal of this step is to create a service that
provides an outline view of documents in your language.
As for the TokenColorer, there is a SAFARI wizard
that creates a skeleton class that contains the generic structure of
the outliner,
and this must be tailored to create an outliner for the specific
language being
defined.
The generated Outliner class extends
org.eclipse.uide.defaults.DefaultOutliner. The principle method
in DefaultOutliner is createOutlinePresentation(),
which retrieves the current AST (from an IParseController) and then
visits
the AST, populating
the outline tree with various TreeItems corresponding to AST nodes of
interest. The visiting is done by an
instance of the
OutlineVisitor class, which is a member of the generated Outliner class
and extends the
AbstractVisitor class.
Some example visit(..) methods for the simple expression language are shown in the generated Outliner.OutlineVisitor class (which may prevent the generated file from compiling). These must be adapted for the specific language being defined.
The generated Outliner class is somewhat dependent
on JikesPG:
- The TokenColorer class implements a Parsersym interface that is
one of the standard, language-specific interfaces generated by JikesPG
- One of its central methods (to set the outline tree) makes use of IToken, which is defined in lpg.lpgjavaruntime
- The createOutlinePresentation(..) method takes an instance of an IParseController which, although it is defined by SAFARI, has a default implementation that makes use of several types generated by JikesPG
Creating a text-folding service
The goal of this step is to create a service that supports the folding of hierarchical syntactic elements in text editors for the language. As with the outlining service, the SAFARI wizard for text folding creates a skeleton class that contains the generic structure to support text folding and this class must be tailored to the specifics of the language being defined.
"File" -> "New"
->
"IDE Language Support" -> "Editor Services" ->
As before, in the wizard assure that the names of
the project and language are correct and, if desired, edit the package
and class name for the implementation.
Hit "Finish" when done.
Creating a reference resolver and hyperlinking service
The goal of this step is to create an implementation for a reference resolution service. This service is not apparent to users, but it enables a user-visible hyperlinking service (that is implemented by a SAFARI library class), and it contributes to the implementation of a user-visible content assist service (and may be used by clients). The SAFARI wizard to generate the reference resolver is called "NewReferenceResolver" and is run by invoking"File" -> "New"
->
"IDE Language Support" -> "Editor Services" ->
The wizard by default generates a referenceReslovers package with a ReferenceResolver class. The principal methods to be implemented are getLinkTarget(node) and getLinkText(node). The user can also implement methods that will perform filtering on given link source nodes to eliminate nodes that aren't appropriate in that role.
Example implementations are provided in the generated class that work with the simple expression language used for examples in the other services. For getLinkTarget(..), the basic logic is to return null for any given node that cannot be a link source, and for a node that can serve as a link sorce, to build a structure of the scopes and declarations of the program (using an AST visitor), then to return the declaration of the given source as found in that structure. Of course, in general, the details of reference resolution are langauge specific, and the amount of programming required to complete the ReferenceResolver may depend on the functionality provided by avaliable language tools such as a compiler. If available language tools already support reference resolution, then the ReferenceResolver may just need to retrieve the relevant information from these tools. On the other hand, if the available language tools do not support reference resolution, then the ReferenceResolver may need to encode the logic necessary to perform resolution.
Once reference resolution is implemented, hyperlinking is automatically enabled in the language editor. The hyperlinking service is implemented by a SAFARI library class and only depends on the ReferenceResolver to become functional. Hyperlinking in a SAFARI editor works as it does in an Eclipse JDT Java editor, by control-clicking on a suitable link target.
Creating a content-proposal service
The content-proposal service (also known as "content assist") is intended to work in a SAFARI editor similarly to the way as it works in the Ecipse JDT Java editor: with the cursor position within or at the end of a word (keyword or identifier), typing control-space will bring up a list of proposals for completing the word, possibly including complex expressions or commands, consistent with the prefix of the word up to the point of the cursor and consistent with the syntax and semantics of the program at that point. Of course, content assist in the JDT Java editor is quite sophisticated. The functionality of this service in a SAFARI editor depends on how the service is implemented for the language of the editor.The SAFARI wizard to generate a content-proposal service is called "NewContentProposer" and is run by invoking
"File" -> "New"
->
"IDE Language Support" -> "Editor Services" ->
As with previous wizards, when you run this one, be sure that the name of the project and language are correct, edit the name of the implementation package and class as needed, and nit "Finish" when done.
The generated implementation skeleton for this service contains code that works for the simple expression language that is used for examples with other services. The main method to be implemented by SAFARI users is getContentProposals(..), which takes a parse controller and an offset and returns an array of completion proposals. The parse controller provides access to the parse tree withiin which proposals are sought, and the offset indicates the point within the source file at which proposals are sought. The basic logic of the example implementation is simple: first get the token that covers the offset in the source file and the prefix in that token up to the offset; then get the AST node that corresponds to that token; then, if the node is of an appropriate type, gather and return the completion proposals for the prefix at that node. To that depth, the logic is largely language-independent. Gathering the completion proposals will usually be much more language specific. In the generated example, the focus is on variable; variables that are visible at the offset point are considered, and completion proposals are constructed for those variables with identifiers that match the prefix under consideration. Implementation for the content proposal service will generally have to address the kind of entities for which proposals are generated (e.g., variables? key words?), the kind of proposals that are returned (e.g., just identifiers or more complex constructs), and the semantics involved in the gathering of prospective proposals (e.g., visibility rules).
Creating
a builder service
To invoke the NewBuilder wizard,
In the wizard:
- Assure that the names of the project and language are entered correctly.
- This wizard requires that an id be
provided for the builder extension and also allows for the provision of
a human-friendly name for the builder extension. The id is
needed because a plugin may have multiple builders and these need to be
distinguishable.
- Although it is not necessary to define a nature to which your builder will be associated, this is a fairly common practice. Whether a builder is configurable depends on the design of the particular builder. If the values for "hasNature" and "isConfigurable" are left blank then they are treated as if false.
- Edit the class field as necessary to
specify the desired package and class name for the implementation.
- The "name" and "value" fields represent a parameter that may be used to pass external information to the builder. The "run" element of the builders schema in org.eclipse.core.resources allows a builder to have some number of parameter elements (0 or more); in each parameter element, the name and value are required. As a simplification, the SAFARI NewBuilder wizard provides exactly one parameter element. Since the name and value field are required, some values must be filled in here, but these values can be ignored by the builder if none are actually needed. (We expect to make the New Builder wizard more flexible in this regard in the future.)
- Finally, if your language has a
translation into Java, and you want to use a Java debugger on it, check
the checkbox "Add SMAP support" if you want to provide source-line
mapping between source in your language and its Java representation.
- Hit "Finish" when done.
- getErrorMarker(..)
- getInfoMarker(..)
- getWarningMarker(..)
- getPlugin(..)
- isSourceFile(..)
- isNonRootSourceFile(..)
- isOutputFolder(..)
- compile(..)
The method isNonRootSourceFile(..) is used to
identify files (like C/C++ ".h" files or JikesPG ".gi" files) that
should not be compiled directly but that should nevertheless be
processed for dependencies.
In any case you will have to edit the
compile(..) method,
which is really the main “build” method.
This will have to call your “compiler” in an appropriate way for
a given source file, retrieve any diagnostic messages that result from
the building, and create problem markers
for those. If your build activity runs
in an external process, or uses java.io to create output files, you
will also
need to call IResource.refresh() on any folders that contain generated
output
files (assuming that they are somewhere in your workspace).
The figure below
/*The Nature class extends org.eclipse.uide.core.ProjectNatureBase and provides some simple method implementations and stubs. These can be used as-is without further modification unless more sophisticated control over building is required for the nature. (With a new nature this is not likely to be true initially.)
* Implementation of the compile method in a SAFARI generated Builder
* for a subset of JavaScript known as “Jsdiv” using a JikesPG-generated
* Parse Controller and the provided MarkerCreator implementation for
* IMessageHandler
*/
protected void compile(final IFile file, IProgressMonitor monitor) {
try {
// Get a Parse Controller to serve as the “compiler”
IParseController parseController = new JsdivParseController();
// Get a MarkerCreator to handle error messages from the parse controller
// and create corresponding marker annotations on the input file
MarkerCreator markerCreator = new MarkerCreator(file, parseController);
// Initialize the Parse Controller
parseController.initialize(
file.getProjectRelativePath().toString(),
file.getProject(), markerCreator);
// Get file contents for parsing
String contents = ContentExtractionUtility.extractContentsToString(file.getLocation().toString());
// Parse the contents
parseController.parse(contents, false, monitor);
// Refresh the parent
doRefresh(file.getParent());
}
}
catch (Exception e) {
JsdivPlugin.getInstance().writeErrorMsg(e.getMessage());
e.printStackTrace();
}
}
Problems with the
MANIFEST.MF file: On some occasions (that seem to be
especially associated with the running of the New Builder
wizard), you may notice an error in your project's MANIFEST.MF
file that is related to the line "Bundle-SymbolicName: leg;
singleton:=true". The error message will say something to
the effect that the attribute singleton
must be true. Of course, it has been set to true.
The problem is that different forms of assignment seem to be preferred
in different circumstances. If you encounter this error, try
changing "singleton:=true"
to "singleton=true" (or
vice versa).
A note on Annotations and Markers
When a SAFARI editor runs, the source text is
parsed frequently and error messages are reported as annotations that
appear in the editor. These annotations are not associated with
the
source file and do not appear in the "Problems" view. When a
SAFARI Builder runs, the editor annotations are deleted and a new set
of marker annotations are created. These annotations also appear
in the editor (and may seem identical to pre-build editor
annotations). However, the marker annotations are
associated with the file and do appear in the "Problems" view.
Building may occur automatically, whenever a file is saved (or a
project is cleaned), or building may occur manually, only when
explicitly invoked by the user. Regardless, problem markers will
appear when builds occur (provided that there are problems). When
a file is opened in an editor the
problem markers are deleted but the file is parsed immediately and new
editor annotations are created.
Creating a nature enabler
Once you have created a builder and corresponding nature for your language, you need to associate the nature with a development project (containing source code in your language) in order for the builder to run automatically on the project. Eclipse does not make it easy to manually associate a new nature to an existing project, so SAFARI provides a "Nature Enabler" wizard that creates a context-menu action that will do this for you automatically.To invoke the Nature Enabler wizard,
In the wizard, assure that the names of the project and language are correct and hit finish.
Once the wizard has run, when you right-click on a project in your development workspace, you should see a menu item that says "Enable
Note that, as currently implemented, the menu actions are made available only for Java projects, so your development project must be a Java project to take advantage of this feature.
Creating a preferences service and pages
SAFARI provides a preference service that is based on the Eclipse Preferences Service (org.eclipse.core.internal.preferences.PreferencesService, since 3.0). The SAFARI model of preferences is basically that of the Preferences Service. This is a layered model, with four intrinsicly supported layers. From most general to most specific, these are:- Default
- Configuration (i.e., workspace configuration)
- Instance (i.e., workspace instance)
- Project
SAFARI initially supports a a style of preference page with four tabs, one for each of the four preference levels. The page and the four tabs (with certain annotations and controls) are generated automatically by the SAFARI New Preference Page wizard. As an alternative, SAFARI also supports the generation of a simple, untabbed preference page.
To invoke the New Preference Page wizard,
In the wizard:
- As usual, assure that the names of the project and language are entered correctly.
- As a language may have multiple
preference pages, you must specify a unique Eclispe identifier and
usre-friendly name for a particular page
- As usual, edit the package and class name for the implementation to be what you want
- The optional "category" field is used to designate an existing preference page, if any, under which the new page will be listed in the Eclipse preferences menu (Window -> Preferences ...). To designate the "parent" page, you must provide its Eclipse id.
- Finally, if you do not wish to
generate a full SAFARI preference page, you can generate a simple
(untabbed) page with some text by filling that text into the
"alternative" field. (If this field is left empty, then the usual
tabbed SAFARI preference page will be generated).
- Hit "Finish" when done.
Additional information about the SAFARI preferences model and service, and about SAFARI preference pages and fields and their implemenation, can be found in Preferences in SAFARI IDEs.
Using your SAFARI IDE
To use your SAFARI IDE, you will typically create a development workspace that contains one or more development projects in which you will write programs in your language, taking advantage of the services provided by your SAFARI IDE.To make a development workspace for your SAFARI IDE, you simply need to import the plugin for your SAFARI IDE into a new or existing workspace. (Note: This will have to be a different workspace from the one in which you developed your SAFARI IDE.)
Once you have imported the plugin for your SAFARI IDE into a workspace, any project in the workspace can serve as a development project for your language. As soon as you start creating source files in the language, in any project, your SAFARI IDE will recognize the files and automatically provide IDE services. (Note that recognition of the files is based on the filename extensions that were provided with the initial language description, so you must name the source files with these extensions.)
The one exception to the automatic provision of IDE services relates to natures and builders. Builders for your language will not run automatically on the files within a project unless the corresponding nature has been associated to the project. You can manually add an entry for your builder and nature to the ".project" file for your development project. (See the .project file for org.eclipse.uide for examples.) However, if your project is a Java project, then SAFARI provides a way to do this automatically, through the Nature Enabler wizard.
To be able to take advantage of the Nature Enabler is one reason that you may want to try out your SAFARI IDE using a Java project for development. If it so happens that your language will be translated to Java, then you'll probably want to use a Java project for that reason (or, to look at it the other way, your SAFARI IDE will work well with any existing Java projects that you may be using for development).
Final words of advice
Be careful entering information into the wizards. The wizards will generally not let you get away without providing information for required fields, but they (and the rest of SAFARI) are not necessarily robust with respect to errors in the provided information. Errors in the provided information may or may not show up directly and may or may not have confusing effects.You can update some of the SAFARI information manually. While the wizards are intended to facilitate the entry of necessary (or at least useful) information at appropriate times and places, you can also manually edit the plugin.xml file to set (or reset) many of the attributes associated with SAFARI extensions. So, if you want to add an icon or validator to an IDE that is already defined, you can do that without rerunning the wizard. (This can help to avoid problems that can occur when the wizards are rerun ...)
Be careful when rerunning the wizards!
- Several of the wizards generate skeleton files that must be tailored by the user. If you rerun one of these wizards, the tailored files will be clobbered and updates to them will be lost. So backup these files where they won't be clobbered.
- Several of the wizards add extensions to the plugin.xml file of
the IDE project. Rerunning these wizards does not remove the
extensions created by previous invocations. So, if you want to
rerun one of these wizards, you should manually delete the
corresponding extension(s) from the plugin.xml file.