Commit d6aa8950 authored by Thanassis Tsiodras's avatar Thanassis Tsiodras
Browse files

ANTLR3 runtime

parent 99e41047
Benjamin Niemann <pink at odahoda dot de>: Main developer of Python target.
Clinton Roy <clinton.roy at gmail dot com>: AST templates and runtime.
[The "BSD licence"]
Copyright (c) 2003-2006 Terence Parr
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
3. The name of the author may not be used to endorse or promote products
derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
include LICENSE AUTHORS ez_setup.py
Metadata-Version: 1.0
Name: antlr_python_runtime
Version: 3.1.3
Summary: Runtime package for ANTLR3
Home-page: http://www.antlr.org/
Author: Benjamin Niemann
Author-email: pink@odahoda.de
License: BSD
Download-URL: http://www.antlr.org/download.html
Description: This is the runtime package for ANTLR3, which is required to use parsers
generated by ANTLR3.
Platform: UNKNOWN
1) ABOUT
========
This is the Python package 'antlr3', which is required to use parsers created
by the ANTLR3 tool. See <http://www.antlr.org/> for more information about
ANTLR3.
2) STATUS
=========
The Python target for ANTLR3 is still in beta. Documentation is lacking, some
bits of the code is not yet done, some functionality has not been tested yet.
Also the API might change a bit - it currently mimics the Java implementation,
but it may be made a bit more pythonic here and there.
WARNING: Currently the runtime library for V3.1 is not compatible with
recognizers generated by ANTLR V3.0.x. If you are an application developer,
then the suggested way to solve this is to package the correct runtime with
your application. Installing the runtime in the global site-packages directory
may not be a good idea.
It is still undetermined, if a future release of the V3.1 runtime will be
compatible with V3.0.x recognizers or if future runtimes V3.2+ will be
compatible with V3.1 recognizers.
Sorry for the inconvenience.
3) DOWNLOAD
===========
This runtime is part of the ANTLR distribution. The latest version can be found
at <http://www.antlr.org/download.html>.
If you are interested in the latest, most bleeding edge version, have a look at
the perforce depot at <http://fisheye2.cenqua.com/browse/antlr>. There are
tarballs ready to download, so you don't have to install the perforce client.
4) INSTALLATION
===============
Just like any other Python package:
$ python setup.py install
See <http://docs.python.org/inst/> for more information.
5) DOCUMENTATION
================
Documentation (as far as it exists) can be found in the wiki
<http://www.antlr.org/wiki/display/ANTLR3/Antlr3PythonTarget>
6) REPORTING BUGS
=================
Please send bug reports to the ANTLR mailing list
<http://www.antlr.org:8080/mailman/listinfo/antlr-interest> or
<pink@odahoda.de>.
Existing bugs may appear someday in the bugtracker:
<http://www.antlr.org:8888/browse/ANTLR>
7) HACKING
==========
Only the runtime package can be found here. There are also some StringTemplate
files in 'src/org/antlr/codegen/templates/Python/' and some Java code in
'src/org/antlr/codegen/PythonTarget.java' (of the main ANTLR3 source
distribution).
If there are no directories 'tests' and 'unittests' in 'runtime/Python', you
should fetch the latest ANTLR3 version from the perforce depot. See section
DOWNLOAD.
You'll need java and ant in order to compile and use the tool.
Be sure to properly setup your CLASSPATH.
(FIXME: is there some generic information, how to build it yourself? I should
point to it to avoid duplication.)
You can then use the commands
$ python setup.py unittest
$ python setup.py functest
to ensure that changes do not break existing behaviour.
Please send patches to <pink@odahoda.de>. For larger code contributions you'll
have to sign the "Developer's Certificate of Origin", which can be found on
<http://www.antlr.org/license.html> or use the feedback form at
<http://www.antlr.org/misc/feedback>.
""" @package antlr3
@brief ANTLR3 runtime package
This module contains all support classes, which are needed to use recognizers
generated by ANTLR3.
@mainpage
\note Please be warned that the line numbers in the API documentation do not
match the real locations in the source code of the package. This is an
unintended artifact of doxygen, which I could only convince to use the
correct module names by concatenating all files from the package into a single
module file...
Here is a little overview over the most commonly used classes provided by
this runtime:
@section recognizers Recognizers
These recognizers are baseclasses for the code which is generated by ANTLR3.
- BaseRecognizer: Base class with common recognizer functionality.
- Lexer: Base class for lexers.
- Parser: Base class for parsers.
- tree.TreeParser: Base class for %tree parser.
@section streams Streams
Each recognizer pulls its input from one of the stream classes below. Streams
handle stuff like buffering, look-ahead and seeking.
A character stream is usually the first element in the pipeline of a typical
ANTLR3 application. It is used as the input for a Lexer.
- ANTLRStringStream: Reads from a string objects. The input should be a unicode
object, or ANTLR3 will have trouble decoding non-ascii data.
- ANTLRFileStream: Opens a file and read the contents, with optional character
decoding.
- ANTLRInputStream: Reads the date from a file-like object, with optional
character decoding.
A Parser needs a TokenStream as input (which in turn is usually fed by a
Lexer):
- CommonTokenStream: A basic and most commonly used TokenStream
implementation.
- TokenRewriteStream: A modification of CommonTokenStream that allows the
stream to be altered (by the Parser). See the 'tweak' example for a usecase.
And tree.TreeParser finally fetches its input from a tree.TreeNodeStream:
- tree.CommonTreeNodeStream: A basic and most commonly used tree.TreeNodeStream
implementation.
@section tokenstrees Tokens and Trees
A Lexer emits Token objects which are usually buffered by a TokenStream. A
Parser can build a Tree, if the output=AST option has been set in the grammar.
The runtime provides these Token implementations:
- CommonToken: A basic and most commonly used Token implementation.
- ClassicToken: A Token object as used in ANTLR 2.x, used to %tree
construction.
Tree objects are wrapper for Token objects.
- tree.CommonTree: A basic and most commonly used Tree implementation.
A tree.TreeAdaptor is used by the parser to create tree.Tree objects for the
input Token objects.
- tree.CommonTreeAdaptor: A basic and most commonly used tree.TreeAdaptor
implementation.
@section Exceptions
RecognitionException are generated, when a recognizer encounters incorrect
or unexpected input.
- RecognitionException
- MismatchedRangeException
- MismatchedSetException
- MismatchedNotSetException
.
- MismatchedTokenException
- MismatchedTreeNodeException
- NoViableAltException
- EarlyExitException
- FailedPredicateException
.
.
A tree.RewriteCardinalityException is raised, when the parsers hits a
cardinality mismatch during AST construction. Although this is basically a
bug in your grammar, it can only be detected at runtime.
- tree.RewriteCardinalityException
- tree.RewriteEarlyExitException
- tree.RewriteEmptyStreamException
.
.
"""
# tree.RewriteRuleElementStream
# tree.RewriteRuleSubtreeStream
# tree.RewriteRuleTokenStream
# CharStream
# DFA
# TokenSource
# [The "BSD licence"]
# Copyright (c) 2005-2008 Terence Parr
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# 3. The name of the author may not be used to endorse or promote products
# derived from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
# OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
# IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
# NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
__version__ = '3.1.3'
def version_str_to_tuple(version_str):
import re
import sys
if version_str == 'HEAD':
return (sys.maxint, sys.maxint, sys.maxint, sys.maxint)
m = re.match(r'(\d+)\.(\d+)(\.(\d+))?(b(\d+))?', version_str)
if m is None:
raise ValueError("Bad version string %r" % version_str)
major = int(m.group(1))
minor = int(m.group(2))
patch = int(m.group(4) or 0)
beta = int(m.group(6) or sys.maxint)
return (major, minor, patch, beta)
runtime_version_str = __version__
runtime_version = version_str_to_tuple(runtime_version_str)
from constants import *
from dfa import *
from exceptions import *
from recognizers import *
from streams import *
from tokens import *
"""Compatibility stuff"""
# begin[licence]
#
# [The "BSD licence"]
# Copyright (c) 2005-2008 Terence Parr
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# 3. The name of the author may not be used to endorse or promote products
# derived from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
# OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
# IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
# NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# end[licence]
try:
set = set
frozenset = frozenset
except NameError:
from sets import Set as set, ImmutableSet as frozenset
try:
reversed = reversed
except NameError:
def reversed(l):
l = l[:]
l.reverse()
return l
"""ANTLR3 runtime package"""
# begin[licence]
#
# [The "BSD licence"]
# Copyright (c) 2005-2008 Terence Parr
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# 3. The name of the author may not be used to endorse or promote products
# derived from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
# OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
# IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
# NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# end[licence]
EOF = -1
## All tokens go to the parser (unless skip() is called in that rule)
# on a particular "channel". The parser tunes to a particular channel
# so that whitespace etc... can go to the parser on a "hidden" channel.
DEFAULT_CHANNEL = 0
## Anything on different channel than DEFAULT_CHANNEL is not parsed
# by parser.
HIDDEN_CHANNEL = 99
# Predefined token types
EOR_TOKEN_TYPE = 1
##
# imaginary tree navigation type; traverse "get child" link
DOWN = 2
##
#imaginary tree navigation type; finish with a child list
UP = 3
MIN_TOKEN_TYPE = UP+1
INVALID_TOKEN_TYPE = 0
# begin[licence]
#
# [The "BSD licence"]
# Copyright (c) 2005-2009 Terence Parr
# All rights reserved.
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# 3. The name of the author may not be used to endorse or promote products
# derived from this software without specific prior written permission.
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
# OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
# IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
# NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# end[licence]
import socket
from antlr3 import Parser, TokenStream, RecognitionException, Token
from antlr3.tree import CommonTreeAdaptor, TreeAdaptor, Tree
class DebugParser(Parser):
def __init__(self, stream, state=None, dbg=None, *args, **kwargs):
# wrap token stream in DebugTokenStream (unless user already did so).
if not isinstance(stream, DebugTokenStream):
stream = DebugTokenStream(stream, dbg)
super(DebugParser, self).__init__(stream, state, *args, **kwargs)
# Who to notify when events in the parser occur.
self._dbg = None
self.setDebugListener(dbg)
def setDebugListener(self, dbg):
"""Provide a new debug event listener for this parser. Notify the
input stream too that it should send events to this listener.
"""
if hasattr(self.input, 'dbg'):
self.input.dbg = dbg
self._dbg = dbg
def getDebugListener(self):
return self._dbg
dbg = property(getDebugListener, setDebugListener)
def beginResync(self):
self._dbg.beginResync()
def endResync(self):
self._dbg.endResync()
def beginBacktrack(self, level):
self._dbg.beginBacktrack(level)
def endBacktrack(self, level, successful):
self._dbg.endBacktrack(level,successful)
def reportError(self, exc):
if isinstance(exc, RecognitionException):
self._dbg.recognitionException(exc)
else:
traceback.print_exc(exc)
class DebugTokenStream(TokenStream):
def __init__(self, input, dbg=None):
self.input = input
self.initialStreamState = True
# Track the last mark() call result value for use in rewind().
self.lastMarker = None
self._dbg = None
self.setDebugListener(dbg)
# force TokenStream to get at least first valid token
# so we know if there are any hidden tokens first in the stream
self.input.LT(1)
def getDebugListener(self):
return self._dbg
def setDebugListener(self, dbg):
self._dbg = dbg
dbg = property(getDebugListener, setDebugListener)
def consume(self):
if self.initialStreamState:
self.consumeInitialHiddenTokens()
a = self.input.index()
t = self.input.LT(1)
self.input.consume()
b = self.input.index()
self._dbg.consumeToken(t)
if b > a+1:
# then we consumed more than one token; must be off channel tokens
for idx in range(a+1, b):
self._dbg.consumeHiddenToken(self.input.get(idx));
def consumeInitialHiddenTokens(self):
"""consume all initial off-channel tokens"""
firstOnChannelTokenIndex = self.input.index()
for idx in range(firstOnChannelTokenIndex):
self._dbg.consumeHiddenToken(self.input.get(idx))
self.initialStreamState = False
def LT(self, i):
if self.initialStreamState:
self.consumeInitialHiddenTokens()
t = self.input.LT(i)
self._dbg.LT(i, t)
return t
def LA(self, i):
if self.initialStreamState:
self.consumeInitialHiddenTokens()
t = self.input.LT(i)
self._dbg.LT(i, t)
return t.type
def get(self, i):
return self.input.get(i)
def index(self):
return self.input.index()
def mark(self):
self.lastMarker = self.input.mark()
self._dbg.mark(self.lastMarker)
return self.lastMarker
def rewind(self, marker=None):
self._dbg.rewind(marker)
self.input.rewind(marker)
def release(self, marker):
pass
def seek(self, index):
# TODO: implement seek in dbg interface
# self._dbg.seek(index);
self.input.seek(index)
def size(self):
return self.input.size()
def getTokenSource(self):
return self.input.getTokenSource()
def getSourceName(self):
return self.getTokenSource().getSourceName()
def toString(self, start=None, stop=None):
return self.input.toString(start, stop)
class DebugTreeAdaptor(TreeAdaptor):
"""A TreeAdaptor proxy that fires debugging ev