ANTLR3 runtime

antlr/antlr_python_runtime-3.1.3/AUTHORS

+Benjamin Niemann <pink at odahoda dot de>: Main developer of Python target.
+Clinton Roy <clinton.roy at gmail dot com>: AST templates and runtime.

antlr/antlr_python_runtime-3.1.3/LICENSE

+[The "BSD licence"]
+Copyright (c) 2003-2006 Terence Parr
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+ 1. Redistributions of source code must retain the above copyright
+    notice, this list of conditions and the following disclaimer.
+ 2. Redistributions in binary form must reproduce the above copyright
+    notice, this list of conditions and the following disclaimer in the
+    documentation and/or other materials provided with the distribution.
+ 3. The name of the author may not be used to endorse or promote products
+    derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
+IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

antlr/antlr_python_runtime-3.1.3/MANIFEST.in

+include LICENSE AUTHORS ez_setup.py
+

antlr/antlr_python_runtime-3.1.3/PKG-INFO

+Metadata-Version: 1.0
+Name: antlr_python_runtime
+Version: 3.1.3
+Summary: Runtime package for ANTLR3
+Home-page: http://www.antlr.org/
+Author: Benjamin Niemann
+Author-email: pink@odahoda.de
+License: BSD
+Download-URL: http://www.antlr.org/download.html
+Description: This is the runtime package for ANTLR3, which is required to use parsers
+        generated by ANTLR3.
+        
+Platform: UNKNOWN

antlr/antlr_python_runtime-3.1.3/README

+1) ABOUT
+========
+
+This is the Python package 'antlr3', which is required to use parsers created
+by the ANTLR3 tool. See <http://www.antlr.org/> for more information about
+ANTLR3.
+
+
+2) STATUS
+=========
+
+The Python target for ANTLR3 is still in beta. Documentation is lacking, some
+bits of the code is not yet done, some functionality has not been tested yet.
+Also the API might change a bit - it currently mimics the Java implementation,
+but it may be made a bit more pythonic here and there.
+
+WARNING: Currently the runtime library for V3.1 is not compatible with
+recognizers generated by ANTLR V3.0.x. If you are an application developer,
+then the suggested way to solve this is to package the correct runtime with
+your application. Installing the runtime in the global site-packages directory
+may not be a good idea.
+It is still undetermined, if a future release of the V3.1 runtime will be
+compatible with V3.0.x recognizers or if future runtimes V3.2+ will be
+compatible with V3.1 recognizers.
+Sorry for the inconvenience.
+
+
+3) DOWNLOAD
+===========
+
+This runtime is part of the ANTLR distribution. The latest version can be found
+at <http://www.antlr.org/download.html>.
+
+If you are interested in the latest, most bleeding edge version, have a look at
+the perforce depot at <http://fisheye2.cenqua.com/browse/antlr>. There are
+tarballs ready to download, so you don't have to install the perforce client.
+
+
+4) INSTALLATION
+===============
+
+Just like any other Python package:
+$ python setup.py install
+
+See <http://docs.python.org/inst/> for more information.
+
+
+5) DOCUMENTATION
+================
+
+Documentation (as far as it exists) can be found in the wiki
+<http://www.antlr.org/wiki/display/ANTLR3/Antlr3PythonTarget>
+
+
+6) REPORTING BUGS
+=================
+
+Please send bug reports to the ANTLR mailing list 
+<http://www.antlr.org:8080/mailman/listinfo/antlr-interest> or
+<pink@odahoda.de>.
+
+Existing bugs may appear someday in the bugtracker:
+<http://www.antlr.org:8888/browse/ANTLR>
+
+
+7) HACKING
+==========
+
+Only the runtime package can be found here. There are also some StringTemplate
+files in 'src/org/antlr/codegen/templates/Python/' and some Java code in
+'src/org/antlr/codegen/PythonTarget.java' (of the main ANTLR3 source
+distribution).
+
+If there are no directories 'tests' and 'unittests' in 'runtime/Python', you
+should fetch the latest ANTLR3 version from the perforce depot. See section
+DOWNLOAD.
+You'll need java and ant in order to compile and use the tool.
+Be sure to properly setup your CLASSPATH.
+(FIXME: is there some generic information, how to build it yourself? I should
+point to it to avoid duplication.)
+
+You can then use the commands
+$ python setup.py unittest
+$ python setup.py functest
+to ensure that changes do not break existing behaviour.
+
+Please send patches to <pink@odahoda.de>. For larger code contributions you'll
+have to sign the "Developer's Certificate of Origin", which can be found on
+<http://www.antlr.org/license.html> or use the feedback form at
+<http://www.antlr.org/misc/feedback>.

antlr/antlr_python_runtime-3.1.3/antlr3/__init__.py

+""" @package antlr3
+@brief ANTLR3 runtime package
+
+This module contains all support classes, which are needed to use recognizers
+generated by ANTLR3.
+
+@mainpage
+
+\note Please be warned that the line numbers in the API documentation do not
+match the real locations in the source code of the package. This is an
+unintended artifact of doxygen, which I could only convince to use the
+correct module names by concatenating all files from the package into a single
+module file...
+
+Here is a little overview over the most commonly used classes provided by
+this runtime:
+
+@section recognizers Recognizers
+
+These recognizers are baseclasses for the code which is generated by ANTLR3.
+
+- BaseRecognizer: Base class with common recognizer functionality.
+- Lexer: Base class for lexers.
+- Parser: Base class for parsers.
+- tree.TreeParser: Base class for %tree parser.
+
+@section streams Streams
+
+Each recognizer pulls its input from one of the stream classes below. Streams
+handle stuff like buffering, look-ahead and seeking.
+
+A character stream is usually the first element in the pipeline of a typical
+ANTLR3 application. It is used as the input for a Lexer.
+
+- ANTLRStringStream: Reads from a string objects. The input should be a unicode
+  object, or ANTLR3 will have trouble decoding non-ascii data.
+- ANTLRFileStream: Opens a file and read the contents, with optional character
+  decoding.
+- ANTLRInputStream: Reads the date from a file-like object, with optional
+  character decoding.
+
+A Parser needs a TokenStream as input (which in turn is usually fed by a
+Lexer):
+
+- CommonTokenStream: A basic and most commonly used TokenStream
+  implementation.
+- TokenRewriteStream: A modification of CommonTokenStream that allows the
+  stream to be altered (by the Parser). See the 'tweak' example for a usecase.
+
+And tree.TreeParser finally fetches its input from a tree.TreeNodeStream:
+
+- tree.CommonTreeNodeStream: A basic and most commonly used tree.TreeNodeStream
+  implementation.
+  
+
+@section tokenstrees Tokens and Trees
+
+A Lexer emits Token objects which are usually buffered by a TokenStream. A
+Parser can build a Tree, if the output=AST option has been set in the grammar.
+
+The runtime provides these Token implementations:
+
+- CommonToken: A basic and most commonly used Token implementation.
+- ClassicToken: A Token object as used in ANTLR 2.x, used to %tree
+  construction.
+
+Tree objects are wrapper for Token objects.
+
+- tree.CommonTree: A basic and most commonly used Tree implementation.
+
+A tree.TreeAdaptor is used by the parser to create tree.Tree objects for the
+input Token objects.
+
+- tree.CommonTreeAdaptor: A basic and most commonly used tree.TreeAdaptor
+implementation.
+
+
+@section Exceptions
+
+RecognitionException are generated, when a recognizer encounters incorrect
+or unexpected input.
+
+- RecognitionException
+  - MismatchedRangeException
+  - MismatchedSetException
+    - MismatchedNotSetException
+    .
+  - MismatchedTokenException
+  - MismatchedTreeNodeException
+  - NoViableAltException
+  - EarlyExitException
+  - FailedPredicateException
+  .
+.
+
+A tree.RewriteCardinalityException is raised, when the parsers hits a
+cardinality mismatch during AST construction. Although this is basically a
+bug in your grammar, it can only be detected at runtime.
+
+- tree.RewriteCardinalityException
+  - tree.RewriteEarlyExitException
+  - tree.RewriteEmptyStreamException
+  .
+.
+
+"""
+
+# tree.RewriteRuleElementStream
+# tree.RewriteRuleSubtreeStream
+# tree.RewriteRuleTokenStream
+# CharStream
+# DFA
+# TokenSource
+
+# [The "BSD licence"]
+# Copyright (c) 2005-2008 Terence Parr
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+# 3. The name of the author may not be used to endorse or promote products
+#    derived from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+# OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+# IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
+# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+# NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+__version__ = '3.1.3'
+
+def version_str_to_tuple(version_str):
+    import re
+    import sys
+
+    if version_str == 'HEAD':
+        return (sys.maxint, sys.maxint, sys.maxint, sys.maxint)
+
+    m = re.match(r'(\d+)\.(\d+)(\.(\d+))?(b(\d+))?', version_str)
+    if m is None:
+        raise ValueError("Bad version string %r" % version_str)
+
+    major = int(m.group(1))
+    minor = int(m.group(2))
+    patch = int(m.group(4) or 0)
+    beta = int(m.group(6) or sys.maxint)
+
+    return (major, minor, patch, beta)
+
+
+runtime_version_str = __version__
+runtime_version = version_str_to_tuple(runtime_version_str)
+
+
+from constants import *
+from dfa import *
+from exceptions import *
+from recognizers import *
+from streams import *
+from tokens import *

antlr/antlr_python_runtime-3.1.3/antlr3/compat.py

+"""Compatibility stuff"""
+
+# begin[licence]
+#
+# [The "BSD licence"]
+# Copyright (c) 2005-2008 Terence Parr
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+# 3. The name of the author may not be used to endorse or promote products
+#    derived from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+# OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+# IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
+# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+# NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+# end[licence]
+
+try:
+    set = set
+    frozenset = frozenset
+except NameError:
+    from sets import Set as set, ImmutableSet as frozenset
+
+
+try:
+    reversed = reversed
+except NameError:
+    def reversed(l):
+        l = l[:]
+        l.reverse()
+        return l
+
+

antlr/antlr_python_runtime-3.1.3/antlr3/constants.py

+"""ANTLR3 runtime package"""
+
+# begin[licence]
+#
+# [The "BSD licence"]
+# Copyright (c) 2005-2008 Terence Parr
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+# 3. The name of the author may not be used to endorse or promote products
+#    derived from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+# OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+# IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
+# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+# NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+# end[licence]
+
+EOF = -1
+
+## All tokens go to the parser (unless skip() is called in that rule)
+# on a particular "channel".  The parser tunes to a particular channel
+# so that whitespace etc... can go to the parser on a "hidden" channel.
+DEFAULT_CHANNEL = 0
+
+## Anything on different channel than DEFAULT_CHANNEL is not parsed
+# by parser.
+HIDDEN_CHANNEL = 99
+
+# Predefined token types
+EOR_TOKEN_TYPE = 1
+
+##
+# imaginary tree navigation type; traverse "get child" link
+DOWN = 2
+##
+#imaginary tree navigation type; finish with a child list
+UP = 3
+
+MIN_TOKEN_TYPE = UP+1
+	
+INVALID_TOKEN_TYPE = 0
+

antlr/antlr_python_runtime-3.1.3/antlr3/dfa.py

+"""ANTLR3 runtime package"""
+
+# begin[licence]
+#
+# [The "BSD licence"]
+# Copyright (c) 2005-2008 Terence Parr
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+# 3. The name of the author may not be used to endorse or promote products
+#    derived from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+# OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+# IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
+# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+# NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+# end[licensc]
+
+from antlr3.constants import EOF
+from antlr3.exceptions import NoViableAltException, BacktrackingFailed
+
+
+class DFA(object):
+    """@brief A DFA implemented as a set of transition tables.
+
+    Any state that has a semantic predicate edge is special; those states
+    are generated with if-then-else structures in a specialStateTransition()
+    which is generated by cyclicDFA template.
+    
+    """
+    
+    def __init__(
+        self,
+        recognizer, decisionNumber,
+        eot, eof, min, max, accept, special, transition
+        ):
+        ## Which recognizer encloses this DFA?  Needed to check backtracking
+        self.recognizer = recognizer
+
+        self.decisionNumber = decisionNumber
+        self.eot = eot
+        self.eof = eof
+        self.min = min
+        self.max = max
+        self.accept = accept
+        self.special = special
+        self.transition = transition
+
+
+    def predict(self, input):
+        """
+        From the input stream, predict what alternative will succeed
+	using this DFA (representing the covering regular approximation
+	to the underlying CFL).  Return an alternative number 1..n.  Throw
+	 an exception upon error.
+	"""
+        mark = input.mark()
+        s = 0 # we always start at s0
+        try:
+            for _ in xrange(50000):
+                #print "***Current state = %d" % s
+                
+                specialState = self.special[s]
+                if specialState >= 0:
+                    #print "is special"
+                    s = self.specialStateTransition(specialState, input)
+                    if s == -1:
+                        self.noViableAlt(s, input)
+                        return 0
+                    input.consume()
+                    continue
+
+                if self.accept[s] >= 1:
+                    #print "accept state for alt %d" % self.accept[s]
+                    return self.accept[s]
+
+                # look for a normal char transition
+                c = input.LA(1)
+
+                #print "LA = %d (%r)" % (c, unichr(c) if c >= 0 else 'EOF')
+                #print "range = %d..%d" % (self.min[s], self.max[s])
+
+                if c >= self.min[s] and c <= self.max[s]:
+                    # move to next state
+                    snext = self.transition[s][c-self.min[s]]
+                    #print "in range, next state = %d" % snext
+                    
+                    if snext < 0:
+                        #print "not a normal transition"
+                        # was in range but not a normal transition
+                        # must check EOT, which is like the else clause.
+                        # eot[s]>=0 indicates that an EOT edge goes to another
+                        # state.
+                        if self.eot[s] >= 0: # EOT Transition to accept state?
+                            #print "EOT trans to accept state %d" % self.eot[s]
+                            
+                            s = self.eot[s]
+                            input.consume()
+                            # TODO: I had this as return accept[eot[s]]
+                            # which assumed here that the EOT edge always
+                            # went to an accept...faster to do this, but
+                            # what about predicated edges coming from EOT
+                            # target?
+                            continue
+
+                        #print "no viable alt"
+                        self.noViableAlt(s, input)
+                        return 0
+
+                    s = snext
+                    input.consume()
+                    continue
+
+                if self.eot[s] >= 0:
+                    #print "EOT to %d" % self.eot[s]
+                    
+                    s = self.eot[s]
+                    input.consume()
+                    continue
+
+                # EOF Transition to accept state?
+                if c == EOF and self.eof[s] >= 0:
+                    #print "EOF Transition to accept state %d" \
+                    #  % self.accept[self.eof[s]]
+                    return self.accept[self.eof[s]]
+
+                # not in range and not EOF/EOT, must be invalid symbol
+                self.noViableAlt(s, input)
+                return 0
+
+            else:
+                raise RuntimeError("DFA bang!")
+            
+        finally:
+            input.rewind(mark)
+
+
+    def noViableAlt(self, s, input):
+        if self.recognizer._state.backtracking > 0:
+            raise BacktrackingFailed
+
+        nvae = NoViableAltException(
+            self.getDescription(),
+            self.decisionNumber,
+            s,
+            input
+            )
+
+        self.error(nvae)
+        raise nvae
+
+
+    def error(self, nvae):
+        """A hook for debugging interface"""
+        pass
+
+
+    def specialStateTransition(self, s, input):
+        return -1
+
+
+    def getDescription(self):
+        return "n/a"
+
+
+##     def specialTransition(self, state, symbol):
+##         return 0
+
+
+    def unpack(cls, string):
+        """@brief Unpack the runlength encoded table data.
+
+        Terence implemented packed table initializers, because Java has a
+        size restriction on .class files and the lookup tables can grow
+        pretty large. The generated JavaLexer.java of the Java.g example
+        would be about 15MB with uncompressed array initializers.
+
+        Python does not have any size restrictions, but the compilation of
+        such large source files seems to be pretty memory hungry. The memory
+        consumption of the python process grew to >1.5GB when importing a
+        15MB lexer, eating all my swap space and I was to impacient to see,
+        if it could finish at all. With packed initializers that are unpacked
+        at import time of the lexer module, everything works like a charm.
+        
+        """
+        
+        ret = []
+        for i in range(len(string) / 2):
+            (n, v) = ord(string[i*2]), ord(string[i*2+1])
+
+            # Is there a bitwise operation to do this?
+            if v == 0xFFFF:
+                v = -1
+
+            ret += [v] * n
+
+        return ret
+    
+    unpack = classmethod(unpack)