Moved to Python3/mypy/pylint/flake8

.flake8

+[flake8]
+ignore = E501,E225,C103
+max-line-length = 160

.gitignore

+*.pyc
+*,cover
+.coverage
+tests-coverage/output
+tests-coverage/smp2.asn
+tests-coverage/datatypessimulink.cat
+tests-coverage/datatypessimulink.pkg
+tests-coverage/Simulink_DataView_asn.m
+*swp

CHANGELOG

+# Change Log
+
+## 2.0.0
+
+- Moved to Python3
+- Added type annotations, checked via mypy
+- Added git commit hooks to check via flake8 AND pylint
+- Configuration files used to customize the checks for the project's
+  naming conventions.
+- Many bugs identified and fixed.
+
+## 1.2.3
+- ctypes backend: emit all ENUMERATED values in DV.py
+
+## 1.2.0
+- Not using SWIG anymore for the Python mappers
+
+## 1.1.2
+- pyside_b_mapper: added combo box for the asn1 value editor
+
+## 1.1.1
+- support empty SEQUENCEs (MyType ::= SEQUENCE {}) - except for Simulink

LICENSE.txt

+(C) Semantix Information Technologies.
+
+Semantix Information Technologies is licensing the code of the
+Data Modelling Tools (DMT) in the following dual-license mode:
+
+Commercial Developer License:
+      The DMT Commercial Developer License is the suggested version
+to use for the development of proprietary and/or commercial software.
+This version is for developers/companies who do not want to comply
+with the terms of the GNU Lesser General Public License version 2.1.
+
+GNU LGPL v. 2.1:
+      This version of DMT is the one to use for the development of
+applications, when you are willing to comply with the terms of the-
+GNU Lesser General Public License version 2.1.
+
+Note that in both cases, there are no charges (royalties) for the
+generated code.
+
+To purchase a commercial developer license (covering the entire
+DMT toolchain), please contact Semantix at: dmt@semantix.gr

Makefile

+PY_SRC:=$(wildcard dmt/asn2dataModel.py dmt/aadl2glueC.py dmt/smp2asn.py dmt/*mappers/[a-zA-Z]*py dmt/commonPy/[a-zA-Z]*py)
+PY_SRC:=$(filter-out dmt/B_mappers/antlr.main.py dmt/A_mappers/Stubs.py, ${PY_SRC})
+
+all:	flake8 pylint mypy
+
+flake8:
+	@echo Performing syntax checks via flake8...
+	@flake8 ${PY_SRC} || exit 1
+
+pylint:
+	@echo Performing static analysis via pylint...
+	@pylint --disable=I --rcfile=pylint.cfg ${PY_SRC}  | sed -n '/^Report/q;p'
+
+mypy:
+	@echo Performing type analysis via mypy...
+	@mypy --check-untyped-defs ${PY_SRC} || exit 1
+
+.PHONY:	flake8 pylint mypy

README.md

+TASTE Data Modelling Technologies
+=================================
+
+These are all the tools used by the [TASTE toolchain](https://taste.tuxfamily.org/)
+to automate handling of the Data Modelling challenges. They include more than two
+dozen codegenerators that create the 'glue' ; that is, the code allowing code
+generated by modelling tools (Simulink, SCADE, OpenGeode, etc) to "speak" to
+one another via ASN.1 marshalling. For the encoders and decoders of the messages
+themselves, we use [ASN1SCC](https://github.com/ttsiodras/asn1scc) - an ASN.1
+compiler geared for safety-critical environments.
+
+For more details, visit the [TASTE site](https://taste.tuxfamily.org/).
+
+Installation
+------------
+
+    $ sudo apt-get install libxslt-dev libxml2-dev python3-pip
+    $ pip3 install --user -r requirements.txt
+    $ make flake8  # optional, check for pep8 compliance
+    $ make pylint  # optional, static analysis with pylint
+    $ make mypy  # optional, type analysis with mypy
+    $ pip3 install --user --upgrade .
+
+Contents
+--------
+
+- **commonPy** (*library*)
+
+    Contains the basic API for parsing ASN.1 (via invocation of 
+    [ASN1SCC](https://github.com/ttsiodras/asn1scc) and simplification
+    of the generated XML AST representation to the Python classes
+    inside `asnAST.py`.
+
+- **asn2aadlPlus** (*utility*)
+
+    Converts the type declarations inside ASN.1 grammars to AADL
+    declarations (used by the Ellidiss tools to design the final systems)
+
+- **asn2dataModel** (*utility*)
+
+    Reads the ASN.1 specification of the exchanged messages, and generates
+    the semantically equivalent Modeling tool/Modeling language declarations
+    (e.g.  SCADE/Lustre, Matlab/Simulink statements, etc). 
+
+    The actual mapping logic exists in plugins, called *A mappers*
+    (`simulink_A_mapper.py` handles Simulink/RTW, `scade6_A_mapper.py`
+    handles SCADE5, `ada_A_mapper.py` generates Ada types,
+    `sqlalchemy_A_mapper.py`, generates SQL definitions via SQLAlchemy, etc)
+
+- **aadl2glueC** (*utility*)
+
+    Reads the AADL specification of the system, and then generates the runtime
+    bridge-code that will map the message data structures from those generated
+    by [ASN1SCC](https://github.com/ttsiodras/asn1scc) to/from those generated
+    by the modeling tool used to functionally model the subsystem (e.g. SCADE,
+    ObjectGeode, Matlab/Simulink, C, Ada, etc).
+
+Contact
+-------
+
+For bug reports, please use the Issue Tracker; for any other communication,
+contact me at:
+
+    Thanassis Tsiodras
+    Real-time Embedded Software Engineer 
+    System, Software and Technology Department
+    European Space Agency
+
+    ESTEC
+    Keplerlaan 1, PO Box 299
+    NL-2200 AG Noordwijk, The Netherlands
+    Athanasios.Tsiodras@esa.int | www.esa.int
+    T +31 71 565 5332

dmt/A_mappers/__init__.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+   asn2dataModel converts ASN.1 modules to a variety of target languages
+"""
+
+from ..commonPy import __version__

dmt/A_mappers/ada_A_mapper.py

+#
+# (C) Semantix Information Technologies.
+#
+# Semantix Information Technologies is licensing the code of the
+# Data Modelling Tools (DMT) in the following dual-license mode:
+#
+# Commercial Developer License:
+#       The DMT Commercial Developer License is the suggested version
+# to use for the development of proprietary and/or commercial software.
+# This version is for developers/companies who do not want to comply
+# with the terms of the GNU Lesser General Public License version 2.1.
+#
+# GNU LGPL v. 2.1:
+#       This version of DMT is the one to use for the development of
+# applications, when you are willing to comply with the terms of the
+# GNU Lesser General Public License version 2.1.
+#
+# Note that in both cases, there are no charges (royalties) for the
+# generated code.
+#
+'''This contains the implementation of model level mapping
+of ASN.1 constructs to C. It is used as a backend of Semantix's
+code generator A.'''
+
+import os
+import sys
+import distutils.spawn as spawn
+
+from ..commonPy.utility import panic
+
+
+def Version():
+    print("Code generator: " + "$Id: ada_A_mapper.py 2382 2012-06-22 08:35:33Z ttsiodras $")  # pragma: no cover
+
+
+# Especially for the C mapper, since we need to pass the complete ASN.1 files list to ASN1SCC,
+# the second param is not asnFile, it is asnFiles
+
+
+def OnStartup(unused_modelingLanguage, asnFiles, outputDir, unused_badTypes):
+    # print "Use ASN1SCC to generate the structures for '%s'" % asnFile
+    asn1SccPath = spawn.find_executable('asn1.exe')
+    if not asn1SccPath:
+        panic("ASN1SCC seems to be missing from your system (asn1.exe not found in PATH).\n")  # pragma: no cover
+    os.system(
+        ("mono " if sys.argv[0].endswith('.py') and sys.platform.startswith('linux') else "") +
+        "\"{}\" -wordSize 8 -typePrefix asn1Scc -Ada -uPER -o \"".format(asn1SccPath) +
+        outputDir + "\" \"" + "\" \"".join(asnFiles) + "\"")
+    os.system("rm -f \"" + outputDir + "\"/*.adb")
+
+
+def OnBasic(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: no cover
+
+
+def OnSequence(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: no cover
+
+
+def OnSet(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: no cover
+
+
+def OnEnumerated(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: no cover
+
+
+def OnSequenceOf(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: no cover
+
+
+def OnSetOf(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: no cover
+
+
+def OnChoice(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: no cover
+
+
+def OnShutdown(unused_badTypes):
+    pass  # pragma: no cover

dmt/A_mappers/c_A_mapper.py

+#
+# (C) Semantix Information Technologies.
+#
+# Semantix Information Technologies is licensing the code of the
+# Data Modelling Tools (DMT) in the following dual-license mode:
+#
+# Commercial Developer License:
+#       The DMT Commercial Developer License is the suggested version
+# to use for the development of proprietary and/or commercial software.
+# This version is for developers/companies who do not want to comply
+# with the terms of the GNU Lesser General Public License version 2.1.
+#
+# GNU LGPL v. 2.1:
+#       This version of DMT is the one to use for the development of
+# applications, when you are willing to comply with the terms of the
+# GNU Lesser General Public License version 2.1.
+#
+# Note that in both cases, there are no charges (royalties) for the
+# generated code.
+#
+'''This contains the implementation of model level mapping
+of ASN.1 constructs to C. It is used as a backend of Semantix's
+code generator A.'''
+
+import os
+import sys
+import distutils.spawn as spawn
+
+from ..commonPy.utility import panic
+
+
+def Version():
+    print("Code generator: " + "$Id: c_A_mapper.py 2382 2012-06-22 08:35:33Z ttsiodras $")  # pragma: no cover
+
+
+# Especially for the C mapper, since we need to pass the complete ASN.1 files list to ASN1SCC,
+# the second param is not asnFile, it is asnFiles
+
+def OnStartup(unused_modelingLanguage, asnFiles, outputDir, unused_badTypes):
+    # print "Use ASN1SCC to generate the structures for '%s'" % asnFile
+    asn1SccPath = spawn.find_executable('asn1.exe')
+    if not asn1SccPath:
+        panic("ASN1SCC seems to be missing from your system (asn1.exe not found in PATH).\n")  # pragma: no cover
+    os.system(
+        ("mono " if sys.argv[0].endswith('.py') and sys.platform.startswith('linux') else "") +
+        "\"{}\" -wordSize 8 -typePrefix asn1Scc -c -uPER -o \"".format(asn1SccPath) +
+        outputDir + "\" \"" + "\" \"".join(asnFiles) + "\"")
+    cmd = 'rm -f '
+    for i in ['real.c', 'asn1crt.c', 'acn.c', 'ber.c', 'xer.c']:
+        cmd += ' "' + outputDir + '"/' + i
+    os.system(cmd)
+    for tmp in asnFiles:
+        os.system("rm -f \"" + outputDir + os.sep + os.path.basename(os.path.splitext(tmp)[0]) + ".c\"")
+
+
+def OnBasic(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: no cover
+
+
+def OnSequence(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: no cover
+
+
+def OnSet(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: no cover
+
+
+def OnEnumerated(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: no cover
+
+
+def OnSequenceOf(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: no cover
+
+
+def OnSetOf(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: no cover
+
+
+def OnChoice(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: no cover
+
+
+def OnShutdown(unused_badTypes):
+    pass  # pragma: no cover

dmt/A_mappers/learn_CHOICE_enums.py

+#!/usr/bin/env python2
+import os
+import sys
+choices = []
+enums = []
+bEnum = False
+
+''' Parse the ASN.1-generated header file and extract the constants used for
+CHOICE determinants (#define det..._PRESENT) and ENUMERATED values
+Emit strings that are appended to DV.py from Makefile.python
+There is no name clash thanks to the rename policy of the ASN.1 compiler
+(a constant cannot be defined twice)
+'''
+
+for line in open(sys.argv[1] + '.h', 'r'):
+    if '_PRESENT' in line and not line.startswith('#define'):
+        choices.append(line.strip().replace(",", ""))
+    elif line.strip().startswith('typedef enum {'):
+        bEnum = True
+    elif line.strip().startswith('}') and bEnum:
+        bEnum = False
+    elif bEnum:
+        enums.append(line.strip().replace(",", "").split("="))
+
+enums_dump = "\n    ".join(
+    'printf("%s = %%d\\n", %s);' % (e, e)
+    for e in choices
+)
+
+enums_dump += "\n    ".join(
+    'printf("%s = %d\\n");' % (name.strip(), int(val))
+    for name, val in enums
+)
+uniq = os.getpid()
+extractor_filename = "/tmp/enums_%d" % uniq
+f = open(extractor_filename + ".c", 'w')
+f.write("""
+#include <stdio.h>
+#include "%(base)s.h"
+
+void main()
+{
+%(enums_dump)s
+}""" % {"enums_dump": enums_dump, "base": sys.argv[1]})
+f.close()
+cmd = "gcc -o %s -I. %s.c" % (extractor_filename, extractor_filename)
+if os.system(cmd) != 0:
+    print("Failed to extract CHOICE enum values...")
+    sys.exit(1)
+os.system(extractor_filename)
+os.unlink(extractor_filename + ".c")
+os.unlink(extractor_filename)

dmt/A_mappers/og_A_mapper.py

+#
+# (C) Semantix Information Technologies.
+#
+# Semantix Information Technologies is licensing the code of the
+# Data Modelling Tools (DMT) in the following dual-license mode:
+#
+# Commercial Developer License:
+#       The DMT Commercial Developer License is the suggested version
+# to use for the development of proprietary and/or commercial software.
+# This version is for developers/companies who do not want to comply
+# with the terms of the GNU Lesser General Public License version 2.1.
+#
+# GNU LGPL v. 2.1:
+#       This version of DMT is the one to use for the development of
+# applications, when you are willing to comply with the terms of the
+# GNU Lesser General Public License version 2.1.
+#
+# Note that in both cases, there are no charges (royalties) for the
+# generated code.
+#
+# This is an evolving test implementation of mapping ASN.1 constructs
+# to SDL. It is used by the backend of Semantix's code generator A.
+
+# import re
+
+from ..commonPy import asnParser
+
+g_outputDir = ""
+g_asnFile = ""
+
+
+def Version():
+    print("Code generator: " + "$Id: og_A_mapper.py 2382 2012-06-22 08:35:33Z ttsiodras $")  # pragma: no cover
+
+
+def OnStartup(unused_modelingLanguage, asnFile, outputDir, unused_badTypes):
+    global g_asnFile
+    g_asnFile = asnFile
+    global g_outputDir
+    g_outputDir = outputDir
+
+
+def OnBasic(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass
+
+
+def OnSequence(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass
+
+
+def OnSet(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: nocover
+
+
+def OnEnumerated(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass
+
+
+def OnSequenceOf(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass
+
+
+def OnSetOf(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: nocover
+
+
+def OnChoice(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass
+
+# obsolete, now the grammar is re-created from the AST (PrintGrammarFromAST)
+#
+# def ClearUp(text):
+#     outputText = ""
+#     lParen = 0
+#     for c in text:
+#         if c == '(':
+#             lParen += 1
+#         if c == ')':
+#             lParen -= 1
+#         if 0 == lParen:
+#             outputText += c.replace('-', '_')
+#         else:
+#             outputText += c
+#     return outputText
+
+
+def OnShutdown(unused_badTypes):
+    # text = open(g_asnFile, 'r').read()
+    # text = re.sub(r'^.*BEGIN', 'Datamodel DEFINITIONS ::= BEGIN', text)
+    # text = re.sub(r'--.*', '', text)
+    outputFile = open(g_outputDir + "DataView.pr", 'w')
+    outputFile.write('Datamodel DEFINITIONS ::= BEGIN\n\n')
+    asnParser.PrintGrammarFromAST(outputFile)
+    outputFile.write('END\n')
+    outputFile.close()

dmt/A_mappers/rtds_A_mapper.py

+#
+# (C) Semantix Information Technologies.
+#
+# Semantix Information Technologies is licensing the code of the
+# Data Modelling Tools (DMT) in the following dual-license mode:
+#
+# Commercial Developer License:
+#       The DMT Commercial Developer License is the suggested version
+# to use for the development of proprietary and/or commercial software.
+# This version is for developers/companies who do not want to comply
+# with the terms of the GNU Lesser General Public License version 2.1.
+#
+# GNU LGPL v. 2.1:
+#       This version of DMT is the one to use for the development of
+# applications, when you are willing to comply with the terms of the
+# GNU Lesser General Public License version 2.1.
+#
+# Note that in both cases, there are no charges (royalties) for the
+# generated code.
+#
+'''Implementation of mapping ASN.1 constructs
+to RTDS. It is used by the backend of Semantix's code generator A.'''
+
+import re
+
+g_outputDir = ""
+g_asnFile = ""
+
+
+def Version():
+    print("Code generator: " + "$Id: og_A_mapper.py 1879 2010-05-17 10:13:12Z ttsiodras $")  # pragma: no cover
+
+
+def OnStartup(unused_modelingLanguage, asnFile, outputDir, unused_badTypes):
+    global g_asnFile
+    g_asnFile = asnFile
+    global g_outputDir
+    g_outputDir = outputDir
+
+
+def OnBasic(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass
+
+
+def OnSequence(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass
+
+
+def OnSet(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: no cover
+
+
+def OnEnumerated(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass
+
+
+def OnSequenceOf(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass
+
+
+def OnSetOf(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass  # pragma: no cover
+
+
+def OnChoice(unused_nodeTypename, unused_node, unused_leafTypeDict):
+    pass
+
+
+# obsolete, now the grammar is re-created from the AST (PrintGrammarFromAST)
+#
+# def ClearUp(text):
+#     outputText = ""
+#     lParen = 0
+#     for c in text:
+#         if c == '(':
+#             lParen += 1
+#         if c == ')':
+#             lParen -= 1
+#         if 0 == lParen:
+#             outputText += c.replace('-', '_')
+#         else:
+#             outputText += c
+#     return outputText
+
+def OnShutdown(unused_badTypes):
+    # text = open(g_asnFile, 'r').read()
+    # text = re.sub(r'^.*BEGIN', 'Datamodel DEFINITIONS ::= BEGIN', text)
+    # text = re.sub(r'--.*', '', text)
+    # outputFile = open(g_outputDir + "DataView.pr", 'w')
+    # outputFile.write('Datamodel DEFINITIONS ::= BEGIN\n\n')
+    # import commonPy.xmlASTtoAsnAST
+    # commonPy.xmlASTtoAsnAST.PrintGrammarFromAST(outputFile)
+    # outputFile.write('END\n')
+    # outputFile.close()
+
+    outputFile = open(g_outputDir + "RTDSdataView.asn", 'w')
+    outputFile.write(re.sub(r'^.*BEGIN', 'RTDSdataView DEFINITIONS ::= BEGIN', open(g_asnFile, 'r').read()))
+    outputFile.close()

dmt/B_mappers/Makefile

+all:	makeParser runTest
+
+makeParser: AadlParser.py
+
+AadlParser.py: aadl.g
+	rm -f AadlLexer.py AadlParser.py
+	runantlr -o . aadl.g || { echo You need to apt-get install antlr ; exit 1 ; }
+
+runTest:
+	./antlr.main.py < PierreToyExampleSystem.aadl

dmt/B_mappers/PierreToyExampleSystem.aadl

+PACKAGE Functional_view
+PUBLIC
+
+SUBPROGRAM Dispatcher
+FEATURES
+    TC : IN PARAMETER DataTypes::T_TC;
+END Dispatcher;
+
+SUBPROGRAM IMPLEMENTATION Dispatcher.AADL
+CALLS {
+    TC_Parser : SUBPROGRAM TC_Parser.AADL;
+    Tc_1 : SUBPROGRAM TC.POS;
+    Tc_2 : SUBPROGRAM TC.PRO;