Commit 61690c88 authored by Thanassis Tsiodras's avatar Thanassis Tsiodras

Moved to Python3/mypy/pylint/flake8

parent 4f60f306
[flake8]
ignore = E501,E225,C103
max-line-length = 160
*.pyc
*,cover
.coverage
tests-coverage/output
tests-coverage/smp2.asn
tests-coverage/datatypessimulink.cat
tests-coverage/datatypessimulink.pkg
tests-coverage/Simulink_DataView_asn.m
*swp
# Change Log
## 2.0.0
- Moved to Python3
- Added type annotations, checked via mypy
- Added git commit hooks to check via flake8 AND pylint
- Configuration files used to customize the checks for the project's
naming conventions.
- Many bugs identified and fixed.
## 1.2.3
- ctypes backend: emit all ENUMERATED values in DV.py
## 1.2.0
- Not using SWIG anymore for the Python mappers
## 1.1.2
- pyside_b_mapper: added combo box for the asn1 value editor
## 1.1.1
- support empty SEQUENCEs (MyType ::= SEQUENCE {}) - except for Simulink
(C) Semantix Information Technologies.
Semantix Information Technologies is licensing the code of the
Data Modelling Tools (DMT) in the following dual-license mode:
Commercial Developer License:
The DMT Commercial Developer License is the suggested version
to use for the development of proprietary and/or commercial software.
This version is for developers/companies who do not want to comply
with the terms of the GNU Lesser General Public License version 2.1.
GNU LGPL v. 2.1:
This version of DMT is the one to use for the development of
applications, when you are willing to comply with the terms of the-
GNU Lesser General Public License version 2.1.
Note that in both cases, there are no charges (royalties) for the
generated code.
To purchase a commercial developer license (covering the entire
DMT toolchain), please contact Semantix at: dmt@semantix.gr
PY_SRC:=$(wildcard dmt/asn2dataModel.py dmt/aadl2glueC.py dmt/smp2asn.py dmt/*mappers/[a-zA-Z]*py dmt/commonPy/[a-zA-Z]*py)
PY_SRC:=$(filter-out dmt/B_mappers/antlr.main.py dmt/A_mappers/Stubs.py, ${PY_SRC})
all: flake8 pylint mypy
flake8:
@echo Performing syntax checks via flake8...
@flake8 ${PY_SRC} || exit 1
pylint:
@echo Performing static analysis via pylint...
@pylint --disable=I --rcfile=pylint.cfg ${PY_SRC} | sed -n '/^Report/q;p'
mypy:
@echo Performing type analysis via mypy...
@mypy --check-untyped-defs ${PY_SRC} || exit 1
.PHONY: flake8 pylint mypy
TASTE Data Modelling Technologies
=================================
These are all the tools used by the [TASTE toolchain](https://taste.tuxfamily.org/)
to automate handling of the Data Modelling challenges. They include more than two
dozen codegenerators that create the 'glue' ; that is, the code allowing code
generated by modelling tools (Simulink, SCADE, OpenGeode, etc) to "speak" to
one another via ASN.1 marshalling. For the encoders and decoders of the messages
themselves, we use [ASN1SCC](https://github.com/ttsiodras/asn1scc) - an ASN.1
compiler geared for safety-critical environments.
For more details, visit the [TASTE site](https://taste.tuxfamily.org/).
Installation
------------
$ sudo apt-get install libxslt-dev libxml2-dev python3-pip
$ pip3 install --user -r requirements.txt
$ make flake8 # optional, check for pep8 compliance
$ make pylint # optional, static analysis with pylint
$ make mypy # optional, type analysis with mypy
$ pip3 install --user --upgrade .
Contents
--------
- **commonPy** (*library*)
Contains the basic API for parsing ASN.1 (via invocation of
[ASN1SCC](https://github.com/ttsiodras/asn1scc) and simplification
of the generated XML AST representation to the Python classes
inside `asnAST.py`.
- **asn2aadlPlus** (*utility*)
Converts the type declarations inside ASN.1 grammars to AADL
declarations (used by the Ellidiss tools to design the final systems)
- **asn2dataModel** (*utility*)
Reads the ASN.1 specification of the exchanged messages, and generates
the semantically equivalent Modeling tool/Modeling language declarations
(e.g. SCADE/Lustre, Matlab/Simulink statements, etc).
The actual mapping logic exists in plugins, called *A mappers*
(`simulink_A_mapper.py` handles Simulink/RTW, `scade6_A_mapper.py`
handles SCADE5, `ada_A_mapper.py` generates Ada types,
`sqlalchemy_A_mapper.py`, generates SQL definitions via SQLAlchemy, etc)
- **aadl2glueC** (*utility*)
Reads the AADL specification of the system, and then generates the runtime
bridge-code that will map the message data structures from those generated
by [ASN1SCC](https://github.com/ttsiodras/asn1scc) to/from those generated
by the modeling tool used to functionally model the subsystem (e.g. SCADE,
ObjectGeode, Matlab/Simulink, C, Ada, etc).
Contact
-------
For bug reports, please use the Issue Tracker; for any other communication,
contact me at:
Thanassis Tsiodras
Real-time Embedded Software Engineer
System, Software and Technology Department
European Space Agency
ESTEC
Keplerlaan 1, PO Box 299
NL-2200 AG Noordwijk, The Netherlands
Athanasios.Tsiodras@esa.int | www.esa.int
T +31 71 565 5332
This diff is collapsed.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
asn2dataModel converts ASN.1 modules to a variety of target languages
"""
from ..commonPy import __version__
#
# (C) Semantix Information Technologies.
#
# Semantix Information Technologies is licensing the code of the
# Data Modelling Tools (DMT) in the following dual-license mode:
#
# Commercial Developer License:
# The DMT Commercial Developer License is the suggested version
# to use for the development of proprietary and/or commercial software.
# This version is for developers/companies who do not want to comply
# with the terms of the GNU Lesser General Public License version 2.1.
#
# GNU LGPL v. 2.1:
# This version of DMT is the one to use for the development of
# applications, when you are willing to comply with the terms of the
# GNU Lesser General Public License version 2.1.
#
# Note that in both cases, there are no charges (royalties) for the
# generated code.
#
'''This contains the implementation of model level mapping
of ASN.1 constructs to C. It is used as a backend of Semantix's
code generator A.'''
import os
import sys
import distutils.spawn as spawn
from ..commonPy.utility import panic
def Version():
print("Code generator: " + "$Id: ada_A_mapper.py 2382 2012-06-22 08:35:33Z ttsiodras $") # pragma: no cover
# Especially for the C mapper, since we need to pass the complete ASN.1 files list to ASN1SCC,
# the second param is not asnFile, it is asnFiles
def OnStartup(unused_modelingLanguage, asnFiles, outputDir, unused_badTypes):
# print "Use ASN1SCC to generate the structures for '%s'" % asnFile
asn1SccPath = spawn.find_executable('asn1.exe')
if not asn1SccPath:
panic("ASN1SCC seems to be missing from your system (asn1.exe not found in PATH).\n") # pragma: no cover
os.system(
("mono " if sys.argv[0].endswith('.py') and sys.platform.startswith('linux') else "") +
"\"{}\" -wordSize 8 -typePrefix asn1Scc -Ada -uPER -o \"".format(asn1SccPath) +
outputDir + "\" \"" + "\" \"".join(asnFiles) + "\"")
os.system("rm -f \"" + outputDir + "\"/*.adb")
def OnBasic(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: no cover
def OnSequence(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: no cover
def OnSet(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: no cover
def OnEnumerated(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: no cover
def OnSequenceOf(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: no cover
def OnSetOf(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: no cover
def OnChoice(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: no cover
def OnShutdown(unused_badTypes):
pass # pragma: no cover
#
# (C) Semantix Information Technologies.
#
# Semantix Information Technologies is licensing the code of the
# Data Modelling Tools (DMT) in the following dual-license mode:
#
# Commercial Developer License:
# The DMT Commercial Developer License is the suggested version
# to use for the development of proprietary and/or commercial software.
# This version is for developers/companies who do not want to comply
# with the terms of the GNU Lesser General Public License version 2.1.
#
# GNU LGPL v. 2.1:
# This version of DMT is the one to use for the development of
# applications, when you are willing to comply with the terms of the
# GNU Lesser General Public License version 2.1.
#
# Note that in both cases, there are no charges (royalties) for the
# generated code.
#
'''This contains the implementation of model level mapping
of ASN.1 constructs to C. It is used as a backend of Semantix's
code generator A.'''
import os
import sys
import distutils.spawn as spawn
from ..commonPy.utility import panic
def Version():
print("Code generator: " + "$Id: c_A_mapper.py 2382 2012-06-22 08:35:33Z ttsiodras $") # pragma: no cover
# Especially for the C mapper, since we need to pass the complete ASN.1 files list to ASN1SCC,
# the second param is not asnFile, it is asnFiles
def OnStartup(unused_modelingLanguage, asnFiles, outputDir, unused_badTypes):
# print "Use ASN1SCC to generate the structures for '%s'" % asnFile
asn1SccPath = spawn.find_executable('asn1.exe')
if not asn1SccPath:
panic("ASN1SCC seems to be missing from your system (asn1.exe not found in PATH).\n") # pragma: no cover
os.system(
("mono " if sys.argv[0].endswith('.py') and sys.platform.startswith('linux') else "") +
"\"{}\" -wordSize 8 -typePrefix asn1Scc -c -uPER -o \"".format(asn1SccPath) +
outputDir + "\" \"" + "\" \"".join(asnFiles) + "\"")
cmd = 'rm -f '
for i in ['real.c', 'asn1crt.c', 'acn.c', 'ber.c', 'xer.c']:
cmd += ' "' + outputDir + '"/' + i
os.system(cmd)
for tmp in asnFiles:
os.system("rm -f \"" + outputDir + os.sep + os.path.basename(os.path.splitext(tmp)[0]) + ".c\"")
def OnBasic(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: no cover
def OnSequence(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: no cover
def OnSet(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: no cover
def OnEnumerated(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: no cover
def OnSequenceOf(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: no cover
def OnSetOf(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: no cover
def OnChoice(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: no cover
def OnShutdown(unused_badTypes):
pass # pragma: no cover
#!/usr/bin/env python2
import os
import sys
choices = []
enums = []
bEnum = False
''' Parse the ASN.1-generated header file and extract the constants used for
CHOICE determinants (#define det..._PRESENT) and ENUMERATED values
Emit strings that are appended to DV.py from Makefile.python
There is no name clash thanks to the rename policy of the ASN.1 compiler
(a constant cannot be defined twice)
'''
for line in open(sys.argv[1] + '.h', 'r'):
if '_PRESENT' in line and not line.startswith('#define'):
choices.append(line.strip().replace(",", ""))
elif line.strip().startswith('typedef enum {'):
bEnum = True
elif line.strip().startswith('}') and bEnum:
bEnum = False
elif bEnum:
enums.append(line.strip().replace(",", "").split("="))
enums_dump = "\n ".join(
'printf("%s = %%d\\n", %s);' % (e, e)
for e in choices
)
enums_dump += "\n ".join(
'printf("%s = %d\\n");' % (name.strip(), int(val))
for name, val in enums
)
uniq = os.getpid()
extractor_filename = "/tmp/enums_%d" % uniq
f = open(extractor_filename + ".c", 'w')
f.write("""
#include <stdio.h>
#include "%(base)s.h"
void main()
{
%(enums_dump)s
}""" % {"enums_dump": enums_dump, "base": sys.argv[1]})
f.close()
cmd = "gcc -o %s -I. %s.c" % (extractor_filename, extractor_filename)
if os.system(cmd) != 0:
print("Failed to extract CHOICE enum values...")
sys.exit(1)
os.system(extractor_filename)
os.unlink(extractor_filename + ".c")
os.unlink(extractor_filename)
#
# (C) Semantix Information Technologies.
#
# Semantix Information Technologies is licensing the code of the
# Data Modelling Tools (DMT) in the following dual-license mode:
#
# Commercial Developer License:
# The DMT Commercial Developer License is the suggested version
# to use for the development of proprietary and/or commercial software.
# This version is for developers/companies who do not want to comply
# with the terms of the GNU Lesser General Public License version 2.1.
#
# GNU LGPL v. 2.1:
# This version of DMT is the one to use for the development of
# applications, when you are willing to comply with the terms of the
# GNU Lesser General Public License version 2.1.
#
# Note that in both cases, there are no charges (royalties) for the
# generated code.
#
# This is an evolving test implementation of mapping ASN.1 constructs
# to SDL. It is used by the backend of Semantix's code generator A.
# import re
from ..commonPy import asnParser
g_outputDir = ""
g_asnFile = ""
def Version():
print("Code generator: " + "$Id: og_A_mapper.py 2382 2012-06-22 08:35:33Z ttsiodras $") # pragma: no cover
def OnStartup(unused_modelingLanguage, asnFile, outputDir, unused_badTypes):
global g_asnFile
g_asnFile = asnFile
global g_outputDir
g_outputDir = outputDir
def OnBasic(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass
def OnSequence(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass
def OnSet(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: nocover
def OnEnumerated(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass
def OnSequenceOf(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass
def OnSetOf(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: nocover
def OnChoice(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass
# obsolete, now the grammar is re-created from the AST (PrintGrammarFromAST)
#
# def ClearUp(text):
# outputText = ""
# lParen = 0
# for c in text:
# if c == '(':
# lParen += 1
# if c == ')':
# lParen -= 1
# if 0 == lParen:
# outputText += c.replace('-', '_')
# else:
# outputText += c
# return outputText
def OnShutdown(unused_badTypes):
# text = open(g_asnFile, 'r').read()
# text = re.sub(r'^.*BEGIN', 'Datamodel DEFINITIONS ::= BEGIN', text)
# text = re.sub(r'--.*', '', text)
outputFile = open(g_outputDir + "DataView.pr", 'w')
outputFile.write('Datamodel DEFINITIONS ::= BEGIN\n\n')
asnParser.PrintGrammarFromAST(outputFile)
outputFile.write('END\n')
outputFile.close()
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
#
# (C) Semantix Information Technologies.
#
# Semantix Information Technologies is licensing the code of the
# Data Modelling Tools (DMT) in the following dual-license mode:
#
# Commercial Developer License:
# The DMT Commercial Developer License is the suggested version
# to use for the development of proprietary and/or commercial software.
# This version is for developers/companies who do not want to comply
# with the terms of the GNU Lesser General Public License version 2.1.
#
# GNU LGPL v. 2.1:
# This version of DMT is the one to use for the development of
# applications, when you are willing to comply with the terms of the
# GNU Lesser General Public License version 2.1.
#
# Note that in both cases, there are no charges (royalties) for the
# generated code.
#
'''Implementation of mapping ASN.1 constructs
to RTDS. It is used by the backend of Semantix's code generator A.'''
import re
g_outputDir = ""
g_asnFile = ""
def Version():
print("Code generator: " + "$Id: og_A_mapper.py 1879 2010-05-17 10:13:12Z ttsiodras $") # pragma: no cover
def OnStartup(unused_modelingLanguage, asnFile, outputDir, unused_badTypes):
global g_asnFile
g_asnFile = asnFile
global g_outputDir
g_outputDir = outputDir
def OnBasic(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass
def OnSequence(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass
def OnSet(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: no cover
def OnEnumerated(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass
def OnSequenceOf(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass
def OnSetOf(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass # pragma: no cover
def OnChoice(unused_nodeTypename, unused_node, unused_leafTypeDict):
pass
# obsolete, now the grammar is re-created from the AST (PrintGrammarFromAST)
#
# def ClearUp(text):
# outputText = ""
# lParen = 0
# for c in text:
# if c == '(':
# lParen += 1
# if c == ')':
# lParen -= 1
# if 0 == lParen:
# outputText += c.replace('-', '_')
# else:
# outputText += c
# return outputText
def OnShutdown(unused_badTypes):
# text = open(g_asnFile, 'r').read()
# text = re.sub(r'^.*BEGIN', 'Datamodel DEFINITIONS ::= BEGIN', text)
# text = re.sub(r'--.*', '', text)
# outputFile = open(g_outputDir + "DataView.pr", 'w')
# outputFile.write('Datamodel DEFINITIONS ::= BEGIN\n\n')
# import commonPy.xmlASTtoAsnAST
# commonPy.xmlASTtoAsnAST.PrintGrammarFromAST(outputFile)
# outputFile.write('END\n')
# outputFile.close()
outputFile = open(g_outputDir + "RTDSdataView.asn", 'w')
outputFile.write(re.sub(r'^.*BEGIN', 'RTDSdataView DEFINITIONS ::= BEGIN', open(g_asnFile, 'r').read()))
outputFile.close()
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
all: makeParser runTest
makeParser: AadlParser.py
AadlParser.py: aadl.g
rm -f AadlLexer.py AadlParser.py
runantlr -o . aadl.g || { echo You need to apt-get install antlr ; exit 1 ; }
runTest:
./antlr.main.py < PierreToyExampleSystem.aadl
PACKAGE Functional_view
PUBLIC
SUBPROGRAM Dispatcher
FEATURES
TC : IN PARAMETER DataTypes::T_TC;
END Dispatcher;
SUBPROGRAM IMPLEMENTATION Dispatcher.AADL
CALLS {
TC_Parser : SUBPROGRAM TC_Parser.AADL;
Tc_1 : SUBPROGRAM TC.POS;
Tc_2 : SUBPROGRAM TC.PRO;