Add orchestrator to gitlab

First version, not functional yet

AUTHORS

+Thanassis Tsiodras <ttsiodras@gmail.com>
+Jerome Hugues <jerome.hugues@isae.fr>
+Maxime Perrotin <Maxime.Perrotin@esa.int>

LICENSE

+                   GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+  This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+  0. Additional Definitions.
+
+  As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+  "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+  An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+  A "Combined Work" is a work produced by combining or linking an
+Application with the Library.  The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+  The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+  The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+  1. Exception to Section 3 of the GNU GPL.
+
+  You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+  2. Conveying Modified Versions.
+
+  If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+   a) under this License, provided that you make a good faith effort to
+   ensure that, in the event an Application does not supply the
+   function or data, the facility still operates, and performs
+   whatever part of its purpose remains meaningful, or
+
+   b) under the GNU GPL, with none of the additional permissions of
+   this License applicable to that copy.
+
+  3. Object Code Incorporating Material from Library Header Files.
+
+  The object code form of an Application may incorporate material from
+a header file that is part of the Library.  You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+   a) Give prominent notice with each copy of the object code that the
+   Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the object code with a copy of the GNU GPL and this license
+   document.
+
+  4. Combined Works.
+
+  You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+   a) Give prominent notice with each copy of the Combined Work that
+   the Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the Combined Work with a copy of the GNU GPL and this license
+   document.
+
+   c) For a Combined Work that displays copyright notices during
+   execution, include the copyright notice for the Library among
+   these notices, as well as a reference directing the user to the
+   copies of the GNU GPL and this license document.
+
+   d) Do one of the following:
+
+       0) Convey the Minimal Corresponding Source under the terms of this
+       License, and the Corresponding Application Code in a form
+       suitable for, and under terms that permit, the user to
+       recombine or relink the Application with a modified version of
+       the Linked Version to produce a modified Combined Work, in the
+       manner specified by section 6 of the GNU GPL for conveying
+       Corresponding Source.
+
+       1) Use a suitable shared library mechanism for linking with the
+       Library.  A suitable mechanism is one that (a) uses at run time
+       a copy of the Library already present on the user's computer
+       system, and (b) will operate properly with a modified version
+       of the Library that is interface-compatible with the Linked
+       Version.
+
+   e) Provide Installation Information, but only if you would otherwise
+   be required to provide such information under section 6 of the
+   GNU GPL, and only to the extent that such information is
+   necessary to install and execute a modified version of the
+   Combined Work produced by recombining or relinking the
+   Application with a modified version of the Linked Version. (If
+   you use option 4d0, the Installation Information must accompany
+   the Minimal Corresponding Source and Corresponding Application
+   Code. If you use option 4d1, you must provide the Installation
+   Information in the manner specified by section 6 of the GNU GPL
+   for conveying Corresponding Source.)
+
+  5. Combined Libraries.
+
+  You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+   a) Accompany the combined library with a copy of the same work based
+   on the Library, uncombined with any other library facilities,
+   conveyed under the terms of this License.
+
+   b) Give prominent notice with the combined library that part of it
+   is a work based on the Library, and explaining where to find the
+   accompanying uncombined form of the same work.
+
+  6. Revised Versions of the GNU Lesser General Public License.
+
+  The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+  Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+  If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.

README.md

+This utility is responsible for all the steps of building TASTE binaries.
+
+Starting from...
+
+    (a) the Interface/Deployment/Data views,
+    (b) the user code of the subsystems,
+
+it will:
+
+    Extract the ASN.1 DataViews
+    Invoke the ASN1 Compiler
+    Unpack any input user code (SCADE, Simulink, RTDS, C, C++, Ada)
+    Invoke the ESA BuildSupport tool
+    Identify the generated Wrappers
+    Invoke Ocarina to generate the containers from the Vertical xform
+    Perform special handling for Ada subsystems
+    Identify the deployment Partition information
+    Detect any GUI subSystems that must be automatically created
+    Create the include paths for compiling C code
+    Create the run-time type converters (ASN.1 <-> SCADE/Simulink/etc)
+    Detect whether Python bridges must be automatically created
+    Compile the user provided code for
+        SCADE systems
+        OpenGEODE systems
+        Simulink systems
+        C and C++ systems
+        Ada Systems
+        RTDS systems
+    Compile the automatically generated drivers for FPGA designs
+    Build the automatically generated GUIs
+    Build the automatically generaetd Python bridges
+    Build the cyclic subsystems
+    Identify and rename any conflicting common symbol in the object files
+    Invoke the Ocarina generated Makefiles (i.e. build and link it all)
+    Gather all executable output into output/binaries folder

orchestrator/__init__.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+    TASTE Orchestrator
+
+    This tool is the script used to build TASTE applications
+
+    Copyright (c) 2008-2015 Neuropublic (formerly Semantix)
+
+    Designed and implemented by Thanassis Tsiodras
+
+    Contact: ttsiodras@gmail.com
+    TASTE contact: maxime.perrotin@esa.int
+
+    License is LGPLv3 - Check the LICENSE file
+"""
+
+
+import taste_orchestrator
+import checkStackUsage
+import patchAPLCs
+
+__version__ = taste_orchestrator.__version__

orchestrator/checkStackUsage.py

+#!/usr/bin/env python
+"""
+Utility to detect recursive calls and calculate total stack usage per function
+(via following the call graph). Works for x86 and SPARC/Leon binaries.
+
+(C) 2011 Thanassis Tsiodras
+Suggestions/comments: ttsiodras@gmail.com
+"""
+
+import os
+import re
+import sys
+import operator
+
+
+class Matcher:
+    """regexp helper"""
+    def __init__(self, pattern, flags=0):
+        self._pattern = re.compile(pattern, flags)
+        self._hit = None
+
+    def match(self, line):
+        self._hit = re.match(self._pattern, line)
+        return self._hit
+
+    def search(self, line):
+        self._hit = re.search(self._pattern, line)
+        return self._hit
+
+    def group(self, idx):
+        return self._hit.group(idx)
+
+
+def CheckForCycles(callGraph, badNodes):
+    """Detect cycles in function call graphs"""
+    def journey(path):
+        node = path[-1]
+        if node not in callGraph:
+            return
+        if callGraph[node] is None:  # has been marked as recursive,
+                                     # so propagate the marking
+            badNodes[:] = path[:]
+            return
+        neighbours = callGraph[node]
+        for neighbour in neighbours:
+            if neighbour in path:
+                badNodes[:] = path[path.index(neighbour):] + [neighbour]
+                return
+            journey(path + [neighbour])
+    for node, v in callGraph.items():
+        if v:  # if not marked as recursive...
+            journey([node])
+            if badNodes:
+                return
+
+
+def findStackUsage(fn, stackUsagePerFunction, callGraph, cache={}):
+    """
+    Calculate the total stack usage of each function,
+    taking into account who it calls
+    """
+    #  pylint: disable=W0102
+    if fn in cache:  # memoization
+        return cache[fn]
+    if fn not in stackUsagePerFunction:
+        return 0
+    if fn not in callGraph or not callGraph[fn]:
+        cache[fn] = stackUsagePerFunction[fn]
+        return stackUsagePerFunction[fn]
+    totalStackUsage = max(  # the largest of the possible call chains
+        ((stackUsagePerFunction[fn] +
+            findStackUsage(x, stackUsagePerFunction, callGraph))
+         for x in callGraph[fn]))
+    cache[fn] = totalStackUsage
+    return totalStackUsage
+
+
+def main():
+    if len(sys.argv) < 2 or not os.path.exists(sys.argv[1]):
+        print "Usage: %s ELFbinary" % sys.argv[0]
+        sys.exit(1)
+    binarySignature = os.popen("file \"%s\"" % sys.argv[1]).readlines()[0]
+    x86 = Matcher(r'ELF 32-bit LSB.*80.86')
+    leon = Matcher(r'ELF 32-bit MSB.*SPARC')
+    if x86.search(binarySignature):
+        objdump = 'objdump'
+        nm = 'nm'
+        functionNamePattern = Matcher(r'^(\S+) <([a-zA-Z0-9_]+?)>:')
+        callPattern = Matcher(r'^.*call\s+\S+\s+<([a-zA-Z0-9_]+)>')
+        stackUsagePattern = Matcher(r'^.*[add|sub]\s+\$(0x\S+),%esp')
+    elif leon.search(binarySignature):
+        objdump = 'sparc-elf-objdump'
+        nm = 'sparc-elf-nm'
+        functionNamePattern = Matcher(r'^(\S+) <([a-zA-Z0-9_]+?)>:')
+        callPattern = Matcher(r'^.*call\s+\S+\s+<([a-zA-Z0-9_]+)>')
+        stackUsagePattern = Matcher(
+            r'^.*save.*%sp, (-([0-9]{2}|[3-9])[0-9]{2}), %sp')
+    else:
+        print "Unknown signature:", binarySignature
+        sys.exit(1)
+
+    # Store .text symbol offsets and sizes (use nm)
+    offsetOfSymbol = {}
+    for line in os.popen(
+            nm + " \"" + sys.argv[1] + "\" | grep ' [Tt] '").readlines():
+        offset, unused, symbol = line.split()
+        offsetOfSymbol[symbol] = int(offset, 16)
+    sizeOfSymbol = {}
+    lastOffset = 0
+    lastSymbol = None
+    for symbol, offset in sorted(
+            offsetOfSymbol.iteritems(), key=operator.itemgetter(1)):
+        if lastSymbol:
+            sizeOfSymbol[lastSymbol] = offset-lastOffset
+        lastSymbol = symbol
+        lastOffset = offset
+    sizeOfSymbol[lastSymbol] = 2**31  # allow last .text symbol to roam free
+
+    # Parse disassembly to create callgraph (use objdump -d)
+    foundStack = False
+    functionName = ""
+    stackUsagePerFunction = {}
+    callGraph = {}
+    insideFunctionBody = False
+
+    offsetPattern = Matcher(r'^([0-9A-Za-z]+):')
+    for line in os.popen(objdump + " -d \"" + sys.argv[1] + "\"").readlines():
+        # Have we matched a function name yet?
+        if functionName != "":
+            # Yes, update "insideFunctionBody" boolean by checking
+            # the current offset against the length of this symbol,
+            # stored in sizeOfSymbol[functionName]
+            offset = offsetPattern.match(line)
+            if offset:
+                offset = int(offset.group(1), 16)
+                if functionName in offsetOfSymbol:
+                    startOffset = offsetOfSymbol[functionName]
+                    insideFunctionBody = \
+                        insideFunctionBody and \
+                        (offset - startOffset) < sizeOfSymbol[functionName]
+
+        # Check to see if we see a new function:
+        # 08048be8 <_functionName>:
+        fn = functionNamePattern.match(line)
+        if fn:
+            offset = int(fn.group(1), 16)
+            functionName = fn.group(2)
+            callGraph.setdefault(functionName, set())
+            # make sure this is the function we found with nm
+            # UPDATE: no, can't do - if a symbol is of local file scope
+            # (i.e. if it was declared with 'static')
+            # then it can appear in multiple places...
+            #
+            #if functionName in offsetOfSymbol:
+            #    if offsetOfSymbol[functionName] != offset:
+            #        print "Weird,", functionName, \
+            #            "is not at offset reported by", nm
+            #        print hex(offsetOfSymbol[functionName]), hex(offset)
+            insideFunctionBody = True
+            foundStack = False
+
+        # If we're inside a function body
+        # (i.e. offset is not out of symbol size range)
+        if insideFunctionBody:
+            # Check to see if we have a call
+            #  8048c0a:       e8 a1 03 00 00       call   8048fb0 <frame_dummy>
+            call = callPattern.match(line)
+            if functionName != "" and call:
+                calledFunction = call.group(1)
+                callGraph[functionName].add(calledFunction)
+
+            # Check to see if we have the first stack reduction opcode
+            #  8048bec:       83 ec 04                sub    $0x46,%esp
+            if not foundStack and functionName != "":
+                stackMatch = stackUsagePattern.match(line)
+                if stackMatch:
+                    # make sure we dont re-update stackusage for this function
+                    foundStack = True
+                    value = stackMatch.group(1)
+                    if value.startswith("0x"):
+                        # sub    $0x46,%esp
+                        value = int(stackMatch.group(1), 16)
+                        if value > 2147483647:
+                            # unfortunately, GCC may also write:
+                            # add    $0xFFFFFF86,%esp
+                            value = 4294967296-value
+                    else:
+                        # save  %sp, -104, %sp
+                        value = -int(value)
+                    stackUsagePerFunction[functionName] = value
+
+    #for fn,v in stackUsagePerFunction.items():
+    #   print fn,v
+    #   print "CALLS:", callGraph[fn]
+
+    # First, detect cycles and remove "bad" nodes from calculations
+    # (recursive calls would lead to infinite stack usage)
+    while True:
+        badNodes = []
+        CheckForCycles(callGraph, badNodes)
+        if not badNodes:
+            break
+        lastStep = badNodes[-1] + " (recursive)"
+        badNodes.pop()
+        badNodes.append(lastStep)
+        print "Detected cycle and will ignore these functions:\n\t", \
+              "\n\t".join(badNodes)
+        for n in set(badNodes):
+            stackUsagePerFunction[n] = None     # marked as recursive
+            callGraph[n] = None                 # marked as recursive
+
+    print "Cumulative stack usage per function:"
+    # Then, navigate the graph to calculate stack needs per function
+    results = []
+    for fn, value in stackUsagePerFunction.items():
+        if value is not None:
+            results.append(
+                (fn, findStackUsage(fn, stackUsagePerFunction, callGraph)))
+        #else:
+        #    results.append((fn, 'recursive'))
+    for fn, value in sorted(results, key=operator.itemgetter(1)):
+        print "%10s: %s" % (value, fn)
+
+if __name__ == "__main__":
+    main()

orchestrator/patchAPLCs.py

+#!/usr/bin/env python
+import sys
+import copy
+import os
+import re
+
+
+def panic(x):
+    if not x.endswith("\n"):
+        x += "\n"
+    sys.stderr.write(x)
+    sys.exit(1)
+
+
+def main():
+    for i in ['OBJCOPY']:
+        if os.getenv(i) == None:
+            panic('You must set the environment variable ' + i + ' - read the instructions. Aborting...')
+    if len(sys.argv) < 5:
+        panic("Usage: " + sys.argv[0] + " dir1 prefix1 dir2 prefix2 <dir3> <prefix3> <...>")
+    i = 1
+    dirs = []
+    while i<len(sys.argv):
+        dirName = sys.argv[i]
+        if not dirName.endswith(os.sep):
+            dirName += os.sep
+        dirs.append([dirName[:], sys.argv[i+1]])
+        i += 2
+    for d in dirs:
+        if not os.path.isdir(d[0]):
+            panic("'%s' is not a directory..." % d[0])
+    symbols = {}
+    for (d, prefix) in dirs:
+        print "Scanning symbols of object files inside:", d
+        symbols[d] = set([])
+        for obj in os.listdir(d):
+            if not obj.endswith(".o"):
+                continue
+            if obj.endswith("C_ASN1_Types.o"):
+                continue
+            for line in os.popen("nm \"%s\"" % os.path.dirname(d) + os.sep + obj).readlines():
+                if line[0] == ' ':
+                    continue
+                line = re.sub(r'^\S+\s+\S+\s+(.*$)', r'\1', line.strip())
+                symbols[d].add(line)
+    for (dirName, prefix) in dirs:
+        print "Creating objcopy commands for object files in:", dirName
+        uniqueSyms = copy.deepcopy(symbols[dirName])
+        for (otherDir, otherPrefix) in dirs:
+            if otherDir == dirName:
+                continue
+            uniqueSyms -= symbols[otherDir]
+        patchSyms = symbols[dirName] - uniqueSyms
+        if len(patchSyms) == 0:
+            print "No patching necessary..."
+            continue
+        objcopyCmds = open(os.path.dirname(dirName) + os.sep + ".." + os.sep + "objcopyCmds", "w")
+        for sym in patchSyms:
+            objcopyCmds.write('%s assert_%s_%s\n' % (sym, prefix, sym))
+        objcopyCmds.close()
+        print "Executing objcopy commands for object files in:", dirName
+        oldpwd = os.getcwd()
+        os.chdir(os.path.dirname(dirName))
+        os.system("for i in *.o ; do \"$OBJCOPY\" --redefine-syms ../objcopyCmds $i $i.new.o && mv $i.new.o $i ; done")
+        os.chdir(oldpwd)
+
+if __name__ == "__main__":
+    main()
+    sys.exit(0)

setup.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+'''
+Setup file for Linux distribution
+Usage:  python setup.py sdist   --> to create a tarball
+        python setup.py install --> to install in python directory
+'''
+# from distutils.core import setup
+
+from setuptools import setup, find_packages
+
+import orchestrator
+
+setup(
+    name='Orchestrator',
+    version=orchestrator.__version__,
+    packages=find_packages(),
+    author='Thanassis Tsiodras',
+    author_email='ttsiodras@gmail.com',
+    description='Builder script for TASTE applications',
+    long_description=open('README.md').read(),
+    include_package_data=True,
+    url='http://taste.tuxfamily.org',
+    classifiers=[
+        'Programming Language :: Python',
+        'License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)',
+        'Operating System :: OS Independent',
+        'Programming Language :: Python :: 2.7'
+    ],
+    entry_points={
+        'console_scripts': [
+            'taste-orchestrator = orchestrator.taste_orchestrator:main',
+            'taste-patch-aplc = orchestrator.patchAPLCs:main',
+            'taste-check-stack-usage = orchestrator.checkStackUsage:main'
+        ]
+    },
+)