Archived
1
0
This repo is archived. You can view files and clone it, but cannot push or open issues or pull requests.
kll/common/expression.py
Jacob Alexander f610d0fb15 KLL Compiler Re-Write
This was many months of efforts in re-designing how the KLL compiler should work.
The major problem with the original compiler was how difficult it was to extend language wise.
This lead to many delays in KLL 0.4 and 0.5 being implemented.

The new design is a multi-staged compiler, where even tokenization occurs over multiple stages.
This allows individual parsing and token regexes to be expressed more simply without affect other expressions.

Another area of change is the concept of Contexts.
In the original KLL compiler the idea of a cache assigned was "hacked" on when I realized the language was "broken" (after nearly finishing the compiler).
Since assignment order is generally considered not to matter for keymappings, I created a "cached" assignment where the whole file is read into a sub-datastructure, then apply to the master datastructure.
Unfortunately, this wasn't really all that clear, so it was annoying to work with.
To remedy this, I created KLL Contexts, which contain information about a group of expressions.
Not only can these groups can be merged with other Contexts, they have historical data about how they were generated allowing for errors very late in processing to be pin-pointed back to the offending kll file.

Backends work nearly the same as they did before.
However, all call-backs for capability evaluations have been removed.
This makes the interface much cleaner as Contexts can only be symbolically merged now.
(Previously datastructures did evaluation merges where the ScanCode or Capability was looked up right before passing to the backend, but this required additional information from the backend).

Many of the old parsing and tokenization rules have been reused, along with the hid_dict.py code.

The new design takes advantage of processor pools to handle multithreading where it makes sense.
For example, all specified files are loaded into ram simulatenously rather than sparingly reading from.
The reason for this is so that each Context always has all the information it requires at all times.

kll
- Program entry point (previously kll.py)
- Very small now, does some setting up of command-line args
- Most command-line args are specified by the corresponding processing stage

common/channel.py
- Pixel Channel container classes

common/context.py
- Context container classes
- As is usual with other files, blank classes inherit a base class
- These blank classes are identified by the class name itself to handle special behaviour
- And if/when necessary functions are re-implemented
- MergeConext class facilitates merging of contexts while maintaining lineage

common/expression.py
- Expression container classes
  * Expression base class
  * AssignmentExpression
  * NameAssociationExpression
  * DataAssociationExpression
  * MapExpression
- These classes are used to store expressions after they have finished parsing and tokenization

common/file.py
- Container class for files being read by the KLL compiler

common/emitter.py
- Base class for all KLL emitters
- TextEmitter for dealing with text file templates

common/hid_dict.py
- Slightly modified version of kll_lib/hid_dict.py

common/id.py
- Identification container classes
- Used to indentify different types of elements used within the KLL language

common/modifier.py
- Container classes for animation and pixel change functions

common/organization.py
- Data structure merging container classes
- Contains all the sub-datastructure classes as well
- The Organization class handles the merge orchestration and expression insertion

common/parse.py
- Parsing rules for funcparserlib
- Much of this file was taken from the original kll.py
- Many changes to support the multi-stage processing and support KLL 0.5

common/position.py
- Container class dealing with physical positions

common/schedule.py
- Container class dealing with scheduling and timing events

common/stage.py
- Contains ControlStage and main Stage classes
  * CompilerConfigurationStage
  * FileImportStage
  * PreprocessorStage
  * OperationClassificationStage
  * OperationSpecificsStage
  * OperationOrganizationStage
  * DataOrganziationStage
  * DataFinalizationStage
  * DataAnalysisStage
  * CodeGenerationStage
  * ReportGenerationStage
- Each of these classes controls the life-cycle of each stage
- If multi-threading is desired, it must be handled within the class
  * The next stage will not start until the current stage is finished
- Errors are handled such that as many errors as possible are recorded before forcing an exit
  * The exit is handled at the end of each stage if necessary
- Command-line arguments for each stage can be defined if necessary (they are given their own grouping)
- Each stage can pull variables and functions from other stages if necessary using a name lookup
  * This means you don't have to worry about over-arching datastructures

emitters/emitters.py
- Container class for KLL emitters
- Handles emitter setup and selection

emitters/kiibohd/kiibohd.py
- kiibohd .h file KLL emitter
- Re-uses some backend code from the original KLL compiler

funcparserlib/parser.py
- Added debug mode control

examples/assignment.kll
examples/defaultMapExample.kll
examples/example.kll
examples/hhkbpro2.kll
examples/leds.kll
examples/mapping.kll
examples/simple1.kll
examples/simple2.kll
examples/simpleExample.kll
examples/state_scheduling.kll
- Updating/Adding rules for new compiler and KLL 0.4 + KLL 0.5 support
2016-09-01 23:48:13 -07:00

631 lines
16 KiB
Python

#!/usr/bin/env python3
'''
KLL Expression Container
'''
# Copyright (C) 2016 by Jacob Alexander
#
# This file is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This file is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this file. If not, see <http://www.gnu.org/licenses/>.
### Imports ###
import copy
from common.id import CapId
### Decorators ###
## Print Decorator Variables
ERROR = '\033[5;1;31mERROR\033[0m:'
WARNING = '\033[5;1;33mWARNING\033[0m:'
### Classes ###
class Expression:
'''
Container class for KLL expressions
'''
def __init__( self, lparam, operator, rparam, context ):
'''
Initialize expression container
@param lparam: LOperatorData token
@param operator: Operator token
@param rparam: ROperatorData token
@param context: Parent context of expression
'''
# First stage/init
self.lparam_token = lparam
self.operator_token = operator
self.rparam_token = rparam
self.context = context # TODO, set multiple contexts for later stages
# Second stage
self.lparam_sub_tokens = []
self.rparam_sub_tokens = []
# Mutate class into the desired type
self.__class__ = {
'=>' : NameAssociationExpression,
'<=' : DataAssociationExpression,
'=' : AssignmentExpression,
':' : MapExpression,
}[ self.operator_type() ]
def operator_type( self ):
'''
Determine which base operator this operator is of
All : (map) expressions are tokenized/parsed the same way
@return Base string representation of the operator
'''
if ':' in self.operator_token.value:
return ':'
return self.operator_token.value
def final_tokens( self, no_filter=False ):
'''
Return the final list of tokens, must complete the second stage first
@param no_filter: If true, do not filter out Space tokens
@return Finalized list of tokens
'''
ret = self.lparam_sub_tokens + [ self.operator_token ] + self.rparam_sub_tokens
if not no_filter:
ret = [ x for x in ret if x.type != 'Space' ]
return ret
def regen_str( self ):
'''
Re-construct the string based off the original set of tokens
<lparam><operator><rparam>;
'''
return "{0}{1}{2};".format(
self.lparam_token.value,
self.operator_token.value,
self.rparam_token.value,
)
def point_chars( self, pos_list ):
'''
Using the regenerated string, point to a given list of characters
Used to indicate where a possible issue/syntax error is
@param pos_list: List of character indices
i.e.
> U"A" : : U"1";
> ^
'''
out = "\t{0}\n\t".format( self.regen_str() )
# Place a ^ character at the given locations
curpos = 1
for pos in sorted( pos_list ):
# Pad spaces, then add a ^
out += ' ' * (pos - curpos)
out += '^'
curpos += pos
return out
def rparam_start( self ):
'''
Starting positing char of rparam_token in a regen_str
'''
return len( self.lparam_token.value ) + len( self.operator_token.value )
def __repr__( self ):
# Build string representation based off of what has been set
# lparam, operator and rparam are always set
out = "Expression: {0}{1}{2}".format(
self.lparam_token.value,
self.operator_token.value,
self.rparam_token.value,
)
# TODO - Add more depending on what has been set
return out
def unique_keys( self ):
'''
Generates a list of unique identifiers for the expression that is mergeable
with other functional equivalent expressions.
This method should never get called directly as a generic Expression
'''
return [ ('UNKNOWN KEY', 'UNKNOWN EXPRESSION') ]
class AssignmentExpression( Expression ):
'''
Container class for assignment KLL expressions
'''
type = None
name = None
pos = None
value = None
## Setters ##
def array( self, name, pos, value ):
'''
Assign array assignment parameters to expression
@param name: Name of variable
@param pos: Array position of the value (if None, overwrite the entire array)
@param value: Value of the array, if pos is specified, this is the value of an element
@return: True if parsing was successful
'''
self.type = 'Array'
self.name = name
self.pos = pos
self.value = value
# If pos is not none, flatten
if pos is not None:
self.value = "".join( str( x ) for x in self.value )
return True
def variable( self, name, value ):
'''
Assign variable assignment parameters to expression
@param name: Name of variable
@param value: Value of variable
@return: True if parsing was successful
'''
self.type = 'Variable'
self.name = name
self.value = value
# Flatten value, often a list of various token types
self.value = "".join( str( x ) for x in self.value )
return True
def __repr__( self ):
if self.type == 'Variable':
return "{0} = {1};".format( self.name, self.value )
elif self.type == 'Array':
return "{0}[{1}] = {2};".format( self.name, self.pos, self.value )
return "ASSIGNMENT UNKNOWN"
def unique_keys( self ):
'''
Generates a list of unique identifiers for the expression that is mergeable
with other functional equivalent expressions.
'''
return [ ( self.name, self ) ]
class NameAssociationExpression( Expression ):
'''
Container class for name association KLL expressions
'''
type = None
name = None
association = None
## Setters ##
def capability( self, name, association, parameters ):
'''
Assign a capability C function name association
@param name: Name of capability
@param association: Name of capability in target backend output
@return: True if parsing was successful
'''
self.type = 'Capability'
self.name = name
self.association = CapId( association, 'Definition', parameters )
return True
def define( self, name, association ):
'''
Assign a define C define name association
@param name: Name of variable
@param association: Name of association in target backend output
@return: True if parsing was successful
'''
self.type = 'Define'
self.name = name
self.association = association
return True
def __repr__( self ):
return "{0} <= {1};".format( self.name, self.association )
def unique_keys( self ):
'''
Generates a list of unique identifiers for the expression that is mergeable
with other functional equivalent expressions.
'''
return [ ( self.name, self ) ]
class DataAssociationExpression( Expression ):
'''
Container class for data association KLL expressions
'''
type = None
association = None
value = None
## Setters ##
def animation( self, animations, animation_modifiers ):
'''
Animation definition and configuration
@return: True if parsing was successful
'''
self.type = 'Animation'
self.association = animations
self.value = animation_modifiers
return True
def animationFrame( self, animation_frames, pixel_modifiers ):
'''
Pixel composition of an Animation Frame
@return: True if parsing was successful
'''
self.type = 'AnimationFrame'
self.association = animation_frames
self.value = pixel_modifiers
return True
def pixelPosition( self, pixels, position ):
'''
Pixel Positioning
@return: True if parsing was successful
'''
for pixel in pixels:
pixel.setPosition( position )
self.type = 'PixelPosition'
self.association = pixels
return True
def scanCodePosition( self, scancodes, position ):
'''
Scan Code to Position Mapping
Note: Accepts lists of scan codes
Alone this isn't useful, but you can assign rows and columns using ranges instead of individually
@return: True if parsing was successful
'''
for scancode in scancodes:
scancode.setPosition( position )
self.type = 'ScanCodePosition'
self.association = scancodes
return True
def __repr__( self ):
if self.type in ['PixelPosition', 'ScanCodePosition']:
output = ""
for index, association in enumerate( self.association ):
if index > 0:
output += "; "
output += "{0}".format( association )
return "{0};".format( output )
return "{0} <= {1};".format( self.association, self.value )
def unique_keys( self ):
'''
Generates a list of unique identifiers for the expression that is mergeable
with other functional equivalent expressions.
'''
keys = []
# Positions require a bit more introspection to get the unique keys
if self.type in ['PixelPosition', 'ScanCodePosition']:
for index, key in enumerate( self.association ):
uniq_expr = self
# If there is more than one key, copy the expression
# and remove the non-related variants
if len( self.association ) > 1:
uniq_expr = copy.copy( self )
# Isolate variant by index
uniq_expr.association = [ uniq_expr.association[ index ] ]
keys.append( ( "{0}".format( key.unique_key() ), uniq_expr ) )
# AnimationFrames are already list of keys
# TODO Reorder frame assignments to dedup function equivalent mappings
elif self.type in ['AnimationFrame']:
for index, key in enumerate( self.association ):
uniq_expr = self
# If there is more than one key, copy the expression
# and remove the non-related variants
if len( self.association ) > 1:
uniq_expr = copy.copy( self )
# Isolate variant by index
uniq_expr.association = [ uniq_expr.association[ index ] ]
keys.append( ( "{0}".format( key ), uniq_expr ) )
# Otherwise treat as a single element
else:
keys = [ ( "{0}".format( self.association ), self ) ]
# Remove any duplicate keys
# TODO Stat? Might be at neat report about how many duplicates were squashed
keys = list( set( keys ) )
return keys
class MapExpression( Expression ):
'''
Container class for KLL map expressions
'''
type = None
triggers = None
operator = None
results = None
animation = None
animation_frame = None
pixels = None
position = None
## Setters ##
def scanCode( self, triggers, operator, results ):
'''
Scan Code mapping
@param triggers: Sequence of combos of ranges of namedtuples
@param operator: Type of map operation
@param results: Sequence of combos of ranges of namedtuples
@return: True if parsing was successful
'''
self.type = 'ScanCode'
self.triggers = triggers
self.operator = operator
self.results = results
return True
def usbCode( self, triggers, operator, results ):
'''
USB Code mapping
@param triggers: Sequence of combos of ranges of namedtuples
@param operator: Type of map operation
@param results: Sequence of combos of ranges of namedtuples
@return: True if parsing was successful
'''
self.type = 'USBCode'
self.triggers = triggers
self.operator = operator
self.results = results
return True
def animationTrigger( self, animation, operator, results ):
'''
Animation Trigger mapping
@param animation: Animation trigger of result
@param operator: Type of map operation
@param results: Sequence of combos of ranges of namedtuples
@return: True if parsing was successful
'''
self.type = 'Animation'
self.animation = animation
self.triggers = animation
self.operator = operator
self.results = results
return True
def pixelChannels( self, pixelmap, trigger ):
'''
Pixel Channel Composition
@return: True if parsing was successful
'''
self.type = 'PixelChannel'
self.pixel = pixelmap
self.position = trigger
return True
def sequencesOfCombosOfIds( self, expression_param ):
'''
Prettified Sequence of Combos of Identifiers
@param expression_param: Trigger or Result parameter of an expression
Scan Code Example
[[[S10, S16], [S42]], [[S11, S16], [S42]]] -> (S10 + S16, S42)|(S11 + S16, S42)
'''
output = ""
# Sometimes during error cases, might be None
if expression_param is None:
return output
# Iterate over each trigger/result variants (expanded from ranges), each one is a sequence
for index, sequence in enumerate( expression_param ):
if index > 0:
output += "|"
output += "("
# Iterate over each combo (element of the sequence)
for index, combo in enumerate( sequence ):
if index > 0:
output += ", "
# Iterate over each trigger identifier
for index, identifier in enumerate( combo ):
if index > 0:
output += " + "
output += "{0}".format( identifier )
output += ")"
return output
def elems( self ):
'''
Return number of trigger and result elements
Useful for determining if this is a trigger macro (2+)
Should always return at least (1,1) unless it's an invalid calculation
@return: ( triggers, results )
'''
elems = [ 0, 0 ]
# XXX Needed?
if self.type == 'PixelChannel':
return tuple( elems )
# Iterate over each trigger variant (expanded from ranges), each one is a sequence
for sequence in self.triggers:
# Iterate over each combo (element of the sequence)
for combo in sequence:
# Just measure the size of the combo
elems[0] += len( combo )
# Iterate over each result variant (expanded from ranges), each one is a sequence
for sequence in self.results:
# Iterate over each combo (element of the sequence)
for combo in sequence:
# Just measure the size of the combo
elems[1] += len( combo )
return tuple( elems )
def trigger_str( self ):
'''
String version of the trigger
Used for sorting
'''
# Pixel Channel Mapping doesn't follow the same pattern
if self.type == 'PixelChannel':
return "{0}".format( self.pixel )
return "{0}".format(
self.sequencesOfCombosOfIds( self.triggers ),
)
def result_str( self ):
'''
String version of the result
Used for sorting
'''
# Pixel Channel Mapping doesn't follow the same pattern
if self.type == 'PixelChannel':
return "{0}".format( self.position )
return "{0}".format(
self.sequencesOfCombosOfIds( self.results ),
)
def __repr__( self ):
# Pixel Channel Mapping doesn't follow the same pattern
if self.type == 'PixelChannel':
return "{0} : {1};".format( self.pixel, self.position )
return "{0} {1} {2};".format(
self.sequencesOfCombosOfIds( self.triggers ),
self.operator,
self.sequencesOfCombosOfIds( self.results ),
)
def unique_keys( self ):
'''
Generates a list of unique identifiers for the expression that is mergeable
with other functional equivalent expressions.
TODO: This function should re-order combinations to generate the key.
The final generated combo will be in the original order.
'''
keys = []
# Pixel Channel only has key per mapping
if self.type == 'PixelChannel':
keys = [ ( "{0}".format( self.pixel ), self ) ]
# Split up each of the keys
else:
# Iterate over each trigger/result variants (expanded from ranges), each one is a sequence
for index, sequence in enumerate( self.triggers ):
key = ""
uniq_expr = self
# If there is more than one key, copy the expression
# and remove the non-related variants
if len( self.triggers ) > 1:
uniq_expr = copy.copy( self )
# Isolate variant by index
uniq_expr.triggers = [ uniq_expr.triggers[ index ] ]
# Iterate over each combo (element of the sequence)
for index, combo in enumerate( sequence ):
if index > 0:
key += ", "
# Iterate over each trigger identifier
for index, identifier in enumerate( combo ):
if index > 0:
key += " + "
key += "{0}".format( identifier )
# Add key to list
keys.append( ( key, uniq_expr ) )
# Remove any duplicate keys
# TODO Stat? Might be at neat report about how many duplicates were squashed
keys = list( set( keys ) )
return keys