KLL Compiler
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
This repo is archived. You can view files and clone it, but cannot push or open issues/pull-requests.

expression.py 16KB

KLL Compiler Re-Write This was many months of efforts in re-designing how the KLL compiler should work. The major problem with the original compiler was how difficult it was to extend language wise. This lead to many delays in KLL 0.4 and 0.5 being implemented. The new design is a multi-staged compiler, where even tokenization occurs over multiple stages. This allows individual parsing and token regexes to be expressed more simply without affect other expressions. Another area of change is the concept of Contexts. In the original KLL compiler the idea of a cache assigned was "hacked" on when I realized the language was "broken" (after nearly finishing the compiler). Since assignment order is generally considered not to matter for keymappings, I created a "cached" assignment where the whole file is read into a sub-datastructure, then apply to the master datastructure. Unfortunately, this wasn't really all that clear, so it was annoying to work with. To remedy this, I created KLL Contexts, which contain information about a group of expressions. Not only can these groups can be merged with other Contexts, they have historical data about how they were generated allowing for errors very late in processing to be pin-pointed back to the offending kll file. Backends work nearly the same as they did before. However, all call-backs for capability evaluations have been removed. This makes the interface much cleaner as Contexts can only be symbolically merged now. (Previously datastructures did evaluation merges where the ScanCode or Capability was looked up right before passing to the backend, but this required additional information from the backend). Many of the old parsing and tokenization rules have been reused, along with the hid_dict.py code. The new design takes advantage of processor pools to handle multithreading where it makes sense. For example, all specified files are loaded into ram simulatenously rather than sparingly reading from. The reason for this is so that each Context always has all the information it requires at all times. kll - Program entry point (previously kll.py) - Very small now, does some setting up of command-line args - Most command-line args are specified by the corresponding processing stage common/channel.py - Pixel Channel container classes common/context.py - Context container classes - As is usual with other files, blank classes inherit a base class - These blank classes are identified by the class name itself to handle special behaviour - And if/when necessary functions are re-implemented - MergeConext class facilitates merging of contexts while maintaining lineage common/expression.py - Expression container classes * Expression base class * AssignmentExpression * NameAssociationExpression * DataAssociationExpression * MapExpression - These classes are used to store expressions after they have finished parsing and tokenization common/file.py - Container class for files being read by the KLL compiler common/emitter.py - Base class for all KLL emitters - TextEmitter for dealing with text file templates common/hid_dict.py - Slightly modified version of kll_lib/hid_dict.py common/id.py - Identification container classes - Used to indentify different types of elements used within the KLL language common/modifier.py - Container classes for animation and pixel change functions common/organization.py - Data structure merging container classes - Contains all the sub-datastructure classes as well - The Organization class handles the merge orchestration and expression insertion common/parse.py - Parsing rules for funcparserlib - Much of this file was taken from the original kll.py - Many changes to support the multi-stage processing and support KLL 0.5 common/position.py - Container class dealing with physical positions common/schedule.py - Container class dealing with scheduling and timing events common/stage.py - Contains ControlStage and main Stage classes * CompilerConfigurationStage * FileImportStage * PreprocessorStage * OperationClassificationStage * OperationSpecificsStage * OperationOrganizationStage * DataOrganziationStage * DataFinalizationStage * DataAnalysisStage * CodeGenerationStage * ReportGenerationStage - Each of these classes controls the life-cycle of each stage - If multi-threading is desired, it must be handled within the class * The next stage will not start until the current stage is finished - Errors are handled such that as many errors as possible are recorded before forcing an exit * The exit is handled at the end of each stage if necessary - Command-line arguments for each stage can be defined if necessary (they are given their own grouping) - Each stage can pull variables and functions from other stages if necessary using a name lookup * This means you don't have to worry about over-arching datastructures emitters/emitters.py - Container class for KLL emitters - Handles emitter setup and selection emitters/kiibohd/kiibohd.py - kiibohd .h file KLL emitter - Re-uses some backend code from the original KLL compiler funcparserlib/parser.py - Added debug mode control examples/assignment.kll examples/defaultMapExample.kll examples/example.kll examples/hhkbpro2.kll examples/leds.kll examples/mapping.kll examples/simple1.kll examples/simple2.kll examples/simpleExample.kll examples/state_scheduling.kll - Updating/Adding rules for new compiler and KLL 0.4 + KLL 0.5 support
hace 7 años
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630
  1. #!/usr/bin/env python3
  2. '''
  3. KLL Expression Container
  4. '''
  5. # Copyright (C) 2016 by Jacob Alexander
  6. #
  7. # This file is free software: you can redistribute it and/or modify
  8. # it under the terms of the GNU General Public License as published by
  9. # the Free Software Foundation, either version 3 of the License, or
  10. # (at your option) any later version.
  11. #
  12. # This file is distributed in the hope that it will be useful,
  13. # but WITHOUT ANY WARRANTY; without even the implied warranty of
  14. # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
  15. # GNU General Public License for more details.
  16. #
  17. # You should have received a copy of the GNU General Public License
  18. # along with this file. If not, see <http://www.gnu.org/licenses/>.
  19. ### Imports ###
  20. import copy
  21. from common.id import CapId
  22. ### Decorators ###
  23. ## Print Decorator Variables
  24. ERROR = '\033[5;1;31mERROR\033[0m:'
  25. WARNING = '\033[5;1;33mWARNING\033[0m:'
  26. ### Classes ###
  27. class Expression:
  28. '''
  29. Container class for KLL expressions
  30. '''
  31. def __init__( self, lparam, operator, rparam, context ):
  32. '''
  33. Initialize expression container
  34. @param lparam: LOperatorData token
  35. @param operator: Operator token
  36. @param rparam: ROperatorData token
  37. @param context: Parent context of expression
  38. '''
  39. # First stage/init
  40. self.lparam_token = lparam
  41. self.operator_token = operator
  42. self.rparam_token = rparam
  43. self.context = context # TODO, set multiple contexts for later stages
  44. # Second stage
  45. self.lparam_sub_tokens = []
  46. self.rparam_sub_tokens = []
  47. # Mutate class into the desired type
  48. self.__class__ = {
  49. '=>' : NameAssociationExpression,
  50. '<=' : DataAssociationExpression,
  51. '=' : AssignmentExpression,
  52. ':' : MapExpression,
  53. }[ self.operator_type() ]
  54. def operator_type( self ):
  55. '''
  56. Determine which base operator this operator is of
  57. All : (map) expressions are tokenized/parsed the same way
  58. @return Base string representation of the operator
  59. '''
  60. if ':' in self.operator_token.value:
  61. return ':'
  62. return self.operator_token.value
  63. def final_tokens( self, no_filter=False ):
  64. '''
  65. Return the final list of tokens, must complete the second stage first
  66. @param no_filter: If true, do not filter out Space tokens
  67. @return Finalized list of tokens
  68. '''
  69. ret = self.lparam_sub_tokens + [ self.operator_token ] + self.rparam_sub_tokens
  70. if not no_filter:
  71. ret = [ x for x in ret if x.type != 'Space' ]
  72. return ret
  73. def regen_str( self ):
  74. '''
  75. Re-construct the string based off the original set of tokens
  76. <lparam><operator><rparam>;
  77. '''
  78. return "{0}{1}{2};".format(
  79. self.lparam_token.value,
  80. self.operator_token.value,
  81. self.rparam_token.value,
  82. )
  83. def point_chars( self, pos_list ):
  84. '''
  85. Using the regenerated string, point to a given list of characters
  86. Used to indicate where a possible issue/syntax error is
  87. @param pos_list: List of character indices
  88. i.e.
  89. > U"A" : : U"1";
  90. > ^
  91. '''
  92. out = "\t{0}\n\t".format( self.regen_str() )
  93. # Place a ^ character at the given locations
  94. curpos = 1
  95. for pos in sorted( pos_list ):
  96. # Pad spaces, then add a ^
  97. out += ' ' * (pos - curpos)
  98. out += '^'
  99. curpos += pos
  100. return out
  101. def rparam_start( self ):
  102. '''
  103. Starting positing char of rparam_token in a regen_str
  104. '''
  105. return len( self.lparam_token.value ) + len( self.operator_token.value )
  106. def __repr__( self ):
  107. # Build string representation based off of what has been set
  108. # lparam, operator and rparam are always set
  109. out = "Expression: {0}{1}{2}".format(
  110. self.lparam_token.value,
  111. self.operator_token.value,
  112. self.rparam_token.value,
  113. )
  114. # TODO - Add more depending on what has been set
  115. return out
  116. def unique_keys( self ):
  117. '''
  118. Generates a list of unique identifiers for the expression that is mergeable
  119. with other functional equivalent expressions.
  120. This method should never get called directly as a generic Expression
  121. '''
  122. return [ ('UNKNOWN KEY', 'UNKNOWN EXPRESSION') ]
  123. class AssignmentExpression( Expression ):
  124. '''
  125. Container class for assignment KLL expressions
  126. '''
  127. type = None
  128. name = None
  129. pos = None
  130. value = None
  131. ## Setters ##
  132. def array( self, name, pos, value ):
  133. '''
  134. Assign array assignment parameters to expression
  135. @param name: Name of variable
  136. @param pos: Array position of the value (if None, overwrite the entire array)
  137. @param value: Value of the array, if pos is specified, this is the value of an element
  138. @return: True if parsing was successful
  139. '''
  140. self.type = 'Array'
  141. self.name = name
  142. self.pos = pos
  143. self.value = value
  144. # If pos is not none, flatten
  145. if pos is not None:
  146. self.value = "".join( str( x ) for x in self.value )
  147. return True
  148. def variable( self, name, value ):
  149. '''
  150. Assign variable assignment parameters to expression
  151. @param name: Name of variable
  152. @param value: Value of variable
  153. @return: True if parsing was successful
  154. '''
  155. self.type = 'Variable'
  156. self.name = name
  157. self.value = value
  158. # Flatten value, often a list of various token types
  159. self.value = "".join( str( x ) for x in self.value )
  160. return True
  161. def __repr__( self ):
  162. if self.type == 'Variable':
  163. return "{0} = {1};".format( self.name, self.value )
  164. elif self.type == 'Array':
  165. return "{0}[{1}] = {2};".format( self.name, self.pos, self.value )
  166. return "ASSIGNMENT UNKNOWN"
  167. def unique_keys( self ):
  168. '''
  169. Generates a list of unique identifiers for the expression that is mergeable
  170. with other functional equivalent expressions.
  171. '''
  172. return [ ( self.name, self ) ]
  173. class NameAssociationExpression( Expression ):
  174. '''
  175. Container class for name association KLL expressions
  176. '''
  177. type = None
  178. name = None
  179. association = None
  180. ## Setters ##
  181. def capability( self, name, association, parameters ):
  182. '''
  183. Assign a capability C function name association
  184. @param name: Name of capability
  185. @param association: Name of capability in target backend output
  186. @return: True if parsing was successful
  187. '''
  188. self.type = 'Capability'
  189. self.name = name
  190. self.association = CapId( association, 'Definition', parameters )
  191. return True
  192. def define( self, name, association ):
  193. '''
  194. Assign a define C define name association
  195. @param name: Name of variable
  196. @param association: Name of association in target backend output
  197. @return: True if parsing was successful
  198. '''
  199. self.type = 'Define'
  200. self.name = name
  201. self.association = association
  202. return True
  203. def __repr__( self ):
  204. return "{0} <= {1};".format( self.name, self.association )
  205. def unique_keys( self ):
  206. '''
  207. Generates a list of unique identifiers for the expression that is mergeable
  208. with other functional equivalent expressions.
  209. '''
  210. return [ ( self.name, self ) ]
  211. class DataAssociationExpression( Expression ):
  212. '''
  213. Container class for data association KLL expressions
  214. '''
  215. type = None
  216. association = None
  217. value = None
  218. ## Setters ##
  219. def animation( self, animations, animation_modifiers ):
  220. '''
  221. Animation definition and configuration
  222. @return: True if parsing was successful
  223. '''
  224. self.type = 'Animation'
  225. self.association = animations
  226. self.value = animation_modifiers
  227. return True
  228. def animationFrame( self, animation_frames, pixel_modifiers ):
  229. '''
  230. Pixel composition of an Animation Frame
  231. @return: True if parsing was successful
  232. '''
  233. self.type = 'AnimationFrame'
  234. self.association = animation_frames
  235. self.value = pixel_modifiers
  236. return True
  237. def pixelPosition( self, pixels, position ):
  238. '''
  239. Pixel Positioning
  240. @return: True if parsing was successful
  241. '''
  242. for pixel in pixels:
  243. pixel.setPosition( position )
  244. self.type = 'PixelPosition'
  245. self.association = pixels
  246. return True
  247. def scanCodePosition( self, scancodes, position ):
  248. '''
  249. Scan Code to Position Mapping
  250. Note: Accepts lists of scan codes
  251. Alone this isn't useful, but you can assign rows and columns using ranges instead of individually
  252. @return: True if parsing was successful
  253. '''
  254. for scancode in scancodes:
  255. scancode.setPosition( position )
  256. self.type = 'ScanCodePosition'
  257. self.association = scancodes
  258. return True
  259. def __repr__( self ):
  260. if self.type in ['PixelPosition', 'ScanCodePosition']:
  261. output = ""
  262. for index, association in enumerate( self.association ):
  263. if index > 0:
  264. output += "; "
  265. output += "{0}".format( association )
  266. return "{0};".format( output )
  267. return "{0} <= {1};".format( self.association, self.value )
  268. def unique_keys( self ):
  269. '''
  270. Generates a list of unique identifiers for the expression that is mergeable
  271. with other functional equivalent expressions.
  272. '''
  273. keys = []
  274. # Positions require a bit more introspection to get the unique keys
  275. if self.type in ['PixelPosition', 'ScanCodePosition']:
  276. for index, key in enumerate( self.association ):
  277. uniq_expr = self
  278. # If there is more than one key, copy the expression
  279. # and remove the non-related variants
  280. if len( self.association ) > 1:
  281. uniq_expr = copy.copy( self )
  282. # Isolate variant by index
  283. uniq_expr.association = [ uniq_expr.association[ index ] ]
  284. keys.append( ( "{0}".format( key.unique_key() ), uniq_expr ) )
  285. # AnimationFrames are already list of keys
  286. # TODO Reorder frame assignments to dedup function equivalent mappings
  287. elif self.type in ['AnimationFrame']:
  288. for index, key in enumerate( self.association ):
  289. uniq_expr = self
  290. # If there is more than one key, copy the expression
  291. # and remove the non-related variants
  292. if len( self.association ) > 1:
  293. uniq_expr = copy.copy( self )
  294. # Isolate variant by index
  295. uniq_expr.association = [ uniq_expr.association[ index ] ]
  296. keys.append( ( "{0}".format( key ), uniq_expr ) )
  297. # Otherwise treat as a single element
  298. else:
  299. keys = [ ( "{0}".format( self.association ), self ) ]
  300. # Remove any duplicate keys
  301. # TODO Stat? Might be at neat report about how many duplicates were squashed
  302. keys = list( set( keys ) )
  303. return keys
  304. class MapExpression( Expression ):
  305. '''
  306. Container class for KLL map expressions
  307. '''
  308. type = None
  309. triggers = None
  310. operator = None
  311. results = None
  312. animation = None
  313. animation_frame = None
  314. pixels = None
  315. position = None
  316. ## Setters ##
  317. def scanCode( self, triggers, operator, results ):
  318. '''
  319. Scan Code mapping
  320. @param triggers: Sequence of combos of ranges of namedtuples
  321. @param operator: Type of map operation
  322. @param results: Sequence of combos of ranges of namedtuples
  323. @return: True if parsing was successful
  324. '''
  325. self.type = 'ScanCode'
  326. self.triggers = triggers
  327. self.operator = operator
  328. self.results = results
  329. return True
  330. def usbCode( self, triggers, operator, results ):
  331. '''
  332. USB Code mapping
  333. @param triggers: Sequence of combos of ranges of namedtuples
  334. @param operator: Type of map operation
  335. @param results: Sequence of combos of ranges of namedtuples
  336. @return: True if parsing was successful
  337. '''
  338. self.type = 'USBCode'
  339. self.triggers = triggers
  340. self.operator = operator
  341. self.results = results
  342. return True
  343. def animationTrigger( self, animation, operator, results ):
  344. '''
  345. Animation Trigger mapping
  346. @param animation: Animation trigger of result
  347. @param operator: Type of map operation
  348. @param results: Sequence of combos of ranges of namedtuples
  349. @return: True if parsing was successful
  350. '''
  351. self.type = 'Animation'
  352. self.animation = animation
  353. self.triggers = animation
  354. self.operator = operator
  355. self.results = results
  356. return True
  357. def pixelChannels( self, pixelmap, trigger ):
  358. '''
  359. Pixel Channel Composition
  360. @return: True if parsing was successful
  361. '''
  362. self.type = 'PixelChannel'
  363. self.pixel = pixelmap
  364. self.position = trigger
  365. return True
  366. def sequencesOfCombosOfIds( self, expression_param ):
  367. '''
  368. Prettified Sequence of Combos of Identifiers
  369. @param expression_param: Trigger or Result parameter of an expression
  370. Scan Code Example
  371. [[[S10, S16], [S42]], [[S11, S16], [S42]]] -> (S10 + S16, S42)|(S11 + S16, S42)
  372. '''
  373. output = ""
  374. # Sometimes during error cases, might be None
  375. if expression_param is None:
  376. return output
  377. # Iterate over each trigger/result variants (expanded from ranges), each one is a sequence
  378. for index, sequence in enumerate( expression_param ):
  379. if index > 0:
  380. output += "|"
  381. output += "("
  382. # Iterate over each combo (element of the sequence)
  383. for index, combo in enumerate( sequence ):
  384. if index > 0:
  385. output += ", "
  386. # Iterate over each trigger identifier
  387. for index, identifier in enumerate( combo ):
  388. if index > 0:
  389. output += " + "
  390. output += "{0}".format( identifier )
  391. output += ")"
  392. return output
  393. def elems( self ):
  394. '''
  395. Return number of trigger and result elements
  396. Useful for determining if this is a trigger macro (2+)
  397. Should always return at least (1,1) unless it's an invalid calculation
  398. @return: ( triggers, results )
  399. '''
  400. elems = [ 0, 0 ]
  401. # XXX Needed?
  402. if self.type == 'PixelChannel':
  403. return tuple( elems )
  404. # Iterate over each trigger variant (expanded from ranges), each one is a sequence
  405. for sequence in self.triggers:
  406. # Iterate over each combo (element of the sequence)
  407. for combo in sequence:
  408. # Just measure the size of the combo
  409. elems[0] += len( combo )
  410. # Iterate over each result variant (expanded from ranges), each one is a sequence
  411. for sequence in self.results:
  412. # Iterate over each combo (element of the sequence)
  413. for combo in sequence:
  414. # Just measure the size of the combo
  415. elems[1] += len( combo )
  416. return tuple( elems )
  417. def trigger_str( self ):
  418. '''
  419. String version of the trigger
  420. Used for sorting
  421. '''
  422. # Pixel Channel Mapping doesn't follow the same pattern
  423. if self.type == 'PixelChannel':
  424. return "{0}".format( self.pixel )
  425. return "{0}".format(
  426. self.sequencesOfCombosOfIds( self.triggers ),
  427. )
  428. def result_str( self ):
  429. '''
  430. String version of the result
  431. Used for sorting
  432. '''
  433. # Pixel Channel Mapping doesn't follow the same pattern
  434. if self.type == 'PixelChannel':
  435. return "{0}".format( self.position )
  436. return "{0}".format(
  437. self.sequencesOfCombosOfIds( self.results ),
  438. )
  439. def __repr__( self ):
  440. # Pixel Channel Mapping doesn't follow the same pattern
  441. if self.type == 'PixelChannel':
  442. return "{0} : {1};".format( self.pixel, self.position )
  443. return "{0} {1} {2};".format(
  444. self.sequencesOfCombosOfIds( self.triggers ),
  445. self.operator,
  446. self.sequencesOfCombosOfIds( self.results ),
  447. )
  448. def unique_keys( self ):
  449. '''
  450. Generates a list of unique identifiers for the expression that is mergeable
  451. with other functional equivalent expressions.
  452. TODO: This function should re-order combinations to generate the key.
  453. The final generated combo will be in the original order.
  454. '''
  455. keys = []
  456. # Pixel Channel only has key per mapping
  457. if self.type == 'PixelChannel':
  458. keys = [ ( "{0}".format( self.pixel ), self ) ]
  459. # Split up each of the keys
  460. else:
  461. # Iterate over each trigger/result variants (expanded from ranges), each one is a sequence
  462. for index, sequence in enumerate( self.triggers ):
  463. key = ""
  464. uniq_expr = self
  465. # If there is more than one key, copy the expression
  466. # and remove the non-related variants
  467. if len( self.triggers ) > 1:
  468. uniq_expr = copy.copy( self )
  469. # Isolate variant by index
  470. uniq_expr.triggers = [ uniq_expr.triggers[ index ] ]
  471. # Iterate over each combo (element of the sequence)
  472. for index, combo in enumerate( sequence ):
  473. if index > 0:
  474. key += ", "
  475. # Iterate over each trigger identifier
  476. for index, identifier in enumerate( combo ):
  477. if index > 0:
  478. key += " + "
  479. key += "{0}".format( identifier )
  480. # Add key to list
  481. keys.append( ( key, uniq_expr ) )
  482. # Remove any duplicate keys
  483. # TODO Stat? Might be at neat report about how many duplicates were squashed
  484. keys = list( set( keys ) )
  485. return keys