This was many months of efforts in re-designing how the KLL compiler should work.
The major problem with the original compiler was how difficult it was to extend language wise.
This lead to many delays in KLL 0.4 and 0.5 being implemented.
The new design is a multi-staged compiler, where even tokenization occurs over multiple stages.
This allows individual parsing and token regexes to be expressed more simply without affect other expressions.
Another area of change is the concept of Contexts.
In the original KLL compiler the idea of a cache assigned was "hacked" on when I realized the language was "broken" (after nearly finishing the compiler).
Since assignment order is generally considered not to matter for keymappings, I created a "cached" assignment where the whole file is read into a sub-datastructure, then apply to the master datastructure.
Unfortunately, this wasn't really all that clear, so it was annoying to work with.
To remedy this, I created KLL Contexts, which contain information about a group of expressions.
Not only can these groups can be merged with other Contexts, they have historical data about how they were generated allowing for errors very late in processing to be pin-pointed back to the offending kll file.
Backends work nearly the same as they did before.
However, all call-backs for capability evaluations have been removed.
This makes the interface much cleaner as Contexts can only be symbolically merged now.
(Previously datastructures did evaluation merges where the ScanCode or Capability was looked up right before passing to the backend, but this required additional information from the backend).
Many of the old parsing and tokenization rules have been reused, along with the hid_dict.py code.
The new design takes advantage of processor pools to handle multithreading where it makes sense.
For example, all specified files are loaded into ram simulatenously rather than sparingly reading from.
The reason for this is so that each Context always has all the information it requires at all times.
kll
- Program entry point (previously kll.py)
- Very small now, does some setting up of command-line args
- Most command-line args are specified by the corresponding processing stage
common/channel.py
- Pixel Channel container classes
common/context.py
- Context container classes
- As is usual with other files, blank classes inherit a base class
- These blank classes are identified by the class name itself to handle special behaviour
- And if/when necessary functions are re-implemented
- MergeConext class facilitates merging of contexts while maintaining lineage
common/expression.py
- Expression container classes
* Expression base class
* AssignmentExpression
* NameAssociationExpression
* DataAssociationExpression
* MapExpression
- These classes are used to store expressions after they have finished parsing and tokenization
common/file.py
- Container class for files being read by the KLL compiler
common/emitter.py
- Base class for all KLL emitters
- TextEmitter for dealing with text file templates
common/hid_dict.py
- Slightly modified version of kll_lib/hid_dict.py
common/id.py
- Identification container classes
- Used to indentify different types of elements used within the KLL language
common/modifier.py
- Container classes for animation and pixel change functions
common/organization.py
- Data structure merging container classes
- Contains all the sub-datastructure classes as well
- The Organization class handles the merge orchestration and expression insertion
common/parse.py
- Parsing rules for funcparserlib
- Much of this file was taken from the original kll.py
- Many changes to support the multi-stage processing and support KLL 0.5
common/position.py
- Container class dealing with physical positions
common/schedule.py
- Container class dealing with scheduling and timing events
common/stage.py
- Contains ControlStage and main Stage classes
* CompilerConfigurationStage
* FileImportStage
* PreprocessorStage
* OperationClassificationStage
* OperationSpecificsStage
* OperationOrganizationStage
* DataOrganziationStage
* DataFinalizationStage
* DataAnalysisStage
* CodeGenerationStage
* ReportGenerationStage
- Each of these classes controls the life-cycle of each stage
- If multi-threading is desired, it must be handled within the class
* The next stage will not start until the current stage is finished
- Errors are handled such that as many errors as possible are recorded before forcing an exit
* The exit is handled at the end of each stage if necessary
- Command-line arguments for each stage can be defined if necessary (they are given their own grouping)
- Each stage can pull variables and functions from other stages if necessary using a name lookup
* This means you don't have to worry about over-arching datastructures
emitters/emitters.py
- Container class for KLL emitters
- Handles emitter setup and selection
emitters/kiibohd/kiibohd.py
- kiibohd .h file KLL emitter
- Re-uses some backend code from the original KLL compiler
funcparserlib/parser.py
- Added debug mode control
examples/assignment.kll
examples/defaultMapExample.kll
examples/example.kll
examples/hhkbpro2.kll
examples/leds.kll
examples/mapping.kll
examples/simple1.kll
examples/simple2.kll
examples/simpleExample.kll
examples/state_scheduling.kll
- Updating/Adding rules for new compiler and KLL 0.4 + KLL 0.5 support
- Updated both FuncMaps to use Soft Replace
- Only used for layers, default and base maps are not affected by soft replace (works like a normal replace)
- Required changing the ScanCode node datastructure
- Interconnect Id's must be stored until the end as it's not possible to calculate the max per node ScanCode until after all the assignments are complete
- Should make future additions more straight-forward (that require per ScanCode information to be stored)
- Name and kll filenames are treated as special variables
- Using the order of the stacks the compilation order can be inferred (useful for debugging)
- Layer names are finally implemented (instead of Layer 1, Layer 2, etc.)
- To better support RAM/Flash packing adding support for first scan code calculation
- This way if the first scan code is high (e.g. 0x40) RAM is not used for the scan codes without keys assigned
- Some initial work for KLL variables
- Will be used to influence runtime and compiler features of the firmware
- Layers never worked previously due to backend bug
- Added base configuration which is automatically clone for replacement, then subtracted for layers if scancode is unused (this allows for partial layer functionality)
- Added default map argument to define a combined map that is different than the typical ANSI map that is used for the base configuration
- Not yet complete
- Partial layers are not compiling properly (adding too many macros, wasting memory)
- Adding example hhkb layout to try out as a layer
- Basic ScanCode to USBCode mapping now generating for the kiibohd controller
- Small fix to template
- Backend
- Macro correlation and transformation
TODO
- More testing
- Analog support
- LED support
- Layer support (only supports default layer currently)
- Not quite complete.
- Most of the parser is done (excluding analog) for 0.3 of the KLL spec
- Transformation and correlation isn't complete yet.
- Backend generation for Kiibohd capabilties is complete.
