cltpt

note that this website was generated using cltpt from org-mode files. so this page/webapp can be a testament to whether org->html conversion works properly (broken text elements is an indicator of parser/conversion failure).

introduction

https://github.com/mahmoodsh36/cltpt is both a tool and a library that is meant to serve as a base for the upcoming organ-mode package for the lem text editor. it is very much a WIP, it is unstable and breaking changes may be introduced without warnings.
the goal of cltpt is to maintain an editor-independent codebase and toolset (the existence of countless parsers for org-mode should be enough a motive for this). the core will be written in common lisp but will be independent of lem's libraries, but ofcourse this is supposed to (mainly) be a package for lem so some parts are bound to be lem-dependent. although lem itself can also be used as a library in common lisp, i think that this is still a good idea.

installation

nixos

if you are on nixos you may use the provided flake. running the commandline tool is as simple as something like:

quicklisp

since the dependencies are all listed in cltpt.asd, cloning the code and then running (ql:quickload :cltpt) in the code directory should be enough to load the library.
to make this perist you may clone the repo into ~/.quicklisp/local-projects/. then you may overwrite main.lisp (which uses asdf directly without quicklisp) to make it work with quicklisp:
then you may use the commandline like: run.sh -h. (in the following sections you may replace cltpt with run.sh and the commands will work.)

example commandline usage

this section will highlight what i think would be the most common usage for the commandline tool.

getting help

to get help you can use:
or help for subcommands:

converting from org-mode to html

convert a single file from org to html (the % part is for "formatting" with lisp code):
convert a bunch of files from org to html:
notice that you can stack multiple instances of the -r (or --rule) argument. this is also possible with the -i (--input) argument (which is used for single files):
this would generate two files, /tmp/file1.html and /tmp/file2.html.

converting from org-mode to latex

notice that this is very similar to the previous commands, with slight differences in the arguments passed to the tool, such as the name of the destination format (the format we want to convert our files to).
convert a single file from org to latex (write it to /tmp/test.tex):
convert a bunch of files from org to latex:

"publish" functionality (webapp generation)

this is different from the simpler convert functionality in that it generates a directory that contains the static files required for a webapp, including css and js files that may belong to a specific theme which the user might choose.
then in /tmp/cltpt-website you may run python -m http.server 8080 -b 0.0.0.0 and visit localhost:8080.

"roam" queries

the library provides an "org-roam-like" interface that works with different methods of identifying different types of objects in your text files. by default it recognizes org-id "identifiers" and links (the ones org-roam use for headers/files). denote-like identifiers and (more importantly) blk-like identifiers which includes the previous ones.
the following command queries a directory of org-mode files and prints the title, id and filepath for each one in the specified format (-o argument).

"agenda" queries

the following command prints an agenda "tree" after retrieving all TODOs from the org files found in the specified directory. the default range displayed is 7 days from now.
these commands will be run in the root directory of the source code of cltpt which contains a few org files for testing/demonstration purposes.
for reference, the above command could be replaced with just the following since test.org is the only file that has agenda entries:
to print an agenda tree for a different range of timestamps, we use:
more options can be found by running
a more "advanced" command might look something like:

roadmap

notice that this roadmap is different from the roadmap for organ-mode.

todos

an X may not mean that the feature is completely implemented but that it is functional for the most part.

• [-] org-header
  • [X] priorities
  • [X] todo state
  • [X] tags
  • [X] properties
  • [X] timestamps, scheduling and deadlines
  • [ ] state history
  • [ ] completion status (e.g. completion percentage of children with tasks etc)
• [ ] org-list
  • [ ] checkboxes
• [-] agenda
  • [X] repeated tasks
  • [X] tags
  • [ ] custom views
  • [ ] task hierarchy in agenda tree
  • [ ] state history tracking
• [ ] markdown
  • [ ] support agenda for markdown
  • [ ] support roam for markdown
• [X] inline lisp execution
• [X] commandline
  • [X] conversion
  • [X] roam
  • [X] agenda
  • [X] advanced conversion with prewritten webapp templates
• [-] conversion (ideas from org-export)
  • [X] org to html
  • [X] org to latex
  • [ ] org to markdown
  • [ ] markdown to org
• [ ] latex
  • [ ] recognize latex links (~\ref~)
  • [ ] recognize latex labels (~\label~)
• [ ] babel
  • [ ] code tangling
  • [ ] code detangling
  • [ ] sessions
  • [ ] data pipelines
  • [ ] library of babel
  • [ ] noweb
• [X] roam (idea from org-roam)
  • [X] node links
    • [X] links to files
    • [X] links to headers
    • [X] links to blocks
• [ ] org-clock
• [X] latex previews for html conversion
• [X] org-attach
• [X] transclusions

org-element support

its not called org-element but a text-object in the source code.

org-element	parsing	highlighting	conversion to html	conversion to latex
list	t		t	t
table	t		t	t
header	t	t	t
link	t		t	t
timestamp	t	t
src-block	t		t	t
export-block	t		t	t
block	t	t	t	t
prop-drawer	t
drawer	t		t	t
latex-env	t		t	t
keyword	t
display-math	t		t	t
inline-math	t		t	t
italic	t		t	t
emph	t		t	t
inline-code	t		t	t
comment	t		t	t
comment-block	t		t	t
web-link	t		t
citation
underline	t		t	t
subscript
superscript
strike-through	t		t	t
footnote
dynamic-block
inline-task
paragraph

babel functionality (code execution)

this section is yet to be written, currently only support for running python code is implemented.

how markup is handled

yet to be written. this section will include:

• what it means for some code to be format-agnostic.
• how escape sequences are handled.

a lisp-based markup

after having used org-mode for years, i was fed up with its syntax. many people tend to say that org has a saner markup syntax than markdown, but i personally never saw the value in the syntax itself but in the rest of the features that org-mode provides.
in my mind the syntax i wanted had to be customizable, extensible, and similar to lisp code so that it is easy to parse. even better, i wanted the syntax to just be lisp code embedded within arbitrary text, so that lisp code is a second-class citizen, unlike in lisp code files.
after a bit of thinking i put something together that i think makes some sense. this section will highlight some example usage of this markup "language".

text macros and blocks

in my mind, a "block" of text is a portion of text that is specified by a opening and closing "tags". for example, the opening tag for a block in org-mode is #+begin_<type>. with this in mind, we consider the following org-mode text block:
with the lisp-based markup cltpt provides, the alternative would simply be:
both the opening and the closing tags are lisp expressions (which we call "text macros") that are evaluated to construct instances of text-object. block-end is a function that returns a value that the parser recognizes as the signal to close the region that was opened by make-block.
if a text macro isnt closed by another, it is taken to be its own text object without a "contained" region. for example, in the following text the macro is simply replaced with the evaluation result during conversion:

html templates

the lisp markup can serve as a templating engine for html files. we can simply write macros that will return read a file and return it as a text-object that simply stores the text that it read from the file, then during conversion, that text will be substituted inplace of the original macro. for example, we can use the following html file which reads another html file:
although this wouldnt work as expected because by default, the contents of the macro are escaped during conversion, so the read contents will not be interpreted as raw contents but as text to escape (for example < gets replaced with < during html conversion). this is easy to overcome by writing a function that returns a text-object that overrides the conversion functionality to prevent escaping from happening, and using the function for the template:

lexer text macros and post-lexer text macros

to be written. good to atleast note for now that % is used for post-lexer macros while # is used for lexer macros. i may consider unifying them in the future.

using the api

iterating through files

the repo https://github.com/mahmoodsh36/template could serve as a good example for how to work with files as it uses the api to generate this webapp.
parsing a file and using its tree can be done like:
iterating through files and their text-object trees can be done like:

iterating through and converting text objects

to iterate through all files, extract latex snippets and convert them into an html file:

working with the code

this will highlight the main concepts and interfaces in the codebase. note that much of the code will likely change in the (near) future, but the main interfaces discussed here will probably remain the same.

the core interface

the core interface cltpt/base contains a few main CLOS interfaces:

• text-object: a type of text-object mainly holds a "combinator rule" that is passed to the combinator later on. this rule is a sexp that is later interpreted as an expression combining different parsing methods and is handled by the combinator during the "lexing" phase.
• text-format.
• document: a special type of text-object that originated from a file and may hold extra metadata to allow us to trace it back to the original file.

and some generic functions of which the most important ones may include:

• convert-tree: convert a text-object from one text-format to another.
• parse-file: parse a file with a given text-format.
• text-object-init: a function that is invoked by the parser after the lexing process has finished.
• text-object-convert: a function that returns a plist telling the conversion process how to proceed for the given text-object instance.

text-object interface

each text object has a "rule" that is passed to the parser. rules are simply trees composed of combinator functions. any class that inherits from text-object needs to have such a rule slot.
a few methods are defined for the text-object interface:

• text-object-init: this is called immediately after parsing and recognizing the text object.
• text-object-finalize: this is called once the final text-object tree has been constructed, each object in the tree is finalized starting from the root (an instance of document).
• text-object-convert: this is called during conversion for each object in the tree, starting from the root. the method returns a plist that determines the behavior of the conversion, this plist is handled by the function convert-tree.

the parser

the lower-level parser (essentially the lexer) is based on a "parser combinator" that tries to simplify and abstract away the work of parsing different types of text objects. before creating instances of text-object's the results of parsing are instances of 'match', which is a tree-based data structure (just like text-object), except that its simpler and holds less properties and functionality.
the basis of both text-object and match is a buffer. a buffer is a tree-based structure (buffers nested within buffers) in which each node essentially pinpoints a region of text. the buffer data structure knows how to adapt to incremental/regional changes in the text and is used extensively in the conversion pipeline.
matches are used to construct text-objects, a match may or may not end up being a text-object depending on whether its id property corresponds to a text-object class.
some related functions are:

• handle-match is a function that takes a match tree and turns it into a text-object tree.
• cltpt/base:parse calls cltpt/combinator:parse, then on each match returned by the combinator it calls handle-match. then it proceeds to finalize the text-object tree.

the conversion pipeline

the conversion functionality starts at convert-tree (convert-document for documents or files) being applied to an instance of a document (which itself is a text-object), and recurses down the text-object tree that is rooted at the document object.
at each step in the recursion a text-object may choose to halt the recursion process and handle the conversion of itself and its children on its own, making this process completely customizable while maintaining an intuitive default behavior that allows for each text-object node in the tree to choose how it should handle the conversion of the text that it spans in the original document.
in convert-tree we call text-object-convert on the object we are given, this function returns metadata telling the conversion pipeline how to proceed. a simple instance of this would be the conversion function for the org-emph type:
the function receives the instance of the destination text-format for conversion, and the text-object instance. the function rewrap-within-tags is there to simplify the conversion/modification of the tags surrounding a text-object's text during conversion. in this case the tags to be changed are forward slashes. for example for html conversion we would be converting /my text/ to <b>my text</b>.
more involved conversion functions may apply their own recursion to apply custom modification to their descendants in the tree and may even choose which children to discard or convert.