Abstract – This is a marky quickstart document for illustrating marky markup using simple examples. The marky source code of this document can be read here. For more information please refer to the marky repository, marky documentation or the simple marky example.

1 Introduction

This is a marky quickstart document for illustrating marky markup using simple examples. This document is the rendered version of the source code presented here. The marky markup is compatible with standard Markdown and can be read as-is. This document represents the output of marky after processing python code, which is embedded into the document itself. In order to understand the examples and see the complete marky syntax, the source code of this file can be read here. The complete documentation of marky is available here

2 Markdown

marky is a Markdown preprocessor allowing to transform Markdown text using python. The preprocessed Markdown text is rendered to pdf and html (other formats using pandoc). pandoc has a powerful set of Markdown extensions supporting structured writing as well as bibliography, figure referencing, table referencing, tex-style equations with referencing etc. (refer to Scientific Writing in Markdown, marky Documentation).

The rendering of Markdown text into html and pdf consists of three steps which are illustrated using the marky documentation md/marky.md.

1. Preprocessing
   (process marky markup, run code, generate content)
   • Input: marky Markdown text: md/marky.md
   • Output: pandoc Markdown text: build/marky.md
2. Linking
   (apply format specific code for html and pdf)
   • Input: pandoc Markdown text: build/marky.md
   • Output
   1. pandoc Markdown text for html: build/marky.html.md
   2. pandoc Markdown text for pdf: build/marky.pdf.md
3. Rendering
   (render html and pdf document using pandoc)
   • Input
   1. pandoc Markdown text for html: build/marky.html.md
   2. pandoc Markdown text for pdf: build/marky.pdf.md
   • Output
   1. html document: html/marky.html
   2. pdf document: pdf/marky.pdf

The whole process is ecapsulated into a python script and a Makefile. Rendering documents using marky requires to write Markdown text and run make all.

3 Automated Reporting

Markdown text with embedded code snippets is a powerful paradigm for automated technical and scientific reporting and possibly other documents. On one hand data can be organized according to the document structure using algorithms embedded in the report itself, and on the other hand the data can be inserted in the report directly from variables. This elliminates the need for manual copying of data into the text and allows to update or reproduce the report automatically for the same and other data. Using the simple marky syntax the user can concentrate on documentation writing from the Markdown perspective and assist the creation of document content using python code snippets.

pandoc filters (refer to Related Work, marky Documentation) allow transforming the document while rendering it. pandoc filters operate on an internal abstract syntax tree (AST) representation, therefore the user must express dynamically created document content as nodes in the format of the AST. marky takes a slight different approach and operates on the Markdown text itself, before it is parsed and rendered.

Python code is embedded into the document with a simple markup syntax similar to Rmarkdown using code blocks and inline expressions. marky parses the code, executes it and writes the results back into the Markdown text. The code can produce output using algorithms or output formatted string variables. marky also allows for the insertion of format dependent raw code in html and tex (for pdf documents).

1. code snippets are embedded into the document text
2. code snippets are executed during preprocessing
3. code produces text for the Markdown document
4. code variables are displayed in the document text
5. format dependent code is applied for html and pdf

4 Download and Run marky

marky is Markdown preprocessor allowing to execute embedded python code in Markdown documents. After preprocessing, a regular Markdown file is present, which is rendered into html and pdf using pandoc. marky handles all this steps using a Makefile. marky is a single-file script which depends on python (>=3.6), pandoc (>=2.11), pyyaml and pandoc-xnos.

Installing Dependencies

pandoc binaries for Debian-based Linux are released here. pyyaml is installed using the linux package manager or pip and pandoc-xnos consists of the components fignos, secnos, eqnos and tablenos which are installed using pip. Depending on the linux installation maybe pip3 has to be used.

pip install pyyaml
pip install pandoc-fignos
pip install pandoc-secnos
pip install pandoc-eqnos
pip install pandoc-tablenos

Download marky Script

marky is downloaded using the following commands.

cd $HOME
git clone https://github.com/lehmann7/marky.git
cd marky

Alternatively, marky can be obtained diretly without git:

cd $HOME
mkdir marky
cd marky
wget https://raw.githubusercontent.com/lehmann7/marky/main/marky.py
chmod +x marky.py

Initialize marky Environment

The marky environment consists of the Makefile and the documentation. The marky Makefile, documentation and quickstart are unpacked from the marky.py script file into the current working directory. The marky environment is initialized using the following commands.

cd $HOME
cd marky
./marky.py --init
WRITE ./md/marky.md
WRITE ./md/marky.mdi
WRITE ./md/marky-src.md
WRITE ./md/quickstart.md
WRITE ./md/quick-src.md
WRITE ./md/example.md
WRITE ./md/example-src.md
WRITE ./data/marky.bib
USAGE
1. `make help`
2. `make all-html httpd`
3. `make all-pdf`

During initialization marky creates two directories md/ and data/. md/ is the directory which contains the Markdown text to be rendered into html and pdf. data/ is the resource directory which contains bibliography, images, videos and other assets.

Render Documentation and Examples

If all dependencies have been installed accordingly and the marky environment is initialized, marky can be used to render a local copy of the documentation, the quickstart and the example.

The following commands render the Markdown text of the documentation.

cd $HOME
cd marky
make all-pdf
make all-html

During make a new directory build/ is created, which contains temporary files (preprocessed Markdown text, linked text for html and pdf). The resulting html and pdf documents are placed inside html/ and pdf/. For rendering pdf a tex environment like texlive needs to be installed. For rendering the html documents, pandoc requires internet access, because java scripts and style sheets are fetched from content delivery networks.

marky Makefile

The marky Makefile coordinates the three steps of the marky document processing pipeline: preprocessing, linking and rendering. The marky Makefile supports several targets for displaying help or rendering all, multiple or specific documents.

Makefile Targets

1. make help: display help message on the console
2. make cheat: display the marky markup Cheat Sheet
3. make scan: scan for new documents md/*.md and update Makefile
4. make all: render all documents md/*.md into html and pdf
5. make all-pdf: render all documents md/*.md into pdf
6. make all-html: render all documents md/*.md into html
7. make httpd: start python webserver in html/
8. make clean: remove all files: build/*, pdf/*, html/*

Make Single Document

When running make all, marky renders all documents, which can be undesirable if only one particular document shall be rendered. By make scan, marky scans the directory md/*.md for new Markdown documents to be processed. For each document, which has been found, marky sets up alias targets in order to debug the preprocessing, linking and rendering of this document.

Assuming the document md/marky.md shall be rendered step by step, marky introduces the following targets.

1. Preprocessing: make md-marky
   
   • md/marky.md -> build/marky.md
2. Linking html: make lhtml-marky
   
   • build/marky.md -> build/marky.html.md
3. Linking pdf: make lpdf-marky
   
   • build/marky.md -> build/marky.pdf.md
4. Rendering html: make html-marky
   • build/marky.html.md -> html/marky.html
5. Rendering pdf: make pdf-marky
   • build/marky.pdf.md -> pdf/marky.pdf

5 Write A New Document

In order to render a new document the Markdown text needs to be saved to a file located in md/example.md which can be found rendered here. The following Markdown snippet can be used as a starting point.

    ---
    title: "`marky` Example"
    date: Date
    author: Name
    link-citations: true
    bibliography: data/marky.bib
    header-includes--pdf: >
       \hypersetup{colorlinks=false,
       allbordercolors={0 0 0},
       pdfborderstyle={/S/U/W 1}}
    header-includes--html: >
       <style>* { box-sizing: border-box; }</style>
    xnos-cleveref: true
    xnos-capitalise: true
    fontsize: 11pt
    
    ---
    
    > **Abstract** -- This is a `marky` example document for
    > illustrating `marky` markup. The `marky` source code of this
    > document can be read [here](example-src.html).
    > For more information please refer to the
    > [`marky` repository](https://github.com/lehmann7/marky),
    > [`marky` documentation](marky.html) or the
    > [`marky` quickstart](quickstart.html).
    
    ---
    
    # Referenced Section {#sec:label}
    
    This is a reference to @sec:label.
    
    ![This is the caption](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg==){#fig:label}
    
    This is a reference to @fig:label.
    
    A  |B  |C  |D
    ---|---|---|---
    000|111|444|555
    222|333|666|777
    
    Table: This is the caption {#tbl:label}
    
    This is a reference to @tbl:label.
    
    $$\mbox{e}^{\mbox{i}\pi}+1=0$${#eq:label}
    
    This is a reference to @eq:label.
    
    This is a citation [@Muller1993].
    
    # `marky` Markup for Execution of Embedded Python Code
    
    **Displayed Code, Executed**
    
    ```!
        import math
        def list_and(l):
            return ", ".join(str(i) for i in l[:-1]) + " and " + str(l[-1])
    
        x = 2
        y = math.sqrt(x)
    ```
    
    **Hidden Code, Executed**
    
    ```!!
        print("Hello Console!")
    ```
    
    **Displayed Code, Not Executed**
    
    ```python
        x = 3
    ```
    
    **Inline Formatted Output**
    
    The square root of $x=`!x`$ is `!y:.3f`.
    
    **Inline Expression**
    
    The first ten numbers are `!list_and(range(10))`.
    
    **Format Links**
    
    ```md
    [Link to document](file.???)
    ```
    
    will be proprocessed into the following text:
    * for `html`: `[Link to document](file.html)`
    * for `pdf`: `[Link to document](file.pdf)`
    
    [Link to this document](example.html)
    
    **Format Codes**
    
    ```!
        def html_FMTCODE(): return "H<sup>T</sup><sub>M</sub>L"
        def pdf_FMTCODE(): return "\LaTeX"
    ```
    
    This is a `?_FMTCODE()` document.
    
    ---
    
    *Thanks for reading, please try `marky`.*
    
    ---
    
    # References {-}
    

6 Code Blocks

Code blocks are embedded in Markdown using fenced code using either the ! or !! flag for displayed and hidden code respectively.

    ```!
    CODE_BLOCK_SHOWN
    ```

    ```!!
    CODE_BLOCK_HIDDEN
    ```

Display and Execute Code

This code block annotated with ! is displayed and executed.

    import math
    x = math.sqrt(2)

Execute Code without Display

The code block annotated with ! is not displayed, but executed.

Display Code but do not Execute

This code block is displayed as python, but not executed.

    z = 0./0.

Using the Python import Statement

Large code blocks can be imported from python modules and the import statement can be used for loading installed libraries.

    import numpy
    import sys
    sys.path.append(".")
    #import module_in_working_directory

7 Inline Code

Inline statements are directly embedded into the text flow using expressions and variables with the corresponding syntax `!EXPRESSION` or `!VARIABLE`. The output of variables can be formatted using the `!VARIABLE[:FORMAT]` or `!EXPRESSION[:FORMAT]` statement according to the python operator {<variable>[:<format>]} implemented in the str.format() specification and the operator {<expression>[:<format>]} implemented in f-strings.

Inline Formatted Output

The result of $\sqrt{2}$ is:

• unformatted output: 1.4142135623730951
• formatted output: 1.4142

The variable x is

• unformatted output: 1.4142135623730951
• formatted output: 1.4142

The variable y = x + 1 is

• unformatted output: 2.414213562373095
• formatted output: 2.4142

(The code block for the definition of y is hidden.)

Inline Expression Output

    x = list(range(1, 11))
    y = [i*i for i in x]

    def list_str(a):
        return [str(i) for i in a]
    def list_and(a):
        return ", ".join(list_str(a[:-1])) + " and " + str(a[-1])

The list can be inserted into the text. The square of the first 10 numbers 1, 2, 3, 4, 5, 6, 7, 8, 9 and 10 is 1, 4, 9, 16, 25, 36, 49, 64, 81 and 100. Square numbers are computed according to y = x².

Inline Statements in Tables

    class square:
        def __init__(self):
            self.x = 0
        def get_x(self):
            return self.x
        def next_y(self):
            y = self.x**2
            self.x += 1
            return y
    sq = square()

The following table is computed according to y = x².

8 Generate Markdown Text

Markdown text can be produced algorithmically from a python algorithm using the _() and __() function. The _() and __() function are special names which are reserved by marky. (refer to Generation of Markdown Text, marky documentation for in-depth explanation).

8.1 The _() Function

The _() function basicly resembled the python print() function. marky does not patch the standard print() function which still displays text in the console and not in the Markdown text. The _() function supports appending text to the previous and the next line of output by using _ as the fist or last parameter.

1. _(_, *args ): append this output to previous output
2. _(_, *args, _): append to previous and next output
3. _( *args, _): append next output to this output
4. _(): disable append flag

Monkey Patch print() Function

The print() function can be monkey patched using the following statment in order to call the _() function instead.

    print("Print", "to",  "console", "!")
    _("Print", "to",  "Markdown", "!")

    print = _ # monkey patch
    print("Print", "to",  "Markdown", "!")

Print to Markdown ! Print to Markdown !

Join Arguments using sep

The signature of the _() is _(*args, sep=" "). sep is used to join the arguments *args into one string.

    _("The first five natural numbers are:")
    _(1, 2, 3, 4, 5, sep=", ")

The first five natural numbers are: 1, 2, 3, 4, 5

Generate a Table with Appending

The append feature is used to create a table.

    _("Column 1", _)
    _("|Column 2", _)
    _("|Column 3", _)
    _()

    _("|".join(["--------"]*3))

    for i in range(5):
        _("% 8d" % (i*3))
        for j in [1, 2]:
            _(_, "|% 8d" % (i*3+j))

Column 1|Column 2|Column 3

——–|——–|——– 0| 1| 2 3| 4| 5 6| 7| 8 9| 10| 11 12| 13| 14

The algorithm produces the following Markdown text.

Column 1|Column 2|Column 3
--------|--------|--------
       0|       1|       2
       3|       4|       5
       6|       7|       8
       9|      10|      11
      12|      13|      14

8.2 The __() Function

Compared to the _() function, the __() function only takes one argument and its purpose is to output a formatted paragraph with indentation. The signature of the __() function is _(arg, crop=True).

Generate a Paragraph with f-Strings

The __() function can be combined with triple quoted block strings and the python 3 f-strings or f"..." string interpolation, refer to f-strings.

    import random
    s = 0
    random.seed(s)
    p = [random.random() for i in range(3)]

    __(f"""
        Parameter one is {p[0]:.3f} and the value depends on the seed
        of the pseudo random number generator, which was chosen
        to be {s}. For the same seed always the same random numbers
        are created. The next two numbers are {p[1]:.3f} and {p[2]:.3f}.
        The sum of the three numbers is {sum(p):.3f} and it is
        {'greater' if sum(p) > 2. else 'lesser or equal'} than two.
    """)

Parameter one is 0.844 and the value depends on the seed of the pseudo random number generator, which was chosen to be 0. For the same seed always the same random numbers are created. The next two numbers are 0.758 and 0.421. The sum of the three numbers is 2.023 and it is greater than two.

Cropping and Indentation of Output

Before the text generated by the __() function is printed into the document. The text is cropped according to the leading white space of the first non-empty line. The leading white space of the first non-empty line is removed from all other lines of the output.

    __("""
        * List Level 1
            * List Level 2
            * List Level 2
                * List Level 3
            * List Level 2
        * List Level 1
        * List Level 1
    """)

• List Level 1
  • List Level 2
  • List Level 2
    • List Level 3
  • List Level 2
• List Level 1
• List Level 1

The code block produces the following output.

* List Level 1
    * List Level 2
    * List Level 2
        * List Level 3
    * List Level 2
* List Level 1
* List Level 1

Disable Cropping of Output

The cropping is disabled using the keyword __(text, crop=False).

    __("""
        * List Level 1
    """, crop=True)

    __("""
        * List Level 2
    """, crop=False)

• List Level 1

  • List Level 2

The code block produces the following output.

* List Level 1
        * List Level 2

9 Format Dependent Links

When writing several Markdown documents often documents are linked between each other using the Markdown link statement [Link Name](file.html) or [Link Name](file.pdf). However, when rendering documents with links into html and pdf the file extension often must be adjusted according to the output format. marky supports the .??? statement, which will be replaced by .html or .pdf depending on the output format.

[Link to document](file.???)

will be proprocessed into the following text:

• for html: [Link to document](file.html)
• for pdf: [Link to document](file.pdf)

Link to this document

10 Format Dependent Code

pandoc Markdown allows to write format specific code within Markdown using html and tex for pdf documents. However, when inserting raw html or raw tex code, the document only can be rendered into html or pdf accordingly. This is only a short summary, for an in-depth explanation of all features please refer to Format Codes, marky documentation.

marky introduces format codes, which are applied during linking after preprocessing. During linking format specific codes for html and pdf are applied in a consistent manner, resulting in documents with Markdown and html or Markdown and tex only. Using this pattern marky documents contain regular Markdown, which can be rendered into html and pdf, as well as format specific codes for tweaking or polishing html and pdf output.

Assuming preprocessing the file md/marky.md, linking format codes results in the two following output files.

1. build/marky.html.md: contains output of html format codes.
2. build/marky.pdf.md: contains output of pdf format codes.

Example 1: Multi-Column Text in pdf and html

Defnition of two format codes mcol_begin and mcol_end, one for the begin of multi column and another for the end of the multi column section. The format codes are appended with _html and _pdf respectively.

    mcol_begin = fmtcode(
        pdf=r"\begin{multicols}{2}",
        html="<div style='column-count: 2;'>"
    )
    mcol_end = fmtcode(
        pdf=r"\end{multicols}",
        html="</div>"
    )

The column-count CSS property requires Internet Explorer>=10, Firefox>=52, Safari>=9, Opera>=37 or Chrome>=50, refer to w3schools. In order to use the multicol tex package, the statement \usepackage{multicol} has to be included in the yaml meta data in the front matter of the Markdown document.

11 Include Statement

marky allows to include other Markdown text using the !!! statement. Please refer to the marky documentation for complete description of the !!! statement. During rendering marky keeps track of included files and creates Makefile rules for dependent make.

    !!! file.mdi

12 Meta Data

marky supports document meta data in Markdown front matter. This feature is not explained in the quickstart. Please refer to the marky documentation for explanation.

    ---
    META_DATA
    ---
    MARKDOWN

13 Escape Markup

The marky markup can be escaped. When markup is escaped marky removes the escape sequence and prints out the unescaped statement.

Thanks for reading, please try marky.