mq Documentation

mq is a command-line tool that processes Markdown using a syntax similar to jq. It’s written in Rust, allowing you to easily slice, filter, map, and transform structured data.

Why mq?

mq makes working with Markdown files as easy as jq makes working with JSON. It’s especially useful for:

• LLM Workflows: Efficiently manipulate and process Markdown used in LLM prompts and outputs
• Documentation Management: Extract, transform, and organize content across multiple documentation files
• Content Analysis: Quickly extract specific sections or patterns from Markdown documents
• Batch Processing: Apply consistent transformations across multiple Markdown files

Features

• Slice and Filter: Extract specific parts of your Markdown documents with ease.
• Map and Transform: Apply transformations to your Markdown content.
• Command-line Interface: Simple and intuitive CLI for quick operations.
• Extensibility: Easily extendable with custom functions.
• Built-in support: Filter and transform content with many built-in functions and selectors.
• REPL Support: Interactive command-line REPL for testing and experimenting.
• IDE Support: VSCode Extension and Language Server Protocol (LSP) support for custom function development.

Getting Started

This section guides you through the installation of mq.

Install

To install mq, you can use cargo:

cargo install --git https://github.com/harehare/mq.git mq-cli
# Installing from cargo is under preparation.
cargo install mq-cli

Docker

$ docker run --rm ghcr.io/harehare/mq:0.1.0-preview

Visual Studio Code Extension

You can install the VSCode extension from the Visual Studio Marketplace.

Playground

An Online Playground is available, powered by WebAssembly.

Example

under preparation

Hello world

# Hello world
select(or(.[], .code, .h)) | upcase() | add(" Hello World")

Markdown TOC

.h
| let link = to_link(add("#", to_text(self)), to_text(self), "")
| if (is_h1()):
  to_md_list(link, 1)
elif (is_h2()):
  to_md_list(link, 2)
elif (is_h3()):
  to_md_list(link, 3)
elif (is_h4()):
  to_md_list(link, 4)
elif (is_h5()):
  to_md_list(link, 5)
else:
  None

Exclude code

select(not(.code))

Extract js code

.code("js")

Extract table

.[1][]

Extract list

.[1]

Extract MDX

select(is_mdx())

Custom function

def snake_to_camel(x):
  let words = split(x, "_")
  | foreach (word, words):
      let first_char = upcase(first(word))
      | let rest_str = downcase(slice(word, 1, len(word)))
      | add(first_char, rest_str);
  | join("");
| snake_to_camel()

Generate sitemap

def sitemap(item, base_url):
  let path = replace(to_text(item), ".md", ".html")
  | let loc = add(base_url, path)
  | s"<url>
  <loc>${loc}</loc>
  <priority>1.0</priority>
</url>";

Reference

This is a reference documentation for the mq.

CLI

The mq command-line interface provides tools for querying and manipulating markdown content. Below is the complete reference for all available commands and options.

Usage: mq [OPTIONS] [QUERY OR FILE] [FILES]... [COMMAND]

Commands:
  repl        Start a REPL session for interactive query execution
  fmt         Format mq or markdown files based on specified formatting options
  completion  Generate shell completion scripts for supported shells
  docs        Show functions documentation for the query
  help        Print this message or the help of the given subcommand(s)

Arguments:
  [QUERY OR FILE]  
  [FILES]...       

Options:
  -f, --from-file                       load filter from the file
  -R, --raw-input                       Reads each line as a string
  -n, --null-input                      Use empty string as the single input value
  -L, --directory <MODULE_DIRECTORIES>  Search modules from the directory
  -M, --module-names <MODULE_NAMES>     Load additional modules from specified files
      --args <NAME> <VALUE>             Sets string that can be referenced at runtime
      --rawfile <NAME> <FILE>           Sets file contents that can be referenced at runtime
      --mdx                             Enable MDX parsing
  -c, --compact-output                  pretty print
  -F, --output-format <OUTPUT_FORMAT>   Compact instead of pretty-printed output [default: markdown] [possible values: markdown, html, text]
  -U, --update                          Update the input markdown
      --unbuffered                      Unbuffered output
      --list-style <LIST_STYLE>         Set the list style for markdown output [default: dash] [possible values: dash, plus, star]
  -o, --output <FILE>                   Output to the specified file
  -v, --verbose...                      Increase logging verbosity
  -q, --quiet...                        Decrease logging verbosity
  -h, --help                            Print help
  -V, --version                         Print version

Examples:

To filter markdown nodes:
$ mq 'query' file.md

To read query from file:
$ mq -f 'file' file.md

To start a REPL session:
$ mq repl

To format mq file:
$ mq fmt --check file.mq

Types and Values

Values

• 42 (a number)
• "Hello, world!" (a string)
• array(1, 2, 3) (an array)
• true, false (a boolean)
• None

Types

Environment Variables

A module handling environment-specific functionality.

• __FILE__: Contains the path to the file currently being processed.

Conditionals and Comparisons

Conditional expressions and comparison operators in mq allow for decision-making based on the evaluation of conditions, enabling dynamic behavior in your queries.

Conditionals

mq supports standard conditional operations through the following functions:

• and(a, b) - Returns true if both a and b are true
• or(a, b) - Returns true if either a or b is true
• not(a) - Returns true if a is false

Examples

# Basic comparisons
and(true, true, true)
# => true
or(true, false, true)
# => true
not(false)
# => true

Comparisons

mq provides comparison functionality through built-in functions.

Basic Comparisons

Standard comparison operators are supported:

• eq(a, b) - Returns true if a equals b
• ne(a, b) - Returns true if a does not equal b
• gt(a, b) - Returns true if a is greater than b
• gte(a, b) - Returns true if a is greater than or equal to b
• lt(a, b) - Returns true if a is less than b
• lte(a, b) - Returns true if a is less than or equal to b

Examples

# Basic comparisons
eq(1, 1)
# => true
gt(2, 1)
# => true
lte("a", "b")
# => true

# String comparisons
eq("hello", "hello")
# => true
gt("xyz", "abc")
# => true

# Numeric comparisons
gte(5.5, 5.0)
# => true
lt(-1, 0)
# => true

# Logical operations
and(true, false)
# => false
or(true, false)
# => true
not(false)
# => true

# Complex conditions
and(gt(x, 0), lt(x, 10))
# =>  true if 0 < x < 10

Syntax

This section outlines the syntax rules in mq, providing a clear reference for writing valid code.

Pipe Operator

A functional operator that allows chaining multiple filter operations together.

Usage

The pipe operator (|) enables sequential processing of filters, where the output of one filter becomes the input of the next filter.

Examples

# Basic pipe usage
42 | add(1) | mul(2)
# => 86

# Multiple transformations
let mul2 = def mul2(x): mul(x, 2);
let gt4 = def gt4(x): gt(x, 4);
array(1, 2, 3) | map(mul2) | filter(gt4)
# => [6]

# Function composition
let double = def _double(x): mul(x, 2);
let add_one = def _add_one(x): add(x, 1);
5 | double(self) | add_one(self)
# => 11

? Operator

The ? operator is a safe navigation operator that provides null-safe operations.

Usage

When applied to a None value, the ? operator prevents errors by returning None instead of raising an exception.

Examples

# Safe access with ? operator
let x = None | x | add?(1)
# => None

# Chaining with ? operator
None | add?(1) | mul?(2)
# => None

# Normal operation when value exists
42 | add?(1)
# => 43

Environment variables

Environment variables can be referenced using $XXX syntax, where XXX represents the name of the environment variable. For example:

• $PATH - References the PATH environment variable
• $HOME - References the HOME environment variable
• $USER - References the current user’s username

This syntax is commonly used in shell scripts and configuration files to access system-level environment variables.

Def Expression

The def expression defines reusable functions with parameters:

Examples

# Function that doubles input
def double(x):
  mul(x, 2);

# Function with conditional logic
def is_positive(x):
  gt(x, 0);

# Composition of functions
def add_then_double(x, y):
  add(x, y) | double(self);

Let Expression

The let expression binds a value to an identifier for later use:

# Binds 42 to x
let x = 42
# Uses x in an expression
let y = add(x, 1)
# Binds `add` function to z
let z = def _add(x): add(x, 1); | z(1)

If Expression

The if expression evaluates a condition and executes code based on the result:

 if (eq(x, 1)):
   "one"
 elif (eq(x, 2)):
   "two"
 else:
   "other"

The if expression can be nested and chained with elif and else clauses. The conditions must evaluate to boolean values.

While Expression

The while loop repeatedly executes code while a condition is true:

let i = 0 |
while (lt(i, 3)):
  let i = add(i, 1) | i;
# => [1, 2, 3]

The while loop in this context returns an array containing all elements processed during the iteration. As the loop executes, it collects each processed value into an array, which is then returned as the final result once the loop condition becomes false.

Key points:

• Creates a new array from loop iterations
• Each loop cycle’s result is added to the array
• Returns the complete array after all iterations
• Similar to map/collect functionality but with while loop control

Until Expression

The until loop repeatedly executes code until a condition becomes true:

let x = 5 |
until(gt(x, 0)):
  let x = sub(x, 1) | x;
# => 0

Until loops are similar to while loops but continue until the condition becomes true instead of while the condition remains true.

Foreach Expression

The foreach loop iterates over elements in an array:

let items = array(1, 2, 3) |
foreach (x, items):
   sub(x, 1);
# => array(0, 1, 2)

Foreach loops are useful for:

• Processing arrays element by element
• Mapping operations across collections
• Filtering and transforming data

Comments

Similar to jq, comments starting with # are doc-comments.

# doc-comment
let value = add(2, 3);

Include

Loads functions from an external file using the syntax include "module_name". The include directive searches for .mq files in the following locations:

• $HOME/.mq - User’s home directory mq folder
• $ORIGIN/../lib/mq - Library directory relative to the source file
• $ORIGIN/../lib - Parent lib directory relative to the source file

include "module_name"

Examples

# Include math functions from math.mq
include "math"
# Now we can use functions defined in math.mq
let result = add(2, 3)

Self

The current value being processed can be referenced as self. When there are insufficient arguments provided in a method call, the current value (self) is automatically passed as the first argument.

Examples

# These expressions are equivalent
"hello" | upcase()
"hello" | upcase(self)

String Interpolation

String Interpolation allow embedding expressions directly inside string literals. In mq, an interpolated string is prefixed with s" and variables can be embedded using ${} syntax.

Syntax

s"text ${ident} more text"

Examples

let name = "Alice"
| let age = 30
| s"Hello, my name is ${name} and I am ${age} years old."
# => Output: "Hello, my name is Alice and I am 30 years old."

Builtin selectors and functions

This page provides an introduction to the built-in selectors and functions available in mq. These are predefined components that you can use in your queries without having to define them yourself.

Builtin functions

Builtin selectors