bench-forgejo/vendor/github.com/yuin/goldmark
zeripath 27757714d0 Change markdown rendering from blackfriday to goldmark (#9533)
* Move to goldmark

Markdown rendering moved from blackfriday to the goldmark.

Multiple subtle changes required to the goldmark extensions to keep
current rendering and defaults.

Can go further with goldmark linkify and have this work within markdown
rendering making the link processor unnecessary.

Need to think about how to go about allowing extensions - at present it
seems that these would be hard to do without recompilation.

* linter fixes

Co-authored-by: Lauris BH <lauris@nix.lv>
2019-12-31 03:53:28 +02:00
..
ast Change markdown rendering from blackfriday to goldmark (#9533) 2019-12-31 03:53:28 +02:00
extension Change markdown rendering from blackfriday to goldmark (#9533) 2019-12-31 03:53:28 +02:00
parser Change markdown rendering from blackfriday to goldmark (#9533) 2019-12-31 03:53:28 +02:00
renderer Change markdown rendering from blackfriday to goldmark (#9533) 2019-12-31 03:53:28 +02:00
text Change markdown rendering from blackfriday to goldmark (#9533) 2019-12-31 03:53:28 +02:00
util Change markdown rendering from blackfriday to goldmark (#9533) 2019-12-31 03:53:28 +02:00
.gitignore Change markdown rendering from blackfriday to goldmark (#9533) 2019-12-31 03:53:28 +02:00
go.mod Change markdown rendering from blackfriday to goldmark (#9533) 2019-12-31 03:53:28 +02:00
go.sum Change markdown rendering from blackfriday to goldmark (#9533) 2019-12-31 03:53:28 +02:00
LICENSE Change markdown rendering from blackfriday to goldmark (#9533) 2019-12-31 03:53:28 +02:00
Makefile Change markdown rendering from blackfriday to goldmark (#9533) 2019-12-31 03:53:28 +02:00
markdown.go Change markdown rendering from blackfriday to goldmark (#9533) 2019-12-31 03:53:28 +02:00
README.md Change markdown rendering from blackfriday to goldmark (#9533) 2019-12-31 03:53:28 +02:00

goldmark

http://godoc.org/github.com/yuin/goldmark https://github.com/yuin/goldmark/actions?query=workflow:test https://coveralls.io/github/yuin/goldmark https://goreportcard.com/report/github.com/yuin/goldmark

A Markdown parser written in Go. Easy to extend, standard compliant, well structured.

goldmark is compliant with CommonMark 0.29.

Motivation

I need a Markdown parser for Go that meets following conditions:

  • Easy to extend.
    • Markdown is poor in document expressions compared with other light markup languages like reStructuredText.
    • We have extensions to the Markdown syntax, e.g. PHP Markdown Extra, GitHub Flavored Markdown.
  • Standard compliant.
    • Markdown has many dialects.
    • GitHub Flavored Markdown is widely used and it is based on CommonMark aside from whether CommonMark is good specification or not.
      • CommonMark is too complicated and hard to implement.
  • Well structured.
    • AST based, and preserves source position of nodes.
  • Written in pure Go.

golang-commonmark may be a good choice, but it seems to be a copy of markdown-it.

blackfriday.v2 is a fast and widely used implementation, but it is not CommonMark compliant and cannot be extended from outside of the package since its AST uses structs instead of interfaces.

Furthermore, its behavior differs from other implementations in some cases, especially regarding lists: (Deep nested lists don't output correctly #329, List block cannot have a second line #244, etc).

This behavior sometimes causes problems. If you migrate your Markdown text to blackfriday-based wikis from GitHub, many lists will immediately be broken.

As mentioned above, CommonMark is too complicated and hard to implement, so Markdown parsers based on CommonMark barely exist.

Features

  • Standard compliant. goldmark gets full compliance with the latest CommonMark spec.
  • Extensible. Do you want to add a @username mention syntax to Markdown? You can easily do it in goldmark. You can add your AST nodes, parsers for block level elements, parsers for inline level elements, transformers for paragraphs, transformers for whole AST structure, and renderers.
  • Performance. goldmark performs pretty much equally to cmark, the CommonMark reference implementation written in C.
  • Robust. goldmark is tested with go-fuzz, a fuzz testing tool.
  • Builtin extensions. goldmark ships with common extensions like tables, strikethrough, task lists, and definition lists.
  • Depends only on standard libraries.

Installation

$ go get github.com/yuin/goldmark

Usage

Import packages:

import (
	"bytes"
	"github.com/yuin/goldmark"
)

Convert Markdown documents with the CommonMark compliant mode:

var buf bytes.Buffer
if err := goldmark.Convert(source, &buf); err != nil {
  panic(err)
}

With options

var buf bytes.Buffer
if err := goldmark.Convert(source, &buf, parser.WithContext(ctx)); err != nil {
  panic(err)
}
Functional option Type Description
parser.WithContext A parser.Context Context for the parsing phase.

Context options

Functional option Type Description
parser.WithIDs A parser.IDs IDs allows you to change logics that are related to element id(ex: Auto heading id generation).

Custom parser and renderer

import (
	"bytes"
	"github.com/yuin/goldmark"
	"github.com/yuin/goldmark/extension"
	"github.com/yuin/goldmark/parser"
	"github.com/yuin/goldmark/renderer/html"
)

md := goldmark.New(
          goldmark.WithExtensions(extension.GFM),
          goldmark.WithParserOptions(
              parser.WithAutoHeadingID(),
          ),
          goldmark.WithRendererOptions(
              html.WithHardWraps(),
              html.WithXHTML(),
          ),
      )
var buf bytes.Buffer
if err := md.Convert(source, &buf); err != nil {
    panic(err)
}

Parser and Renderer options

Parser options

Functional option Type Description
parser.WithBlockParsers A util.PrioritizedSlice whose elements are parser.BlockParser Parsers for parsing block level elements.
parser.WithInlineParsers A util.PrioritizedSlice whose elements are parser.InlineParser Parsers for parsing inline level elements.
parser.WithParagraphTransformers A util.PrioritizedSlice whose elements are parser.ParagraphTransformer Transformers for transforming paragraph nodes.
parser.WithASTTransformers A util.PrioritizedSlice whose elements are parser.ASTTransformer Transformers for transforming an AST.
parser.WithAutoHeadingID - Enables auto heading ids.
parser.WithAttribute - Enables custom attributes. Currently only headings supports attributes.

HTML Renderer options

Functional option Type Description
html.WithWriter html.Writer html.Writer for writing contents to an io.Writer.
html.WithHardWraps - Render new lines as <br>.
html.WithXHTML - Render as XHTML.
html.WithUnsafe - By default, goldmark does not render raw HTML and potentially dangerous links. With this option, goldmark renders these contents as written.

Built-in extensions

Attributes

parser.WithAttribute option allows you to define attributes on some elements.

Currently only headings support attributes.

Attributes are being discussed in the CommonMark forum. This syntax may possibly change in the future.

Headings

## heading ## {#id .className attrName=attrValue class="class1 class2"}

## heading {#id .className attrName=attrValue class="class1 class2"}
heading {#id .className attrName=attrValue}
============

Typographer extension

Typographer extension translates plain ASCII punctuation characters into typographic punctuation HTML entities.

Default substitutions are:

Punctuation Default entity
' &lsquo;, &rsquo;
" &ldquo;, &rdquo;
-- &ndash;
--- &mdash;
... &hellip;
<< &laquo;
>> &raquo;

You can overwrite the substitutions by extensions.WithTypographicSubstitutions.

markdown := goldmark.New(
	goldmark.WithExtensions(
		extension.NewTypographer(
			extension.WithTypographicSubstitutions(extension.TypographicSubstitutions{
				extension.LeftSingleQuote:  []byte("&sbquo;"),
				extension.RightSingleQuote: nil, // nil disables a substitution
			}),
		),
	),
)

Security

By default, goldmark does not render raw HTML and potentially dangerous URLs. If you need to gain more control over untrusted contents, it is recommended to use an HTML sanitizer such as bluemonday.

Benchmark

You can run this benchmark in the _benchmark directory.

against other golang libraries

blackfriday v2 seems to be fastest, but it is not CommonMark compliant, so the performance of blackfriday v2 cannot simply be compared with that of the other CommonMark compliant libraries.

Though goldmark builds clean extensible AST structure and get full compliance with CommonMark, it is reasonably fast and has lower memory consumption.

goos: darwin
goarch: amd64
BenchmarkMarkdown/Blackfriday-v2-12                  326           3465240 ns/op         3298861 B/op      20047 allocs/op
BenchmarkMarkdown/GoldMark-12                        303           3927494 ns/op         2574809 B/op      13853 allocs/op
BenchmarkMarkdown/CommonMark-12                      244           4900853 ns/op         2753851 B/op      20527 allocs/op
BenchmarkMarkdown/Lute-12                            130           9195245 ns/op         9175030 B/op     123534 allocs/op
BenchmarkMarkdown/GoMarkdown-12                        9         113541994 ns/op         2187472 B/op      22173 allocs/op

against cmark (CommonMark reference implementation written in C)

----------- cmark -----------
file: _data.md
iteration: 50
average: 0.0037760639 sec
go run ./goldmark_benchmark.go
------- goldmark -------
file: _data.md
iteration: 50
average: 0.0040964230 sec

As you can see, goldmark performs pretty much equally to cmark.

Extensions

goldmark internal(for extension developers)

Overview

goldmark's Markdown processing is outlined as a bellow diagram.

            <Markdown in []byte, parser.Context>
                           |
                           V
            +-------- parser.Parser ---------------------------
            | 1. Parse block elements into AST
            |   1. If a parsed block is a paragraph, apply 
            |      ast.ParagraphTransformer
            | 2. Traverse AST and parse blocks.
            |   1. Process delimiters(emphasis) at the end of
            |      block parsing
            | 3. Apply parser.ASTTransformers to AST
                           |
                           V
                      <ast.Node>
                           |
                           V
            +------- renderer.Renderer ------------------------
            | 1. Traverse AST and apply renderer.NodeRenderer
            |    corespond to the node type

                           |
                           V
                        <Output>

Parsing

Markdown documents are read through text.Reader interface.

AST nodes do not have concrete text. AST nodes have segment information of the documents. It is represented by text.Segment .

text.Segment has 3 attributes: Start, End, Padding .

TODO

See extension directory for examples of extensions.

Summary:

  1. Define AST Node as a struct in which ast.BaseBlock or ast.BaseInline is embedded.
  2. Write a parser that implements parser.BlockParser or parser.InlineParser.
  3. Write a renderer that implements renderer.NodeRenderer.
  4. Define your goldmark extension that implements goldmark.Extender.

Donation

BTC: 1NEDSyUmo4SMTDP83JJQSWi1MvQUGGNMZB

License

MIT

Author

Yusuke Inuzuka