Fixes #26548
This PR refactors the rendering of markup links. The old code uses
`strings.Replace` to change some urls while the new code uses more
context to decide which link should be generated.
The added tests should ensure the same output for the old and new
behaviour (besides the bug).
We may need to refactor the rendering a bit more to make it clear how
the different helper methods render the input string. There are lots of
options (resolve links / images / mentions / git hashes / emojis / ...)
but you don't really know what helper uses which options. For example,
we currently support images in the user description which should not be
allowed I think:
<details>
<summary>Profile</summary>
https://try.gitea.io/KN4CK3R
</details>
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
In #26365 issue references were disabled entirely for documents,
intending to match GitHub behavior. However cross-references do appear
to work in documents on GitHub.
This is useful for example to write release notes in a markdown document
and reference issues. While the simpler syntax may create links when not
intended, hopefully the cross-reference syntax is unique enough to avoid
it.
Fix #28526, regression of
* #26365
(although the author of #26365 has recent activities, but there is no
response for the regression, so I proposed this quick fix and keep the
fix simple to make it easier to backport to 1.21)
- Currently the repository description uses the same sanitizer as a
normal markdown document. This means that element such as heading and
images are allowed and can be abused.
- Create a minimal restricted sanitizer for the repository description,
which only allows what the postprocessor currently allows, which are
links and emojis.
- Added unit testing.
- Resolves https://codeberg.org/forgejo/forgejo/issues/1202
- Resolves https://codeberg.org/Codeberg/Community/issues/1122
(cherry picked from commit 631c87cc2347f0036a75dcd21e24429bbca28207)
Co-authored-by: Gusted <postmaster@gusted.xyz>
- In org mode you can specify an description for media via the following
syntax `[[description][media link]]`. The description is then used as
title or alt.
- This patch fixes the rendering of the description by seperating the
description and non-description cases and using `org.String()`.
- Added unit tests.
- Inspired by
6eb20dbda9/org/html_writer.go (L406-L427)
- Resolves https://codeberg.org/Codeberg/Community/issues/848
(cherry picked from commit 8b8aab83113b34bade61964e2097ed497abc39e9)
Co-authored-by: Gusted <postmaster@gusted.xyz>
- Currently the post processing will transform all issue indexes (such as `#6`) into a clickable link.
- This makes sense in an situation like issues or PRs,
where referencing to other issues is quite common
and only referencing their issue index is an handy and efficient way to do it.
- Currently this is also run for documents
(which is the user profile and viewing rendered files),
but in those situations it's less common to reference issues by their index and instead could mean something else.
- This patch disables this post processing for issue index for documents. Matches Github's behavior.
- Added unit tests.
- Resolves https://codeberg.org/Codeberg/Community/issues/1120
Co-authored-by: Gusted <postmaster@gusted.xyz>
Regression: https://github.com/go-gitea/gitea/pull/24805
Closes: #25945
- Disallow `javascript`, `vbscript` and `data` (data uri images still
work) url schemes even if all other schemes are allowed
- Fixed older `cbthunderlink` tests
---------
Co-authored-by: delvh <dev.lh@web.de>
# The problem
There were many "path tricks":
* By default, Gitea uses its program directory as its work path
* Gitea tries to use the "work path" to guess its "custom path" and
"custom conf (app.ini)"
* Users might want to use other directories as work path
* The non-default work path should be passed to Gitea by GITEA_WORK_DIR
or "--work-path"
* But some Gitea processes are started without these values
* The "serv" process started by OpenSSH server
* The CLI sub-commands started by site admin
* The paths are guessed by SetCustomPathAndConf again and again
* The default values of "work path / custom path / custom conf" can be
changed when compiling
# The solution
* Use `InitWorkPathAndCommonConfig` to handle these path tricks, and use
test code to cover its behaviors.
* When Gitea's web server runs, write the WORK_PATH to "app.ini", this
value must be the most correct one, because if this value is not right,
users would find that the web UI doesn't work and then they should be
able to fix it.
* Then all other sub-commands can use the WORK_PATH in app.ini to
initialize their paths.
* By the way, when Gitea starts for git protocol, it shouldn't output
any log, otherwise the git protocol gets broken and client blocks
forever.
The "work path" priority is: WORK_PATH in app.ini > cmd arg --work-path
> env var GITEA_WORK_DIR > builtin default
The "app.ini" searching order is: cmd arg --config > cmd arg "work path
/ custom path" > env var "work path / custom path" > builtin default
## ⚠️ BREAKING
If your instance's "work path / custom path / custom conf" doesn't meet
the requirements (eg: work path must be absolute), Gitea will report a
fatal error and exit. You need to set these values according to the
error log.
----
Close #24818
Close #24222
Close #21606
Close #21498
Close #25107
Close #24981
Maybe close #24503
Replace #23301
Replace #22754
And maybe more
Fixes #25160.
`data-source-position` of checkboxes in a task list was incorrect
whenever there was YAML front matter. This would result in issue content
or PR descriptions getting corrupted with random `x` or space characters
when a user checked or unchecked a task.
This change makes the CSS for `<video>` in markup match that of `<img>`,
and also allows additional attributes to be used. This way the width,
padding, alignment should work equally well for both.
The old code has a lot of technical debts, eg: `repo/wiki/view.tmpl` /
`Iterate`
This PR improves the Wiki TOC display and improves the code.
---------
Co-authored-by: delvh <dev.lh@web.de>
Remove `[repository.editor] PREVIEWABLE_FILE_MODES` setting that seemed
like it was intended to support this but did not work. Instead, whenever
viewing a file shows a preview, also have a Preview tab in the file
editor.
Add new `/markup` web and API endpoints with `comment`, `gfm`,
`markdown` and new `file` mode that uses a file path to determine the
renderer.
Remove `/markdown` web endpoint but keep the API for backwards and
GitHub compatibility.
## ⚠️ BREAKING ⚠️
The `[repository.editor] PREVIEWABLE_FILE_MODES` setting was removed.
This setting served no practical purpose and was not working correctly.
Instead a preview tab is always shown in the file editor when supported.
---------
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
This branch continues the work of #23092 and attempts to rid the
codebase of any `nil` contexts when using a `RenderContext`.
Anything that renders markdown or does post processing may call
`markup.sha1CurrentPatternProcessor()`, and this runs
`git.OpenRepository()`, which needs a context. It will panic if the
context is `nil`. This branch attempts to _always_ include a context
when creating a `RenderContext` to prevent future crashes.
Co-authored-by: Kyle D <kdumontnu@gmail.com>
As you can imagine, for the Blender development process it is rather
nice to be able to include videos in issues, pull requests, etc.
This PR allows the `<video>` HTML tag to be used in MarkDown, with the
`src`, `autoplay`, and `controls` attributes.
## Help Needed
To have this fully functional, personally I feel the following things
are still missing, and would appreciate some help from the Gitea team.
### Styling
Some CSS is needed, but I couldn't figure out which of the LESS files
would work. I tried `web_src/less/markup/content.less` and
`web_src/less/_base.less`, but after running `make` the changes weren't
seen in the frontend.
This I would consider a minimal set of CSS rules to be applied:
```css
video {
max-width: 100%;
max-height: 100vh;
}
```
### Default Attributes
It would be fantastic if Gitea could add some default attributes to the
`<video>` tag. Basically `controls` should always be there, as there is
no point in disallowing scrolling through videos, looping them, etc.
### Integration with the attachments system
Another thing that could be added, but probably should be done in a
separate PR, is the integration with the attachments system. Dragging in
a video should attach it, then generate the appropriate MarkDown/HTML.
Some bugs caused by less unit tests in fundamental packages. This PR
refactor `setting` package so that create a unit test will be easier
than before.
- All `LoadFromXXX` files has been splited as two functions, one is
`InitProviderFromXXX` and `LoadCommonSettings`. The first functions will
only include the code to create or new a ini file. The second function
will load common settings.
- It also renames all functions in setting from `newXXXService` to
`loadXXXSetting` or `loadXXXFrom` to make the function name less
confusing.
- Move `XORMLog` to `SQLLog` because it's a better name for that.
Maybe we should finally move these `loadXXXSetting` into the `XXXInit`
function? Any idea?
---------
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: delvh <dev.lh@web.de>
These functions don't examine contents, just filenames, so they don't
fit in well in a markup module.
This was originally part of
https://github.com/go-gitea/gitea/pull/22177.
Signed-off-by: Nick Guenther <nick.guenther@polymtl.ca>
Fixes #22628
This PR adds cross references for commits by using the format
`owner/repo@commit` . References are rendered like
[go-gitea/lgtm@6fe88302](#dummy).
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
When using an external renderer, STDOUT is expected to be HTML. But
anything written to STDERR is currently ignored. In cases where the
renderer fails, I would like to log any error messages that the external
program outputs to STDERR.
- Remove code that isn't being used.
Found this is my stash from a few weeks ago, not sure how I found this
in the first place.
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Change all license headers to comply with REUSE specification.
Fix #16132
Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Previously mentioning a user would link to its profile, regardless of
whether the user existed. This change tests if the user exists and only
if it does - a link to its profile is added.
* Fixes #3444
Signed-off-by: Yarden Shoham <hrsi88@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
It distractingly shows up on unit tests
* Looks like a leftover from #20571
Signed-off-by: Yarden Shoham <hrsi88@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
There is a small bug in #20571 whereby `$a a$b b$` will not be correctly
detected as a math inline block of `a a$b b`. This PR fixes this.
Also reenable test cases as per #21340
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
The behaviour of `PreventSurroundingPre` has changed in
https://github.com/alecthomas/chroma/pull/618 so that apparently it now
causes line wrapper tags to be no longer emitted, but we need some form
of indication to split the HTML into lines, so I did what
https://github.com/yuin/goldmark-highlighting/pull/33 did and added the
`nopWrapper`.
Maybe there are more elegant solutions but for some reason, just
splitting the HTML string on `\n` did not work.
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
This PR adds mathematical rendering with KaTeX.
The first step is to add a Goldmark extension that detects the latex
(and tex) mathematics delimiters.
The second step to make this extension only run if math support is
enabled.
The second step is to then add KaTeX CSS and JS to the head which will
load after the dom is rendered.
Fix #3445
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* feat: extend issue template for yaml
* feat: support yaml template
* feat: render form to markdown
* feat: support yaml template for pr
* chore: rename to Fields
* feat: template unmarshal
* feat: split template
* feat: render to markdown
* feat: use full name as template file name
* chore: remove useless file
* feat: use dropdown of fomantic ui
* feat: update input style
* docs: more comments
* fix: render text without render
* chore: fix lint error
* fix: support use description as about in markdown
* fix: add field class in form
* chore: generate swagger
* feat: validate template
* feat: support is_nummber and regex
* test: fix broken unit tests
* fix: ignore empty body of md template
* fix: make multiple easymde editors work in one page
* feat: better UI
* fix: js error in pr form
* chore: generate swagger
* feat: support regex validation
* chore: generate swagger
* fix: refresh each markdown editor
* chore: give up required validation
* fix: correct issue template candidates
* fix: correct checkboxes style
* chore: ignore .hugo_build.lock in docs
* docs: separate out a new doc for merge templates
* docs: introduce syntax of yaml template
* feat: show a alert for invalid templates
* test: add case for a valid template
* fix: correct attributes of required checkbox
* fix: add class not-under-easymde for dropzone
* fix: use more back-quotes
* chore: remove translation in zh-CN
* fix EasyMDE statusbar margin
* fix: remove repeated blocks
* fix: reuse regex for quotes
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Support localized README
* Slightly simplify getting the readme file and add some tests. Ensure that i18n also
works for docs/ etc.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update modules/markup/renderer.go
* Update modules/markup/renderer.go
* Update modules/markup/renderer.go
Co-authored-by: Andrew Thornton <art27@cantab.net>
Use body text color in for links in the repository files table
Issue/PR links (`.ref-issue`) will not be affected, as seen in other git services.
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
- This code is only valid when `refNumeric` exist(otherwise we didn't find
such numeric PR and can skip that check) and give a free-pas to the "BEFORE" check when
`ref` is nil.
- Resolves #20109
* Prototyping
* Start work on creating offsets
* Modify tests
* Start prototyping with actual MPH
* Twiddle around
* Twiddle around comments
* Convert templates
* Fix external languages
* Fix latest translation
* Fix some test
* Tidy up code
* Use simple map
* go mod tidy
* Move back to data structure
- Uses less memory by creating for each language a map.
* Apply suggestions from code review
Co-authored-by: delvh <dev.lh@web.de>
* Add some comments
* Fix tests
* Try to fix tests
* Use en-US as defacto fallback
* Use correct slices
* refactor (#4)
* Remove TryTr, add log for missing translation key
* Refactor i18n
- Separate dev and production locale stores.
- Allow for live-reloading in dev mode.
Co-authored-by: zeripath <art27@cantab.net>
* Fix live-reloading & check for errors
* Make linter happy
* live-reload with periodic check (#5)
* Fix tests
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
* clean git support for ver < 2.0
* fine tune tests for markup (which requires git module)
* remove unnecessary comments
* try to fix tests
* try test again
* use const for GitVersionRequired instead of var
* try to fix integration test
* Refactor CheckAttributeReader to make a *git.Repository version
* update document for commit signing with Gitea's internal gitconfig
* update document for commit signing with Gitea's internal gitconfig
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Allow render HTML with css/js external links
* Fix bug because of filename escape chars
* Fix lint
* Update docs about new configuration item
* Fix bug of render HTML in sub directory
* Add CSP head for displaying iframe in rendering file
* Fix test
* Apply suggestions from code review
Co-authored-by: delvh <dev.lh@web.de>
* Some improvements
* some improvement
* revert change in SanitizerDisabled of external renderer
* Add sandbox for iframe and support allow-scripts and allow-same-origin
* refactor
* fix
* fix lint
* fine tune
* use single option RENDER_CONTENT_MODE, use sandbox=allow-scripts
* fine tune CSP
* Apply suggestions from code review
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Automatically add sidebar in the wiki view containing a TOC for the wiki page.
Make the TOC collapsable
Signed-off-by: Andrew Thornton <art27@cantab.net>
When Gitea is running as PID 1 git will occassionally orphan child processes leading
to (defunct) processes. This PR simply sets Setpgid to true on these child processes
meaning that these defunct processes will also be correctly reaped.
Fix #19077
Signed-off-by: Andrew Thornton <art27@cantab.net>
Remove two unmaintained vendor packages `i18n` and `paginater`. Changes:
* Rewrite `i18n` package with a more clear fallback mechanism. Fix an unstable `Tr` behavior, add more tests.
* Refactor the legacy `Paginater` to `Paginator`, test cases are kept unchanged.
Trivial enhancement (no breaking for end users):
* Use the first locale in LANGS setting option as the default, add a log to prevent from surprising users.
The main purpose is to refactor the legacy `unknwon/com` package.
1. Remove most imports of `unknwon/com`, only `util/legacy.go` imports the legacy `unknwon/com`
2. Use golangci's depguard to process denied packages
3. Fix some incorrect values in golangci.yml, eg, the version should be quoted string `"1.18"`
4. Use correctly escaped content for `go-import` and `go-source` meta tags
5. Refactor `com.Expand` to our stable (and the same fast) `vars.Expand`, our `vars.Expand` can still return partially rendered content even if the template is not good (eg: key mistach).
* Add tests for references with dashes
This commit adds tests for full URLs referencing repos names and user
names containing a dash.
* Extend regex to match URLs to repos/users with dashes
* Don't panic & allow shorter sha1
- Don't panic when the full regex isn't matched and allow the usage of a
shorter sha1 being used.
- Resolves #18471
* Update modules/markup/html.go
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
This PR continues the work in #17125 by progressively ensuring that git
commands run within the request context.
This now means that the if there is a git repo already open in the context it will be used instead of reopening it.
Signed-off-by: Andrew Thornton <art27@cantab.net>
This PR registers requests with the process manager and manages hierarchy within the processes.
Git repos are then associated with a context, (usually the request's context) - with sub commands using this context as their base context.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Prevent double sanitize.
* Use SanitizeReaderToWriter.
At the moment `actualRender` uses `SanitizeReader` to sanitize the output. But `SanitizeReader` gets called in `markup.render` too so the output gets sanitized twice.
I moved the `SanitizeReader` call into `RenderRaw` because this method does not use `markup.render`. I would like to remove the `RenderRaw`/`RenderRawString` methods too because they are only called from tests, the fuzzer and the `/markup/raw` api endpoint. This endpoint is not in use so I think we could remove them. If we really in the future need a method to render markdown without PostProcessing we could achieve this with a more flexible `renderer.NeedPostProcess` method.
* Add copy button to markdown code blocks
Done mostly in JS because I think it's better not to try getting buttons
past the markup sanitizer.
* add svg module tests
* fix sanitizer regexp
* remove outdated comment
* vertically center button in issue comments as well
* add comment to css
* fix undefined on view file line copy
* combine animation less files
* Update modules/markup/markdown/markdown.go
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* add test for different sizes
* add cloneNode and add tests for it
* use deep clone
* remove useless optional chaining
* remove the svg node cache
* unify clipboard copy string and i18n
* remove unused var
* remove unused localization
* minor css tweaks to the button
* comment tweak
* remove useless attribute
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Fixes #16558 CSV delimiter determiner
* Fixes #16558 - properly determine CSV delmiiter
* Moves quoteString to a new function
* Adds big test with lots of commas for tab delimited csv
* Adds comments
* Shortens the text of the test
* Removes single quotes from regexp as only double quotes need to be searched
* Fixes spelling
* Fixes check of length as it probalby will only be 1e4, not greater
* Makes sample size a const, properly removes truncated line
* Makes sample size a const, properly removes truncated line
* Fixes comment
* Fixes comment
* tests for FormatError() function
* Adds logic to find the limiter before or after a quoted value
* Simplifies regex
* Error tests
* Error tests
* Update modules/csv/csv.go
Co-authored-by: delvh <dev.lh@web.de>
* Update modules/csv/csv.go
Co-authored-by: delvh <dev.lh@web.de>
* Adds comments
* Update modules/csv/csv.go
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: delvh <dev.lh@web.de>
When rendering source in org mode there is a mistake in the highlight code that
causes a panic.
This PR fixes this.
Fix #17139
Signed-off-by: Andrew Thornton <art27@cantab.net>
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* In Render tolerate not being passed a context
It is possible for RenderString to be passed to an external renderer if markdown
is set to be rendered by an external renderer. No context is currently sent to these
meaning that this will error out.
Fix #16835
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add Context to Repo calls for RenderString
All calls from routers can easily add the context - so add it.
Signed-off-by: Andrew Thornton <art27@cantab.net>
The fuzzer found an issue with the issue pattern processor where there is a spurious
path.Clean which does not need to be there. This PR also sets the default AppURL for
the fuzzer too.
Signed-off-by: Andrew Thornton <art27@cantab.net>
