Colour console output stripped from both Julia backends

[MichaelHatherly]

Bug description

Both the Jupyter-based backend and https://github.com/PumasAI/QuartoNotebookRunner.jl cannot print colour output to stdout in notebook cells. Originally seen in PumasAI/QuartoNotebookRunner.jl#94 (comment).

The first image is using QNR. The output from the first two cells is expected to have some amount of colour highlighting. None is rendered. Note that QNR is returning the expected JSON output with ANSI codes embedded:

julia> json.cells[end-4].outputs[1].text
"\e[32m\e[1mStatus\e[22m\e[39m `~/personal/quartos/example/Project.toml`\n  \e[90m[23fbe1c1] \e[39mLatexify v0.16.6\n  \e[90m[6
099a3de] \e[39mPythonCall v0.9.24\n  \e[90m[6f49c342] \e[39mRCall v0.14.6\n"

it just gets stripped afterwards, presumably somewhere in quarto? Worth noting that the output of the third cell is a stacktrace that does manage to print the correct colours in it's output. I assume these are hitting different code paths?

This image is from the jupyter backend, same issue appears here:

Steps to reproduce

QNR backend:

---
title: "Julia"
engine: "julia"
---

```{julia}
import Pkg
Pkg.status()
```

```{julia}
printstyled("Hello, World!", color=:red)
```

```{julia}
#| error: true
div(1, 0)
```

Jupyter backend:

---
title: "Julia"
jupyter: julia-1.10
---

```{julia}
import Pkg
Pkg.status()
```

```{julia}
printstyled("Hello, World!", color=:red)
```

```{julia}
#| error: true
div(1, 0)
```

Actual behavior

No response

Expected behavior

Colour output should print correctly in cell outputs.

Your environment

• OS: macOS 15.2

Quarto check output

Quarto 1.6.39
[✓] Checking environment information...
      Quarto cache location: /Users/mike/Library/Caches/quarto
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.4.0: OK
      Dart Sass version 1.70.0: OK
      Deno version 1.46.3: OK
      Typst version 0.11.0: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 1.6.39
      Path: /Applications/quarto/bin

[✓] Checking tools....................OK
      TinyTeX: v2023.12
      Chromium: (not installed)

[✓] Checking LaTeX....................OK
      Using: TinyTex
      Path: /Users/mike/Library/TinyTeX/bin/universal-darwin
      Version: 2023

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.9.2
      Path: /Users/mike/.local/share/mise/installs/python/3.9/bin/python3
      Jupyter: 5.7.2
      Kernels: julia-1.11, julia-1.10, julia-1.9, python3

[✓] Checking Jupyter engine render....OK

[✓] Checking R installation...........OK
      Version: 4.4.2
      Path: /opt/homebrew/Cellar/r/4.4.2_2/lib/R
      LibPaths:
        - /opt/homebrew/lib/R/4.4/site-library
        - /opt/homebrew/Cellar/r/4.4.2_2/lib/R/library
      knitr: (None)
      rmarkdown: (None)

      The knitr package is not available in this R installation.
      Install with install.packages("knitr")
      The rmarkdown package is not available in this R installation.
      Install with install.packages("rmarkdown")

[MichaelHatherly]

I can hack around this by outputting the stdout to a Julia Text object with

```{julia}
import Pkg
Text() do io
    Pkg.status(; io)
end
```

```{julia}
Text() do io
    printstyled(io, "Hello, World!", color=:red)
end
```

which gives

The reason for this is that these use two different fields of the ipynb JSON to store the result.

julia> json.cells[end-4].outputs[1].data["text/plain"]
"\e[32m\e[1mStatus\e[22m\e[39m `~/personal/quartos/example/Project.toml`\n  \e[90m[23fbe1c1] \e[39m
Latexify v0.16.6\n  \e[90m[6099a3de] \e[39mPythonCall v0.9.24\n  \e[90m[6f49c342] \e[39mRCall v0.14
.6\n

Ideally both fields should support colour output.

[mcanouil]

For reference:

• Reveal.js with Julia: escape codes are not handled in output of executable code blocks  #929

[mcanouil]

At least errors go through ANSI processing with Jupyter or Julia as the engine.

• Process ANSI escape code in non-HTML format for Jupyter errors #10347

Input	Output
--- title: "Python" engine: jupyter --- ```{python} #| error: true 1 + "2" ```
--- title: "Julia" engine: jupyter --- ```{julia} #| error: true 1 + "2" ```
--- title: "Julia" engine: julia --- ```{julia} #| error: true 1 + "2" ```

[cderv]

@MichaelHatherly I see this in the intermediate .md output for this cell

```{julia}
printstyled("Hello, World!", color=:red)
```

::: {#4 .cell execution_count=1}
``` {.julia .cell-code}
printstyled("Hello, World!", color=:red)
```

::: {.cell-output .cell-output-stdout}
```
Hello, World!
```
:::
:::

So no ANSI character or color in the output that Quarto gets - processing must happen in TS before any Lua processing

Note that QNR is returning the expected JSON output with ANSI codes embedded:
(...)
it just gets stripped afterwards, presumably somewhere in quarto?

We use colors.stripColor in several places, but we do keep ansi code in some cases like

quarto-cli/src/core/jupyter/jupyter.ts

Lines 1879 to 1898 in ffb8c3d

	} else {
	if (options.toHtml) {
	if (lines.some(hasAnsiEscapeCodes)) {
	const html = await Promise.all(
	lines.map(convertToHtmlSpans),
	);
	return mdMarkdownOutput(
	[
	"\n::: {.ansi-escaped-output}\n```{=html}\n<pre>",
	...html,
	"</pre>\n```\n:::\n",
	],
	);
	} else {
	return mdCodeOutput(lines);
	}
	} else {
	return mdCodeOutput(lines.map(colors.stripColor));
	}
	}

Worth noting that the output of the third cell is a stacktrace that does manage to print the correct colours in it's output. I assume these are hitting different code paths?

and in Error output as you found

quarto-cli/src/core/jupyter/jupyter.ts

Lines 1794 to 1821 in ffb8c3d

	async function mdOutputError(
	output: JupyterOutputError,
	options: JupyterToMarkdownOptions,
	) {
	const traceback = output.traceback.join("\n");
	if (
	!options.toHtml ||
	(!hasAnsiEscapeCodes(output.evalue) && !hasAnsiEscapeCodes(traceback))
	) {
	if (output.traceback.length > 0) {
	return mdCodeOutput([
	output.ename + ": " + output.evalue + "\n" + traceback,
	]);
	} else {
	return mdCodeOutput([
	output.ename + ": " + output.evalue,
	]);
	}
	}
	const tracebackHtml = await convertToHtmlSpans(traceback);
	return mdMarkdownOutput(
	[
	"\n::: {.ansi-escaped-output}\n```{=html}\n<pre>",
	tracebackHtml,
	"</pre>\n```\n:::\n",
	],
	);
	}

as pointed out above special support was added in #10347

So it is either not working as expected for Julia engine - or we need to handle it differently. 🤔

[MichaelHatherly]

Thanks for those pointers on where to look. A few minor changes seem to be getting it working as I would expect it to look.

I'll open a PR when I have a chance to.

[cderv]

Great !

I don't think this is tied to Julia in Jupyter. We do handle ANSI only a few places, and stdout output (like Pkg.status() or printstyled("Hello, World!", color=:red) are) are not handled.

quarto-cli/src/core/jupyter/jupyter.ts

Lines 1621 to 1634 in a0ff13f

	if (output.output_type === "stream") {
	const stream = output as JupyterOutputStream;
	if (asis && stream.name === "stdout") {
	let text: string[] = [];
	if (typeof stream.text === "string") {
	text = [stream.text];
	} else {
	text = stream.text;
	}
	md.push(text.join(""));
	} else {
	md.push(mdOutputStream(stream));
	}
	} else if (output.output_type === "error") {

quarto-cli/src/core/jupyter/jupyter.ts

Lines 1769 to 1792 in a0ff13f

	function mdOutputStream(output: JupyterOutputStream) {
	let text: string[] = [];
	if (typeof output.text === "string") {
	text = [output.text];
	} else {
	text = output.text;
	}
	// trim off warning source line for notebook
	if (output.name === "stderr") {
	if (text[0]) {
	const firstLine = text[0].replace(
	/<ipython-input.*?>:\d+:\s+/,
	"",
	);
	return mdCodeOutput(
	[firstLine, ...text.slice(1)].map(colors.stripColor),
	);
	}
	}
	// normal default handling
	return mdCodeOutput(text.map(colors.stripColor));
	}

We'll wait on your PR if you want to tackle this. Thanks !